write some codes

1 files changed, 121 insertions, 3 deletions

diff --git a/01-autoassociative-memory.txt b/01-autoassociative-memory.txt
index 715133a..85871ba 100644
--- a/01-autoassociative-memory.txt
+++ b/01-autoassociative-memory.txt
@@ -1,8 +1,126 @@
 * Autoassociative memory
 
-is where a partial, or distorted memory produces a previously known memory. Our starting point here is the open access conference paper Krotov & Hopfield 2016, Dense Associative Memory for Pattern Recognition.
+is where a partial, or distorted memory produces a previously known
+memory. Our starting point here is the open access conference paper
+Krotov & Hopfield 2016, Dense Associative Memory for Pattern
+Recognition.
 
-Hopfield networks are a deep single layer of spiking neurons connected in a way that closely tracks some models of animal neurons, formulated as a Lyapunov function which means if we think long enough, we will converge to a specific memory. The conference paper presents the mathematics for what are called modern hopfield networks- which use rectified polynomial activation functions in the Lyapunov condition, which results in them being able to store very many memories compared to the number of neurons. This can be thought of as a polynomial resulting in very different heights quickly, for relatively small linear distances between memories. This is why rectified polynomials can be used rather than a rectified linear function, otherwise popular in deep learning.
+Hopfield networks are a deep single layer of spiking neurons connected
+in a way that closely tracks some models of animal neurons, formulated
+as a Lyapunov function which means if we think long enough, we will
+converge to a specific memory.
 
-sm0l networks are desirable for us.
+* Wait, what?
+Instead of starting from the beginning or the end I'm going to start
+from the middle. The following is a useful common lisp implementation
+of Hopfield nets using a rectified exponential function. Details of
+the common lisp package are in the appendix.
 
+The rest of this book will be a deconstruction of this function with
+reimplementation in interlisp as a lispusers contribution.
+
+This lisp is intended to both be useable and offputting motivating the
+adoption of the interlisp programming system, and a set of logical and
+stylistic changes and optimisations.
+#+begin_src lisp
+  (defun closure-hop-net (number-bits)
+    (let ((potential (make-array number-bits :element-type '(integer 0 1)))
+	  (memories (list))
+	  (formatting-output "Potential: ~s~%Memories:~%~{~s~%~}"))
+      (labels ((rect-poly-2 (x) (if (< x 0) 0 (expt x 2)))
+	       (b2s (x) (expt -1 (1+ x)))
+	       (signed-idx (sgn memory idx)
+		 (rect-poly-2
+		  (+ (* sgn (b2s (aref memory idx)))
+		     (loop for pot across potential
+			   for mem across memory
+			   for count from 0
+			   summing
+			   (cond
+			     ((= count idx) 0)
+			     (t (* (b2s mem) (b2s pot))))))))
+	       (local-update (idx)
+		 (setf
+		  (aref potential idx)
+		  (if
+		   (minusp
+		    (loop for memory in memories
+			  summing
+			  (- (signed-idx +1 memory idx)
+			     (signed-idx -1 memory idx))))
+		   0 1))))
+	(lambda (&key push-memory pop-memory format update set-potential)
+	  (cond
+	    (set-potential (setf potential set-potential))
+	    (push-memory (push push-memory memories))
+	    (pop-memory (pop memories))
+	    (format (when (stringp format) (setf formatting-output format))
+		    (format nil formatting-output potential memories))
+	    (update (if (numberp update) (local-update update)
+			(loop for idx below (length potential)
+			      do (local-update idx)))))))))
+#+end_src
+what a mouthful. Garbage collected bit-array closures. Updates are
+performed from raw memories every time, which is a niche use case.
+
+However it is a hopfield net and does work.
+
+* The xor example
+All the common lisp 2e compilers use asdf. asdf package details for
+binry-hop are in the appendix. The following was performed
+interactively using the popular sbcl compiler hosted on
+tilde.institute.
+#+begin_src lisp
+  (require :asdf)
+  (require "binry-hop")
+  (in-package :hop)
+  (defparameter *eg* (closure-hop-net 3))
+  (dolist (mem '(#*000 #*110 #*101 #*011)) (funcall *eg* :push-memory mem))
+  (funcall *eg* :set-potential #*100)
+  (print (funcall *eg* :format t))
+#+end_src
+Outputting
+#+begin_example
+  Potential: #*100
+  Memories:
+  #*011
+  #*101
+  #*110
+  #*000
+#+end_example
+We are using the first two bits as the XOR inputs, and the third bit
+as the output. Our memories for the XOR truth table, but our input is
+incorrect according to our memories.
+#+begin_src lisp
+  (funcall *eg* :update 2)
+  (funcall *eg* :format t)
+#+end_src
+We did it!
+#+begin_example
+  Potential: #*101
+  Memories:
+  #*011
+  #*101
+  #*110
+  #*000
+#+end_example
+A hopfield net converges to one of its memories, and stops changing
+for one of its inputs. The updates can be performed breadth first or
+randomly (Monte Carlo). In this case we cheated and just updated the
+only bit we were interested in (the output bit).
+
+Over the next chapters I will first explore and develop programming a
+simple game in terms of hopfield nets, then rewriting, and begin
+exploring once comfortably inside the interlisp programming system.
+
+* Side note on a feature of common lisp
+
+One intended use case for common lisp is portability. The same common
+lisp code is expected to be common between very different
+environments. One effect of this is that while it is easy to port
+interlisp into common lisp by editing out particular interlisp
+features, it is relatively difficult to adapt common lisp to the more
+idiomatic interlisp. However this experience is expected to be
+instructive. The above common lisp package can be used freely in lisp,
+including embedding in C as it stands; but we will not try and explore
+in C.