#!/usr/bin/env python # coding: utf-8 # # SYDE 556/750: Simulating Neurobiological Systems # # ### Symbols and symbolic-like representations # ## Symbols and Symbol-like representation in neurons # # - We've seen how to represent vectors in neurons # - And how to compute functions on those vectors # - And dynamical systems # - But how can we do anything like human language? # - How could we represent the fact that "the number after 8 is 9" # - Or "dogs chase cats" # - Or "Anne knows that Bill thinks that Charlie likes Dave" # - Does the NEF help us at all with this? # - Or is this just too hard a problem yet? # ### Traditional Cognitive Science # # - Lots of theories that work with structured information like this # - Pretty much all of them use some representation framework like this: # - `after(eight, nine)` # - `chase(dogs, cats)` # - `knows(Anne, thinks(Bill, likes(Charlie, Dave)))` # # - Cognitive models manipulate these sorts of representations # - mental arithmetic # - driving a car # - using a GUI # - parsing language # - etc etc # - Seems to match well to behavioural data, so something like this should be right # - So how can we do this in neurons? # # - This is a hard problem # - Jackendoff (2002) posed this as a major problem, identifying four linguistic challenges for cognitive neuroscience # - The Binding Problem # - The Problem of 2 # - The Problem of Variables # - Working Memory vs Long-term memory # - The Binding Problem # - Suppose you see a red square and a blue circle # - How do you keep these ideas separate? How does "red" get bound with "square", and how is it kept separate from "blue" and "circle"? # - The Problem of 2 # - "The little star's beside the big star" # - How do we keep those two uses of "star" separate? # - The Problem of Variables # - Words seem to have types (nouns, verbs, adjectives, etc, etc) # - How do you make use of that? # - E.g. "blue *NOUN*" is fine, but "blue *VERB*" is not (or is very different) # - Working memory vs Long-term memory # - We can both use sentences (working memory) and store them indefinitely (long-term memory) # - How do these transfer back and forth? # - What are they in the brain? This seems to require moving from storing something in neural activity to storing it in connection weights # # ### Possible solutions # # 1. Oscillations # - "red square and blue circle" # - Different patterns of activity for RED, SQUARE, BLUE, and CIRCLE # - Have the patterns for RED and SQUARE happen, then BLUE and CIRCLE, then back to RED and SQUARE # - More complex structures possible too: # - E.g. the LISA architecture # # # # - Problems # - What controls this oscillation? # - How is it parsed? (i.e., mapped to sets of oscillators, noise) # - How do we deal with the exponentional explosion of nodes needed? # - How to deal with A*B B*C (but not C*A) # # '2. Implementing Symbol Systems in Neurons # - Build a general-purpose symbol-binding system # - Lots of temporary pools of neurons # - Ways to temporarily associate them with particular concepts # - Ways to temporarily associate pools together # - Neural Blackboard Architecture # # # # - Problems # - Very particular structure (doesn't seem to match biology) # - Uses a very large number of neurons (~500 million) to be flexible enough for simple sentences # - And that's just to represent the sentence, never mind controlling and manipulating it # # # '3. Vector operators # - Paul Smolensky [1990](http://verbs.colorado.edu/~llbecker/papers/Smolensky-TensorProductVariableBinding.pdf) suggests using a mathematical operator called a 'tensor product' to bind vectors together. # - The idea is that this operator can 'bind' and its inverse 'unbind' these vectors together. # - It provides an algebraic way of specifying symbolic structures in a vector space. # - If you can represent vector spaces and operators in neurons (e.g. NEF), then it provides a way of working with symbolic structures in neurons. # # # # - Problems # - Every time you bind vectors, the result has $D^2$ as many dimensions. # - It scales extremely poorly: the dimensionality of a structure representation is the base space to the power of depth, i.e. $D_S = D^{depth+1}$ # # # ### A deeper issue # - If we're just implementing something like the original symbol system in neurons, then we're reducing the neural aspects to *mere implementational details*. # - That is, if we had a perfectly good way to implement symbol structures in neurons, then we could take existing cognitive theories based on symbol structures and implement them in neurons. # - But so what? That would be conceding the point that the neurons don't matter -- you don't *need* neurons to understand cognition. # - Sure, they're useful for testing some aspects, like what firing patterns should be. # - But it's more for completeness sake than for a better understanding. # - (Same as wondering how a neuron gets implemented in terms of atoms or quarks. Not that important for function.) # - This is why the majority of cognitive scientists don't worry about neurons. # ### Semantic pointers # # - Tensor products are on the right track, if we can solve the scaling problem. # - 'How to build a brain' exploits a compression operator introduced by Tony Plate for cognitive modeling called 'circular convolution' $\circledast$ (this operator has been used in signal processing for a long time). # - In the book, I suggest that neurally realized compressed vector representations are prevalent in the brain, and call such representations 'semantic pointers'. # - Pointers: because they can be used like computer science pointers to be efficient references to complex representations. # - Semantic: because (unlike CS pointers), their content is derived (through compression) from semantically related representations. # # # - This provides something that's similar to the symbolic approach, but much more tied to biology. # - Most of the same capabilities as the classic symbol systems. # - But not all. # - Based on vectors and functions on those vectors. # - There is a vector for each concept (and maybe 'sub concepts'). # - Build up structures by doing math on those vectors. # # # - Example # - "blue square and red circle" # - Can't just do BLUE+SQUARE+RED+CIRCLE. # - Why? # - Need some other operation as well. # - Requirements: # - Input 2 vectors, get a new vector as output. # - Reversible (given the output and one of the input vectors, generate the other input vector). # - Output vector is highly dissimilar to either input vector. # - Unlike addition, where the output is highly similar. # # - Circular convoluation can act as such a function. # - (There are many other such functions: XOR, Multiply, etc... collectively this kind of vector space binding to represent structures are sometimes called 'Vector Symbolic Architectures' (Gayler, 2003).) # # - Why circular convolution? # - Extensively studied (Plate, 1997: Holographic Reduced Representations). # - Easy to approximately invert (circular correlation). # # - Examples: # - `BLUE` $\circledast$ `SQUARE + RED` $\circledast$ `CIRCLE` # - `DOG` $\circledast$ `AGENT + CAT` $\circledast$ `THEME + VERB` $\circledast$ `CHASE` # # # - Unbinding: # # # # # # # - Can also think of this operator as a compressed outer product # # # - Lots of nice properties: # - Can store complex structures: # - `after(eight, nine)` # - `NUMBER` $\circledast$ `EIGHT + NEXT` $\circledast$ `NINE` # - `knows(Anne, thinks(Bill, likes(Charlie, Dave)))` # - `SUBJ` $\circledast$ `ANNE + ACT` $\circledast$ `KNOWS + OBJ` $\circledast$ `(SUBJ` $\circledast$ `BILL + ACT` $\circledast$ `THINKS + OBJ` $\circledast$ `(SUBJ` $\circledast$ `CHARLIE + ACT` $\circledast$ `LIKES + OBJ` $\circledast$ `DAVE))` # - But gracefully degrades! # - As representation gets more complex, the accuracy of breaking it apart decreases. # - Keeps similarity information. # - If `RED` is similar to `PINK` then `RED` $\circledast$ `CIRCLE` is similar to `PINK` $\circledast$ `CIRCLE`. # # - But rather complicated: # - Seems like a weird operation for neurons to do. # ### Circular convolution in the NEF # # - Or is it? # - Circular convolution is a whole bunch ($D^2$) of multiplies. # - Like any convolution, it can also be written as a Fourier transform, an elementwise multiply, and another Fourier transform: # - i.e. $A \circledast B = IDFT(DFT(A).DFT(B)))$ # - The discrete fourier transform is just a linear operation, hence a matrix. # - So that's just $D$ pairwise multiplies. # - In fact, circular convolution turns out to be *exactly* what the NEF shows neurons are good at (i.e., low order polynomials). # ### Jackendoff's Challenges # # - As pointed out in [Vector Symbolic Architectures Answer Jackendoff's Challenges for Cognitive Neuroscience].(http://arxiv.org/ftp/cs/papers/0412/0412059.pdf) # - The Binding Problem: # - There is a lot of "binding" (structurally combining items) in linguistics (more than vision) . # - `RED` $\circledast$ `CIRCLE + BLUE` $\circledast$ `TRIANGLE` # - After it is bound, we can ask "what color is the circle by doing $\circledast$ `CIRCLE'`. # - where `'` is "inverse". # - (`RED` $\circledast$ `CIRCLE + BLUE` $\circledast$ `TRIANGLE`) $\circledast$ `CIRCLE'` # - = `RED` $\circledast$ `CIRCLE` $\circledast$ `CIRCLE' + BLUE` $\circledast$ `TRIANGLE` $\circledast$ `CIRCLE'` # - = `RED + BLUE` $\circledast$ `TRIANGLE` $\circledast$ `CIRCLE'` # - = `RED + noise` # - $\approx$ `RED` # # - The Problem of 2: # - How can we distinguish two uses of the same concept in a sentence? # - "The little star's beside the big star." # - `OBJ1` $\circledast$ `(TYPE` $\circledast$ `STAR + SIZE` $\circledast$ `LITTLE) + OBJ2` $\circledast$ `(TYPE` $\circledast$ `STAR + SIZE` $\circledast$ `BIG) + BESIDE` $\circledast$ `OBJ1` $\circledast$ `OBJ2` # - Notice that `BESIDE` $\circledast$ `OBJ1` $\circledast$ `OBJ2` = `BESIDE` $\circledast$ `OBJ2` $\circledast$ `OBJ1` # - And that we can distinguish the two uses of star. # - The Problem of Variables: # - How can we manipulate abstract place holders? # - `S` = `RED` $\circledast$ `NOUN` # - `VAR` = `BALL` $\circledast$ `NOUN'` # - `S` $\circledast$ `VAR` = `RED` $\circledast$ `BALL` # - Binding in Working Memory vs Long-term memory: # - How can we use and manipulate the same structures in both WM and LTM? # - Vectors are what we work with (activity of neurons). # - Functions are long-term memory (connection weights). # - Functions return (and modify) vectors. # ### Symbol-like manipulation # # - Can do a lot of standard symbol operations. # - Sometimes you have to explicitly bind and unbind to manipulate the data. # - Less accuracy for more complex structures. # - *But we can also do more with these representations* # - Sometimes you don't have to explicitly bind/unbind to manipulate. # - Can define other vector operations not available for symbols... # # # # ### Raven's Progressive Matrices # # - An IQ test that's generally considered to be the best at measuring general-purpose "fluid" intelligence: # - Nonverbal (so it's not measuring language skills, and fairly unbiased across cultures, hopefully). # - Fill in the blank: given eight possible answers, pick one. # # # # - This is not an actual question on the test: # - The test is copyrighted. # - They don't want the test to leak out, since it's been the same set of 60 questions since 1936. # - But they do look like that. # # - How can we model people doing this task? # - A fair number of different attempts: # - None neural. # - Generally use the approach of building in a large set of different types of patterns to look for, and then trying them all in turn. # - Which seems wrong for a test that's supposed to be about *flexible, fluid* intelligence. # # - Does this vector approach offer an alternative? # # # - First we need to represent the different patterns as a vector. # - This is a hard image interpretation problem. # - Still ongoing work here. # - So we'll skip it and start with things in vector form. # # # # - How do we represent a picture? # - `SHAPE` $\circledast$ `ARROW + NUMBER` $\circledast$ `ONE + DIRECTION` $\circledast$ `UP` # - Can do variations like this for all the pictures. # - Fairly standard with most assumptions about how people represent complex scenes. # - But that part is not being modelled (yet!). # # - We have shown that it's possible to build these sorts of representations up directly from visual stimuli. # - With a very simple vision system that can only recognize a few different shapes. # - And where items have to be shown sequentially as it has no way of moving its eyes. # ## Examples in Spaun # # - Let's look at a simpler but related example: # - Binding used for building a list to remember # In[14]: from IPython.display import YouTubeVideo YouTubeVideo('U_Q6Xjz9QHg', width=720, height=400, loop=1, autoplay=0, playlist='U_Q6Xjz9QHg') # - The memory of the list is built up by using a basal ganglia action selection system to control feeding values into an integrator # - The thought bubble shows how close the decoded values are to the ideal # - Notice the forgetting! # # # # - The same system can be used to do a version of the Raven's Matrices task # In[15]: from IPython.display import YouTubeVideo YouTubeVideo('Q_LRvnwnYp8', width=720, height=400, loop=1, autoplay=0, playlist='Q_LRvnwnYp8') # - `S1 = ONE` $\circledast$ `P1` # - `S2 = ONE` $\circledast$ `P1 + ONE` $\circledast$ `P2` # - `S3 = ONE` $\circledast$ `P1 + ONE` $\circledast$ `P2 + ONE` $\circledast$ `P3` # - `S4 = FOUR` $\circledast$ `P1` # - `S5 = FOUR` $\circledast$ `P1 + FOUR` $\circledast$ `P2` # - `S6 = FOUR` $\circledast$ `P1 + FOUR` $\circledast$ `P2 + FOUR` $\circledast$ `P3` # - `S7 = FIVE` $\circledast$ `P1` # - `S8 = FIVE` $\circledast$ `P1 + FIVE` $\circledast$ `P2` # # - what is `S9`? # How does Spaun make a guess at the end? # # - Let's figure out what the transformation is # - `T1 = S2` $\circledast$ `S1'` # - `T2 = S3` $\circledast$ `S2'` # - `T3 = S5` $\circledast$ `S4'` # - `T4 = S6` $\circledast$ `S5'` # - `T5 = S8` $\circledast$ `S7'` # # - `T = (T1 + T2 + T3 + T4 + T5)/5` # - `S9 = S8` $\circledast$ `T` # # - `S9 = FIVE` $\circledast$ `P1 + FIVE` $\circledast$ `P2 + FIVE` $\circledast$ `P3` # # - This becomes a novel way of manipulating structured information # - Exploiting the fact that it is a vector underneath # - [A spiking neural model applied to the study of human performance and cognitive decline on Raven's Advanced Progressive Matrices](http://www.sciencedirect.com/science/article/pii/S0160289613001542) # ### Using Nengo # - How can we use Nengo to perform these kinds of operations? # - There's a 'SPA' module you can import, and it lets you do all kinds of symbol-like manipulations (and other cognitive processing). # - Let's try answering simple questions about the bindings in a representation. # - I just stole this from the 'Question answering' example in Nengo. # In[38]: get_ipython().run_line_magic('pylab', 'inline') import nengo import nengo_spa as spa # In[41]: # Number of dimensions for the Semantic Pointers dimensions = 32 model = spa.Network(label="Simple question answering") with model: colour_in = spa.Transcode(colour_input, output_vocab=dimensions) shape_in = spa.Transcode(shape_input, output_vocab=dimensions) cue = spa.Transcode(cue_input, output_vocab=dimensions) conv = spa.State(dimensions) out = spa.State(dimensions) # Connect the buffers colour_in * shape_in >> conv conv * ~cue >> out # The input will switch every 0.5 seconds between `RED` and `BLUE`. In the same way the shape input switches between `CIRCLE` and `SQUARE`. Thus, the network will bind alternatingly `RED * CIRCLE` and `BLUE * SQUARE` for 0.5 seconds each. # # The cue for deconvolving bound semantic pointers cycles through `CIRCLE`, `RED`, `SQUARE`, and `BLUE` within one second. # # In[42]: def colour_input(t): if (t // 0.5) % 2 == 0: return 'RED' else: return 'BLUE' def shape_input(t): if (t // 0.5) % 2 == 0: return 'CIRCLE' else: return 'SQUARE' def cue_input(t): sequence = ['0', 'CIRCLE', 'RED', '0', 'SQUARE', 'BLUE'] idx = int((t // (1. / len(sequence))) % len(sequence)) return sequence[idx] # In[43]: with model: model.config[nengo.Probe].synapse = nengo.Lowpass(0.03) p_colour_in = nengo.Probe(colour_in.output) p_shape_in = nengo.Probe(shape_in.output) p_cue = nengo.Probe(cue.output) p_conv = nengo.Probe(conv.output) p_out = nengo.Probe(out.output) # In[44]: from nengo_gui.ipython import IPythonViz IPythonViz(model, "configs/blue_red_spa.py.cfg") # In[45]: sim = nengo.Simulator(model) sim.run(3.) # In[46]: plt.figure(figsize=(10, 10)) vocab = model.vocabs[dimensions] plt.subplot(5, 1, 1) plt.plot(sim.trange(), spa.similarity(sim.data[p_colour_in], vocab)) plt.legend(vocab.keys(), fontsize='x-small') plt.ylabel("colour") plt.subplot(5, 1, 2) plt.plot(sim.trange(), spa.similarity(sim.data[p_shape_in], vocab)) plt.legend(vocab.keys(), fontsize='x-small') plt.ylabel("shape") plt.subplot(5, 1, 3) plt.plot(sim.trange(), spa.similarity(sim.data[p_cue], vocab)) plt.legend(vocab.keys(), fontsize='x-small') plt.ylabel("cue") plt.subplot(5, 1, 4) for pointer in ['RED * CIRCLE', 'BLUE * SQUARE']: plt.plot(sim.trange(), vocab.parse(pointer).dot(sim.data[p_conv].T), label=pointer) plt.legend(fontsize='x-small') plt.ylabel("convolved") plt.subplot(5, 1, 5) plt.plot(sim.trange(), spa.similarity(sim.data[p_out], vocab)) plt.legend(vocab.keys(), fontsize='x-small') plt.ylabel("output") plt.xlabel("time [s]"); # In[37]: sum(ens.n_neurons for ens in model.all_ensembles) #Total number of neurons # In[ ]: