Iterative Retrieval and Block Coding in Autoassociative and Heteroassociative Memory

Neural associative memories (NAM) are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Gripon and Berrou (2011) investigated NAM employing block coding, a particular sparse coding method, and reported a significant increase in storage capacity. Here we verify and extend their results for both heteroassociative and recurrent autoassociative networks. For this we provide a new analysis of iterative retrieval in finite autoassociative and heteroassociative networks that allows estimating storage capacity for random and block patterns. Furthermore, we have implemented various retrieval algorithms for block coding and compared them in simulations to our theoretical results and previous simulation data. In good agreement of theory and experiments, we find that finite networks employing block coding can store significantly more memory patterns. However, due to the reduced information per block pattern, it is not possible to significantly increase stored information per synapse. Asymptotically, the information retrieval capacity converges to the known limits C=ln2 approximate to 0.69 and C=(ln2)/4 approximate to 0.17 also for block coding. We have also implemented very large recurrent networks up to n=2 center dot 106 neurons, showing that maximal capacity C approximate to 0.2 bit per synapse occurs for finite networks having a size n approximate to 105 similar to cortical macrocolumns.