PyLissom documentation

The LISSOM family of self-organizing computational models aims to replicate the detailed development of the visual cortex of humans.

PyLissom is a PyTorch extension implementing the LISSOM networks. It’s split in two parts: the core nn and optim packages, which implement the LISSOM network itself, and the datasets, models, and utils packages. Some of the datasets, models and utils of PyLissom were inspired by Topographica, a former implementation of the LISSOM networks oriented in its design to the neuroscience community. Instead, PyLissom was designed for a hybrid use case of the machine learning and the neuroscience communities.

Installation instructions

Linux (tested on Ubuntu 16.04):

conda create -n MLOSS18

source activate MLOSS18

conda install pytorch==0.1.12

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pylissom

The code is hosted in pypi: https://test.pypi.org/project/pylissom/

Getting Started

There are Jupyter notebooks with tutorials at the github’s page of the project. If Github is not rendering them, we leave these links at your disposal:

Linear modules

Lissom modules

Optimizers

Orientation Maps and pylissom tools

The main features provided are:

LISSOM’s activation

Consisting of several layers following the torch.nn.Module interface. Found in pylissom.nn.modules.

LISSOM’s hebbian learning mechanism and others

Implemented following the torch.optim.Optimizer interface. Found in pylissom.optim.

Configuration and model building tools

Make it easy to track and change layer’s hyperparameters. Based in the configobj and yaml config libraries. Examples of config files and code can be found in pylissom.models and pylissom.utils.config.

Common Guassian stimuli for LISSOM experiments

Following the torch.utils.data.Dataset interface. Uses the popular scikit-image and cv2 libs. Found in pylissom.datasets and pylissom.utils.stimuli module.

Plotting helpers

For displaying LISSOM layers weights and activations mainly in Jupyter Notebooks. Uses matplotlib. Found in pylissom.utils.plotting.

Training objects for simplifying torch boilerplate.

Found in pylissom.utils.training.pipeline module.

UML diagrams

I present an auto-generated UML class diagram of the main classes in pylissom, starting with the modules and supervised learning classes. The arrows with hollow triangle tips represent inheritance, and the black filled diamond tips represent the composition relationships, with the parameters name in the containing class in green.

WARNING: the plot was drawn statically so it may be out of date with the source code.

Modules (neural layers) classes

At the top, we see that all classes inherit from Module, the pytorch abstract class for network layers. Reading from the left side of the plot we see the Linear module, also a pytorch class, which only applies a linear transformation to the incoming data: \(y = Ax + b\).

I implemented the LISSOM connection fields inheriting from such Linear module, in the DifferenceOfGaussiansLinear, GaussianLinear, GaussianCloadLinear and Cortex classes.

A Composition design pattern was used for the higher and lower level layers. Complex classes receive as parameters for initialization simpler layers and thus are associated with them in the UML diagram. The higher level classes inherit from the Module base class, such as Lissom, LGN, and ReducedLissom, and are associated to the linear modules and the PiecewiseSigmoid class that implements the LISSOM activation.

For example, the ReducedLissom is the name given in by the LISSOM creators to the brain V1, which has afferent, excitatory and inhibitory modules, being Cortex classes. And the Lissom class is the V1 plus two LGN maps representing the on and off channels of the brain Visual Cortex. In the figure are also the OrientationMap class, and the Pipeline class.

In the next Fig. are the dataset, configuration and optimizers classes:

Optimizer, dataset and configuration classes

Reading from the left, we see the pytorch Dataset abstract class, representing the input data of the neural network models. Dataset is a python iterable, the reason why it doesn’t have any attributes or functions. I provide the OrientedGaussians and ThreeDotFaces as datasets and other utils that I used that a Lissom researcher will find useful as a starting point, representing common stimuli for a Lissom network.

Next to the right, we see the classes that solve the configuration problem of LISSOM. The EvalConfigObj and EvalConfigYaml are the two classes I provide to the end user, extending the yaml and ConfigObj libraries, and are used by default in the canned models package I also provide.

And finally, the optimizer classes implement Lissom learning. The pytorch Optimizer abstract base class is the interface to neural layer learning. The CortexHebbian class is the most important, it implements the Lissom learning with the Hebbian rule for a set of neural connections such as the afferent, excitatory or inhibitory.

In consequence, with three CortexHebbian we can have a ReducedLissom (V1) optimizer, which I called ReducedLissomHebbian. The ConnectionDeath and NeighborsDecay are other optimizers I provide that are common to LISSOM.

The Visual Cortex

Brief review of the visual cortex and the visual maps in the cortex.

Human visual system

During visual perception, light entering the eye is detected by the retina, an array of photoreceptors and related cells on the inside of the rear surface of the eye. The cells in the retina encode the light levels at a given location as patterns of electrical activity in neurons called ganglion cells. Output from the ganglion cells travels through neural connections to the lateral geniculate nucleus of the thalamus (LGN). From the LGN, the signals continue to the primary visual cortex (V1). V1 is the first cortical site of visual processing; the previous areas are termed subcortical. The output from V1 goes on to many different higher cortical areas, including areas that underlie object and face processing.

Human visual pathways (top view). Diagram of the main feedforward pathways in the human visual system.

Lateral Geniculate Nucleus (LGN)

The Lateral Geniculate Nucleus of the thalamus is the first step of visual processing. The LGN’s neurons do a process akin to edge detection between bright and dark areas. Some neurons at the LGN prefer bright over dark, or vice versa, being called ON or OFF cells.

ON cell in retina or LGN

OFF cell in retina or LGN

Primary Visual Cortex (V1)

The V1 is the first cortical site of visual processing. Input from the thalamus goes through afferent connections to V1, and the output goes on to many different higher cortical areas, including areas that underlie object and face processing. The neurons form local connections within V1 (long-range lateral connections) or connect to higher visual processing areas.

The cells themselves can prefer several features, for example, preferring one eye or the other, the orientation of the stimulus and its direction of movement, color combinations(such as red/green or blue/yellow borders), disparity (relative positions on the two retinas), etc.

Receptive field in V1. Starting in V1, most cells in primates have orientation-selective RFs. The V1 RFs can be classified into a few basic spatial types. The figure shows a two-lobe arrangement, favoring a \(45^\circ\) edge with dark in the upper left and light in the lower right

At a given location on the cortical sheet, the neurons in a vertical section through the cortex respond most strongly to the same eye of origin, stimulus orientation, spatial frequency, and direction of movement. It is customary to refer to such a section as a column . Nearby columns generally have similar, but not identical, preferences; slightly more distant columns have more dissimilar preferences. Preferences repeat at regular intervals (approximately 1–2 mm) in every direction, which ensures that each type of preference is represented across the retina. This arrangement of preferences forms a smoothly varying map for each dimension.

Visual maps

The term visual map refers to the existence of a non-random relationship between the positions of neurons in the visual centers of the brain (e.g. in the visual cortex) and the values of one or more of the receptive field properties of those neurons . The term is usually qualified by reference to the property concerned. For example, stimulus orientation is represented across the cortex in an orientation map of the retinal input. In an orientation map, each location on the retina is mapped to a region on the map, with each possible orientation at that retinal location represented by different but nearby orientation-selective cells. The figure below displays an example orientation map from monkey cortex.

Orientation map in the macaque. Orientation preference map observed in area V1 of macaque monkey. Each neuron is colored according to the orientation it prefers, using the color key on the right. Nearby neurons in the map generally prefer similar orientations, forming groups of the same color called iso-orientation patches. Scale bar = 1 mm.

Lissom

The LISSOM model focuses on V1 and the structures to which it connects. This note explains the key concepts of the LISSOM networks: its architecture, activation and learning rule.

LISSOM principles

LISSOM is a laterally connected map model of the Primary Visual Cortex . The model is based on self-organizing maps, but its connectivity, activation, and learning mechanisms are designed to capture the essential biological processes in more detail.

LISSOM is based on five principles:

1. The central layout of the LISSOM model is a two-dimensional array of computational units, corresponding to vertical columns in the cortex. Such columns act as functional units in the cortex, responding to similar inputs, and therefore form an appropriate level of granularity for a functional model.
2. Each unit receives input from a local anatomical receptive field in the retina, mediated by the ON-center and OFF-center channels of the LGN. Such connectivity corresponds to the neural anatomy; it also allows modeling a large area of the visual cortex and processing large realistic visual inputs, which in turn allows studying higher visual function such as visual illusions, grouping, and face detection.
3. The cortical units are connected with excitatory and inhibitory lateral connections that adapt as an integral part of the self-organizing process.
4. The units respond by computing a weighted sum of their input, limited by a logistic (sigmoid) nonlinearity. This is a standard model of computation in the neuronal units that matches their biological characteristics well.
5. The learning is based on Hebbian adaptation with divisive normalization. Hebbian learning is well supported by neurobiological data and biological experiments have also suggested how normalization could occur in animals.

In other words, LISSOM takes the central idea of self-organizing maps (1), and implements it at a level of known visual cortex structures (2 and 3) and processes (4 and 5). Although each of these principles has been tested in other models, their combination is novel and allows LISSOM to account for a wide range of phenomena in the development, plasticity, and function of the primary visual cortex.

The LISSOM model

Basic LISSOM model of the primary visual cortex. The core of the LISSOM model consists of a two-dimensional array of computational units representing columns in V1. These units receive input from the retinal receptors through the ON/OFF channels of the LGN, and from other columns in V1 through lateral connections. The solid circles and lines delineate the receptive fields of two sample units in the LGN and one in V1, and the dashed circle in V1 outlines the lateral connections of the V1 unit. The LGN and V1 activation in response to a sample input on the retina is displayed in gray-scale coding from white to black (low to high).

The V1 network in LISSOM is a sheet of N x N interconnected computational units, or “neurons”. Because the focus is on the two-dimensional organization of the cortex, each neuron in V1 corresponds to a vertical column of cells through the six layers of the biological cortex. This columnar organization helps make the problem of simulating such a large number of neurons tractable, and is viable because the cells in a column generally fire in response to the same inputs. The activity of each neuron is represented by a continuous number within [0..1]. Therefore, it is important to keep in mind that LISSOM neurons are not strictly identifiable with single cells in the biological cortex; instead, LISSOM models biological mechanisms at an aggregate level.

Each cortical neuron receives external input from two types of neurons in the LGN: ON-center and OFF-center. The LGN neurons in turn receive input from a small area of the retina, represented as an R x R array of photoreceptor cells. The afferent input connections from the retina to LGN and LGN to V1 are all excitatory. In addition to the afferent connections, each cortical neuron has reciprocal excitatory and inhibitory lateral connections with other neurons. Lateral excitatory connections have a short range, connecting only close neighbors in the map. Lateral inhibitory connections run for long distances, but may be patchy, connecting only selected neurons.

The ON and OFF neurons in the LGN represent the entire pathway from photoreceptor output to the V1 input, including the ON/OFF processing in the retinal ganglion cells and the LGN. Although the ON and OFF neurons are not always physically separated in the biological pathways, for conceptual clarity they are divided into separate channels in LISSOM. Each of these channels is further organized into an L x L array corresponding to the retinotopic organization of the LGN. For simplicity and computational efficiency, only single ON and OFF channels are used in LISSOM, but multiple channels could be included to represent different spatial frequencies. Also, the photoreceptors are uniformly distributed over the retina; since the inputs are relatively small in the most common LISSOM experiments, the fovea/periphery distinction is not crucial for the basic model.

Each neuron develops an initial response as a weighted sum (scalar product) of the activation in its afferent input connections. The lateral interactions between cortical neurons then focus the initial activation pattern into a localized response on the map. After the pattern has stabilized, the connection weights of cortical neurons are modified. As the self-organization progresses, these neurons grow more nonlinear and weak connections die off. The result is a self-organized structure in a dynamic equilibrium with the input.

The following subsections describe the specific components of the LISSOM model in more detail. They focus on the basic version of the model trained with unoriented Gaussian inputs, to highlight the basic principles as clearly as possible.

Connections to the LGN

LISSOM focuses on learning at the cortical level, so all connections to neurons in the ON and OFF channels are set to fixed strengths.

The strengths were chosen to approximate the receptive fields that have been measured in adult LGN cells, using a standard difference-of-Gaussians model. First, the center of each LGN receptive field is mapped to the location in the retina corresponding to the location of the LGN unit. This mapping ensures that the LGN will have the same two-dimensional topographic organization as the retina. Using that location as the center, the weights are then calculated from the difference of two normalized Gaussians. More precisely, the weight \(L_{xy,ab}\) from receptor (x, y) in the receptive field of an ON-center cell (a, b) with center \((x_c,y_c)\) is given by the following equation, where \(\sigma_c\) determines the width of the central Gaussian and \(\sigma_s\) the width of the surround Gaussian:

\[L_{xy,ab}=\frac{\exp(-\frac{(x-x_c)^2+(y-y_c)^2}{\sigma_c^2})}{\sum_{uv}\exp(-\frac{(u-x_c)^2+(v-y_c)^2}{\sigma_c^2})} - \frac{\exp(-\frac{(x-x_c)^2+(y-y_c)^2}{\sigma_s^2})}{\sum_{uv}\exp(-\frac{(u-x_c)^2+(v-y_c)^2}{\sigma_s^2})}\]

The weights for an OFF-center cell are the negative of the ON-center weights, i.e. they are calculated as the surround minus the center. shows examples of such ON and OFF receptive fields. Note that even though the OFF cells have the same weights as ON cells (differing only by the sign), their activities are not redundant. Since the firing rates in biological systems cannot be negative, each cell is thresholded to have only positive activations. As a result, the ON and OFF cells will never be active at the same cortical location. They therefore provide complementary information, both in the model and in the visual system. Separating the ON and OFF channels in this way makes it convenient to compare the model with experimental results.

ON neuron

OFF neuron

Connections in the Cortex

In contrast to the fixed connection weights in the LGN, all connections in cortical regions in LISSOM are modifiable by neural activity. They are initialized according to the gross anatomy of the visual cortex, with weight values that provide a neutral starting point for self-organization.

Each neuron’s afferent receptive field center is located randomly within a small radius of its optimal position, i.e. the point corresponding to the neuron’s location in the cortical sheet. The neuron is connected to all ON and OFF neurons within radius rA from the center. For proper self-organization to occur, the radius rA must be large compared with the scatter of the centers, and the RFs of neighboring neurons must overlap significantly, as they do in the cortex.

Lateral excitatory connections are short range, connecting each neuron to itself and to its neighbors within a close radius. The extent of lateral excitation should be comparable to the activity correlations in the input. Lateral inhibitory connections extend in a larger radius, and also include connections from the neuron itself and from its neighbors. The range of lateral inhibition may vary as long as it is greater than the excitatory radius. This overall center–surround pattern is crucial for self-organization, and approximates the lateral interactions that take place at high contrasts in the cortex.

Initial V1 afferent and lateral weights. The initial incoming weights of a sample neuron at the center of V1 are plotted in gray-scale coding from white to black (low to high).

Response Generation

Before each input presentation, the activities of all units in the LISSOM network are initialized to zero. The system then receives input through activation of the retinal units. The activity propagates through the ON and OFF channels of the LGN to the cortical network, where the neurons settle the initial activation through the lateral connections, as will be described in detail below.

Retinal Activation

An input pattern is presented to the LISSOM model by activating the photoreceptor units in the retina according to the gray-scale values in the pattern. shows a basic input pattern consisting of multiple unoriented Gaussians. To generate such input patterns, the activity for photoreceptor cell (x, y) is calculated according to:

\[\chi_{xy} = \max_{k} \exp (-\frac{(x-x_{c,k})^2+(y-y_{c,k})^2}{\sigma^2_u})\]

where \((x_{c,k},y_{c,k})\) specifies the center of Gaussian \(k\) and \(\sigma_u\) its width. At each iteration, \(x_{c,k}\) and \(y{c,k}\) are chosen randomly within the retinal area; \(\sigma_u\) is usually constant.

Example input and response. At each self-organization iteration in LISSOM, the photoreceptors in the retina are activated with two unoriented Gaussians.

LGN Activation

The cells in the ON and OFF channels of the LGN compute their responses as a squashed weighted sum of activity in their receptive fields (). More precisely, the response \(\xi_{ab}\) of ON or OFF-center cell \((a, b)\) is calculated as

\[\label{fig:lgnactivation} \xi_{ab} = \sigma(\gamma_L \sum_{xy}\chi_{xy}L_{xy,ab})\]

where \(X_{xy}\) is the activation of cell \((x, y)\) in the receptive field of \((a, b)\), \(L_{xy,ab}\) is the afferent weight from \((x, y)\) to \((a, b)\), and \(\gamma_L\) is a constant scaling factor. The squashing function \(\sigma(\cdot)\) () is a piecewise linear approximation of the sigmoid activation function:

\[\begin{split}\sigma(s) = \begin{cases} 0 & s \leq \theta_l\\ (s-\theta_l)/(\theta_u - \theta_l) & \theta_l < s < \theta_u\\ 1 & s \geq \theta_u \end{cases}\end{split}\]

As in other models, this approximation is used because it implements the essential thresholding and saturation behavior, and can be computed more quickly than a smooth logistic function.

Changing \(\gamma_L\) in by a factor \(m\) is equivalent to dividing \(\Theta_l\) and \(\Theta_u\) by \(m\). Even so, \(\gamma_L\) is treated as a separate parameter to make it simpler to use the same values of \(\Theta_l\) and \(\Theta_u\) for different networks. The specific value of \(\gamma_L\) is set manually so that the LGN outputs approach 1.0 in the highest-contrast regions of typical input patterns. This allows each subsequent level to use similar parameter values in general, other than \(\gamma_L\).

Because of its DoG-shaped receptive field, an LGN neuron will respond whenever the input pattern is a better match to the central portion of the RF than to the surrounding portion. The positive and negative portions of the RF thus have a push– pull effect. That is, even if an input pattern activates the ON portion of the LGN RF, the neuron will not fire unless the OFF portion is not activated. This balance ensures that the neurons will remain selective for edges over a wide range of brightness levels. This push–pull effect is crucial when natural images are used as input to the model. Overall, the LGN neurons respond to image contrast, subject to the minimum and maximum activity values enforced by the activation function.

Neuron activation function :math:`sigma(s)`. The neuron requires an input as large as the threshold \(\sigma_l\) before responding, and saturates at the ceiling \(\sigma_u\). The output activation values are limited to [0..1]. This activation function is an efficient approximation of the logistic (sigmoid) function.

Cortical Activation

The cortical activation mechanism is similar to that of the LGN, but extended to support self-organization and to include lateral interactions. The total activation is computed by combining the afferent and lateral contributions. First, the afferent stimulation \(s_{ij}\) of V1 neuron \((i, j)\) is calculated as a weighted sum of activations in its receptive fields on the LGN:

\[\label{fig:corticalactivation} s_{ij} = \gamma_A\left(\sum_{ab \in \,ON} \xi_{ab}A_{ab,ij} + \sum_{ab \in \,OFF} \xi_{ab}A_{ab,ij}\right)\]

where \(\xi_{ab}\) is the activation of neuron (a, b) in the receptive field of neuron (i, j) in the ON or OFF channels, \(A_{ab,ij}\) is the corresponding afferent weight, and \(\gamma_A\) is a constant scaling factor. The afferent stimulation is squashed using the sigmoid activation function, forming the neuron’s initial response as

\[\eta_{ij}(0) = \sigma (s_{ij})\]

After the initial response, lateral interaction sharpens and strengthens the cortical activity over a very short time scale. At each of these subsequent discrete time steps, the neuron combines the afferent stimulation \(s\) with lateral excitation and inhibition:

\[\label{fig:steptcorticalactivation} \eta_{ij}(t) = \sigma \left(s_{ij} + \gamma_E\sum_{kl} \eta_{kl}(t-1)E_{kl,ij} - \gamma_I\sum_{kl} \eta_{kl}(t-1)I_{kl,ij}\right)\]

where \(\eta_{kl}(t - 1)\) is the activity of another cortical neuron (k, l) during the previous time step, \(E_{kl,ij}\) is the excitatory lateral connection weight on the connection from that neuron to neuron (i, j), and \(I_{kl,ij}\) is the inhibitory connection weight. All connection weights have positive values. The scaling factors \(\gamma_E\) and \(\gamma_I\) represent the relative strengths of excitatory and inhibitory lateral interactions, which determine how easily the neuron reaches full activation.

The cortical activity pattern starts out diffuse and spread over a substantial part of the map (). Within a few iterations of , it converges into a small number of stable focused patches of activity, or activity bubbles (). Such settling results in a sparse final activation, which allows representing visual information efficiently. It also ensures that nearby neurons have similar patterns of activity and therefore encode similar information, as seen in the cortex.

Learning

Self-organization of the connection weights takes place in successive input iterations. Each iteration consists of presenting an input image, computing the corresponding settled activation patterns in each neural sheet, and modifying the weights.Weak lateral connections are periodically removed, modeling connection death in biological systems.

Weight Adaptation

After the activity has settled, the connection weights of each cortical neuron are modified. Both the afferent and lateral weights adapt according to the same biologically motivated mechanism: the Hebb rule with divisive postsynaptic normalization:

\[\label{fig:hebbianrule} w'_{pq,ij} = \frac{w_{pq,ij} + \alpha X_{pq}\eta_{ij}}{\sum_{uv}(w_{uv,ij} + \alpha X_{uv}\eta_{ij})}\]

where \(w_{pq,ij}\) is the current afferent or lateral connection weight (either \(A\), \(E\) or \(I\)) from (p, q) to (i, j), \(w'_{pq,ij}\) is the new weight to be used until the end of the next settling process, \(\alpha\) is the learning rate for each type of connection (\(\alpha_A\) for afferent weights, \(\alpha_E\) for excitatory, and \(\alpha_I\) for inhibitory), \(X_{pq}\) is the presynaptic activity after settling (\(\xi\) for afferent, \(\eta\) for lateral), and \(\eta_{ij}\) stands for the activity of neuron (i, j) after settling. Afferent inputs (i.e. both ON and OFF channels together), lateral excitatory inputs, and lateral inhibitory inputs are normalized separately.

In line with the Hebbian principle, when the presynaptic and postsynaptic neurons are frequently simultaneously active, their connection becomes stronger. As a result, the neurons learn correlations in the input patterns. Normalization prevents the weight values from increasing without bounds; this process corresponds to redistributing the weights so that the sum of each weight type for each neuron remains constant. Such normalization can be seen as an abstraction of neuronal regulatory processes.

Connection Death

Modeling connection death in the cortex, lateral connections in the LISSOM model survive only if they represent significant correlations among neuronal activity. Once the map begins to organize, most of the long-range lateral connections link neurons that are no longer simultaneously active. Their weights become small, and they can be pruned without disrupting self-organization.

The parameter \(t_d\) determines the onset of connection death. At \(t_d\), lateral connections with strengths below a threshold wd are eliminated. From \(t_d\) on, more weak connections are eliminated at intervals \(\Delta t_d\) during the self-organizing process. Eventually, the process reaches an equilibrium where the mapping is stable and all lateral weights stay above \(w_d\). The precise rate of connection death is not crucial to selforganization, and in practice it is often sufficient to prune only once, at \(t_d\).

Most long-range connections are eliminated this way, resulting in patchy lateral connectivity similar to that observed in the visual cortex. Since the total synaptic weight is kept constant, inhibition concentrates on the most highly correlated neurons, resulting in effective suppression of redundant activation. The short-range excitatory connections link neurons that are often part of the same bubble. They have relatively large weights and are rarely pruned.

Parameter Adaptation

The above processes of response generation, weight adaptation, and connection death are sufficient to form ordered afferent and lateral input connections like those in the cortex. However, the process can be further enhanced with gradual adaptation of lateral excitation, sigmoid, and learning parameters, resulting in more refined final maps.

As the lateral connections adapt, the activity bubbles in the cortex will become more focused, resulting in fine-tuning the map. As in other self-organizing models (such as SOM), this process can be accelerated by gradually decreasing the excitatory radius until it covers only the nearest neighbors. Such a decrease helps the network develop more detailed organization faster.

Gradually increasing the sigmoid parameters \(\Theta_l\) and \(\Theta_u\) produces a similar effect. The cortical neurons become harder to activate, further refining the response. Also, the learning rates \(\alpha_A\), \(\alpha_E\) and \(\alpha_I\) can be gradually reduced.

Such parameter adaptation models the biological processes of maturation that take place independently from input-driven self-organization, leading to loss of plasticity in later life.

Supervised Learning

In addition to providing a precise understanding of the mechanisms underlying visual processing in the brain, LISSOM can serve as a foundation for artificial vision systems. Such systems have the advantage that they are likely to process visual information the same way humans do, which makes them appropriate for many practical applications.

First, LISSOM networks can be used to form efficient internal representations for pattern recognition applications. A method must be developed for automatically identifying active areas in the maps and assigning labels to neural populations that respond to particular stimulus features. One particularly elegant approach is to train another neural network to do the interpretation. By adding backprojections from the interpretation network back to the map, a supervised process could be implemented. The backprojections learn which units on the map are statistically most likely to represent the category; they can then activate the correct LISSOM units even for slightly unusual inputs, resulting in more robust recognition.

Second, a higher level network (such as multiple hierarchically organized LISSOM networks) can serve for object recognition and scene analysis systems, performing rudimentary segmentation and binding. Object binding and object segmentation are thought to depend on specific long-range lateral interactions, so in principle a stacking network is an appropriate architecture for the task. At the lowest level, preliminary features such as contours would be detected, and at each successively higher level, the receptive fields cover more area in the visual space, eventually representing entire objects. A high-level recognition system could then operate on these representations to perform the actual object recognition and scene interpretation.

pylissom

pylissom.math module

This module contains auxiliary math functions.

pylissom.math.euclidian_distances(x, y, mu_x, mu_y)

This function implements the euclidean distance between two 2-dimensional vectors.

Parameters:

• x – first element of the first vector
• y – second element of the first vector
• mu_x – first element of the second vector
• mu_y – second element of the second vector

Returns:

Euclidean distance

pylissom.math.euclidean_distance_general(x, y)

This function implements the euclidean distance between two n-dimensional vectors as numpy arrays.

Parameters:

• x – First vector (numpy array)
• y – Second vector (numpy array)

Returns:

euclidean distance

pylissom.math.gaussian(x, y, mu_x, mu_y, sigma, sigma_y=None)

This function implements a circular gaussian function.

Parameters:

• x –
• y –
• mu_x – Center
• mu_y – Center
• sigma –
• sigma_y –

Returns:

Gaussian

pylissom.math.normalize(matrix, norm=1, axis=1)

This function implements a normalization of the row or column vectors of a matrix (by default, normalizes the columns and uses norm 1).

Parameters:

• matrix – input matrix
• norm – Dimension of the norm
• axis – 0 is column, 1 is row

Returns:

A matrix normalized by columns or rows

pylissom.datasets

Submodules

pylissom.datasets.datasets module

Extends torchvision.datasets with two common Lissom stimuli, Oriented Gaussians and “Gaussian” Faces

pylissom.datasets.datasets.get_dataset(train, args)

class pylissom.datasets.datasets.RandomDataset(length)

Bases: torch.utils.data.Dataset

Abstract Dataset representing random samples, subclasses must implement pylissom.datasets.RandomDataset._gen()

class pylissom.datasets.datasets.OrientatedGaussians(size, length, gaussians=2)

Bases: pylissom.datasets.datasets.RandomDataset

Dataset of random Oriented Gaussians samples, as used in Computional Maps in the Visual Cortex

class pylissom.datasets.datasets.ThreeDotFaces(size, length, faces=2)

Bases: pylissom.datasets.datasets.RandomDataset

Dataset of random Faces made of Three Gaussians Disks, as used in Computional Maps in the Visual Cortex

class pylissom.datasets.datasets.CKDataset(path_images='/home/hbari/data/X.npy', path_labels='/home/hbari/data/y.npy', path_subjects='/home/hbari/data/subjs.npy')

Bases: torch.utils.data.Dataset

pylissom.datasets.datasets.train_test_ck_samplers(ck_dataset, train, train_pct=0.5)

pylissom.datasets.datasets.subj_indep_train_test_samplers(subjs, pct)

pylissom.models

Submodules

pylissom.models.models module

Extens model_zoo and torchvision.models with some functions that read a config file and output some Lissom Modules, i.e. Lissom, ReducedLissom or LGN maps

pylissom.models.models.get_reduced_lissom(retinal_density='DEFAULT', cortical_density='DEFAULT', rlissom_params='rlissom', optim_params='optim', cfg_path=None)

pylissom.models.models.get_lgn(retinal_density='DEFAULT', lgn_density='DEFAULT', on=False, lgn_params='lgn', cfg_path=None)

pylissom.models.models.get_lissom(retinal_density='DEFAULT', lgn_density='DEFAULT', cortical_density='DEFAULT', lgn_params='lgn', rlissom_params='rlissom', optim_params='optim', cfg_path=None)

pylissom.models.models.get_net(net_input_shape, classes)

pylissom.models.models.get_supervised(retinal_density='DEFAULT', lgn_density='DEFAULT', cortical_density='DEFAULT', lgn_params='lgn', rlissom_params='rlissom', optim_params='optim', cfg_path=None, classes=10)

pylissom.nn.functional

Submodules

pylissom.nn.functional.functions module

pylissom.nn.functional.functions.linear_decay(w, start, epoch, final_epoch)

pylissom.nn.functional.functions.kill_neurons(w, threshold)

pylissom.nn.functional.functions.piecewise_sigmoid(min_theta, max_theta, inp)

pylissom.nn.functional.functions.check_compatible_mul(module_one, module_two)

Checks that two modules have correct sizes for matrix multiplication

pylissom.nn.functional.functions.check_compatible_add(module_one, module_two)

Checks that two modules have correct sizes for matrix addition

pylissom.nn.functional.weights module

This module contains functions that modify the weights of the neural network.

pylissom.nn.functional.weights.apply_fn_to_weights_between_maps(in_features, out_features, fn, **kwargs)

The goal of this function is to apply a function fn, to all the elements of an array of dimension rows_dims_source x rows_dims_source (the lower array) centered on an element of the superior array. The elements of the array would be the weights of the superior layer, with the inferior layer, i.e., it modifies the weights of each one of the neurons of the superior layer, with respect to all the neurons of the inferior layer.

ASSUMES SQUARE MAPS PROBLEMAS? OJO QUE EL STEP PUEDE SER UN FLOAT

Parameters:

• in_features –
• out_features –
• fn – The function applied to the weights
• **kwargs – Optional parameters to fn

Returns:

An array containing the new weights of the superior layer.

pylissom.nn.functional.weights.get_gaussian_weights

Returns a Tensor of size in_features x out_features with each column weight representing a Gaussian Disk

pylissom.nn.functional.weights.circular_mask

Creates a boolean mask representing valid connective radius

pylissom.nn.functional.weights.apply_circular_mask_to_weights(matrix, radius)

This functions applies a circular mask to a matrix of weights. The weights of the neurons that are more far than the radius, will have its weight set to zero.

Parameters:

• matrix – Tensor of weights. The rows are the neurons. The columns the weights of the neuron.
• radius – The radius of neighborhood.

Returns:

pylissom.nn.modules

Extends torch.nn with Lissom layers, split in the simpler Linear module and the higher-level Lissom module

pylissom.nn.modules.register_recursive_forward_hook(module, hook)

Adds a forward hook to all modules in module

pylissom.nn.modules.named_apply(mod, fn, prefix)

Like torch.nn.Module.apply() but with named children

pylissom.nn.modules.input_output_hook(module, input, output)

pylissom.nn.modules.register_recursive_input_output_hook(module)

Adds a hook to module so it saves in memory input and output in each forward pass

Submodules

pylissom.nn.modules.linear module

class pylissom.nn.modules.linear.GaussianLinear(in_features, out_features, sigma=1.0)

Bases: torch.nn.Linear

Applies a linear transformation to the incoming data: \(y = Ax + b\)

where A is a Gaussian matrix

Parameters:sigma - (-) –

class pylissom.nn.modules.linear.GaussianCloudLinear(in_features, out_features, sigma=1.0)

Bases: pylissom.nn.modules.linear.GaussianLinear

Applies a linear transformation to the incoming data: \(y = Ax + b\)

where A is a Gaussian matrix multiplied with Gaussian Noise

Parameters:sigma - (-) –

class pylissom.nn.modules.linear.PiecewiseSigmoid(min_theta=0.0, max_theta=1.0)

Bases: torch.nn.Module

Applies a piecewise approximation of the sigmoid function \(f(x) = 1 / ( 1 + exp(-x))\)

The formula is as follows: TODO :param - min_theta -: :param - max_theta -:

forward(input)

class pylissom.nn.modules.linear.UnnormalizedDifferenceOfGaussiansLinear(in_features, out_features, on, sigma_surround, sigma_center=1.0)

Bases: torch.nn.Linear

NOT USED, only for example in notebooks

pylissom.nn.modules.lissom module

class pylissom.nn.modules.lissom.Cortex(in_features, out_features, radius, sigma=1.0)

Bases: pylissom.nn.modules.linear.GaussianCloudLinear

Applies a linear transformation to the incoming data: \(y = Ax + b\)

where A is a Gaussian Cloud with a connective radius

This module is primarily used to build a :py:class`ReducedLissom`

Parameters:

• radius - (-) –
• sigma - (-) –

class pylissom.nn.modules.lissom.DifferenceOfGaussiansLinear(in_features, out_features, on, radius, sigma_surround, sigma_center=1.0)

Bases: torch.nn.Linear

Applies a linear transformation to the incoming data: \(y = Ax + b\), where A is a Difference of Gaussians with a connective radius:

\[\begin{equation*} \text{out}_ab = \sigma(\phi_L \sum_(xy) \text{input}_xy L_xy,ab) \end{equation*}\]

Parameters:

• on - Defines if the substraction goes sorround gaussian - center gaussian or the other way around (-) –
• radius - (-) –
• sigma_surround - (-) –
• sigma_center - (-) –

class pylissom.nn.modules.lissom.Mul(number)

Bases: torch.nn.Module

Represents a layer than only multiplies the input by a constant, used in pylissom.nn.modules.LGN

forward(input)

class pylissom.nn.modules.lissom.LGN(in_features, out_features, on, radius, sigma_surround, sigma_center=1.0, min_theta=0.0, max_theta=1.0, strength=1.0, diff_of_gauss_cls=<class 'pylissom.nn.modules.lissom.DifferenceOfGaussiansLinear'>, pw_sigmoid_cls=<class 'pylissom.nn.modules.linear.PiecewiseSigmoid'>)

Bases: torch.nn.Sequential

Represents an LGN channel, can be ON or OFF

The transformation applied can be described as:

\[\begin{equation*} \text{out}_ab = \sigma(\phi_L \sum_(xy) \text{input}_xy L_xy,ab) \end{equation*}\]

where \(\sigma\) is the piecewise sigmoid, \(N\) is foo

It inherits from Sequential because an LGN is in essence a composition of several transformations

• afferent_module

Parameters:

• on - (-) –
• radius - (-) –
• sigma_surround - (-) –
• sigma_center - (-) –
• strength - (-) –
• min_theta - (-) –
• max_theta - (-) –

class pylissom.nn.modules.lissom.ReducedLissom(afferent_module, excitatory_module, inhibitory_module, min_theta=1.0, max_theta=1.0, settling_steps=10, afferent_strength=1.0, excitatory_strength=1.0, inhibitory_strength=1.0, pw_sigmoid_cls=<class 'pylissom.nn.modules.linear.PiecewiseSigmoid'>)

Bases: torch.nn.Module

Represents a Reduced Lissom consisting of afferent, excitatory and inhibitory modules

The transformation applied can be described as:

\[\begin{equation*} n_ij = \sigma(s_ij + \phi_E \sum_(kl) n_kl (t-1) E_kl,ij - \phi_I \sum_(kl) n_kl (t-1) E_kl,ij I_lk,ij) \end{equation*}\]

where \(\sigma\) is the piecewise sigmoid, \(N\) is foo

• afferent_module

Parameters:

• afferent_module - (-) –
• excitatory_module - (-) –
• inhibitory_module - (-) –
• afferent_strength - (-) –
• excitatory_strength - (-) –
• inhibitory_strength - (-) –
• min_theta - (-) –
• max_theta - (-) –
• settling_steps - (-) –

forward(cortex_input)

class pylissom.nn.modules.lissom.Lissom(on, off, v1)

Bases: torch.nn.Module

Represents a Full Lissom, with ON/OFF channels and a V1 ( ReducedLissom )

The transformation applied can be described as:

\[\begin{equation*} \text{out} = \text{v1}(\text{on}(input) + \text{off}(input)) \end{equation*}\]

:param - on - an ON LGN map: :param - off - an OFF LGN map: :param - v1 - a ReducedLissom map:

Shape:

• TODO

forward(input)

pylissom.optim

Submodules

pylissom.optim.optimizers module

Extends the torch.optim.Optimizer class with Lissom optimizers, mainly Hebbian Learning

class pylissom.optim.optimizers.CortexOptimizer(cortex)

Bases: torch.optim.Optimizer

Abstract torch.optim.Optimizer that can only be used with pylissom.nn.modules.Cortex

class pylissom.optim.optimizers.SequentialOptimizer(*optimizers)

Bases: object

Similar to torch.nn.Sequential but for optimizers, used to contain pylissom.optim.optimizers.CortexHebbian for ReducedLissom modules

step()

zero_grad()

class pylissom.optim.optimizers.CortexHebbian(cortex, learning_rate)

Bases: pylissom.optim.optimizers.CortexOptimizer

Implements hebbian learning over a pylissom.nn.modules.Cortex weights

The formula is as follows:

..math:

\begin{equation*}
\text{w\'}_pq,ij = \text{w\'}_pq,ij + \alpha X_pq n_ij
\end{equation*}

:param - cortex - pylissom.nn.modules.Cortex map to apply formula: :param - learning_rate -:

step(**kwargs)

class pylissom.optim.optimizers.ReducedLissomHebbian(afferent_mod, inhibitory_mod, excitatory_mod, aff_params, inhib_params, excit_params)

Bases: pylissom.optim.optimizers.SequentialOptimizer

class pylissom.optim.optimizers.CortexPruner(cortex, pruning_step=2000)

Bases: pylissom.optim.optimizers.CortexOptimizer

Abstract class that prunes the weights in each step, subclasses must implement pylissom.optim.optimizers.CortexPruner._prune()

:param - cortex - pylissom.nn.modules.Cortex map to apply formula: :param - pruning_step -:

step(**kwargs)

class pylissom.optim.optimizers.ConnectionDeath(cortex, pruning_step=2000, connection_death_threshold=0.0025)

Bases: pylissom.optim.optimizers.CortexPruner

Prunes the weights that are less than a threshold

:param - cortex - pylissom.nn.modules.Cortex map to apply formula: :param - pruning_step -: :param - connection_death_threshold -:

class pylissom.optim.optimizers.NeighborsDecay(cortex, pruning_step=2000, decay_fn=<function linear_decay>, final_epoch=8.0)

Bases: pylissom.optim.optimizers.CortexPruner

Reduces the connective radius of each neuron

:param - cortex - pylissom.nn.modules.Cortex map to apply formula: :param - pruning_step -: :param - decay_fn - Default = linear_decay: :param - final_epoch - necessary for the linear function:

pylissom.utils

Submodules

pylissom.utils.helpers module

pylissom.utils.helpers.save_model(model, optimizer=None, fname='best_model.pth.tar')

pylissom.utils.helpers.load_model(fname='model.pth.tar')

pylissom.utils.helpers.debug()

Calls a debugger that works with jupyter notebooks

class pylissom.utils.helpers.TimeIt

Bases: object

At instantiation starts a timer and prints value when end() is called

end()

pylissom.utils.orientation_maps module

Provides some helpers to calculate Orientation Preferences of a Lissom Network

class pylissom.utils.orientation_maps.OrientationMap(model, inputs, use_tqdm_notebook=True)

Bases: object

maximum_activations(model, inputs)

calculate_keys_activations(model, inputs)

get_orientation_map

static orientation_hist(orientation_map)

get_orientation_hist

pylissom.utils.orientation_maps.plot_orientation_map(orientation_map)

pylissom.utils.orientation_maps.plot_orientation_hist(orientation_hist)

pylissom.utils.orientation_maps.metrics_orientation_hist(orientation_hist)

pylissom.utils.orientation_maps.get_oriented_lines(size, orientations=180)

pylissom.utils.orientation_maps.numpy_dict_to_tensors(d)

pylissom.utils.stimuli module

Provides several functions that create and manipulate matrices representing different stimuli, mainly guassians disks TODO: only use cv2 or scikit-image, not both

pylissom.utils.stimuli.translate(bar, step=1, over_x=True)

pylissom.utils.stimuli.random_translation(img, x_offset=5, y_offset=5)

pylissom.utils.stimuli.generate_horizontal_bar(size)

pylissom.utils.stimuli.generate_gaussian(shape, mu_x=0.0, mu_y=0.0, sigma_x=1.0, sigma_y=None)

pylissom.utils.stimuli.rotations(img, num=13)

Returns: Returns a dictionary of len 180 / num, representing {rotation_degrees: rotated_img}

pylissom.utils.stimuli.gaussian_generator(size, mu_x, mu_y, sigma_x, sigma_y, orientation)

Parameters:orientation – It’s actually redundant because orientation is a function of the sigmas, but make it easier to use

Returns: A numpy matrix representing a gaussian of shape = (size, size)

pylissom.utils.stimuli.generate_random_gaussian(size)

Returns: A numpy matrix with a random gaussian of shape = (size, size)

pylissom.utils.stimuli.random_gaussians_generator(size, gaussians=1)

Parameters:

• size – img will have shape = (size, size)
• gaussians – How many gaussians per matrix

Returns: Yields a squared numpy matrix with gaussians disks

pylissom.utils.stimuli.generate_random_faces(size)

pylissom.utils.stimuli.generate_three_dots(size, mu_x, mu_y, sigma_x, orientation)

Returns: A numpy matrix with 3 gaussian disks representing a face

pylissom.utils.stimuli.faces_generator(size, num=1)

Parameters:

• size – img will have shape = (size, size)
• gaussians – How many faces per matrix

Returns: Yields a squared numpy matrix with 3-gaussians faces

pylissom.utils.stimuli.combine_matrices(*matrices)

Returns: Merges matrices in one matrix using the maximum values in each pixel

pylissom.utils.stimuli.sine_grating(size, freq, phase)

pylissom.utils.config

Submodules

pylissom.utils.config.config module

Extends two common configuration libraries, configobj and yaml, to support execution of arbitrary code. This lets the user use mathematical formulas in their config file, common in Lissom experimentation.

class pylissom.utils.config.config.EvalConf

Bases: object

eval_dict()

class pylissom.utils.config.config.EvalConfigYaml(infile='/home/docs/checkouts/readthedocs.org/user_builds/pylissom/checkouts/latest/pylissom/utils/config/modules_params.yaml', user_values=None)

Bases: pylissom.utils.config.config.EvalConf

Uses yaml.add_constructor() to add arbitrary code execution to yaml file

eval_dict()

static anchors(kwargs)

class pylissom.utils.config.config.EvalConfigObj(infile=None, user_values=None, **kwargs)

Bases: configobj.ConfigObj, pylissom.utils.config.config.EvalConf

Inherits configobj.ConfigObj and adds arbitrary code execution to config file

eval_dict()

static recursive_eval(section)

pylissom.utils.config.config.global_config(conf_obj=True, *args, **kwargs)

pylissom.utils.plotting

Submodules

pylissom.utils.plotting.layers module

Some functions to plot specifically weights and activations of Lissom modules

pylissom.utils.plotting.layers.images_matrix(matrix, range_interval=None)

pylissom.utils.plotting.layers.weights_to_numpy_matrix(weights, values_range)

pylissom.utils.plotting.layers.plot_layer_weights(layer, use_range=True, recursive=False, prefix='')

Assumes layer parameters are weights and plots them :param layer: Plots weights (parameters) of this layer :param use_range: Normalize the image values between (0, 1) or (-1, 1) if layer is pylissom.nn.modules.lissom.DifferenceOfGaussiansLinear :param recursive: Plot weights of children modules recursively :param prefix: Title of plot

pylissom.utils.plotting.layers.simple_plot_layer_activation(layer, prefix='')

pylissom.utils.plotting.layers.plot_layer_activation(layer, prefix='')

Plots input and activation of layer and children modules recursively

Assumes layer has input and output parameters defined (probably defined with pylissom.nn.modules.register_recursive_input_output_hook()

Parameters:

• layer – Layer to plot
• prefix – Title of plot

pylissom.utils.plotting.matrix module

Some plotting functions that receive numpy matrices as input

pylissom.utils.plotting.matrix.plot_matrix(img, vmin=0, vmax=1)

Plots a numpy matrix in grayscale normalizing with vmin, vmax

pylissom.utils.plotting.matrix.plot_list_matrices(imgs)

pylissom.utils.plotting.matrix.plot_dict_matrices(imgs)

Parameters:imgs – Dictionary of {title: numpy matrix} items

pylissom.utils.plotting.matrix.tensor_to_numpy_matrix(tensor, shape)

Returns: Numpy version of tensor with shape = shape

pylissom.utils.plotting.matrix.plot_tensor(tensor, shape, vmin=0, vmax=1)

pylissom.utils.training

Submodules

pylissom.utils.training.cross_validation module

class pylissom.utils.training.cross_validation.CVSubjectIndependent(ck_dataset, k=5)

Bases: object

train_val_samplers()

test_sampler()

pylissom.utils.training.cross_validation.run_cross_validation(model_fn, ck_dataset, cv_sampler, args)

pylissom.utils.training.grid_search module

class pylissom.utils.training.grid_search.GridSearch(model_fn, param_grid, train_loader=None, test_loader=None, epochs=1, **kwargs)

Bases: object

run()

pylissom.utils.training.grid_search.run_lgn_grid_search(input_shape, lgn_shape, args)

pylissom.utils.training.grid_search.run_cortex_grid_search(input_shape, cortex_shape, args)

pylissom.utils.training.grid_search.run_lissom_grid_search(input_shape, lgn_shape, cortex_shape, args)

pylissom.utils.training.grid_search.run_supervised_grid_search(input_shape, lgn_shape, cortex_shape, args)

pylissom.utils.training.pipeline module

class pylissom.utils.training.pipeline.Pipeline(model, optimizer=None, loss_fn=None, log_interval=10, dataset_len=None, cuda=False, prefix='', use_writer=False)

Bases: object

train(train_data_loader, epoch)

test(test_data_loader, epoch)

static process_input(inp, normalize=False)

accuracy(data_loader)

Contribute

• Source Code: https://github.com/hernanbari/pylissom

License

The project is licensed under the GPLv3 license.

Indices and tables

• Index
• Module Index