nengo_mpi¶

Installing nengo_mpi¶

sudo apt-get install libboost-dev libatlas-base-dev

git clone https://github.com/nengo/nengo_mpi.git
cd nengo_mpi

cp conf/mpich.cfg mpi.cfg

cp conf/mpich_trusty.cfg mpi.cfg

pip install --user .

How It Works¶

Here we attempt to give a rough idea of how nengo_mpi works under the hood, and, in particular, how it achieves parallelization. nengo_mpi is based heavily on the reference implementation of nengo. The reference implementation works by converting a high-level neural model specification into a low-level computation graph. The computation graph is a collection of operators and signals. In short, signals store data, and operators perform computation on signals and store the results in other signals. To run the simulation, nengo simply executes each operator in the computation graph once per time step. For a concrete example of how this works, consider the following simple nengo script:

import nengo

model = nengo.Network()
with model:
    A = nengo.Ensemble(n_neurons=50, dimensions=1)
    B = nengo.Ensemble(n_neurons=50, dimensions=1)
    conn = nengo.Connection(A, B)

sim = nengo.Simulator(model)
sim.run(time_in_seconds=1.0)

The conversion from the high-level specification (e.g. the nengo Network stored in the variable model) to computation graph is called the build step, and takes place in the line sim = nengo.Simulator(model). The generated computation graph looks something like this:

A few signals and operators whose purposes are somewhat opaque have been omitted here for clarity. Now suppose that we’re impatient and find that the call to sim.run is too slow. We can easily parallelize the simulation step by making use of nengo_mpi. Making the few necessary changes, we end up with the following script:

import nengo
import nengo_mpi

model = nengo.Network()
with model:
    A = nengo.Ensemble(n_neurons=50, dimensions=1)
    B = nengo.Ensemble(n_neurons=50, dimensions=1)
    nengo.Connection(A, B)

# assign the ensembles to different processors
assignments = {A: 0, B: 1}
sim = nengo_mpi.Simulator(model, assignments=assignments)

sim.run(time_in_seconds=1.0)

Now ensembles A and B will be simulated on different processors, and we should get a factor of 2 speedup in running the simulation (though it will hardly be perceptible given how tiny our network is). nengo_mpi will produce a computation graph quite similar to the one produced by vanilla nengo, except it will use operators that are implemented in C++ rather than python, and will add a few new operators to achieve the inter-process communication:

The MPISend operator stores the index of the processor to send its data to, and likewise the MPIRecv operator stores the index of the processor to receive data from. Moreover, they both share a “tag”, a unique identifier which bonds the two operators together and ensures that the data from the MPISend operator gets sent to the correct MPIRecv operator. This basic pattern can be scaled up to simulate very large networks on thousands of processors.

Some readers may have noticed something odd by now: it may seem like it would be impossible to achieve accelerated performance from the set-up depicted in the above diagrams. In particular, it seems as if the operators on processor 1 will need to wait for the results from processor 0, so the computation is still ultimately a serial one, just that now we have added inter-process communication in the pipeline to slow things down.

This turns out not to be the case, because the Synapse operator is special in that it is what we call an “update” operator. Update operators break the computation graph up into independently-simulatable components. In the first diagram, the DotInc operator in ensemble B performs computation on the value of the Input signal from the previous time-step [1]. Thus, the operators in ensemble B do not need to wait for the operators in ensemble A and the connection, since the values from the previous time-step should already be available. Likewise, in the second diagram, the MPIRecv operator actually receives data from the previous time-step. Thanks to this mechanism, we are in fact able to achieve large-scale parallelization, demonstrated empirically by our Benchmarks.

Workflows¶

There are two distinct ways to use nengo_mpi.

sim = nengo_mpi.Simulator(model, partitioner=partitioner, save_file="model.net")

python nengo_script.py

mpirun -np NP nengo_mpi model.net 1.0

mpirun -np NP nengo_mpi --log results.h5 model.net 1.0

nengo_cpp --log results.h5 model.net 1.0

Modules¶

nengo_mpi is composed of several fairly separable modules.

Benchmarks¶

Benchmarks testing the simulation speed of nengo_mpi were performed with 3 different machines and 3 different large-scale spiking neural networks. The machines used were a home PC with a Quad-Core 3.6GHz Intel i7-4790 and 16 GB of RAM, Scinet’s General Purpose Cluster, and Scinet’s 4 rack Blue Gene/Q. We tested nengo_mpi using different numbers of processors, and also tested the reference implementation of nengo (on the home PC only) for comparison.

Stream Network¶

The stream network exhibits a simple connectivity structure, and is intended to be close to the optimal configuration for executing a simulation quickly In parallel using nengo_mpi. In particular, the ratio of the amount of communication vs computation per step is relatively low. The network takes a single parameter \(n\) giving the total number of neural ensembles. The network contains \(\sqrt{n}\) different “streams”, where a stream is a collection of \(\sqrt{n}\) ensembles connected in a circular fashion (so each ensemble has 1 incoming and 1 outgoing connection). Each ensemble is \(4\)-dimensional and contains \(200\) LIF neurons, and we vary \(n\) as the independent variable in the graphs below. The largest network contains \(2^{12} = 4096\) ensembles, for a total of \(200 * 4096 = 819,200\) neurons. In every case, the ensembles are distributed evenly amongst the processors. Each execution consists of 5 seconds of simulated time, and each data point is the result of 5 separate executions.

Random Graph Network¶

The random graph network is constructed by choosing a fixed number of ensembles, and then randomly choosing ensemble-to-ensemble connections to insantiate until a desired proportion of the total number of possible connections is reached. In all cases, we use \(1024\) ensembles, and we vary the proportion of connections. This network is intended to show how the performance of nengo_mpi scales as the ratio of communication to computation increases, and investigate whether it is always a good idea to add more processors. Adding more processors typically increases the amount of inter-processor communication, since it increases the likelihood that any two ensembles are simulated on different processors. Therefore, if communication is the bottleneck, then adding more processors will tend to decrease performance.

Each ensemble is 2-dimensional and contains 100 LIF neurons, and each connection computes the identity function. With \(n\) ensembles, there are \(n^2\) possible connections (since we allow self-connections and the connections are directed). Therefore in the most extreme case we have have \(0.2 \times 1024^2 \approx 209,715\) connections, each relaying a 2-dimensional value. The number of such connections that need to engage in inter-processor communication is a function of the number of processors used in the simulation, and the particular random connectivity structure that arose. Each execution consists of 5 seconds of simulated time, and each data point is the average of executions on 5 separate networks with different randomly chosen connectivity structure.

SPAUN¶

SPAUN (Semantic Pointer Architecture Unified Network) is a large-scale, functional model of the human brain developed at the CNRG. It is composed entirely of spiking neurons, and can perform eight separate cognitive tasks without modification. The original paper can be found here. SPAUN is an extremely large-scale spiking neural network (currently approaching 4 million neurons when using 512-dimensional semantic pointers) with very complex connectivity, and represents a somewhat more realistic test than the more contrived examples used above. In the plots below we vary the dimensionality of the semantic pointers, the internal representations used by SPAUN.

Clearly the larger clusters are providing less of a benefit here. The hypothesized reason is that thus far we have been unable to split SPAUN up into sufficiently small components than can be simulated independently. There are some components with many thousands of neurons on them. Thus the limiting factor for the speed of simulation is how quickly an individual processor is able to simulate one of these large components. We can likely remedy this by playing around with the connectivity of SPAUN and finding ways to reduce the maximum component size.

FAQ¶

Is there any build step parallelization?

No, nengo_mpi only provides parallelization for the simulation step. The build step is where all the really difficult stuff happens, which, for instance, makes an Ensemble act like an Ensemble. Therefore, nengo_mpi simply uses vanilla nengo’s builder, which runs serially in python.

During an invocation such as:

mpirun -np 8 python -m nengo_mpi nengo_script.py

the build step is performed entirely by the process with index 0.

It is definitely possible to create a parallelized version of the builder. However, that should probably use a more python-friendly, platform-agnostic technology than MPI (something like ZeroMQ). In other words, thats another project.

What is the difference between a cluster a component, a partition, a chunk, a process, a processor, and a node? I’ve seen all these words used in the code with apparently similar meanings.

All these terms do in fact have precise meanings in the context of nengo_mpi. They can nicely be divided up into terms that apply at build time and terms that apply at simulation time.

• Build Time

  • A cluster (distinct from a cluster of machines in high-performance computing) is a group of nengo objects that must be simulated together, for any of a number of reasons (see the class NengoObjectCluster in partition/base.py). The most prominent reason is that there is an path of Connections between the two objects that does not have a synapse (since synapses are the main source of “update” operators; see How It Works). Another common reason is that the two objects are connected by a Connection which has a learning rule. The partitioning step applies a partitioning function to a graph whose nodes are clusters.
  • A component (as in a component of a partition) is a group of clusters that will be simulated together. Components are computed by the partitioning step. When creating an instance of nengo_mpi.Simulator, we typically specify the number of components that we want the network to be divided into. When nengo_mpi saves a network to file for communication with the C++ code, each component is stored separately.
  • A partition is a collection of components. The goal of the partitioning step is to create a partition of the set of clusters, in the sense used here. High-quality partitions are those which do not assign drastically different amounts of work to different components, and which minimize the amount of communication between components.

• Simulation Time

  • A process is, of course, an OS abstraction for a line of computation. A processor is a physical computation device. Processes run on processors. It is generally possible to run a nengo_mpi simulation using more processes than there are processors available on the machine, however the amount of parallelization we can obtain is determined by the number of physical processors (though hyperthreading can increase the effective number of processors). The number of processes used to run a simulation is specified by the -np <NP> command-line argument when calling mpi_run.
  • A chunk (see chunk.hpp) is the C++ code’s abstraction for a collection of nengo objects (actually, signals and operators corresponding to those objects) that are being simulated by a single process. There is a one-to-one relationship between chunks and processes. One of the first things that each process does is create a chunk.
  • The relationship between chunks/processes and components is as follows. At build time the network is divided into some specified number of components by partitioning. At simulation time, some specified number of chunks/processes will be active. Components are assigned to chunks/processes in a round-robin fashion. For example, if there are 4 chunks/processes active and the network to simulate has 7 components, then process 0 simulates components 0 and 4, process 1 simulates 1 and 5, etc. If the network instead had only 3 components, then process 3 would be left without anything to simulate, which is perfectly OK.
  • In the world of High-Performance Computing, a node (distinct from a nengo Node) is a physical computer consisting of some number of processors. On the General Purpose Cluster there are 8 processors per node and on Bluegene/Q there are 16 (that becomes 16 for GPC and 64 for BGQ once hyperthreading is taken into account). When running on one of these high-performance clusters, jobs are assigned computational resources in units of nodes rather than processors.