Motivation
Early test setups during the development of neuromorphic hardware often do not allow to test the hardware in the same fashion as it will be used in its intended applications (e.g. transistor level simulations of single neuron vs. running large networks). Nevertheless it is important (already during during early stages of development) to have an idea on how well the designed circuits will perform later. Therefore it is convenient to have a set of small scale experiments (e.g. single neuron experiments) that can be run early and give an estimate on the performance of the hardware. Of course these experiments depend strongly on the intended application. If your hardware should be e.g. a simulation platform for neural networks in the cortex, a single neuron should (if stimulated with the same input) behave similar to a biological neuron in the cortex. Since most hardware platforms are not designed for one single purpose, there should be a collection of different single neuron experiments that test the hardware performance with respect to many different fields of applications. This workshop is a platform to collect such experiments from different groups of hardware designers and modelers of neural networks and allows to test multiple hardware devices in these experiments.
Challenges
Each challenge should consist of 3 parts: The input data for the neuron, the output data of a reference neuron and a measure to evaluate the performace of a hardware neuron. Also input and output data should be split into a test and a training set. The training set can be used to find the appropriate hardware paramerters and the test set is used to evaluate the final performance. All participants of the workshop are encouraged to add their own challenges.
Bio-Challenge
Hardware platforms that are designed as emulation platforms for biological neural networks need to produce biologically plausible output. This challenge tests the ability of the hardware to reproduce biological measurements. It is taken from the Single-Neuron Modeling Competition 2007 hosted by the INCF and organized by Renaud Jolivet, Richard Naud, Wulfram Gerstner et al. (Data taken from: Rauch A, La Camera G, Lüscher HR, Senn W, Fusi S 2003 Neocortical pyramidal cells respond as integrate-and-fire neurons to in vivo-like input currents. J Neurophysiol 90:1598–1612). Details on the original challenge can be found e.g. in Jolivet, Renaud, et al. "A benchmark test for a quantitative assessment of simple neuron models." Journal of neuroscience methods 169.2 (2008): 417-424.
Data
The aim is to predict, for given current inputs of a neuron, the times of the spikes produced by that neuron.The data consists of a training set and a test set. The training set contains 8 different input currents that were injected into the same neuron and the measured output spikes and voltage. The injection of each input current was repeated 4 times to account for the variations in the biological neurons' behaviour. This data set should be used to find the optimal parameter settings of the hardware platform to reproduce the biological spike times. The test set contains 4 new input currents and the spike times of the biological neuron.This data is used to evaluate the performance of the hardware when the optimal parameter settings are found.
Performance Measure
The measure proposed in the Single Neuron Modeling Competition 2007 is the Gamma-Measure. This measure is fast and easy to compute but has some disadvantages that will be discussed during the workshop. Other options such as the Victor-Purpura Measure, the Van Rossum Distance, the Schreiber Measure and the SPIKE-Distance will be assessed.
AdEx-Challenge
This challenge is inspired by the Heidelberg hardware systems that are (depending on the system) designed to follow the equations of a leaky integrate and fire neuron (LIF) or an adaptive exponential integrate and fire neuron (AdEx). This challenge tests how similar a hardware neuron behaves to a simulated AdEx neuron when presented with high frequent, random spike input.
Data
The data consists of a training set (7 different inputs) and a test set (4 different inputs). The inputs are each made of 2 poisson input spike trains, one excitatory and one inhibitory. The frequency of the poisson spike trains is the same for all sets, but the weights of the synaptic connection to the reference neuron changed for each input. The used synaptic weights are included in the data sets. Since the reference was created using a deterministic simulation (PyNN and NEST), the neuron was only stimulated once with each input. Additionally, the AdEx parameters that were used to create the references were included. As in the Bio-Challenge the training set should be used to tune the hardware and the test set is used to evaluate its performance.
Performance Measure
The performance measure will be the same as the measure chosen for the Bio-Challenge.
adex_challenge.tar.gz (2.1 MB)
LIF-Sampling Challenges
LIF-Sampling is the theory which describes how a well-contructed network of LIF neurons can sample from the probability distribution of a Boltzmann machine (for details see: Petrovici MA, Bill J, Bytschok I, Schemmel J, Karlheinz Meier. Stochastic inference with deterministic spiking neurons. arXiv:1311.3211). Hence, this idea opens up the powerful concept of Boltzmann machines known from machine learning and promises very interesting applications for neural networks. For LIF-sampling each neurons of a network recieves a Poissonian background noise and information coding spikes from other neurons. The simulatneous presence of background noise and information coding spikes are one of the essential points in the theory. Further, as a spike time based coding it requires a precise reproduction of the spike-times from the benchmarked neurons.
A) Activation function
LIF sampling requires that each neuron has a symmetric logistic activation function. That is, that the probability for a neuron to spike is a logistic function of the free mean membrane potential. Here, the mean free membrane potential means the mean membrane potential if spiking is eliminated, e.g. the spiking threshold is set to a very high value. Spiking probability means the probability of a neuron to be in the refractory state, i.e. spiked state, if the threshold potential is set to its normal value. The mean free membrane potential is usually controlled by the leakage potential or by a contant external current.
The challenge is easily said: Reach a symmetric logistic activation function for a single neuron!
B) Input/Output challange
This challange follows the same idea as the BioChallange.
Data
The data was created by simulating a minimalistic example of LIF-sampling. The same neuron was placed into 9 different network setups for the training set and into further 4 different setups for the test set. The simulation was done using PyNN and NEST.
Performance Measure
A priori we suggest that measures that consider spike times are best suited as a LIF-Sampling benchmark.
Dendritic Prediction Challenge
This challenge has shown that we are capable of testing a newly implemented spiking learning rule - Dendritic Prediction of Somatic Current. The analyzed neural model is comprising of two segments each producing output currents. The error between them is driving synaptic weights updates. The goal is that dendrite and soma reach the same state - output the same currents.
In the following figure there is presented an experiment designed to test the performance of the two synapses input.
The dendritic prediction experiment
Input of one of the dendritic synapses is aligned to the teacher signal, the other is shifted. We wexpect the weight of the first synapse to get elevated while the other one to depress.
In the following oscilloscope snapshots we present the recorings of two synaptic weights, where one of them stays high after the teaching phase, where the other drops to the low state - gets supresed.
The dendritic prediction experiment
The on-chip synaptic weight recordings.
The recordings data come from the Infra - most recent chip generation born in Zurich, INI.
Spike timing measures
The following scripts are implementations of the Gamma measure (used in the original single neuron challenge) and of 4 other wide spread spike measures: Van Rossum, Schreiber, Victor Purpura and SPIKE distance. Links the the original publications can be found in the scripts.
At Scholarpedia short descriptions of the most popular measures can be found as well as links to the original publications. Everyone is encouraged to add either a script or description of other measures or a testbench to evaluate the measures above!
Investigated Backends
The following paragraphs describe the platforms/hardware systems taking part in the challenges described above. All participants of the workshop are encouraged to add their platform/system with a brief description.
Simulations
The circuits in neuromorphic hardware are designed to follow the equations of a certain neuron model. In order to assess the possibilities of the hardware neuron it is also important to know how good the model itself is suited for a certain application. This can be done by software simulations of the neuron model not including any hardware specific properties (such as e.g. limitations in parameter range or bandwidth). We are planing to test the AdEx- and LIF-Neuron in our challenges using PyNN simulations with NEST as backend. All other neuron simulations are welcome as well.
Heidelberg Hardware Systems
The Electronic Vision(s) group (Heidelberg, Kirchoff Institute for Physics) develops highly accelerated, mixed signal neuromorphic hardware. The single neurons and synapses are fabricated as analog circuits that communicate with each other via digital signals. In the scope of the research projects BrainScaleS and Human Brain Project the group develops a wafer scale system with around 200 000 neurons and 44 million synapses per wafer. The system serves as a modeling platform for neural networks and it can be used via high-level PyNN interface.
The Spikey and WaferScaleS systems are also publicly available with a Human Brain Project Identity Account. For an invitation please contact the organizers (Laura Kriener, Akos Kungl) of the workshop. Additionally a limited number of Spikeys will be available in Capo Caccia.
Spikey
The Spikey single-chip neuromorphic setup contains 384 LIF-neurons with can be each connected via approximately 100 000 conductance based synapses. The synapses also feature short term (STP) and spike timing dependent plasticity (STDP).
HICANNv4 - Chip
The High Input Count Analog Neural Network (HICANN) chip is the building block of the large scale wafer systems. A single chip contains 512 AdEx-neurons and ca. 115 000 conductance based synapses. As in the Spikey chip the synapses feature both STP and STDP.
HICANN-DLS - Test Chip
The HICANN-DLS chip is planned to be the successor of the HICANN chips. The currently available test chip contains 32 LIF neurons (AdEx circuits are to be added in the future). The chip features a Plasticity Processing Unit (PPU) that allows the implementation of more complex plasticity mechanisms.
For more details see: Programmable Plasticity on Neuromorphic Hardware (Workshop by Christian Pehle)
Spinnacker
The APT (Advanced Processors Group) at the University of Manchester devleops the SpiNNaker ackitecture, which forms one of the ditgital neuromorphic hardware platforms. The SpiNNaker machine consists of large numbers of ARM cores which are wired together using a novel communication fabric which supports neural network simulation. The neurons and synapses are repesented as software programmes which run on the arm cores, where they communciate though the use of multicast packets. In the scope of the research projects BrainScaleS and Human Brain Project the APT group developed 1200 48 chip based SpiNNaker boards which can be wired together to form bigger machines. The currently largest avilable machine is a 600 board machine which can protentially provide around 460 million simple neurons, each with a fan in of 10000 other neurons. The current software supports 118 million simple neurons each with fan in of 10000 other neurons. This system serves as a modelling platofrm for neural networks and it can be used via high-level PyNN interface. It also supports none neural application simulation though our own Graph based interface which is documented here: http://spinnakermanchester.github.io/2016.001.AnotherFineProductFromTheNonsenseFactory/
The SpiNNaker system is also publicly available with a Human Brain Project Identity Account. For an invitation please contact the SpiNNaker team. Additionally there is a SpiNNaker machine available for use in Capo Caccia.
Models supported
The SpiNNaker software supports the following basic models:
- If Curr Exp
- If Curr Duel Exp
- If Cond Exp
- If Cond Delta
- IZK Cond Exp
- IZK Curr Ezp
- IZK curr Delta
The SpiNNaker software currently supprots the follwoing plasticity which can be attached to the basic neuron models as required (apart from the IZK [ask Micheal Hopkins why]).
Timing based
- Mad nearest pair
- mad pair
- mad pfister triplet
- nearest pair
- pair
- pfister triplet
Weight based
- Additive
- Multipicatve
ROLLS
The experiments were carried out using the latest generation of our neuromorphic processors, which is represented by the CxQuad chip. This is a multi-neuron chip with hierarchical on-chip routing, which comprises 4 cores, with 256 neurons each.
The neurons circuits implemented on the chip are based on the adaptive exponential I&F circuits described in the "Neuromorphic Silicon Neurons" paper published in Frontiers in Neuromorphic Engineering in 2013.
Results
In this workshop we gathered the single neuron challenges described above. They are suited to estimate the ability of neurons of different platforms to perform various different tasks.
Bio Challenge
The goal of this challenge was to reproduce spike times of a biological neuron which was stimulated by a current. The analog hardware developed in Heidelberg and Zürich both do not allow current input. On Spinnaker it is not yet implemented. A try to quickly implement it during the workshop failed due to time constraints. Therefore this challenge could only be carried out in software simulations. The following plots show exemplary comparisons of spike times of the biological neuron and simulations of a AdEx and LIF neuron. The parameters of the AdEx Neuron were obtained using a genetic algorithm. The LIF parameters were chosen to imitate the AdEx neuron.
We see that for some data sets the AdEx and the LIF neuron are qualitatively able to reproduce the biological spike times, for others (see last plot) their behaviour differs strongly. This might be due to the strongly differing nature of the input in this data set (different mean and standard deviation of the ornstein uhlenbeck input current).
AdEx Challenge
The goal in the AdEx Challenge was to reproduce spike times of a simulated AdEx neuron as exactly as possible (using the same input spikes). The following plots show the spike times of the AdEx-Reference, a simulated LIF-Neuron and the spikes produced by a LIF-Neuron on Spinnaker.
We see that for some inputs the resulting spike trains are very similar, for other the Spinnaker produces much less spikes. This could be explained by the fact, that Spinnaker uses a larger time step (0.1ms) than the simulation (0.01ms).
The challenge was also carried out on the analog hardware systems of Zürich and Heidelberg: Since fine tuning of hardware parameters is very time consuming, we can only present preliminary results on one data set.
Rolls
We used the data set from the neuromorphic challenge <1>, and made a first attempt at finding appropriate synapse and neuron parameters by iterative hand-made tuning operations. For a proper solutions, we plan to follow a more systematic approach based on automated software (Python) tuning procedures.
The challenge was to find a set of paramters that would best approximate the timing os spikes of a reference target. To view the reference target on the oscilloscopio, we transmitted the given spike times to a neuron on core 1 of the chip. We injected the given excitatory and inhibitory spike trains to corresponding synapse circuits on a neuron on core 0 , and tweaked parameters for thaose circuits.
Since we were told that the original model used adaptation and both excitatory and inhibitory inputs, we turned on the spike-frequency adaptation circuits of the Silicon neuron as well.
Examples of spike-trains measured from the chip (purple traice), and compared to the reference spike-train (green traice) are shown in the figures below
On the Spikey we managed to obtain a spike train which is rather similar to the simulated reference (also only for one data set) but we observed a very large trial to trail variability. After tuning the parameters the voltage trace resembles the trace simulated with Nest rather closely as can be seen in the picture below.
The large trial to trial variability can be explained by the fact, that in this challenge many small interactions (psps) accumulate. Many small variabilities sum up and have a large impact. Therefore the results are unreliable.
For fewer but much stronger interactions the trial to trial variability affects results only weakly. An example for that is a XOR-Network implemented during the workshop on the spikey chip. Its performance is nearly unaffected by triall to trial variations. The network produces an output spike (o) if a spike occurs on only one of the inputs (i1, i2).
LIF sampling on Spinnaker
Currently TSO-mechanism is not implemeted on Spinakker, although it is required by the LIF-sampling theory. In order to investigate this restriction we compared LIF-sampling with NEST simulator with and without TSO and Spinnaker on a small proof-of-concept Boltzmann machine.
