Video Time Encoding and Decoding Machines¶
This package provides code to encode and decode videos with time encoding machines consisting of Gabor or center-surround receptive fields followed by Integrate-and-fire neurons [1] , [2] . It supports both the pseudoinverse algorithm, described in [1] , [2] and recurrent neural networks method, described in [3] for decoding.
Additional time encoding and decoding algorithms implemented in Python are available in the Time Encoding and Decoding (TED) Toolkit.
Contents¶
Installation Instructions¶
Quick Installation¶
If you have pip installed, you should be able to install the latest stable release of vtem by running the following:
pip install vtem
All dependencies should be automatically downloaded and installed if they are not already on your system.
Obtaining the Latest Software¶
The latest stable and development versions of vtem can be downloaded from GitHub
Online documentation for vtem is available at http://bionet.github.io/vtem/index.html/
Installation Dependencies¶
vtem requires that the following software packages be installed:
- Python 2.7 or later.
- setuptools 0.6c10 or later.
- NumPy 1.6.0 or later.
- CUDA SciKit 0.042 or greater
- PyTables 2.4.0 or greater
- PyCUDA 2011.1 or later
- NVIDIA CUDA Toolkit 4.0 or later.
To support reading and writing video files directly, the package requires
To build the documentation, the following packages are also required:
- Sphinx 1.1.0 or later.
Running the demos¶
To run the demos in the package as it is, please download the demo files and unarchive it in the demos folder.
To run the demos on other videos, the filename can be provided with the -i switch. All options supported can be viewed by:
python <demo_filename> --help
If you have OpenCV installed, all video files encoded with codecs that have been installed with OpenCV should be supported.
To run without OpenCV support, please load the video into MATLAB and save it using the scripts provided in the matlab_h5 folder of the package.
Platform Support¶
The software has been developed and tested on Linux; it should also work on other Unix-like platforms supported by the above packages. Parts of the package may work on Windows as well, but remain untested.
Building and Installation¶
To build and install the toolbox, download and unpack the source release and run:
python setup.py install
from within the main directory in the release. To rebuild the documentation, run:
python setup.py build_sphinx
Getting Started¶
Sample codes for the package are located in the /demos subdirectory. For a detailed description of the package modules, please refer the reference section of the package documentation.
Reference¶
vtem.vtem¶
This module provides the functions to encode a video using time encoding machines and store the resultant spikes and also the parameters of the VTEM in h5 files.
VTEM_IAF | Encode a video with IAF neurons and the specified Receptive field |
VTEM_Gabor_IAF | Encode a video with IAF neurons and gabor receptive field |
VTEM_CS_IAF | Encode a video with IAF neurons and gabor receptive field |
vtem.vtdm¶
This module provides the functions to recover a video enocded using time encoding machines.
decode_video | Reconstruct video using VTDM with dirichlet kernel, assuming IAF neurons |
vtem.ensemble_encode¶
This module provides the functions to encode the output of a receptive field using an ensemble of neurons. Only Integrate and fire neurons are supported in the current release
IAF_encode¶
- class vtem.ensemble_encode.IAF_encode(num_neurons, dt, dtype=<type 'numpy.float64'>)¶
Methods
encode(neural_inputs[, startbias, avg_rate]) Encode with IAFs load_parameters([h_kappa, h_bias, h_delta, ...]) Load encoding parameters to GPU reset_timer() Reset the time_count to zeros set_initial_value() Set integration from zeros set_parameters() Set the parameters using default values set_time_count() Set time_count zeros - __init__(num_neurons, dt, dtype=<type 'numpy.float64'>)¶
Population encoding with IAF neurons
Parameters: num_neurons : integer
number of neurons to encode
dt : float
time interval between two consecutive samples in the input for each neuron
dtype : np.float64 or np.float32, optional
dtype of spike interval to be stored (np.float32 not yet tested) If not specified, will be set to np.float64
- encode(neural_inputs, startbias=0, avg_rate=0.1)¶
Encode with IAFs
Parameters: neural_inputs : PitchArray
PitchArray of shape (num_samples, num_neurons) containing inputs to all neurons
startbias : integer, optional
the neuron index corresponding to first column of neural_inputs
avg_rate: float, optional
average spiking rate assumed for neurons, will allocate memory num_samples/avg_rate for each neuron for storing spikes
If not specified, will be set to 0.1
Returns: spikes: ndarray of self.dtype
stores the spikes for one neuron after another
spike_count: ndarray of int32 of size num_neurons
indicates the number of spikes generated by each neuron
Notes
spikes for neuron j can be accessed by
cum_count = np.concatenate((np.zeros(1,np.int32),np.cumsum(spike_count))) tk = spikes[cum_count[j]:cum_count[j+1]]
- load_parameters(h_kappa=None, h_bias=None, h_delta=None, h_sigma=None, h_time_count=None, h_v0=None)¶
Load encoding parameters to GPU
h_kappa, h_bias, h_delta can be set up using default values (using None), or specified together (using ndarrays of dtype)
h_time_count can be set up using default values (using None), or specified values(using ndarrays of dtype)
h_v0 can be set up using default values (using None), or specified values(using ndarrays of dtype)
- reset_timer()¶
Reset the time_count to zeros
- set_initial_value()¶
Set integration from zeros
- set_parameters()¶
Set the parameters using default values
Will set kappa’s to be 1.0, deltas to be 0.03, bias to be 0.8
- set_time_count()¶
Set time_count zeros
IAF_encode_rt¶
- class vtem.ensemble_encode.IAF_encode_rt(num_neurons, dt, dtype=<type 'numpy.float64'>)¶
Methods
encode(neural_inputs[, startbias, avg_rate]) Encode with IAFs with random thresholds load_parameters([h_kappa, h_bias, h_delta, ...]) Load encoding parameters to GPU reset_timer() reseting the time_count to zeros set_initial_value() set integration from zeros set_parameters() Set the parameters using default values set_time_count() Set time_count to zeros - __init__(num_neurons, dt, dtype=<type 'numpy.float64'>)¶
Population encoding with IAF neurons with random thresholds
Parameters: num_neurons : integer
number of neurons to encode
dt : float
time interval between two consecutive samples in the input for each neuron
dtype : type, optional
dtype of spike interval to be stored ( np.float32 not tested)
- encode(neural_inputs, startbias=0, avg_rate=0.1)¶
Encode with IAFs with random thresholds
Parameters: neural_inputs : PitchArray
PitchArray of shape (num_samples, num_neurons) containing inputs to all neurons
startbias : integer
the neuron index corresponding to first column of neural_inputs
avg_rate : float
average spiking rate assumed for neurons, will allocate memory num_samples/avg_rate for each neuron for storing spikes
Returns: spikes : ndarray of self.dtype
stores the spikes for one neuron after another
spike_count : ndarray of int32 of size num_neurons
indicates the number of spikes generated by each neuron
Notes
spikes for neuron j can be accessed by
cum_count = np.concatenate((np.zeros(1,np.int32),np.cumsum(spike_count))) tk = spikes[cum_count[j]:cum_count[j+1]]
- load_parameters(h_kappa=None, h_bias=None, h_delta=None, h_sigma=None, h_delta_value=None, h_time_count=None, h_v0=None)¶
Load encoding parameters to GPU
h_kappa, h_bias, h_delta can be set up using default values (using None), or specified together (using ndarrays of dtype)
h_time_count can be set up using default values (using None), or specified values(using ndarrays of dtype)
h_v0 can be set up using default values (using None), or specified values(using ndarrays of dtype)
- reset_timer()¶
reseting the time_count to zeros
- set_initial_value()¶
set integration from zeros
- set_parameters()¶
Set the parameters using default values
Will set kappa’s to be 1.0, deltas to be 0.03, bias to be 0.8
- set_time_count()¶
Set time_count to zeros
vtem.vrf¶
This module provides the functions to compute the output at the receptive field when presented with a video.
vrf_gabor¶
- class vtem.vrf.vrf_gabor(shape, dtype=<type 'numpy.float64'>, dx=0.0625, dy=0.0625, scale=1, domain=None)¶
Gabor RFs
Methods
compute_Ds(Mx, My) Returns the dirichlet coefficients of all gabor filters with order Mx, My compute_Dsw(d_Ds, Mx, My, h_norm) Compute the weighting matrix of the “correlation” between each two RFs compute_dirich_space(d_Ds, Mx, My, Px, Py, ...) Compute the spatial reconstruction functions compute_dirich_space_fft(d_Ds, Mx, My, Px, ...) filter(V) Filter a video V generate_visual_receptive_fields([...]) Generate a batch of gabor filters from parameters set in self get_gabor_parameters([dilations, ...]) Create a set of parameters for a family of gabor filters load_parameters([num_neurons, h_alpha, h_l, ...]) Load gabor parameters to GPU - __init__(shape, dtype=<type 'numpy.float64'>, dx=0.0625, dy=0.0625, scale=1, domain=None)¶
Parameters: shape : list
2-tuple indicating the shape (number of pixels in (height,width) / (y,x) ) of RF
dtype : type, optional
numpy dtype to use for RF calculations. If not specified , will default to np.float64
dx : float
spatial resolution in x direction, distance between two pixels
dy: float
spatial resolution in y direction, distance between two pixels
scale : float
scale up the shape by scale and scale down dx,dy by scale effectively increase the resoltuion of the RFs
domain : list
list of 2, [a,b], specifying the domain to cover RFs in default parameter settings a in x direction, b in y direction
Notes
The coordinate system is given by the following:
row (width / X) major -----------------------------------------> width | Y | ^--------------------> | |--------------------> | |--------------------> | |--------------------> X | | v height
- compute_Ds(Mx, My)¶
Returns the dirichlet coefficients of all gabor filters with order Mx, My in the format of PitchArray with shape (num_neurons, 2*Mx+1, 2*My+1)
- compute_Dsw(d_Ds, Mx, My, h_norm)¶
Compute the weighting matrix of the “correlation” between each two RFs
Parameters: d_Ds : PitchArray
containing dirichlet coefficient most possibly created by compute_Ds
Mx : integer
order in the x dimension
My : integer
order in the y dimension
Returns: PitchArray with shape (num_neurons, num_neurons)
- compute_dirich_space(d_Ds, Mx, My, Px, Py, Sx, Sy, Wx, Wy, x_start=None, y_start=None)¶
Compute the spatial reconstruction functions
Parameters: d_Ds : PitchArray
containing dirichlet coefficient most possibly created by compute_Ds
Mx : integer
Order in the x dimension
My : integer
Order in the y dimension
Px : integer
number of pixels in reconstruction functions in the x dimension
Py : integer
number of pixels in reconstruction functions in the y dimension
Sx : float
spatial domain in degree of reconstruction functions in x direction
Sy : float
spatial domain in degree of reconstruction functions in y direction
Wx : float
spatial bandwidth in x direction
Wy : float
spatial bandwidth in y direction
x_start : float
indicating the starting degree in x direction
y_start : float
indicating the starting degree in y direction
output: PitchArray with shape (num_neurons, Px, Py)
- filter(V)¶
Filter a video V Must set up parameters of gabor first
Parameters: V : 3D ndarray, with shape (num_frames, Px, Py)
Returns: The filtered output by the gabor filters specified in self
output is a PitchArray with shape (num_neurons, num_frames)
jth row of which is the output of jth gabor filter
- generate_visual_receptive_fields(startbias=0, N_filters=None, x_start=None, y_start=None)¶
Generate a batch of gabor filters from parameters set in self
Parameters: start_bias : integer, optional
start the the (start_bias)th filter
N_filters : integer, optional
number of filters to generate
x_start : float, optional
indicating the starting degree in x direction
y_start : float, optional
indicating the starting degree in y direction
- get_gabor_parameters(dilations=[1, 0, -1, -2, -3], num_rotations=8, alpha0=2.0, b0=1.5, b0scalebase=1.4142135623730951, KAPPA=2.5)¶
Create a set of parameters for a family of gabor filters
Parameters: dilation : list
a list containing the dilation parameter m
num_rotations : integer
number of rotations of each (dilation, translation_x, translation_y) pair
alpha0 : np.float64
base factor of dilation
b0 : np.float64
base factor of translation
b0scalebase : np.float64
effective translation in each dilation will be b0 * (alpha0**dilations[i]) * b0scalebase**(-dilations[i])
KAPPA : float
spatial bandwidth
- load_parameters(num_neurons=None, h_alpha=None, h_l=None, h_x0=None, h_y0=None, h_ab=None, KAPPA=2.5, set=0)¶
Load gabor parameters to GPU
num_neurons, h_alpha, h_l, h_x0, h_y0, h_ab must be specified together or not specified at all
Parameters: num_neurons : integer
total number of neurons
h_alpha : ndarray of float64
containing dilation parameters alpha = alpha0**(m)
h_l : ndarray of float64
containing rotation parameters l (angles)
h_x0 : ndarray of float64
containing translation parameters x0
h_y0 : ndarray of float64
containing translation parameters y0
h_ab : ndarray of int32
containing 0 or 1, 0 for real part, 1 for imaginary part jth gabor filter will be generated according to h_alpha[j], h_l[j], h_n[j], h_k[j] and h_ab[j]
KAPPA : float
spatial bandwidth
set : integer
0 if self parameters has not been set, 1 if they have been set by get_gabor_parameters or other manner)
vrf_cs¶
- class vtem.vrf.vrf_cs(shape, dtype=<type 'numpy.float64'>, dx=0.0625, dy=0.0625, scale=1, domain=None)¶
Centre Surround RFs
Methods
compute_Ds(Mx, My) Parameters: compute_Dsw(d_Ds, Mx, My, h_norm) Compute the weighting matrix of the “correlation” between each two RFs compute_dirich_space(d_Ds, Mx, My, Px, Py, ...) Compute the spatial reconstruction functions compute_dirich_space_fft(d_Ds, Mx, My, Px, ...) filter(V) Filter a video V generate_visual_receptive_fields([...]) Generate a batch of centre surround filters from parameters set in self get_cs_parameters([dilations, alpha0, b0, ...]) Create a set of parameters for a family of Centre Surround filters load_parameters([num_neurons, h_alpha, ...]) Load Centre Surround parameters to GPU - __init__(shape, dtype=<type 'numpy.float64'>, dx=0.0625, dy=0.0625, scale=1, domain=None)¶
Parameters: shape : list
2-tuple indicating the shape (number of pixels in (height,width) / (y,x) ) of RF
dtype : type, optional
numpy dtype to use for RF calculations. If not specified , will default to np.float64
dx : float
spatial resolution in x direction, distance between two pixels
dy: float
spatial resolution in y direction, distance between two pixels
scale : float
scale up the shape by scale and scale down dx,dy by scale effectively increase the resoltuion of the RFs
domain : list
list of 2, [a,b], specifying the domain to cover RFs in default parameter settings a in x direction, b in y direction
Notes
The coordinate system is given by the following:
row (width / X) major -----------------------------------------> width | Y | ^--------------------> | |--------------------> | |--------------------> | |--------------------> X | | v height
- compute_Ds(Mx, My)¶
Parameters: Mx : integer
Order in the x dimension
My : integer
Order in the y dimension
Returns: The dirichlet coefficients of all gabor filters with order Mx, My
in the format of PitchArray with shape (num_neurons, 2*Mx+1, 2*My+1)
- compute_Dsw(d_Ds, Mx, My, h_norm)¶
Compute the weighting matrix of the “correlation” between each two RFs
Parameters: d_Ds : PitchArray
containing dirichlet coefficient most possibly created by compute_Ds
Mx : integer
order in the x direction
My : integer
order in the y direction
Returns: PitchArray with shape (num_neurons, num_neurons)
- compute_dirich_space(d_Ds, Mx, My, Px, Py, Sx, Sy, Wx, Wy, x_start=None, y_start=None)¶
Compute the spatial reconstruction functions
Parameters: d_Ds : PitchArray
containing dirichlet coefficient most possibly created by compute_Ds
Mx : integer
Order in the x direction
My : integer
Order in the y direction
Px : integer
number of pixels in reconstruction functions in x direction
Py : integer
number of pixels in reconstruction functions in y direction
Sx : float
spatial domain in degree of reconstruction functions in x direction
Sy : float
spatial domain in degree of reconstruction functions in y direction
Wx : float
spatial bandwidth in x direction
Wy : float
spatial bandwidth in y direction
x_start : float, optional
indicating the starting degree in x direction
y_start : float, optional
indicating the starting degree in y direction
output: PitchArray with shape (num_neurons, Px, Py)
- filter(V)¶
Filter a video V Must set up parameters of CS RF first
Parameters: V : 3D ndarray, with shape (num_frames, Px, Py)
Returns: the filtered output by the gabor filters specified in self
output is a PitchArray with shape (num_neurons, num_frames),
jth row of which is the output of jth gabor filter
- generate_visual_receptive_fields(startbias=0, N_filters=None, x_start=None, y_start=None)¶
Generate a batch of centre surround filters from parameters set in self
Parameters: start_bias : integer, optional
start the the (start_bias)th filter
N_filters : integer, optional
generate N_filters filters
x_start : float
indicating the starting degree in x direction
y_start : float
indicating the starting degree in y direction
- get_cs_parameters(dilations=[1, 0, -1, -2, -3], alpha0=2.0, b0=1.0, b0scalebase=1.0, sigma_center=0.5, sigma_surround=0.8)¶
Create a set of parameters for a family of Centre Surround filters
Parameters: dilation : list, optional
a list containing the dilation parameter m
num_rotations : list, optional
number of rotations of each (dilation, translation_x, translation_y) pair
alpha0 : np.float64, optional
base factor of dilation
b0 : np.float64, optional
base factor of translation
b0scalebase : np.float64, optional
effective translation in each dilation will be b0 * (alpha0**dilations[i]) * b0scalebase**(-dilations[i])
sigma_center : float, optional
standard deviation of the center
sigma_surround : float, optional
standard deviation of the surround
- load_parameters(num_neurons=None, h_alpha=None, h_x0=None, h_y0=None, set=0, sigma_center=0.5, sigma_surround=0.8)¶
Load Centre Surround parameters to GPU
num_neurons, h_alpha, h_l, h_x0, h_y0, h_ab must be specified together or unspecified together
Parameters: num_neurons : integer
total number of neurons
h_alpha : ndarray of float64
containing dilation parameters alpha = alpha0**(m)
h_l : ndarray of float64
containing rotation parameters l (angles)
h_x0 : ndarray of float64
containing translation parameters x0
h_y0 : ndarray of float64
containing translation parameters y0
h_ab : ndarray of int32
containing 0 or 1, 0 for real part, 1 for imaginary part jth gabor filter will be generated according to h_alpha[j], h_l[j], h_n[j], h_k[j] and h_ab[j]
set : integer
0 if self parameters has not been set, 1 if they have been set by get_gabor_parameters or other manner)
sigma_center : float, optional
standard deviation of the center
sigma_surround : float, optional
standard deviation of the surround
Authors & Acknowledgments¶
This software was written and packaged by Nikul H. Ukani and Yiyin Zhou, currently at the Bionet Group [1] at Columbia University, under the supervision of Prof. Aurel A. Lazar.
[1] | http://bionet.ee.columbia.edu/ |
License¶
Copyright (c) 2013-2014 Nikul H. Ukani and Yiyin Zhou. All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
- Neither the name of Nikul H. Ukani, Yiyin Zhou, nor the names of any contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Index¶
[1] | (1, 2) Video Time Encoding Machines, Aurel A. Lazar and Eftychios A. Pnevmatikakis, IEEE Transactions on Neural Networks, Volume 22 , Number 3 , pp. 461-473 , March 2011 |
[2] | (1, 2) Encoding Natural Scenes with Neural Circuits with Random Thresholds, Aurel A. Lazar, Eftychios A. Pnevmatikakis and Yiyin Zhou, Vision Research, Volume 50, Number 22, pp. 2200-2212, October 2010, Special Issue on Mathematical Models of Visual Coding |
[3] | Massively Parallel Neural Encoding and Decoding of Visual Stimuli, Aurel A. Lazar and Yiyin Zhou, Neural Networks, Volume 32, pp. 303-312, August 2012, Special Issue: IJCNN 2011 |