Requirements¶

Fuel’s requirements are

• PyYAML, to parse the configuration file
• six, to support both Python 2 and 3 with a single codebase
• h5py and PyTables for the HDF5 storage back-end
• pillow, providing PIL for image preprocessing
• Cython, for fast extensions
• pyzmq, to efficiently send data across processes
• picklable_itertools, for supporting iterator serialization
• SciPy, to read from MATLAB’s .mat format
• requests, to download canonical datasets

nose2 is an optional requirement, used to run the tests.

Development¶

If you want to work on Fuel’s development, your first step is to fork Fuel on GitHub. You will now want to install your fork of Fuel in editable mode. To install in your home directory, use the following command, replacing USER with your own GitHub user name:

$ pip install -e git+git@github.com:USER/fuel.git#egg=fuel[test,docs] --src=$HOME

As with the usual installation, you can use --user or --no-deps if you need to. You can now make changes in the fuel directory created by pip, push to your repository and make a pull request.

If you had already cloned the GitHub repository, you can use the following command from the folder you cloned Fuel to:

$ pip install -e file:.#egg=fuel[test,docs]

Fuel contains Cython extensions, which need to be recompiled if you update the Cython .pyx files. Each time these files are modified, you should run:

$ python setup.py build_ext --inplace

Division of labour¶

There are four basic tasks that Fuel needs to handle:

• Interface with the data, be it on disk or in memory.
• Decide which data points to visit, and in which order.
• Iterate over the selected data points.
• At each iteration step, apply some transformation to the selected data points.

Each of those four tasks is delegated to a particular class of objects, which we’ll be introducing in order.

Schematic overview of Fuel¶

For the more visual people, here’s a schematic view of how the different components of Fuel interact together. Dashed lines are optional.

Datasets: interfacing with data¶

The Dataset class is responsible for interfacing with the data and handling data access requests. Subclasses of Dataset specialize in certain types of data.

Datasets contain one or more sources of data, such as an array of images, a list of labels, a dictionary specifying an ontology, etc. Each source in a dataset is identified by a unique name.

All datasets have the following attributes:

• sources: tuple of source names indicating what the dataset will provide when queried for data.
• provides_sources: tuple of source names indicating what sources the dataset is able to provide.
• axis_labels: dict mapping each source name to a tuple of axis labels, or None. Not all source names need to appear in the axis labels dictionary.

Some datasets also have a num_examples attribute telling how many examples the dataset provides.

>>> from collections import OrderedDict
>>> from fuel.datasets import IterableDataset
>>> dataset = IterableDataset(
...     iterables=OrderedDict([('features', features), ('targets', targets)]),
...     axis_labels=OrderedDict([('features', ('batch', 'height', 'width')),
...                              ('targets', ('batch', 'index'))]))

>>> print('Provided sources are {}.'.format(dataset.provides_sources))
Provided sources are ('features', 'targets').
>>> print('Sources are {}.'.format(dataset.sources))
Sources are ('features', 'targets').
>>> print('Axis labels are {}.'.format(dataset.axis_labels))
Axis labels are OrderedDict([('features', ('batch', 'height', 'width')), ('targets', ('batch', 'index'))]).
>>> print('Dataset contains {} examples.'.format(dataset.num_examples))
Dataset contains 8 examples.

>>> state = dataset.open()
>>> print(state.__class__.__name__)
imap

>>> while True:
...     try:
...         print(dataset.get_data(state=state))
...     except StopIteration:
...         print('Iteration over')
...         break
(array([[ 47, 211],
       [ 38,  53]]), array([0]))
(array([[204, 116],
       [152, 249]]), array([3]))
(array([[143, 177],
       [ 23, 233]]), array([0]))
(array([[154,  30],
       [171, 158]]), array([1]))
(array([[236, 124],
       [ 26, 118]]), array([2]))
(array([[186, 120],
       [112, 220]]), array([2]))
(array([[ 69,  80],
       [201, 127]]), array([2]))
(array([[246, 254],
       [175,  50]]), array([3]))
Iteration over

>>> state = dataset.reset(state=state)
>>> print(dataset.get_data(state=state))
(array([[ 47, 211],
       [ 38,  53]]), array([0]))

>>> dataset.close(state=state)

>>> from fuel.datasets import IndexableDataset
>>> dataset = IndexableDataset(
...     indexables=OrderedDict([('features', features), ('targets', targets)]),
...     axis_labels=OrderedDict([('features', ('batch', 'height', 'width')),
...                              ('targets', ('batch', 'index'))]))

>>> state = dataset.open()
>>> print('State is {}.'.format(state))
State is None.
>>> print(dataset.get_data(state=state, request=[0, 1]))
(array([[[ 47, 211],
        [ 38,  53]],

       [[204, 116],
        [152, 249]]]), array([[0],
       [3]]))
>>> dataset.close(state=state)

>>> restricted_dataset = IndexableDataset(
...     indexables=OrderedDict([('features', features), ('targets', targets)]),
...     axis_labels=OrderedDict([('features', ('batch', 'height', 'width')),
...                              ('targets', ('batch', 'index'))]),
...     sources=('features',))
>>> print(restricted_dataset.provides_sources)
('features', 'targets')
>>> print(restricted_dataset.sources)
('features',)
>>> state = restricted_dataset.open()
>>> print(restricted_dataset.get_data(state=state, request=[0, 1]))
(array([[[ 47, 211],
        [ 38,  53]],

       [[204, 116],
        [152, 249]]]),)
>>> restricted_dataset.close(state=state)

Iteration schemes: which examples to visit¶

Encapsulating and accessing our data is good, but if we’re to integrate it into a training loop, we need to be able to iterate over the data. For that, we need to decide which indices to request and in which order. This is accomplished via an IterationScheme subclass.

At its most basic level, an iteration scheme is responsible, through its get_request_iterator() method, for building an iterator that will return requests. Here are some examples:

>>> from fuel.schemes import (SequentialScheme, ShuffledScheme,
...                           SequentialExampleScheme, ShuffledExampleScheme)
>>> schemes = [SequentialScheme(examples=8, batch_size=4),
...            ShuffledScheme(examples=8, batch_size=4),
...            SequentialExampleScheme(examples=8),
...            ShuffledExampleScheme(examples=8)]
>>> for scheme in schemes:
...     print(list(scheme.get_request_iterator()))
[[0, 1, 2, 3], [4, 5, 6, 7]]
[[7, 2, 1, 6], [0, 4, 3, 5]]
[0, 1, 2, 3, 4, 5, 6, 7]
[7, 2, 1, 6, 0, 4, 3, 5]

We can therefore use an iteration scheme to visit a dataset in some order.

>>> state = dataset.open()
>>> scheme = ShuffledScheme(examples=dataset.num_examples, batch_size=4)
>>> for request in scheme.get_request_iterator():
...     data = dataset.get_data(state=state, request=request)
...     print(data[0].shape, data[1].shape)
(4, 2, 2) (4, 1)
(4, 2, 2) (4, 1)
>>> dataset.close(state)

Data streams: automating the iteration process¶

Iteration schemes offer a more convenient way to visit the dataset than accessing the data by hand, but we can do better: the act of getting a fresh state from the dataset, getting a request iterator from the iteration scheme, using both to access the data and closing the state is repetitive. To automate this, we have data streams, which are subclasses of AbstractDataStream.

The most common AbstractDataStream subclass is DataStream. It is instantiated with a dataset and an iteration scheme, and returns an epoch iterator through its get_epoch_iterator() method, which iterates over the dataset in the order defined by the iteration scheme.

>>> from fuel.streams import DataStream
>>> data_stream = DataStream(dataset=dataset, iteration_scheme=scheme)
>>> for data in data_stream.get_epoch_iterator():
...     print(data[0].shape, data[1].shape)
(4, 2, 2) (4, 1)
(4, 2, 2) (4, 1)

Transformers: apply some transformation on the fly¶

Some AbstractDataStream subclasses take data streams as input. We call them transformers, and they enable us to build complex data preprocessing pipelines.

Transformers are Transformer subclasses, which is itself an AbstractDataStream subclass. Here are some commonly used ones:

• Flatten: flattens the input into a matrix (for batch input) or a vector (for examplewise input).
• ScaleAndShift: scales and shifts the input by scalar quantities.
• Cast: casts the input into some data type.

As an example, let’s standardize the images we have by substracting their mean and dividing by their standard deviation.

>>> from fuel.transformers import ScaleAndShift
>>> # Note: ScaleAndShift applies (batch * scale) + shift, as
>>> # opposed to (batch + shift) * scale.
>>> scale = 1.0 / features.std()
>>> shift = - scale * features.mean()
>>> standardized_stream = ScaleAndShift(data_stream=data_stream,
...                                     scale=scale, shift=shift,
...                                     which_sources=('features',))

The resulting data stream can be used to iterate over the dataset just like before, but this time features will be standardized on-the-fly.

>>> for batch in standardized_stream.get_epoch_iterator():
...     print(batch)
(array([[[ 0.18530572, -1.54479571],
        [ 0.42249705,  0.24111545]],

       [[-1.30760439,  0.98059429],
        [-1.43317627, -1.2238898 ]],

       [[ 1.46892937,  1.58054882],
        [ 0.47830677, -1.2657471 ]],

       [[ 0.63178351, -0.28907693],
        [-0.40069638,  1.10616617]]]), array([[1],
       [0],
       [3],
       [2]]))
(array([[[ 1.32940506, -0.2332672 ],
        [-1.60060544, -0.31698179]],

       [[ 0.03182898,  0.50621164],
        [-1.64246273,  1.28754777]],

       [[ 0.88292727, -0.34488665],
        [ 0.15740086,  1.51078666]],

       [[-1.00065091, -0.84717417],
        [ 0.84106998, -0.19140991]]]), array([[2],
       [0],
       [3],
       [2]]))

Now, let’s imagine that for some reason (e.g. running Theano code on GPU) we need features to have a data type of float32. We can cast them on-the-fly with a Cast transformer.

>>> from fuel.transformers import Cast
>>> cast_standardized_stream = Cast(
...     data_stream=standardized_stream,
...     dtype='float32', which_sources=('features',))

As you can see, Fuel makes it easy to chain transformations to form a preprocessing pipeline. The complete pipeline now looks like this:

>>> data_stream = Cast(
...     ScaleAndShift(
...         DataStream(
...             dataset=dataset, iteration_scheme=scheme),
...         scale=scale, shift=shift, which_sources=('features',)),
...     dtype='float32', which_sources=('features',))

Going further¶

You now know enough to find your way around Fuel. Here are the next steps:

• Learn how to use built-in datasets.
• Learn how to import your own data in Fuel.
• Learn how to extend Fuel to suit your needs.

Environment variable¶

In order for Fuel to know where to look for its data, the data_path configuration variable has to be set inside ~/.fuelrc. It’s expected to be a sequence of paths separated by an OS-specific delimiter (: for Linux and OSX, ; for Windows):

# ~/.fuelrc
data_path: "/first/path/to/my/data:/second/path/to/my/data"

When looking for a specific file (e.g. mnist.hdf5), Fuel will search each of these paths in sequence, using the first matching file that it finds.

This configuration variable can be overridden by setting the FUEL_DATA_PATH environment variable:

$ export FUEL_DATA_PATH="/first/path/to/my/data:/second/path/to/my/data"

Let’s now change directory for the rest of this tutorial:

$ cd $FUEL_DATA_PATH

Download a built-in dataset¶

We’re going to download the raw data files for the MNIST dataset with the fuel-download script that was installed with Fuel:

$ fuel-download mnist

The script is pretty simple: you call it and pass it the name of the dataset you’d like to download. In order to know which datasets are available to download via fuel-download, type

$ fuel-download -h

You can pass dataset-specific arguments to the script. In order to know which arguments are accepted, append -h to your dataset choice:

fuel-download mnist -h

Two arguments are always accepted:

• -d DIRECTORY : define where the dataset files will be downloaded. By default, fuel-download uses the current working directory.
• --clear : delete the dataset files instead of downloading them, if they exist.

Convert downloaded files¶

You should now have four new files in your directory:

• train-images-idx3-ubyte.gz
• train-labels-idx1-ubyte.gz
• t10k-images-idx3-ubyte.gz
• t10k-labels-idx1-ubyte.gz

Those are the original files that can be downloaded off Yann Lecun’s website. We now need to convert those files into a format that the MNIST dataset class will recognize. This is done through the fuel-convert script:

$ fuel-convert mnist

This will generate an mnist.hdf5 file in your directory, which the MNIST class recognizes.

Once again, the script accepts dataset-specific arguments which you can discover by appending -h to your dataset choice:

fuel-convert mnist -h

Two arguments are always accepted:

• -d DIRECTORY : where fuel-convert should look for the input files.
• -o OUTPUT_FILE : where to save the converted dataset.

Let’s delete the raw input files, as we don’t need them anymore:

$ fuel-download mnist --clear

Inspect Fuel-generated dataset files¶

Six months from now, you may have a bunch of dataset files lying on disk, each with slight differences that you can’t identify or reproduce. At that time, you’ll be glad that fuel-info exists.

When a dataset is generated through fuel-convert, the script tags it with what command was issued to generate the file and what were the versions of relevant parts of the library at that time.

You can inspect this metadata calling fuel-info and passing an HDF5 file as argument:

$ fuel-info mnist.hdf5

Metadata for mnist.hdf5
=======================

The command used to generate this file is

    fuel-convert mnist

Relevant versions are

    H5PYDataset     0.1
    fuel.converters 0.1

Working with external packages¶

By default, Fuel looks for downloaders and converters in the fuel.downloaders and fuel.converters modules, respectively, but you’re not limited to that.

Fuel can be told to look into additional modules by setting the extra_downloaders and extra_converters configuration variables in ~/.fuelrc. These variables are expected to be lists of module names.

For instance, suppose you’d like to include the following modules:

• package1.extra_downloaders
• package2.extra_downloaders
• package1.extra_converters
• package2.extra_converters

You should include the following in your ~/.fuelrc:

# ~/.fuelrc
extra_downloaders:
- package1.extra_downloaders
- package2.extra_downloaders
extra_converters:
- package1.extra_converters
- package2.extra_converters

These configuration variables can be overridden through the FUEL_EXTRA_DOWNLOADERS and FUEL_EXTRA_CONVERTERS environment variables, which are expected to be strings of space-separated module names, like so:

export FUEL_EXTRA_DOWNLOADERS="package1.extra_downloaders package2.extra_downloaders"
export FUEL_EXTRA_CONVERTERS="package1.extra_converters package2.extra_converters"

This feature lets external developers define their own Fuel dataset downloader/converter packages, and also makes working with private datasets more straightforward.

A toy example¶

We’ll start this tutorial by creating bogus data sources that could be lying around on disk.

>>> import numpy
>>> numpy.save(
...     'train_vector_features.npy',
...     numpy.random.normal(size=(90, 10)).astype('float32'))
>>> numpy.save(
...     'test_vector_features.npy',
...     numpy.random.normal(size=(10, 10)).astype('float32'))
>>> numpy.save(
...     'train_image_features.npy',
...     numpy.random.randint(2, size=(90, 3, 5, 5)).astype('uint8'))
>>> numpy.save(
...     'test_image_features.npy',
...     numpy.random.randint(2, size=(10, 3, 5, 5)).astype('uint8'))
>>> numpy.save(
...     'train_targets.npy',
...     numpy.random.randint(10, size=(90, 1)).astype('uint8'))
>>> numpy.save(
...     'test_targets.npy',
...     numpy.random.randint(10, size=(10, 1)).astype('uint8'))

Our goal is to process these files into a format that can be natively imported in Fuel.

HDF5 datasets¶

The best-supported way to load data in Fuel is through the H5PYDataset class, which wraps HDF5 files using h5py.

This is the class that’s used for most built-in datasets. It makes a series of assumptions about the structure of the HDF5 file which greatly simplify things if your data happens to meet these assumptions:

• All data is stored into a single HDF5 file.
• Data sources reside in the root group, and their names define the source names.
• Data sources are not explicitly split into separate HDF5 datasets or separate HDF5 files. Instead, splits are defined in the split attribute of the root group. It’s expected to be a 1D numpy array of compound dtype with seven fields, organized as follows:
  1. split : string identifier for the split name
  2. source : string identifier for the source name
  3. start : start index (inclusive) of the split in the source array, used if indices is a null reference.
  4. stop : stop index (exclusive) of the split in the source array, used if indices is a null reference.
  5. indices : h5py.Reference, reference to a dataset containing subset indices for this split/source pair. If it’s a null reference, start and stop are used.
  6. available : boolean, False is this split is not available for this source
  7. comment : comment string

Converting the toy example¶

Let’s now convert our bogus files into a format that’s digestible by H5PYDataset.

We first load the data from disk.

>>> train_vector_features = numpy.load('train_vector_features.npy')
>>> test_vector_features = numpy.load('test_vector_features.npy')
>>> train_image_features = numpy.load('train_image_features.npy')
>>> test_image_features = numpy.load('test_image_features.npy')
>>> train_targets = numpy.load('train_targets.npy')
>>> test_targets = numpy.load('test_targets.npy')

We then open an HDF5 file for writing and create three datasets in the root group, one for each data source. We name them after their source name.

>>> import h5py
>>> f = h5py.File('dataset.hdf5', mode='w')
>>> vector_features = f.create_dataset(
...     'vector_features', (100, 10), dtype='float32')
>>> image_features = f.create_dataset(
...     'image_features', (100, 3, 5, 5), dtype='uint8')
>>> targets = f.create_dataset(
...     'targets', (100, 1), dtype='uint8')

Notice how the number of examples we specify (100) in the shapes is the sum of the number of training and test examples. We’ll be filling the first 90 rows with training examples and the last 10 rows with test examples.

>>> vector_features[...] = numpy.vstack(
...     [train_vector_features, test_vector_features])
>>> image_features[...] = numpy.vstack(
...     [train_image_features, test_image_features])
>>> targets[...] = numpy.vstack([train_targets, test_targets])

H5PYDataset allows us to label axes with semantic information. We record that information in the HDF5 file through dimension scales.

>>> vector_features.dims[0].label = 'batch'
>>> vector_features.dims[1].label = 'feature'
>>> image_features.dims[0].label = 'batch'
>>> image_features.dims[1].label = 'channel'
>>> image_features.dims[2].label = 'height'
>>> image_features.dims[3].label = 'width'
>>> targets.dims[0].label = 'batch'
>>> targets.dims[1].label = 'index'

This particular choice of label is arbitrary. Nothing in Fuel forces you to adopt any labeling convention. Note, however, that certain external frameworks that rely on Fuel may impose some restrictions on the choice of labels.

The last thing we need to do is to give H5PYDataset a way to recover what the splits are. This is done by setting the split attribute of the root group.

>>> split_array = numpy.empty(
...     6,
...     dtype=numpy.dtype([
...         ('split', 'a', 5),
...         ('source', 'a', 15),
...         ('start', numpy.int64, 1),
...         ('stop', numpy.int64, 1),
...         ('indices', h5py.special_dtype(ref=h5py.Reference)),
...         ('available', numpy.bool, 1),
...         ('comment', 'a', 1)]))
>>> split_array[0:3]['split'] = 'train'.encode('utf8')
>>> split_array[3:6]['split'] = 'test'.encode('utf8')
>>> split_array[0:6:3]['source'] = 'vector_features'.encode('utf8')
>>> split_array[1:6:3]['source'] = 'image_features'.encode('utf8')
>>> split_array[2:6:3]['source'] = 'targets'.encode('utf8')
>>> split_array[0:3]['start'] = 0
>>> split_array[0:3]['stop'] = 90
>>> split_array[3:6]['start'] = 90
>>> split_array[3:6]['stop'] = 100
>>> split_array[:]['indices'] = h5py.Reference()
>>> split_array[:]['available'] = True
>>> split_array[:]['comment'] = '.'.encode('utf8')
>>> f.attrs['split'] = split_array

We created a 1D numpy array with six elements. The dtype for this array is a compound type: every element of the array is a tuple of (str, str, int, int, h5py.Reference, bool, str). The length of each string element has been chosen to be the maximum length we needed to store: that’s 5 for the split element ('train' being the longest split name) and 15 for the source element ('vector_features' being the longest source name). We didn’t include any comment, so the length for that element was set to 1. Due to a quirk in pickling empty strings, we put '.' as the comment value.

H5PYDataset expects the split attribute of the root node to contain as many elements as the cartesian product of all sources and all splits, i.e. all possible split/source combinations. Sometimes, no data is available for some source/split combination: for instance, the test set may not be labeled, and the ('test', 'targets') combination may not exist. In that case, you can set the available element for that combination to False, and H5PYDataset will ignore it.

Don’t worry too much about indices; we’ll get back to that later. For the moment, all you need to know is that since our splits are contiguous, we don’t need that feature and therefore put empty references.

The method described above does the job, but it’s not very convenient. An even simpler way of achieving the same result is to call create_split_array().

>>> from fuel.datasets.hdf5 import H5PYDataset
>>> split_dict = {
...     'train': {'vector_features': (0, 90), 'image_features': (0, 90),
...               'targets': (0, 90)},
...     'test': {'vector_features': (90, 100), 'image_features': (90, 100),
...              'targets': (90, 100)}}
>>> f.attrs['split'] = H5PYDataset.create_split_array(split_dict)

The create_split_array() method expects a dictionary mapping split names to dictionaries. Those dictionaries map source names to tuples of length 2, 3 or 4. The first two elements correspond to the start and stop indexes. The other two elements are optional and correspond to the indices reference and the comment, respectively. The method will create the array behind the scenes, choose the string lengths automatically and populate it with the information in the split dictionary. If a particular split/source combination isn’t present, its available attribute is set to False, which allows us to specify only what’s actually present in the HDF5 file we created.

We flush, close the file and voilà!

>>> f.flush()
>>> f.close()

Playing with H5PYDataset datasets¶

Let’s explore what we can do with the dataset we just created.

The simplest thing is to load it by giving its path and a tuple of split names:

>>> train_set = H5PYDataset('dataset.hdf5', which_sets=('train',))
>>> print(train_set.num_examples)
90
>>> test_set = H5PYDataset('dataset.hdf5', which_sets=('test',))
>>> print(test_set.num_examples)
10

Passing more than one split name would cause the splits to be concatenated. The available data sources would be the intersection of the sources provided by each split.

You can further restrict which examples are used by providing a slice object or a list of indices as the subset argument.

>>> train_set = H5PYDataset(
...     'dataset.hdf5', which_sets=('train',), subset=slice(0, 80))
>>> print(train_set.num_examples)
80
>>> valid_set = H5PYDataset(
...     'dataset.hdf5', which_sets=('train',), subset=slice(80, 90))
>>> print(valid_set.num_examples)
10

The available data sources are defined by the names of the datasets in the root node of the HDF5 file, and H5PYDataset automatically picked them up for us:

>>> print(train_set.provides_sources)
('image_features', 'targets', 'vector_features')

It also parsed axis labels, which are accessible through the axis_labels property, which is a dict mapping source names to a tuple of axis labels:

>>> print(train_set.axis_labels['image_features'])
('batch', 'channel', 'height', 'width')
>>> print(train_set.axis_labels['vector_features'])
('batch', 'feature')
>>> print(train_set.axis_labels['targets'])
('batch', 'index')

We can request data as usual:

>>> handle = train_set.open()
>>> data = train_set.get_data(handle, slice(0, 10))
>>> print((data[0].shape, data[1].shape, data[2].shape))
((10, 3, 5, 5), (10, 1), (10, 10))
>>> train_set.close(handle)

We can also request just the vector features:

>>> train_vector_features = H5PYDataset(
...     'dataset.hdf5', which_sets=('train',), subset=slice(0, 80),
...     sources=['vector_features'])
>>> handle = train_vector_features.open()
>>> data, = train_vector_features.get_data(handle, slice(0, 10))
>>> print(data.shape)
(10, 10)
>>> train_vector_features.close(handle)

Loading data in memory¶

Reading data off disk is inefficient compared to storing it in memory. Large datasets make it inevitable, but if your dataset is small enough that it fits into memory, you should take advantage of it.

In H5PYDataset, this is accomplished via the load_in_memory constructor argument. It has the effect of loading just what you requested, and nothing more.

>>> in_memory_train_vector_features = H5PYDataset(
...     'dataset.hdf5', which_sets=('train',), subset=slice(0, 80),
...     sources=['vector_features'], load_in_memory=True)
>>> data, = in_memory_train_vector_features.data_sources
>>> print(type(data)) 
<... 'numpy.ndarray'>
>>> print(data.shape)
(80, 10)

Non-contiguous splits¶

Sometimes it’s not possible to store the different splits contiguously. In that case, you’ll want to use the indices field of the H5PYDataset split array. A non-empty reference in that field overrides the start and stop fields, and the dataset the reference points to is used to determine the indices for that split/source pair.

Suppose that you’d like to use the even examples as your training set and the odd examples as your test set. We’ll start with the HDF5 file we populated earlier and manipulate its split attribute.

>>> f = h5py.File('dataset.hdf5', mode='a')
>>> f['train_indices'] = numpy.arange(0, 100, 2)
>>> train_ref = f['train_indices'].ref
>>> f['test_indices'] = numpy.arange(1, 100, 2)
>>> test_ref = f['test_indices'].ref
>>> split_dict = {
...     'train': {'vector_features': (-1, -1, train_ref),
...               'image_features': (-1, -1, train_ref),
...               'targets': (-1, -1, train_ref)},
...     'test': {'vector_features': (-1, -1, test_ref),
...              'image_features': (-1, -1, test_ref),
...              'targets': (-1, -1, test_ref)}}
>>> f.attrs['split'] = H5PYDataset.create_split_array(split_dict)
>>> f.flush()
>>> f.close()

We created two new datasets containing even and odd indices from 0 to 99, respectively, and passed references to these datasets in the split dict. In that case, the value we pass to start and stop really doesn’t matter, so we arbitrarily chose -1 for both.

Let’s check that the training and test set do contain even and odd examples:

>>> train_set = H5PYDataset(
...     'dataset.hdf5', which_sets=('train',), sources=('vector_features',))
>>> handle = train_set.open()
>>> print(
...     numpy.array_equal(
...         train_set.get_data(handle, slice(0, 50))[0],
...         numpy.vstack(
...             [numpy.load('train_vector_features.npy'),
...              numpy.load('test_vector_features.npy')])[::2]))
True
>>> train_set.close(handle)
>>> test_set = H5PYDataset(
...     'dataset.hdf5', which_sets=('test',), sources=('vector_features',))
>>> handle = test_set.open()
>>> print(
...     numpy.array_equal(
...         test_set.get_data(handle, slice(0, 50))[0],
...         numpy.vstack(
...             [numpy.load('train_vector_features.npy'),
...              numpy.load('test_vector_features.npy')])[1::2]))
True
>>> test_set.close(handle)

Variable-length data¶

H5PYDataset also supports variable length data. Let’s update the image features to reflect that:

>>> sizes = numpy.random.randint(3, 9, size=(100,))
>>> train_image_features = [
...     numpy.random.randint(256, size=(3, size, size)).astype('uint8')
...     for size in sizes[:90]]
>>> test_image_features = [
...     numpy.random.randint(256, size=(3, size, size)).astype('uint8')
...     for size in sizes[90:]]

In this new example, images have random shapes ranging from 3x3 pixels to 8x8 pixels.

First, we put the vector features and the targets inside the HDF5 file as before:

>>> f = h5py.File('dataset.hdf5', mode='w')
>>> f['vector_features'] = numpy.vstack(
...     [numpy.load('train_vector_features.npy'),
...      numpy.load('test_vector_features.npy')])
>>> f['targets'] = numpy.vstack(
...     [numpy.load('train_targets.npy'),
...      numpy.load('test_targets.npy')])
>>> f['vector_features'].dims[0].label = 'batch'
>>> f['vector_features'].dims[1].label = 'feature'
>>> f['targets'].dims[0].label = 'batch'
>>> f['targets'].dims[1].label = 'index'

We now have to put the variable-length images inside the HDF5 file. We can’t do that directly, since HDF5 and h5py don’t support multi-dimensional ragged arrays. However, there is support for 1D ragged arrays. Instead, we’ll flatten the images before putting them in the HDF5 file:

>>> all_image_features = train_image_features + test_image_features
>>> dtype = h5py.special_dtype(vlen=numpy.dtype('uint8'))
>>> image_features = f.create_dataset('image_features', (100,), dtype=dtype)
>>> image_features[...] = [image.flatten() for image in all_image_features]
>>> image_features.dims[0].label = 'batch'

If you’re feeling lost, have a look at the dedicated tutorial on variable-length data.

The images are now in the HDF5 file, but that doesn’t help us unless we can recover their original shape. For that, we’ll create a dimension scale that we’ll attach to the 'image_features' dataset using the name 'shapes' (use this exact name):

>>> image_features_shapes = f.create_dataset(
...     'image_features_shapes', (100, 3), dtype='int32')
>>> image_features_shapes[...] = numpy.array(
...     [image.shape for image in all_image_features])
>>> image_features.dims.create_scale(image_features_shapes, 'shapes')
>>> image_features.dims[0].attach_scale(image_features_shapes)

We’d also like to tag those variable-length dimensions with semantic information. We’ll create another dimension scale that we’ll attach to the 'image_features' dataset using the name 'shape_labels' (use this exact name):

>>> image_features_shape_labels = f.create_dataset(
...     'image_features_shape_labels', (3,), dtype='S7')
>>> image_features_shape_labels[...] = [
...     'channel'.encode('utf8'), 'height'.encode('utf8'),
...     'width'.encode('utf8')]
>>> image_features.dims.create_scale(
...     image_features_shape_labels, 'shape_labels')
>>> image_features.dims[0].attach_scale(image_features_shape_labels)

The H5PYDataset class will handle things from there on. When image features are loaded, it will retrieve their shapes and do the reshape automatically.

Lastly, we create the split dictionary exactly as before:

>>> split_dict = {
...     'train': {'vector_features': (0, 90), 'image_features': (0, 90),
...               'targets': (0, 90)},
...     'test': {'vector_features': (90, 100), 'image_features': (90, 100),
...              'targets': (90, 100)}}
>>> f.attrs['split'] = H5PYDataset.create_split_array(split_dict)
>>> f.flush()
>>> f.close()

That’s it. Now let’s kick the tires a little. The axis labels appear as they should:

>>> train_set = H5PYDataset(
...     'dataset.hdf5', which_sets=('train',), sources=('image_features',))
>>> print(train_set.axis_labels['image_features'])
('batch', 'channel', 'height', 'width')

H5PYDataset retrieves images of different shapes and automatically unflattens them:

>>> handle = train_set.open()
>>> images, = train_set.get_data(handle, slice(0, 10))
>>> train_set.close(handle)
>>> print(images[0].shape, images[1].shape)
(3, 6, 6) (3, 8, 8)

The object returned by get_data is a 1D numpy array of objects:

>>> print(type(images), images.dtype, images.shape) 
<... 'numpy.ndarray'> object (10,)

Toy example¶

For this tutorial, we’ll implement the venerable Iris dataset in Fuel. This dataset features 150 examples split into three classes (50 examples per class), and each example consists of four features.

For the purpose of demonstration, we’ll split the dataset into a training set (100 examples), a validation set (20 examples) and a test set (30 examples). We’ll pretend the test set doesn’t have any label information available, as is often the case for machine learning competitions.

Download code¶

The Iris dataset is contained in a single file, iris.data, which we’ll need to make available for users to download.

The preferred way of downloading dataset files in Fuel is the fuel-download script. Dataset implementations include a function for downloading their required files in the fuel.downloaders subpackage. In order to make that function accessible to fuel-download, they need to include it in the all_downloaders attribute of the fuel.downloaders subpackage.

The function accepts an argparse.ArgumentParser instance as input and should return a downloading function. Put the following piece of code inside the fuel.downloaders.iris module (you’ll have to create it):

from fuel.downloaders.base import default_downloader

def fill_subparser(subparser):
    subparser.set_defaults(
        urls=['https://archive.ics.uci.edu/ml/machine-learning-databases/'
              'iris/iris.data'],
        filenames=['iris.data'])
    return default_downloader

You should also register Iris as a downloadable dataset via the all_downloaders attribute. It’s a tuple of pairs of name and subparser filler function. Here’s an example of how the fuel.downloaders init file might look:

from fuel.downloaders import binarized_mnist
from fuel.downloaders import iris

all_downloaders = (
    ('binarized_mnist', binarized_mnist.fill_subparser),
    ('iris', iris.fill_subparser))

A lot is going on in these few lines of code, so let’s break it down.

In order to be more flexible, the fuel-download script uses subparsers. This lets each dataset define their own set of arguments. If you registered it properly, the function you just defined will get called and be given its own subparser to fill. Users will then be able to type the fuel-download iris command and iris.data will be downloaded.

When the fuel-download iris command is typed, the download script will call the function returned by fill_subparser and give it the argparse.Namespace instance containing all parsed command line arguments. That function is responsible for downloading the data.

We used the default_downloader() convenience function as our download function. It expects the parsed arguments to contain a list of URLs and a list of filenames, and downloads each URL, saving it under its corresponding filename. This is why we set the urls and filenames default arguments.

If your use case is more exotic, you can just as well define your own download function. Be aware of the following default arguments:

• directory : in which directory the files need to be saved
• clear : if True, your download function is expected to remove the downloaded files from directory.

Conversion code¶

In order to minimize the amount of code we have to write, we’ll subclass H5PYDataset. This means we’ll have to create an HDF5 file to store our data. For more information, see the dedicated tutorial on how to create an H5PYDataset-compatible HDF5 file.

Much like for downloading data files, the preferred way of converting data files in Fuel is through the fuel-convert script. Its implementation is very similar to fuel-download. The arguments to be aware of in the subparser are

• directory : in which directory the input files reside
• output-directory : where to save the converted dataset

The converter function should return a tuple containing path(s) to the converted dataset(s).

Put the following piece of code inside the fuel.converters.iris module (you’ll have to create it):

import os

import h5py
import numpy

from fuel.converters.base import fill_hdf5_file


def convert_iris(directory, output_directory, output_filename='iris.hdf5'):
    output_path = os.path.join(output_directory, output_filename)
    h5file = h5py.File(output_path, mode='w')
    classes = {'Iris-setosa': 0, 'Iris-versicolor': 1, 'Iris-virginica': 2}
    data = numpy.loadtxt(
        os.path.join(directory, 'iris.data'),
        converters={4: lambda x: classes[x]},
        delimiter=',')
    numpy.random.shuffle(data)
    features = data[:, :-1].astype('float32')
    targets = data[:, -1].astype('uint8')
    train_features = features[:100]
    train_targets = targets[:100]
    valid_features = features[100:120]
    valid_targets = targets[100:120]
    test_features = features[120:]
    data = (('train', 'features', train_features),
            ('train', 'targets', train_targets),
            ('valid', 'features', valid_features),
            ('valid', 'targets', valid_targets),
            ('test', 'features', test_features))
    fill_hdf5_file(h5file, data)
    h5file['features'].dims[0].label = 'batch'
    h5file['features'].dims[1].label = 'feature'
    h5file['targets'].dims[0].label = 'batch'
    h5file['targets'].dims[1].label = 'index'

    h5file.flush()
    h5file.close()

    return (output_path,)

def fill_subparser(subparser):
    return convert_iris

We used the convenience fill_hdf5_file() function to populate our HDF5 file and create the split array. This function expects a tuple of tuples, one per split/source pair, containing the split name, the source name, the data array and (optionally) a comment string.

We also used H5PYDataset‘s ability to extract axis labels to add semantic information to the axes of our data sources. This allowed us to specify that target values are categorical ('index‘). Note that you can use whatever label you want in Fuel, although certain frameworks using Fuel may have some hard-coded assumptions about which labels to use.

As for the download code, you should register Iris as a convertible dataset via the all_converters attribute of the fuel.converters subpackage. Here’s an example of how the init file might look:

from fuel.converters import binarized_mnist
from fuel.converters import iris

all_converters = (
    ('binarized_mnist', binarized_mnist.fill_subparser),
    ('iris', iris.fill_subparser))

Dataset subclass¶

Let’s now implement the H5PYDataset subclass that will interface with our newly-created HDF5 file.

One advantage of subclassing H5PYDataset is that the amount of code to write is very minimal:

from fuel.datasets import H5PYDataset
from fuel.utils import find_in_data_path


class Iris(H5PYDataset):
    filename = 'iris.hdf5'

    def __init__(self, which_sets=which_sets, **kwargs):
        kwargs.setdefault('load_in_memory', True)
        super(Iris, self).__init__(
            file_or_path=find_in_data_path(self.filename),
            which_sets=which_sets, **kwargs)

Our subclass is just a thin wrapper around the H5PYDataset class that defines the data path and switches the load_in_memory argument default to True (since this dataset easily fits in memory). Everything else is handled by the superclass.

Putting it together¶

We now have everything we need to start playing around with our new dataset implementation.

Try downloading and converting the data file:

cd $FUEL_DATA_PATH
fuel-download iris
fuel-convert iris
fuel-download iris --clear
cd -

You can now use the Iris dataset like you would use any other built-in dataset:

>>> from fuel.datasets.iris import Iris 
>>> train_set = Iris(('train',))
>>> print(train_set.axis_labels['features'])
('batch', 'feature')
>>> print(train_set.axis_labels['targets'])
('batch', 'index')
>>> handle = train_set.open()
>>> data = train_set.get_data(handle, slice(0, 10))
>>> print((data[0].shape, data[1].shape))
((10, 4), (10, 1))
>>> train_set.close(handle)

Working with external packages¶

To distribute Fuel-compatible downloaders and converters independently from Fuel, you have to write your own modules or subpackages which will define all_downloaders and all_converters. Here is how the Iris downloader and converter might look like if you were to include them as part of the my_fuel package:

# my_fuel/downloaders/iris_downloader.py
from fuel.downloaders.base import default_downloader

def fill_subparser(subparser):
    subparser.set_defaults(
        urls=['https://archive.ics.uci.edu/ml/machine-learning-databases/'
              'iris/iris.data'],
        filenames=['iris.data'])
    return default_downloader

# my_fuel/downloaders/__init__.py
from my_fuel.downloaders import iris

all_downloaders = (('iris', iris.fill_subparser),)

# my_fuel/converters/iris.py
import os

import h5py
import numpy

from fuel.converters.base import fill_hdf5_file


def convert_iris(directory, output_directory, output_filename='iris.hdf5'):
    output_path = os.path.join(output_directory, output_filename)
    h5file = h5py.File(output_path, mode='w')
    classes = {'Iris-setosa': 0, 'Iris-versicolor': 1, 'Iris-virginica': 2}
    # ...

def fill_subparser(subparser):
    return convert_iris

# my_fuel/converters/__init__.py
from my_fuel.converters import iris

all_converters = (('iris', iris.fill_subparser),)

In order to use the downloaders and converters defined in my_fuel, users would then have to set the extra_downloaders and extra_converters configuration variables inside ~/.fuelrc like so:

extra_downloaders: ['my_fuel.downloaders']
extra_converters: ['my_fuel.converters']

or set the FUEL_EXTRA_DOWNLOADERS and FUEL_EXTRA_CONVERTERS environment variables like so (this would override the extra_downloaders and extra_converters configuration variables):

$ export FUEL_EXTRA_DOWNLOADERS=my_fuel.downloaders
$ export FUEL_EXTRA_CONVERTERS=my_fuel.converters

New dataset classes¶

New dataset classes are implemented by subclassing Dataset and implementing its get_data() method. If your dataset interacts with stateful objects (e.g. files on disk), then you should also override the open() and close() methods.

If your data fits in memory, you can save yourself some time by inheriting from IndexableDataset. In that case, all you need to do is load the data as a dict mapping source names to their corresponding data and pass it to the superclass as the indexables argument.

For instance, here’s how you would implement a specialized class to interface with .npy files.

>>> from collections import OrderedDict
>>> import numpy
>>> from six import iteritems
>>> from fuel.datasets import IndexableDataset
>>>
>>> class NPYDataset(IndexableDataset):
...     def __init__(self, source_paths, **kwargs):
...         indexables = OrderedDict(
...             [(source, numpy.load(path)) for
...              source, path in iteritems(source_paths)])
...         super(NPYDataset, self).__init__(indexables, **kwargs)

Here’s this class in action:

>>> numpy.save('npy_dataset_features.npy',
...            numpy.arange(40).reshape((10, 4)))
>>> numpy.save('npy_dataset_targets.npy',
...            numpy.arange(10).reshape((10, 1)))
>>> dataset = NPYDataset(OrderedDict([('features', 'npy_dataset_features.npy'),
...                                   ('targets', 'npy_dataset_targets.npy')]))
>>> state = dataset.open()
>>> print(dataset.get_data(state=state, request=[0, 1, 2, 3]))
(array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15]]), array([[0],
       [1],
       [2],
       [3]]))
>>> dataset.close(state)

New iteration schemes¶

New iteration schemes are implemented by subclassing IterationScheme and implementing a get_request_iterator() method, which should return an iterator that returns lists of indices.

Two subclasses of IterationScheme typically serve as a basis for other iteration schemes: IndexScheme (for schemes requesting examples) and BatchScheme (for schemes requesting batches). Both subclasses are instantiated by providing a list of indices or the number of examples to visit (meaning that examples 0 through examples - 1 would be visited), and BatchScheme accepts an additional batch_size argument.

Here’s how you would implement an iteration scheme that iterates over even examples:

>>> from fuel.schemes import IndexScheme, BatchScheme
>>> # `iter_` : A picklable version of `iter`
>>> from picklable_itertools import iter_, imap
>>> # Partition all elements of a sequence into tuples of length at most n
>>> from picklable_itertools.extras import partition_all

>>> class ExampleEvenScheme(IndexScheme):
...     def get_request_iterator(self):
...         indices = list(self.indices)[::2]
...         return iter_(indices)
>>> class BatchEvenScheme(BatchScheme):
...     def get_request_iterator(self):
...         indices = list(self.indices)[::2]
...         return imap(list, partition_all(self.batch_size, indices))

Here are the two iteration scheme classes in action:

>>> print(list(ExampleEvenScheme(10).get_request_iterator()))
[0, 2, 4, 6, 8]
>>> print(list(BatchEvenScheme(10, 2).get_request_iterator()))
[[0, 2], [4, 6], [8]]

New transformers¶

An important thing to know about data streams is that they distinguish between two types of outputs: single examples, and batches of examples. Depending on your choice of iteration scheme, a data stream’s produces_examples property will either be True (it produces examples) or False (it produces batches).

Transformers are aware of this, and as such implement two distinct methods: transform_example() and transform_batch(). A new transformer is typically implemented by subclassing Transformer and implementing one or both of these methods.

As an example, here’s how you would double the value of some 'features' data source.

>>> from fuel.transformers import Transformer
>>>
>>> class FeaturesDoubler(Transformer):
...     def __init__(self, data_stream, **kwargs):
...         super(FeaturesDoubler, self).__init__(
...             data_stream=data_stream,
...             produces_examples=data_stream.produces_examples,
...             **kwargs)
...
...     def transform_example(self, example):
...         if 'features' in self.sources:
...             example = list(example)
...             index = self.sources.index('features')
...             example[index] *= 2
...             example = tuple(example)
...         return example
...
...     def transform_batch(self, batch):
...         if 'features' in self.sources:
...             batch = list(batch)
...             index = self.sources.index('features')
...             batch[index] *= 2
...             batch = tuple(batch)
...         return batch

Since we wish to support both batches and examples, we’ll declare our output type to be the same as our data stream’s output type.

If you were to build a transformer that only works on batches, you would pass produces_examples=False and implement only transform_batch(). If anyone tried to use your transformer on an example data stream, an error would automatically be raised.

Let’s test our doubler on some dummy dataset.

>>> from fuel.schemes import SequentialExampleScheme, SequentialScheme
>>> from fuel.streams import DataStream
>>>
>>> dataset = IndexableDataset(
...     indexables=OrderedDict([
...         ('features', numpy.array([1, 2, 3, 4])),
...         ('targets', numpy.array([-1, 1, -1, 1]))]))
>>> example_scheme = SequentialExampleScheme(examples=dataset.num_examples)
>>> example_stream = FeaturesDoubler(
...     data_stream=DataStream(
...         dataset=dataset, iteration_scheme=example_scheme))
>>> batch_scheme = SequentialScheme(
...     examples=dataset.num_examples, batch_size=2)
>>> batch_stream = FeaturesDoubler(
...     data_stream=DataStream(
...         dataset=dataset, iteration_scheme=batch_scheme))
>>> print([example for example in example_stream.get_epoch_iterator()])
[(2, -1), (4, 1), (6, -1), (8, 1)]
>>> print([batch for batch in batch_stream.get_epoch_iterator()])
[(array([2, 4]), array([-1,  1])), (array([6, 8]), array([-1,  1]))]

You may have noticed that in this example the transform_example() and transform_batch() implementations are the same. In such cases, you can subclass from AgnosticTransformer instead. It requires that you implement a transform_any() method, which will be called by both transform_example() and transform_batch().

>>> from fuel.transformers import AgnosticTransformer
>>>
>>> class FeaturesDoubler(AgnosticTransformer):
...     def __init__(self, data_stream, **kwargs):
...         super(FeaturesDoubler, self).__init__(
...             data_stream=data_stream,
...             produces_examples=data_stream.produces_examples,
...             **kwargs)
...
...     def transform_any(self, data):
...         if 'features' in self.sources:
...             data = list(data)
...             index = self.sources.index('features')
...             data[index] *= 2
...             data = tuple(data)
...         return data

So far so good, but our transformer applies the same transformation to every source in the dataset. What if we want to be more general and be able to select which sources we want to process with our transformer?

Transformers which are applied sourcewise like our doubler should usually subclass from SourcewiseTransformer. Their constructor takes an additional which_sources keyword argument specifying which sources to apply the transformer to. It’s expected to be a tuple of source names. If which_sources is None, then the transformer is applied to all sources. Subclasses of SourcewiseTransformer should implement a transform_source_example() method and/or a transform_source_batch() method, which apply on an individual source.

There also exists an AgnosticSourcewiseTransformer class for cases where the example and batch implementations of a sourcewise transformer are the same. This class requires a transform_any_source() method to be implemented.

>>> from fuel.transformers import AgnosticSourcewiseTransformer
>>>
>>> class Doubler(AgnosticSourcewiseTransformer):
...     def __init__(self, data_stream, **kwargs):
...         super(Doubler, self).__init__(
...             data_stream=data_stream,
...             produces_examples=data_stream.produces_examples,
...             **kwargs)
...
...     def transform_any_source(self, source, _):
...         return 2 * source

Let’s try this implementation on our dummy dataset.

>>> target_stream = Doubler(
...     data_stream=DataStream(
...         dataset=dataset,
...         iteration_scheme=batch_scheme),
...     which_sources=('targets',))
>>> all_stream = Doubler(
...     data_stream=DataStream(
...         dataset=dataset,
...         iteration_scheme=batch_scheme),
...     which_sources=None)
>>> print([batch for batch in target_stream.get_epoch_iterator()])
[(array([1, 2]), array([-2,  2])), (array([3, 4]), array([-2,  2]))]
>>> print([batch for batch in all_stream.get_epoch_iterator()])
[(array([2, 4]), array([-2,  2])), (array([6, 8]), array([-2,  2]))]

Finally, what if we not only want to select at runtime which sources the transformation applies to, but also the transformer itself? This is what the Mapping transformer solves. In addition to a data stream, its constructor accepts a mapping argument, which is expected to be a function. This function which will be applied to data coming from the stream.

Here’s how you would implement the feature doubler using Mapping.

>>> from fuel.transformers import Mapping
>>>
>>> features_index = dataset.sources.index('features')
>>> def double(data):
...     data = list(data)
...     data[features_index] *= 2
...     return tuple(data)
>>> mapping_stream = Mapping(
...     data_stream=DataStream(
...         dataset=dataset, iteration_scheme=batch_scheme),
...     mapping=double)
>>> print([batch for batch in mapping_stream.get_epoch_iterator()])
[(array([2, 4]), array([-1,  1])), (array([6, 8]), array([-1,  1]))]