Welcome to tflon’s documentation!¶

What is tflon?¶

Tflon is an up-and-coming deep learning toolkit designed to streamline tensorflow model and data pipeline construction while enabling complex, dynamically structured operations on data to foster innovative deep learning research.

Currently, tflon’s most developed use case is executing computations on graph-structure data, such as chemical data, using the wave architecture.

Ultimately, tflon aims to provide three key features.

A flexible data input model, which uses python multiprocessing to enable throughput without necessitating custom tensorflow extensions for handling complex data.
A modular model construction framework, which supports complex, dynamically structured operations on data, and can be mixed with pure tensorflow, or alternative APIs such as keras or sonnet.
Seamless, intuitive integration with other tools including: data storage (pyarrow, parquet), distributed execution (horovod, MPI), profiling (pyflame, tfprof), resource monitoring (psutil, GPUtil), and others.

Currently, we are still a long way off from achieving these goals. If you are interested in contributing, please let us know!

Table of Contents¶

Introduction¶

Tutorials¶

Tutorial: Welcome¶

Overview¶

In this tutorial, we walk through how to build a logistic regressor in Tflon using Wisconsin Diagnostic Breast Cancer dataset as our example. We will build on this example extensively to demonstrate how to fully utilize the Tflon deep learning package and extend it with custom features. We will assume a working knowledge of pandas and object oriented programming in python. Futher, we recommend Deep Learning by Benjio as this tutorial doesn’t serve as a replacement for solid practical and theoretical understanding of deep learning.

Setup¶

Clone the latest tflon repository and ensure the following requirements:

pandas 0.21.0+
tensorflow 1.6.0+
horovod 0.13.4+
mpi4py 3.0.0+
mpirun 2.1.0+

Instalation¶

You can find the source code at https://bitbucket.org/mkmatlock/tflon, and install it using Mercurial:

hg install https://<your username>@bitbucket.org/mkmatlock/tflon/src/default/
pip install ./tflon

Imports¶

Before using tflon in python, we use the following lines to import the necessary dependancies

import pandas as pd
import tensorflow as tf
import tflon

Environment Reset¶

By the nature of Tensorflow which maintains a stateful representation of the network being trained. In order to instantiate another network, we have to reset the system with

tflon.system.reset()

This will reset the tensorflow stored network. So, that you can define and train another model within the same python file or jupyter notebook kernel.

Example Dataset¶

For the purposes of this tutorial, we will be using the Wisconsin Diagnostic Breast Cancer (WDBC) dataset found at https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data

The data has 30 real valued features including

radius (mean of distances from center to points on the perimeter)
texture (standard deviation of gray-scale values)
perimeter
area
smoothness (local variation in radius lengths)
compactness (perimeter^2 / area - 1.0)
concavity (severity of concave portions of the contour)
concave points (number of concave portions of the contour)
symmetry
fractal dimension (“coastline approximation” - 1)

We seek to construct a classifier to determine if the biometrics correspond to a malignant (212) or benign (357) using the 569 samples.

Logistic Regressor¶

Check out http://www.win-vector.com/blog/2011/09/the-simpler-derivation-of-logistic-regression/ if you’re not familiar with it. Basically, you can think of it as a best linear fit of the log odds ratio minimizing the pseudo-\(R^2 = 1-\frac {\text{deviance}}{\text{null deviance}}\).

From the provided link:

Logistic regression models are multiplicative in their inputs.
The exponent of each coefficient tells you how a unit change in that input variable affects the odds ratio of the response being true.
Logistic regression is coordinate-free: translations, rotations, and rescaling of the input variables will not affect the resulting probabilities.
Logistic regression preserves the marginal probabilities of the training data.
Overly large coefficient magnitudes, overly large error bars on the coefficient estimates, and the wrong sign on a coefficient could be indications of correlated inputs.
Coefficients that tend to infinity could be a sign that an input is perfectly correlated with a subset of your responses. Or put another way, it could be a sign that this input is only really useful on a subset of your data, so perhaps it is time to segment the data.
Pseudo-\(R^2\) is a useful goodness-of-fit heuristic.

Pre-Proccessing¶

The data is in the form of comma separated values(csv): ID, Malignant/Benign, Features. Using pandas, we extract the targets (‘targ’) with a binary encoding 1 for malignant and 0 otherwise. Informally, the targets are the observed labels we are trying to learn and ‘desc’ for the descriptors which are the features we are learning from.

df = pd.read_csv("./wdbc.data" , header=None)
df[1] = df[1].apply(lambda x : 1 if (x == 'M') else 0)
df = df.iloc[:,1:]
targ, desc = tflon.data.Table(df).split([1])

Then, we create a tflon Tablefeed which essentially stores the ‘targ’ and ‘desc’ as a dictionary making for efficient training.

feed = tflon.data.TableFeed({'desc':desc, 'targ':targ})

Defining a Model¶

For this example, we are simply building a ridge logistic regressor so it suffices to feed the normalized input into a single dense neuron with a sigmoidal activation.

To define a new model, we define a new subclass of tflon.model.Model and override the _model attribute. Using add_input and add_target, we indicate that the input should have 30 features and a single target. Again, a target is the observed labels that we want our network to learn from.

class MyModel(tflon.model.Model):
    def _model(self):
        I = self.add_input('desc', shape=[None, 30])
        T = self.add_target('targ', shape=[None, 1])

Architecture¶

Now in defining the actual architecture, we appeal to the toolkit subpackage which has layers available for us.

WIn = tflon.toolkit.WindowInput()
Dense = tflon.toolkit.Dense(1)
net = WIn(Dense)
out = net(I)

To be more concise, we can stack layers ontop of each other with the ‘|’ symbol allowing us to construct L which is the instantitation of the defined architecture

net = tflon.toolkit.WindowInput() |\
      tflon.toolkit.Dense(1)

out = net(I)

In the case of the WindowInput function, it normalizes each feature with the following function

\[\frac {x - min}{max-min}\]

As for Dense(\(x\)), it simply constructs fully connected neuron layer with \(x\) neurons. For us, we only use 1 neuron, so \(x = 1\).

To be truly minimalistic, this layer can be skipped all together:

net= tflon.toolkit.Dense(1)

This is the main area of design for customizing the model that we build. Additionally, the recommend way of adding new custom functionality to tflon is by adding a new layer with the ‘|’ symbol. We’ll discuss in more detail how to define new layers here

Prediction¶

The prediction is the final pass made on the output which in our case by choosing sigmoidal activation we result in a logistic regressor.

self.add_output( 'pred', tf.nn.sigmoid(out) )

Objective Function¶

In a logistic regressor there is a objective function or a cost function that is minimiazed to definine the best fit. The L2 is an artifact of our choice to build a ridge regressor.

self.add_loss( 'xent', tflon.toolkit.xent_uniform_sum(T, out) )
self.add_loss( 'l2', tflon.toolkit.l2_penalty(self.weights) )

Checkout the full list of loss functions in the toolkit here.

Metrics¶

Finally, we define some metrics that allow us to judge the performance of our logistic regressor. In particular, here we have used the Area Under the Reciever Operating Characteristic curve (AUC) as our metric which gauges sensitivity (true positive rate) and specificity (false positive rate). We can then retrieve the value later as will be mentioned shortly.

self.add_metric( 'auc', tflon.toolkit.auc(T, out) )

Putting it together¶

``` python class MyModel(tflon.model.Model): def _model(self): I = self.add_input(‘desc’, shape=[None, 30]) T = self.add_target(‘targ’, shape=[None, 1])

net = tflon.toolkit.WindowInput() |\
      tflon.toolkit.Dense(1)
out = net(I)

self.add_output( 'pred', tf.nn.sigmoid(out) )
self.add_loss( 'xent', tflon.toolkit.xent_uniform_sum(T, out) )
self.add_loss( 'l2', tflon.toolkit.l2_penalty(self.weights) )
self.add_metric( 'auc', tflon.toolkit.auc(T, out) )
```

Train the Model¶

To train the model, we first instantiate the model and define our trainer:

python LR = MyModel() trainer = tflon.train.OpenOptTrainer( iterations=100)

Then exactly synonomous to tensorflow, we execute the training from within a session which can be thought of as denoting an environment after which background bookkeeping will be erased.

with tf.Session():
    LR.fit( feed, trainer, restarts=2 )
    metrics = LR.evaluate(feed)
    print "AUC:", metrics['auc']

Note that here we are able to use evaluate to retrieve our AUC.asdf

Save and Load Trained Model¶

Once we’re trained a model we want to be able to reuse it in the future. So it’s crucial that we mention how to save and load models here.

To save a model we simply use the save function within the session but after we do the fit. This will produce a pickle file

LR.save("saved_lr.pkl")

Then next time when we want to load the model back, we can simply use load. Note that you have to manually initialize the model before use.

LR = tflon.model.Model.load("saved_lr.pkl")
with tf.Session():
    LR.initialize()
    x = LR.evaluate(feed)
    print "AUC:", metrics['auc']

Putting it all together¶

Congratulations our ridge logistic regressor is ready!

import pandas as pd
import tensorflow as tf
import tflon

class MyModel(tflon.model.Model):
    def _model(self):
        I = self.add_input('desc', shape=[None, 30])
        T = self.add_target('targ', shape=[None, 1])

        net = tflon.toolkit.WindowInput() |\
              tflon.toolkit.Dense(1)
        out = net(I)

        self.add_output( 'pred', tf.nn.sigmoid(out) )
        self.add_loss( 'xent', tflon.toolkit.xent_uniform_sum(T, out) )
        self.add_loss( 'l2', tflon.toolkit.l2_penalty(self.weights) )
        self.add_metric( 'auc', tflon.toolkit.auc(T, out) )

if __name__=='__main__':

    df = pd.read_csv("./wdbc.data" , header=None)
    df[1] = df[1].apply(lambda x : 1 if (x == 'M') else 0)
    df = df.iloc[:,1:]
    targ, desc = tflon.data.Table(df).split([1])
    feed = tflon.data.TableFeed({'desc':desc, 'targ':targ})

    LR = MyModel()

    trainer = tflon.train.OpenOptTrainer( iterations=100)

    with tf.Session():
        LR.fit( feed, trainer, restarts=2 )
        LR.save("saved_lr.p")
        metrics = LR.evaluate(feed)
        print "AUC:", metrics['auc']

Sample Output¶

2018-06-11 13:28:38.304284: I Found a solution with loss 7.27e+01 AUC: 0.99299717

Tutorial: Neural Network¶

Overview¶

In this section, we are building a neural network with minor tweaks to the ridge logistic regressor in the previous section.

Architecture¶

We’re going to add a simple fully connected (dense) layer between the input and the target neuron, which will allow our predictor to be more able to represent the data. Here’s a diagram of our desired model:

In tflon, it’s as simple as adding

tflon.toolkit.Dense(5, activation=tf.tanh)

Giving us:

class NNModel(tflon.model.Model):
    def _model(self):
        I = self.add_input('desc', shape=[None, 30])
        T = self.add_target('targ', shape=[None, 1])

        net = tflon.toolkit.WindowInput() |\
              tflon.toolkit.Dense(5, activation=tf.tanh) |\
              tflon.toolkit.Dense(1)
        out = net(I)

        self.add_output( 'pred', tf.nn.sigmoid(out) )
        self.add_loss( 'xent', tflon.toolkit.xent_uniform_sum(T, out) )
        self.add_loss( 'l2', tflon.toolkit.l2_penalty(self.weights) )
        self.add_metric( 'auc', tflon.toolkit.auc(T, out) )

Variants¶

Multiple output regression¶

For many Neural Networks we would like to predict multiple targets real valued targets. This can be done convienently using Dense and modifying add_target. Since it’s a real value regressor, we will used the common least mean squared error instead of crossentropy.

class MNNModel(tflon.model.Model):
    def _model(self):
        I = self.add_input('desc', shape=[None, 29])
        T = self.add_target('targ', shape=[None, 2])

        net = tflon.toolkit.WindowInput() |\
              tflon.toolkit.Dense(5, activation=tf.tanh) |\
              tflon.toolkit.Dense(2)
        out = net(I)

        self.add_output( 'pred', tf.nn.sigmoid(out) )
        self.add_loss('loss', tf.reduce_sum(tf.square(pred - out)))
        self.add_loss( 'l2', tflon.toolkit.l2_penalty(self.weights) )
        self.add_metric( 'auc', tflon.toolkit.auc(T, out) )

For use with the breast cancer data, the following line in the preprocessing will also need to be modified to set two columns for the target instead of one.

targ, desc = tflon.data.Table(df).split([2])

Generalization¶

The goal in building deep learning models is to train the network to best respond to unobserved data. Thus, it’s important to construct a model that doesn’t overfit or ‘memorizes’ the training data.

Validation and Test Set¶

Holdout: It’s best practice to reserve a holdout set partitioned before any parameter tuning. This ensures that at the end this dataset can be used as an unbiased estimate of the generalizaiton error. We refer to this set as the holdout/test set and the rest the training set.

As expected, this can easily be achieved by adding the following line after we define our feed. We holdout 10% for testing.

feed = tflon.data.TableFeed({'desc':desc, 'targ':targ})
IDs = np.random.randint(569 -1, shape=569//10)
testing =  feed.holdout(IDs)

Then, we can evaluate the result. test_result is an array of the model’s predictions while test_AUC provides the AUC.

with tf.Session():
    LR.fit( feed, trainer, restarts=2 )
    test_result = NN.infer(testing,'pred')
    test_AUC = NN.evaluate(testing)['AUC']

Crossvalidation: Simply put crossvalidation splits the data into fragments using each fragment as validation once. This allows us to use all the data in the training set for validation when evaluating the effect of different parameter settings. We will not show it explicitly, but effectively we implement this by applying holdout to different subsets.

AUC: 0.9991015

In [ ]:

import pandas as pd
import tensorflow as tf
import tflon
tflon.system.reset()
class LRModel(tflon.model.Model):
    def _model(self):
        I = self.add_input('desc', shape=[None, 29])
        T = self.add_target('targ', shape=[None, 1])

        net = tflon.toolkit.WindowInput() |\
                tflon.toolkit.Apply(lambda x: tf.nn.convolution())|\
        tflon.toolkit.Dense(5, activation=tf.tanh) | tflon.toolkit.Dense(1)
        L = net(I)

        self.add_output( 'pred', L )
        self.add_loss('loss', tf.reduce_sum(tf.square(pred - out)))
        self.add_loss( 'L2', tflon.toolkit.l2_penalty(self.weights))
        self.add_metric( 'auc', tflon.toolkit.auc(T, L) )

df = pd.read_csv("~/wdbc.data" , header=None)
df[1] = df[1].apply(lambda x : 1 if (x == 'M') else 0)
df = df.iloc[:,2:]
targ, desc = tflon.data.Table(df).split([1])
feed = tflon.data.TableFeed({'desc':desc, 'targ':targ})

LR = LRModel()

trainer = tflon.train.OpenOptTrainer( iterations=100)

with tf.Session():
    LR.fit( feed, trainer, restarts=2 )
    metrics = LR.evaluate(feed)
    print "AUC:", metrics['auc']

Tutorial: Inputs¶

Overview¶

Currently this page is a stub intended to introduce some examples of importing graphs and nested tables. If this is of interest to you, feel free to peruse the documentation to find the functions you need.

Tutorial: Convolutional NN¶

Overview¶

In this section, we are building a convolutional NN to demonstrate how to integrate tensorflow functionality into tflon workflow.

Architecture¶

Although the Breast Cancer Data set is not natural for convolution, convolution is particularly well suited for processing data such as images where there is translational symmetry. For our example, we add convolution, exponential linear unit (elu), and max pooling preceeding our dense layers.

A simple way of doing this is to leverage the implementations tensorflow. Further, this example allows us to demonstrate how to seamlessly integrate tensorflow into our framework.

Model¶

We’re going to add an additional convolutional-maxpooling layer to our single layered neural network replacing the normalization layer.

In short, convolution is achieved by augmenting filters of smaller but fixed dimensions across the input features. By filter we mean a fully connected neurons over a subset of the input.

As mentioned in the Welcome tutorial, each layer in the definition expressed as a nested function. Thus, seek to apply tensorflow functions to the input before passing it to the toolkit modules. The key layers we’re adding are conv2d , exponential linear unit (elu), and max_pool. We recommend reading about the functions in tensorflow if you are not familiar with convolution.

For reference, the conv2d has the following signature:

tf.nn.conv2d( input, filter=[filter_height, filter_width, in_channels, out_channels], strides, padding )

In our example, we use a 3x3 filter striding by only 1 in each direction so that we saturate the input. Futher, we arbitrarially assume the image is single channel (ie gray scale). Then the output of the layer can have multiple channels representing the network learning to recognize certain visual patterns. In our case, we arbitrarially allow for 16 features.

weight = self.get_weight('w' , [3,3,1,16])

In the past, we defined the net function which we applied to the input.

out = net(I)

Now instead, we apply the tensorflow functions to the input producing nxt before passing it to net to add the dense layers.

out = net(nxt)

Giving us:

class ConvNN(tflon.model.Model):
   def _model(self):

       width = 16
       depth = 1
       acti = tf.nn.elu
       num_pixel_targets = 1

       I = self.add_input('images', shape=[None, None, None, depth])
       T = self.add_target('solution', shape=[None, None, None, num_pixel_targets])

       Lin = 1
       Lout = width
       weight = self.get_weight('w' , [3,3,Lin,Lout])
       bias = self.get_bias('b', [Lout])

       conv = tf.nn.conv2d(I, weight, [1,1,1,1], padding='SAME')
       nonlin = tf.nn.elu(conv+bias)
       conv_out = tf.nn.max_pool(nonlin, [1,2,2,1], [1,1,1,1], padding='SAME')

       net = tflon.toolkit.Dense(5, activation=acti) \
             | tflon.toolkit.Dense(1, activation = acti, name='output')

       out=net(conv_out)
       self.add_output('solution', out)
       self.add_loss('loss', tf.reduce_sum(tf.square(T - out)))

Custom Parameters¶

Now we introduct a final modification that allows us to customize the model. We migrate our hardcoded arbitrary choices including width so that we can customize it when we call it.

model = ConvNN(width= 32, activation=tf.nn.relu)

To complement this change, we must inform the _model to look for parameter values using self.inject_parameters().

class ConvNN(tflon.model.Model):
   def _model(self):
       self.inject_parameters()
       num_pixel_targets = 1

       I = self.add_input('images', shape=[None, None, None, self.depth])

       T = self.add_target('solution', shape=[None, None, None, num_pixel_targets])

       Lin = 1
       Lout = self.width
       weight = self.get_weight('w' , [3,3,Lin,Lout])
       bias = self.get_bias('b', [Lout])
       conv = tf.nn.conv2d(I, weight, [1,1,1,1], padding='SAME')
       nonlin = tf.nn.elu(conv+bias)
       conv_out = tf.nn.max_pool(nonlin, [1,2,2,1], [1,1,1,1], padding='SAME')

       net = tflon.toolkit.Dense(5, activation=self.activation) \
             | tflon.toolkit.Dense(1, activation = self.activation, name='output')

       out=net(conv_out)
       self.add_output('solution', out)
       self.add_loss('loss', tf.reduce_sum(tf.square(T - out)))

   def _parameters(self):
       return {'width': 16,
               'depth':1,
               'activation': tf.nn.elu,}

AUC: 0.9991015

Tutorial: Distributed¶

Overview¶

In this section, we are building a distributed neural network. The key part that needs to be handled from the user perspective is how the data will be divided when passed to different computing units.

The underlying distributed API is a wrapper over Horovod the open source component of Uber’s Michelangelo. https://github.com/uber/horovod Michealangelo was designed to deploy and scale distributed deep learning projects over multiple GPUs in a network.

The remainder of the distributed book-keeping is managed with the MPI package http://mpi4py.scipy.org/docs/. The beauty of MPI is that it handles both multithreaded and multiserver synchronization ( for the purposes here we’ll call both distributed even though the term is traditionally applied to just the latter).

Schema¶

In short, the _schema method in model describes where the data is stored and how the retrieve it. This means as it is set up right now, we need to create a folder with the data broken up into different files (shards). Then define a corresponding _schema which overrides the default.

Shard Example¶

Say we originally have the following directory structure:

data
-->full_data.tsv

Then, we can divide the full_data.tsv into by using unix head and tail:

head -100 data/full_data.tsv > data/shard0/shard0.tsv
tail -n +101 > data/shard1/shard1.tsv

This gives us the following directory structure:

data
--->full_data.tsv
--->shard0
---/--->shard0.tsv
--->shard1
---/--->shard1.tsv

This lets us train simultaneously on shard0 and shard1

In our example, we define tsv_reader, which accesses the data with pandas and provide for the descriptors and target the following three arguments: file name, table class, and helper function in our case tsv_reader.

def _schema(self):
    tsv_reader = lambda fpath: pd.read_csv(fpath, sep='\t',
                               dtype={'ID':str}).set_index('ID')
    return {'desc': ('descriptors.tsv', tflon.data.Table, tsv_reader),
            'targ': ('targets.tsv', tflon.data.Table, tsv_reader)}

Distributed Feeds and Distributed Trainer¶

Thankfully, most of hardwork has already been done. It simply suffices to interface the functionality from tflon.distributed

Setup¶

First, we need to initialize horovod and gpu resources with a 1000ms timeout if training gets stalled:

tflon.data.TensorQueue.DEFAULT_TIMEOUT=1000
config = tflon.distributed.init_distributed_resources()

Trainer¶

We use the built-in DistributedTrainer which interfaces just like the ordinary trainer, OpenOptTrainer. According to the creators, Adam Optimization takes advantage of

"AdaGrad (Duchi et al., 2011), which works well with sparse gradients,
and RMSProp (Tieleman & Hinton, 2012), which works well in on-line and non-stationary
settings"

This explicitly is implemented as

trainer = tflon.distributed.DistributedTrainer( tf.train.AdamOptimizer(1e-3),
            iterations=1000 )

Feed¶

For defining the feed, we use make_distributed_table_feed. The first argument is the path to the directory containing the shards(subdirectories with partitions of the data). The second argument denotes the scheme. The master_table argument is simply indicating which key in the table to use as the index. By default, the feed will divide up the shards evenly between available processors.

feed = tflon.distributed.make_distributed_table_feed(
        resource_filename('tflon_test.data', 'distributed'),
        NN.schema, master_table='desc', partition_strategy='all' )

Executing the model¶

Since the distributed table is implemented with MPI, you will need to run it with mpirun and in our case two threads:

mpirun -np 2 python ./distributed.py

The table will not work properly and throw an error if mpi threads have no data, so the number of threads requested with the np option must be less than the number of shards

Warning¶

Due to the way MPI is designed, the is_master with anything that involved distributed data. For example, anything during the training phase. Only use is_master post-training. In our example, we use it for the evaluation.

Example¶

Below is an example of putting all these changes together:

import tflon
import tensorflow as tf
import pandas as pd
from pkg_resources import resource_filename

class NeuralNet(tflon.model.Model):
    def _schema(self):
        tsv_reader = lambda fpath: pd.read_csv(fpath, sep='\t',
                                    dtype={'ID':str}).set_index('ID')
        return {'desc': ('descriptors.tsv', tflon.data.Table, tsv_reader),
                'targ': ('targets.tsv', tflon.data.Table, tsv_reader)}

    def _model(self):
        I = self.add_input('desc', shape=[None, 210])
        T = self.add_target('targ', shape=[None, 1])
        net = tflon.toolkit.WindowInput() |\
              tflon.toolkit.Dense(20, activation=tf.tanh) |\
              tflon.toolkit.Dense(5, activation=tf.tanh) |\
              tflon.toolkit.Dense(1)
        L = net(I)

        self.add_output( "pred", tf.nn.sigmoid(L) )
        self.add_loss( "xent", tflon.toolkit.xent_uniform_sum(T, L) )
        self.add_loss( "l2", tflon.toolkit.l2_penalty(self.weights) )
        self.add_metric( 'auc', tflon.toolkit.auc(T, L) )

if __name__=='__main__':
    tflon.data.TensorQueue.DEFAULT_TIMEOUT=1000
    # Initialize horovod and setup gpu resources
    config = tflon.distributed.init_distributed_resources()

    graph = tf.Graph()
    with graph.as_default():
        # Add a model instance
        NN = NeuralNet(use_gpu=True)

        # Create the distributed trainer
        trainer = tflon.distributed.DistributedTrainer( tf.train.AdamOptimizer(1e-3),
                    iterations=1000 )

    # Create the data feed, use the same feed for all process instances
    # tflon.distributed.DistributedTable adds MPI synchronization to the Table API min and max ops
    # Usually, different data would be loaded on each process (see tflon.distributed.make_distributed_table_feed)
    feed = tflon.distributed.make_distributed_table_feed(
            resource_filename('tflon_test.data', 'distributed'),
            NN.schema, master_table='desc', partition_strategy='all' )

    with tf.Session(graph=graph, config=config):
        # Train with minibatch size 100
        NN.fit( feed.shuffle(batch_size=100), trainer, restarts=2, source_tables=feed )

        # Perform inference on the master process
        if tflon.distributed.is_master():
            metrics = NN.evaluate( feed )
            print "AUC:", metrics['auc']

AUC: 0.9991015

In [ ]:

Tutorial: Wave Architecture¶

Overview¶

In this section, we demonstrate our home grown wave architecture implemented in tflon. This example demonstrates a non trivial use case with a more complex architecture than we had previously.

Architecture¶

Here we implement it with a built in Graph Recurrent Neural Network (GRNN) which is a fundamental unit introduced for the construction of Wave. In short, Wave aspires to improve long range information propogation. This improvement was demonstrated to be particularly useful for operating on graphs representing molecules.

See our paper for more details.

Tutorial: Extending Tflon¶

Overview¶

In this section we demonstrate how to add additional functionality to tflon. For now there’s also just miscellaneous tips that haven’t found a home yet

Tensorflow Basics¶

It’s a good idea to look at https://tensorflow.org as a reference:

Roughly speaking, Tensorflow represents the network as a graph: collection of neurons and the connections between neurons. This graph of modular pieces can then be distributed across multiple computational units and coordinated in order to optimize training and applying the network.

Pitfall¶

By the nature of Tensorflow which maintains a stateful representation of the network being trained. In order to instantiate another network, we have to reset the system with

tflon.system.reset()

This will automatically reset the tensorflow stored network. So, that you can define and train another model within the same python file.

Code Structure¶

Before we add anything to tflon, let’s understand how the code base is organized. Here is a control flow diagram for tflon execution:

First off adding Module should be done by implementing the tflon.toolkit.Module interface

from tflon.toolkit import Module

In particular, there are three main functions in Module that are useful to override: - build: called when the module is initialized (where static variables are populated)

call: called when the network is constructed (gets called )
show_data: gets called before data is fed into the module (useful for preprocessing, eg WindowInput)
featurize: reshapes the data per batch

build and call have no default so any class that implements the module interface must have these two functions defined.

show_data and featurize are called when MyModel.fit is called

tflon/toolkit/modules.py has a great example of implementing custom modules

Hint: Since __getattr__ in Module searches the Tower instance, effectively all the attributes of Tower can be thought of as part of Module.

Example with Highway¶

We use Highway as an example of how to add a module. The source code is in tflon/toolkit/module.py

See a description of the algorithm here: https://arxiv.org/abs/1505.00387.

class Highway(Module):

    def build(self, block):
        self._gate = Dense(self.input_shapes[0][-1],
            activation=tf.nn.sigmoid, bias_initializer=tf.constant_initializer(-1.))
        self._block = block

    def call(self, inputs):
        hidden = self._block(inputs)
        gate = self._gate(inputs)
        return hidden * gate + inputs * (1-gate)

Incidentally, we can just use the default show_data and featurize

Crash Course on @property¶

Highly recommend looking at this description if you’re not familiar with @property in python

https://www.programiz.com/python-programming/property

Key Takeaways - ‘_var’ is a convention and python doesn’t treat it any different than any other variable - ‘@’ is a decorator so this

@property
def fn():
    return

becomes

def fn():
    return
fn = property(fn)

fn = property(fn) takes fn and reassigns fn to be have like a private variable
- defines a _fn as well as implicitly defines set and get functions -in partictular, the body of fn becomes the get function
- so now self._fn contains the implicit data value
- but self.fn will apply the body of fn

In [ ]:

API documentation¶

Chemistry¶

Molecule API¶

class tflon.chem.molecule.Molecule(graph, node_properties=[], edge_properties=[])¶

Defines a networkx based representation of a molecule for dynamic ML computations

classmethod from_json(serialized)¶

Generate a networkx representation of a serialized json format molecule

Parameters:	serialized (str) – The serialized json representation

heavy_atoms¶: Get a list of heavy atom indices

heavy_bonds¶

Get a list of heavy bonds

Returns:	A list of tuples of atom indices

pymol¶: Convert this Molecule to a pybel.Molecule

topological_groups¶

Compute topological groups by exhaustive path enumeration (warning – still very slow, use pymol_topological_groups instead)

Returns:	a dictionary with keys of atom indices and values topological group (int)
Return type:	atom_topology (dict)

tflon.chem.molecule.atom_to_parquet(molecules, data, output_fn=None, numeric_ids=False, topological_index=False)¶

Generate a parquet file containing atom-level data from a pandas.DataFrame.

The input atom-level data has indexes of the form molecule_id.atom_index, with one atom per row

Each column of the data is condensed to a list of lists format, with all atoms for a given molecule appearing on one row, with the molecule ID as row ID.

Atoms are sorted by their index. Values in each list appear in this order. Atoms with missing data appear as NaN. This may occur with data files generated without hydrogens.

Keyword Arguments:
Parameters:	molecules (DataFrame) – A dataframe containing tflon.chem.Molecule objects data (DataFrame) – Pandas dataframe containing atom-level data output_fn (str) – Path to an output parquet file
	numeric_ids (bool) – Expect molecule_id to be the index of the molecule instead of the title (default = False) topological_index (bool) – Expect the atom index of the input data frame contain the topological index (T) in the format molID.T.atomID (default = False)

tflon.chem.molecule.bond_to_parquet(molecules, data, output_fn, numeric_ids=False, lonepair=False, topological_index=False)¶

Generate a parquet file containing bond-level data from a pandas.DataFrame.

The input bond-level data has indexes of the form molecule_id.atom1_index.atom2_index, with one bond per row

Each column of the data is condensed to a list of lists format, with all bonds for a given molecule appearing on one row, with the molecule ID as row ID.

Bonds are sorted by the natural sort on tuples of atom pair indices (i, j), with i < j. Values in each list appear in this order. Bonds with missing data appear as NaN. This may occur with data files generated without hydrogens.

Keyword Arguments:
Parameters:	molecules (DataFrame) – A dataframe containing tflon.chem.Molecule objects data (DataFrame) – Pandas dataframe containing bond-level data output_fn (str) – Path to an output parquet file
	numeric_ids (bool) – Expect molecule_id to be the index of the molecule instead of the title (default = False) topological_index (bool) – Expect the atom index of the input data frame contain the topological index (T) in the format molID.T.atomID (default = False)

tflon.chem.molecule.pymol_to_json(mol, addhs=True, kekulize=True, lonepair=False, coordinates=False)¶

Convert pybel Molecule to json format for storage and import to Molecule class.

If hydrogens are interspersed, pybel preserves the correct atom ordering.

Molecules must be properly kekulized, as only bond types 1, 2, and 3 are accepted.

Keyword Arguments:
Parameters:	mol (pybel.Molecule) – A molecule object obtained from pybel.read*
	addhs (bool) – Add explicit hydrogens (default=True) kekulize (bool) – Kekulize the molecule (default=True) lonepair (bool) – Add dummy atoms and bonds for lone pairs (default=False) coordinates (bool) – whether to include coordinates in the json representation, this will override addhs (default=False)

tflon.chem.molecule.pymol_topological_groups(mol)¶: A faster implementation of topological indexes than Molecule.topological_groups

tflon.chem.molecule.rdmol_to_json(mol, addhs=True, kekulize=True, coordinates=False)¶

Convert rdmol to json format for storage and import to Molecule class.

Please note: If hydrogens are interspersed, rdkit breaks any expected ordering constraints. If this is a problem, use pybel with pymol_to_json.

Molecules must be properly kekulized, as only bond types 1, 2, and 3 are accepted.

Keyword Arguments:
Parameters:	mol (rdmol) – A molecule object obtained from an rdkit reader
	addhs (bool) – Add explicit hydrogens (default=True) kekulize (bool) – Kekulize the molecule (default=True)

tflon.chem.molecule.sdf_to_parquet(sdf_fn, output_fn=None, numeric_ids=False, removehs=True, addhs=True, lonepair=False, coordinates=False)¶

Generate a parquet file containing tflon.chem.Molecule json blobs from an SDF.

The resulting file contains a single column ‘Structures’, indexed by pybel.Molecule.title

Parameters:

sdf_fn (str) – Path to the sdf file

Keyword Arguments:

output_fn (str) – Output parquet file path (default=None)
numeric_ids (bool) – Typecast pybel.Molecule.title to int before writing index
removehs (bool) – Remove hydrogens (default=True)
addhs (bool) – Add missing hydrogens (default=True)
lonepair (bool) – Add dummy atoms and bonds for lone pairs (default=False)
coordinates (bool) – Add coordinates to the json molecule representations, overrides removehs (default=False)

class tflon.chem.data.MoleculeTable(data)¶

Convert a DataFrame containing serialized json format molecule structures to tflon.chem.Molecule objects.

See documentation of tflon.graph.GraphTable for instantiation

Toolkit API¶

class tflon.chem.toolkit.AtomProperties(*args, **kwargs)¶

build(molecule_table, atom_types=['H', 'B', 'C', 'N', 'O', 'F', 'P', 'S', 'Cl', 'As', 'Se', 'Br', 'I'], symbols=True)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call()¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

class tflon.chem.toolkit.BondLength(*args, **kwargs)¶

build(graph_table)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call()¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

class tflon.chem.toolkit.GraphConverter(*args, **kwargs)¶

build(molecule_table)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(bond_properties=None)¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

Data pipelines¶

Table data types¶

class tflon.data.tables.BlockReduceTable(data)¶

Specifies blocks of a tensor for segment operations.

Input dataframes must contain a single column of list-type elements containing the size of each reduction block.

class tflon.data.tables.DenseNestedTable(data)¶

Table which converts dataframes containing list type elements into 2-d tensors by flattening lists within a column across rows.

All lists in a row must have the same length.

For example, for molecule atom data, the following form is used:

ID col1 col2 col3 mol1 [atom1 atom2 …] [atom1 atom2 …] [atom1 atom2 …]

class tflon.data.tables.PaddedNestedTable(data, padded_width=None, dtype=<type 'numpy.float32'>)¶

Table which supports storing ragged arrays using a padding strategy. Converts to a sparse output.

Input dataframes must contain a single column of list-type entries.

class tflon.data.tables.RaggedTable(data, dtype=<type 'numpy.float32'>)¶

Table for managing ragged arrays without using a padding strategy. Values are converted to RaggedTensorValue for input via a RaggedTensor placeholder,which can be sliced by the ragged array index number to extract individual ragged elements.

Dataframes must contain a single column of list-type elements.

class tflon.data.tables.SparseIndexTable(data, code)¶

Used for indexing very large tensors, such as a corpus of words, or a list of diagnosis codes.

This can be used to gather slices from a very large embedding tensor.

Input dataframes must contain a single column of list-type entries containing integer indexes.

Converts to a sparse binary matrix.

class tflon.data.tables.Table(data)¶: The basic building block of the tflon data model. Tables are essentially a wrapper for pandas DataFrames that specify the transformation mapping the dataframe onto input tensors of a model. Examples should have at most a single row per table, with a single index value. Multiple extensions to Table are available to handle specific data types. For example, node features of a graph can be handled by DenseNestedTable, which represents each graph as a single row with nested lists for node features.

Parquet helper functions¶

tflon.data.parquet.build_dataframe(columns, dtypes, index_col='ID', index_type=<type 'int'>)¶

Build a dataframe with pre-specified column types.

Keyword Arguments:
Parameters:	columns (list) – String names of columns dtypes – If list or tuple, dtypes for columns in same ordering If dict, dtypes for columns with column names as keys Otherwise, dtypes is assumed to be a type to assign all columns:
	index_col (str) – The name of the index index_type (type) – The type of the index

tflon.data.parquet.read_parquet(source, columns=None, nthreads=4, use_pandas_metadata=True)¶

Read data from a parquet file

Keyword Arguments:
Parameters:	source (str) – A filepath to read parquet data
	columns (None or list) – A list of columns to read. If None, read all columns (default=None) nthreads (int) – Number of threads for reading large parquet files (default=4) use_pandas_metadata (bool) – (default=True)
Returns:	pandas.DataFrame

tflon.data.parquet.write_parquet(df, destination)¶

Read data from a pandas.DataFrame to parquet

Parameters:	df (pandas.DataFrame) – A pandas dataframe containing pyarrow parquet-compatible data destination (str) – A filepath to write parquet data

Data feeds¶

class tflon.data.feeds.Fetchable¶

Interface for implementing data sources such as queues and loaders.

Fetchable.fetch returns a dictionary of data inputs.

class tflon.data.feeds.FetchableGroup(fetchables=[])¶: Interface for implementing grouped data sources, calls Fetchable.fetch on each member fetchable to construct a dictionary of outputs

class tflon.data.feeds.IndexQueue(IDlist, epoch_limit=9223372036854775807, shuffle=True)¶

IndexQueue provides a thread-safe, optionally eternal, optionally randomized iterator over IDs in a dataset.

The number of epochs can be tracked with the IndexQueue.epochs attribute

class tflon.data.feeds.PersistentTensorManager(feed)¶: Deletes persistent tensors loaded into a session after they have been expired.

class tflon.data.feeds.QueueCoordinator(model, source, queue, processes, timeout=1, limit=10)¶

Construct a coordinator that orchestrates subprocess featurizers

Parameters:

model (tflon.Model) – Source for featurize preprocessing
source (iterator) – Iterable which returns tensor dictionaries
queue (TensorQueue) – A tensorflow queue for loading session feed_dict
processes (int) – Number of processes
timeout (float) – Timeout for queue ops, in seconds (default=1)
limit (int) – Maximum number of queue elements per subprocess

class tflon.data.feeds.RaggedTensor(values, lengths, dense_shape)¶

Implementation of ragged tensor input placeholder. This is an alternative to SparseTensor, which uses a tensor of ragged lengths used for slicing variable length vectors.

This is not currently designed to be used directly in any ops, but should be used by selecting vectors with RaggedTensor.slice

Parameters:	values (tensor-like) – Values of the sparse tensor of dimension V dense_shape (tensor-like) – The dense shape of the tensor ragged_shapes (tensor-like) – A tensor of ragged shapes with number of dimensions N-1

class tflon.data.feeds.RaggedTensorValue(values, lengths, dense_shape)¶

Value wrapper used to pass data to RaggedTensor model inputs.

Parameters:	values (np.array) – A 1-D numpy array containing the values of the ragged tensor lengths (np.array) – A 1-D array specifying the length of each vector in the ragged tensor dense_shape (tuple) – The dense shape of the ragged tensor, usually equal to (max(lengths), len(lengths))

class tflon.data.feeds.SparseTensor(indices, values, dense_shape, infered_shape)¶

This is an extension to tf.SparseTensor which enables shape inference for e.g setting weight shapes in matmul ops.

Parameters:	indices (tensor-like) – 2-D tensor of int64 indices, locations of non-zero values values (tensor-like) – 1-D tensor of non-zero values dense_shape (tensor-like) – The final dense shape of the tensor at feed time. Differs from infered_shape infered_shape (tensor-like) – The infered shape of the sparse tensor, returned by SparseTensor.get_shape()

get_shape()¶

Get the TensorShape representing the shape of the dense tensor.

Returns:	A TensorShape object.

class tflon.data.feeds.TableFeed(table_map, master_table=None, aligned=False)¶

The standard high level data preprocessor for tflon. Use this to define input data to Model.fit and Model.infer

Keyword Arguments:
Parameters:	table_map (dict) – Dictionary of str -> tflon.data.Table objects defining the raw data tables
	master_table (str) – The key for the table defining the master index

align()¶

Align all the examples by the master table index.

Skip if already aligned.

batch(indices)¶

Get a subset of examples as a new TableFeed

Parameters:	indices (iterable) – The indices of examples to include in the batch

drop(drop_set, errors='ignore')¶

Drop a specified set of examples from this feed

Parameters:	drop_set (iterable) – Indices of examples to drop

epoch()¶: Get the current data epoch as reached by TableFeed.shuffle

holdout(holdout_set)¶

Return a specified holdout set of examples as a new TableFeed and drop those examples from this feed

Parameters:	holdout_set (iterable) – Indices of holdout examples

index¶: Get the master index of this table feed

iterate(batch_size)¶

Construct an in-order iterator over mini-batches of examples. Limited to one epoch.

Parameters:	batch_size (int) – The number of examples per batch

merge(*feeds)¶: Merge this table feed with other feeds

sample()¶: Return a function which can be used to sample batches of varying size from this feed

shuffle(batch_size)¶

Construct an eternal iterator over shuffled mini-batches of examples.

Parameters:	batch_size (int) – The number of examples per batch

xval(folds=None, groups=None, shuffle=True)¶

Generate datasets for xval

Keyword Arguments:
	folds (int) – The number of folds, if None, then perform leave-one-out (default=None) groups (DataFrame) – A dataframe containing a single column specifying holdout groups for each fold (folds are numbered from 0 to n-1) shuffle (boolean) – Whether to shuffle the data or use the index ordering (default=True)

Returns: iterable: A generator yielding tuples of (holdout, remainder)

class tflon.data.feeds.TensorLoader(tower, device=None, name='TensorLoader')¶

Loads data tensors into the session, avoiding costly data copying in feed_dict.

This class is used for training with global optimizers, or other data operations. TensorLoader is flexible, in that a subset of inputs to the model may be passed to TensorLoader.load

See TensorQueue and QueueCoordinator for training models with stochastic minibatch optimizers.

>>> TL = TensorLoader( tower, name="TensorLoader_%s" % (tower.name) )
>>> B = datapipeline.next()
>>> TL.load(B)
>>> S.run( tower.get_outputs(), feed_dict=TL.feed() )

fetch()¶: Get the last generated feed dictionary

load(tensor_map)¶

Loads a tensor dict into the tensorflow session and return a feed_dict with tensor handles.

TensorLoader keeps a handle to loaded tensors to prevent garbage collection between session evaluations.

Parameters:	tensor_map (dict) – (name, tensor) pairs containing input/target names and compatible data (e.g numpy arrays)
Returns:	A dictionary of (name, persistent tensor handle) key value pairs

class tflon.data.feeds.TensorQueue(tower, capacity=10, staging=False, timeout=None, device=None, name='TensorQueue')¶

class tflon.data.feeds.ThreadCoordinator(model, source, queue, timeout=1)¶

Construct a coordinator that performs featurization on a thread

Parameters:	model (tflon.Model) – Source for featurize preprocessing source (iterator) – Iterable which returns tensor dictionaries queue (TensorQueue) – A tensorflow queue for loading session feed_dict

tflon.data.feeds.convert_sparse_matrix_to_sparse_tensor(X)¶: Convert sparse scipy matrices to tf.SparseTensorValue. csr and csc matrices are converted to coo first.

tflon.data.feeds.convert_to_feedable(X)¶

Converts input to a type valid for input to the tensorflow.Session.run feed_dict argument

Parameters:	X – A feedable type (valid types: tflon.data.Table, numpy.ndarray, scipy.sparse.csr_matrix, scipy.sparse.coo.coo_matrix, tf.Tensor, tf.SparseTensorValue)
Returns:	np.array, tf.Tensor, or tf.SparseTensorValue
Raises:	`ValueError` – If X cannot be converted to a feedable type

Schema definitions¶

class tflon.data.schema.Schema(schema)¶

Wrapper for schema specifications returned by tflon.model.Model.schema

Parameters: schema (dict) – dictionary of str -> (filenames, type[, reader]). Keys are string table names corresponding to model inputs, filenames is a string or list of strings specifying the names of files to be loaded from the shard directory, type is a class inheriting from Table, reader (optional) is a function with signature reader(filename) -> pandas.DataFrame (default = tflon.data.read_parquet).

load(*directories)¶

Load tables from a shard using this schema.

Keyword Arguments:
Parameters:	directories (str*) – paths to the shard directories containing serialized tables
	reader (callable) – A function used for loading serialized tables. Signature: reader(filename), default: tflon.data.read_parquet
Returns:	Map of name -> Table, the loaded table data
Return type:	dict

Output transformations¶

class tflon.data.output.CopyIndex(template_table, columns=None)¶

Add an index to a 2-D tensor

Parameters:	template_table (str) – The name of the input table used to set the indexes

class tflon.data.output.DenseToNested(template_table, columns=None)¶

Convert a 2-D tensor to a nested tensor, the number of entries in each nested list is decided by a template table

Parameters:	template_table (str) – The name of the input table used to set nest sizes and indexes

tflon.data.output.IdentityTransform¶: alias of tflon.data.output.CopyIndex

class tflon.data.output.TensorTransform¶: Interface class for output tensor transformations.

Distributed API¶

class tflon.distributed.distributed.DistributedTable(table)¶: Table wrapper implementing mpi Reduce ops for table data.

class tflon.distributed.distributed.DistributedTrainer(optimizer, iterations, **kwargs)¶: This trainer adds horovod DistributedOptimizer wrapper to a tensorflow optimizer, and handles broadcasting initialized model states.

tflon.distributed.distributed.get_rank()¶: Returns rank (the index for distributed thread)

tflon.distributed.distributed.init_distributed_resources()¶: Initialize mpi and Horovod for distributed training. This should be called before any tflon or tensorflow calls.

tflon.distributed.distributed.is_master()¶: Check whether the current process is the rank 0 mpi process.

tflon.distributed.distributed.make_distributed_table_feed(root, schema, master_table=None, partition_strategy='mod')¶

Load data shards for distributed training.

Keyword Arguments:
Parameters:	root (str) – The root directory containing shards. Each shard should be a subdirectory of this directory. schema (tflon.data.Schema) – Schema mapping files to named tables as returned by tflon.model.Model.schema
	partition_strategy (str) – Strategy for dividing shards among nodes. Supported values include: * ‘mod’: For process rank r and number n, divide shards evenly by distributing every r+n-th shard * ‘all’: Replicate all shards on all processes

Usage Examples¶

Basic neural network¶

Distributed training¶

# Example code showing how to run distributed training across multiple GPUs or nodes
# using horovod and MPI

import tflon
import tensorflow as tf
import pandas as pd
from pkg_resources import resource_filename

class NeuralNet(tflon.model.Model):
    def _model(self):
        I = self.add_input('desc', shape=[None, 210])
        T = self.add_target('targ', shape=[None, 1])

        net = tflon.toolkit.WindowInput() |\
              tflon.toolkit.Dense(20, activation=tf.tanh) |\
              tflon.toolkit.Dense(5, activation=tf.tanh) |\
              tflon.toolkit.Dense(1)
        L = net(I)

        self.add_output( "pred", tf.nn.sigmoid(L) )
        self.add_loss( "xent", tflon.toolkit.xent_uniform_sum(T, L) )
        self.add_loss( "l2", tflon.toolkit.l2_penalty(self.weights) )
        self.add_metric( 'auc', tflon.toolkit.auc(T, L) )

if __name__=='__main__':
    tflon.data.TensorQueue.DEFAULT_TIMEOUT=1000
    # Initialize horovod and setup gpu resources
    config = tflon.distributed.init_distributed_resources()

    graph = tf.Graph()
    with graph.as_default():
        # Add a model instance
        NN = NeuralNet(use_gpu=True)

        # Create the distributed trainer
        trainer = tflon.distributed.DistributedTrainer( tf.train.AdamOptimizer(1e-3), iterations=1000 )

    # Create the data feed, use the same feed for all process instances
    # tflon.distributed.DistributedTable adds MPI synchronization to the Table API min and max ops
    # Usually, different data would be loaded on each process (see tflon.distributed.make_distributed_table_feed)
    tsv_reader = lambda fpath: pd.read_csv(fpath, sep='\t', dtype={'ID':str}).set_index('ID')
    schema = NN.schema.map(desc=('descriptors.tsv', tsv_reader), targ=('targets.tsv', tsv_reader))

    # Look at tflon_test/data/distributed to see how shards are organized on disk
    feed = tflon.distributed.make_distributed_table_feed( resource_filename('tflon_test.data', 'distributed'), schema, master_table='desc',partition_strategy='all' )

    with tf.Session(graph=graph, config=config):
        # Train with minibatch size 100
        NN.fit( feed.shuffle(batch_size=100), trainer, restarts=2, source_tables=feed )

        # Perform inference on the master process
        if tflon.distributed.is_master():
            auc = NN.evaluate( feed, query='auc' )
            print "AUC:", auc
            assert auc > 0.8

Molecule neighborhood convolution¶

Graphs¶

Graph base class¶

class tflon.graph.graph.DiGraph(digraph, node_properties=[], edge_properties=[])¶

Wrapper for directed graph support. Casts digraph to undirected graph and provides edge_direction query.

Parameters:	digraph (networkx.DiGraph) – A directed graph

edge_direction(u, v)¶

Return the direction of an edge relative to a query

Parameters:

u – Origin node index
v – Destination node index

Returns:

If u->v and v->u are both edges, returns 0: If only u->v is an edge, returns 1 If only v->u is an edge, returns -1

Return type:

char

Raise:: KeyError: If neither u->v nor v->u are edges

classmethod from_json(serialized)¶

Generate a networkx representation of a serialized json format graph

Parameters:	serialized (str) – The serialized json representation

get_edges(u, v)¶

Get all edges between nodes u and v

Parameters:	u – Node index v – Node index
Returns:	a generator yielding edges
Return type:	generator

multiplicity(u, v)¶

Count the number of edges (1 or 2) between adjacent vertices u and v

Parameters:	u – Origin node index v – Destination node index
Returns:	The number of edges between u and v
Return type:	int

node_edges(u)¶

Get all the edges into or out of a given node

Parameters:	u – Node index
Returns:	A list of edges
Return type:	list

port_number(u, e)¶: Assigns Digraph graph edge to one of four ports

class tflon.graph.graph.ViGraph(graph, parent_graph, node_properties=[], edge_properties=[], preserve_bonds=False, single_virtual_port=False)¶

Wrapper for virtual graph support.

Parameters:	graph (networkx.Graph) – An undirected graph bonds (IndexMap, list-type) – Bonding edge indices edge_properties (list-type) (optional) – node_properties (list-type) (optional) –

bfs(roots)¶

Return a block-partitioned BFS of all subgraphs

Parameters:	roots (list) – A list of starting nodes, one for each connected component

centroids()¶: Choose a random node from the center of each connected component (nodes with eccentricity equal to radius)

port_number(u, e)¶: Assigns Virtual graph edge to one of four ports: virtual/bonding +- tree/cross edge

Graph data pre-processors¶

tflon.graph.data.DenseEdgesToTableTransform¶: alias of tflon.graph.data.EdgesToNested

tflon.graph.data.DenseNodesToTableTransform¶: alias of tflon.graph.data.NodesToNested

class tflon.graph.data.DiGraphTable(data)¶

Convert a DataFrame containing serialized json format digraph structures to tflon.graph.DiGraph objects.

See documentation of tflon.graph.GraphTable for instantiation

class tflon.graph.data.EdgesToNested(graph_table, columns=None)¶

class tflon.graph.data.GatherTable(data)¶: Table type which manages indices for gather operations

class tflon.graph.data.GraphElementsToNested(graph_table, etype, columns=None)¶

Convert a 2-D tensor to a nested tensor, the number of entries in each nested list is decided by the corresponding number of graph elements

Parameters:	graph_table (str) – The name of the table containing tflon.graph.Graph objects.

class tflon.graph.data.GraphTable(data, gtype=<class 'tflon.graph.graph.Graph'>)¶

Convert a DataFrame containing serialized json format graph structures to tflon.graph.Graph objects.

Keyword Arguments:
Parameters:	data (pandas.DataFrame) – The input dataframe, with one column containing serialized json molecules
	convert (bool) – Whether to convert from json representation, if False expects dataframe of tflon.graph.Graph objects (default=True)

class tflon.graph.data.NodesToNested(graph_table, columns=None)¶

class tflon.graph.data.PoolTable(data)¶: Table type which manages pooling indices for graph convolution. A pool is a collection of indices which are used to gather from a tensor and then reduce to a tensor of the same size as the original.

class tflon.graph.data.RaggedIndexTable(data, dtype=<type 'numpy.float32'>)¶: Similar to GatherTable, but stores ragged arrays of indices, used for pulling tensor elements for updates on dynamic data structures

class tflon.graph.data.RaggedReduceTable(data, dtype=<type 'numpy.float32'>)¶: Similar to BlockReduceTable, but stores ragged arrays of indices, used for reducing tensor elements for updates on dynamic data structures

class tflon.graph.data.SparseRowSelector(data)¶: Table to convert row indices to a sparse binary matrix for dynamic partition operations.

class tflon.graph.data.SparseToDenseRowSelector(data)¶: Extends SparseRowSelector to convert a sparse row selection to a dense matrix prior to feed

Graph operations toolkit¶

class tflon.graph.toolkit.EdgeProperties(*args, **kwargs)¶

build(*args, **kwargs)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

class tflon.graph.toolkit.ElementProperties(*args, **kwargs)¶

build(graph_table, properties, etype, dtype=tf.float32)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call()¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

class tflon.graph.toolkit.Encoder(*args, **kwargs)¶

Implementation of the set2set recurrent encoder network: https://arxiv.org/abs/1511.06391

This encoder is used to produce a graph-level encoding for predicting graph-level properties.

Other Parameters:
Parameters:	lstm_cell (tf.nn.rnn_cell.RNNCell) – An instance of a recursive unit used for encoding weight_network (callable) – A function used to calculate attention values from node states
	iterations (int) – The number of encoder steps to apply (default=10)

build(graph_table, lstm_cell, weight_network)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(nodes)¶

Perform graph encoding

Parameters:	nodes (tensor-like) – Node properties
Returns:	Graph-level encodings
Return type:	Tensor

class tflon.graph.toolkit.Neighborhood(*args, **kwargs)¶

build(graph_table)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(nodes)¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

class tflon.graph.toolkit.NodeProperties(*args, **kwargs)¶

build(*args, **kwargs)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

class tflon.graph.toolkit.NodeToEdge(*args, **kwargs)¶

Calculate edge representations from pairs of node states using an order invariant weave op.

Other Parameters:
	activation (callable) – An activation function (default=tf.nn.relu) width (int) – The number of output features, if -1, then output size = input size (default=-1)

build(graph_table)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(nodes)¶

Parameters:	nodes (tensor-like) – Node properties
Returns:	edge properties
Return type:	Tensor

class tflon.graph.toolkit.Pool(*args, **kwargs)¶

Graph pooling. Pools all nodes with a given depth (in number of edges traversed). Only nodes at the specified depth are pooled.

Keyword Arguments:
Parameters:	depth (int) – The depth to find nodes for pooling.
	reduce (callable) – A reduction function with signature `func(tensor, segments)` (Default: tf.segment_sum) * tensor: N-D tensor for reduction * segments: 1-D tensor of segment indices weighting (callable) – A weighting function with signature: `func(tensor, segments)` (Default: None) * tensor: N-D tensor for weight calculation * segments: 1-D tensor of segment indices

build(graph_table, depth, reduction=<function segment_sum>, weighting=None)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(nodes)¶

Parameters:	nodes (Tensor) – node properties of dimension N x …
Returns:	same shape as nodes
Return type:	Tensor

class tflon.graph.toolkit.Reduce(*args, **kwargs)¶

Graph reduce. Pools all nodes or edges within a given graph into a single output.

Keyword Arguments:

reduction (callable) – A reduction function with signature func(tensor, segments) (Default: tf.segment_sum) * tensor: N-D tensor for reduction * segments: 1-D tensor of segment indices
weighting (callable) – A weighting function with signature: func(tensor, segments) (Default: None) * tensor: N-D tensor for weight calculation * segments: 1-D tensor of segment indices
element_type – reduce over nodes or edges, defaults = node

batch_size¶: Get the expected output tensor size

build(graph_table, reduction=<function segment_sum>, weighting=None, element_type='node')¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(nodes)¶

Parameters:	nodes (Tensor) – node properties of dimension N x …
Returns:	of shape G x … where G is the number of distinct graphs in the batch
Return type:	Tensor

inverse(reduced)¶

Broadcast graph-level output of this op back to the node level

Parameters:	reduced (Tensor) – The output of the reduction op
Returns:	of shape N x … where N is the number of nodes in the original batch
Return type:	Tensor

Graph recurrent networks (Wave)¶

class tflon.graph.grnn.DynamicPasses(*args, **kwargs)¶

build(graph_table)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(node_states, edge_states=None)¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

class tflon.graph.grnn.GRNN(*args, **kwargs)¶

Graph recursive neural network (wave network).

Keyword Arguments:
	num_ports (int) – Number of ports for mixgate (number of edge types * 2) (default=2) edge_properties (bool) – Whether edges have properties (default=False)

Other Arguments:: dagger (callable): A function which returns node blocks, ancestors, and edge types for scheduling graph recursion state updates (default=Graph.bfs) rooter (callable): A function which returns the starting node for the dagger (default=Graph.centroids)

build(graph_table, num_ports=2, edge_properties=False)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(inputs=None, forward_cell=None, backward_cell=None, edge_properties=None)¶

Perform a single forward-backward pass of graph recursion

Keyword Arguments:
Parameters:	inputs (tensor-like) – The node properties/states forward_cell (callable) – RNNCell to use for the forward pass, usually an instance of tf.nn.rnn_cell.RNNCell, or a callable accepting two arguments (inputs, states) backward_cell (callable) – RNNCell to use for the backward pass, usually an instance of tf.nn.rnn_cell.RNNCell, or a callable accepting two arguments (inputs, states)
	edge_properties (tensor-like) – Edge properties for the graph, if available (default=None)
Returns:	The updated node states
Return type:	Tensor

wrap_cell(*args, **kwargs)¶

Add a mix gate to an RNNCell

Parameters:	cell (RNNCell) – An RNNCell to wrap **kwargs – Additional arguments passed to the GRNNCell constructor
Returns:	A graph recurrent cell
Return type:	GRNNCell

class tflon.graph.grnn.GRNNCell(*args, **kwargs)¶

An RNNCell wrapper which applies the mix gate to combine multiple states prior to a state update

Parameters:

rnn_cell (callable) – A cell matching the tf.nn.rnn_cell.RNNCell API, which is used to compute the state update
ports (int) – The number of mix gate ports
edge_properties (bool) – Whether edge properties are included in the mix gate calculation

Other Parameters:

activation (callable) – An activation function for the mix gate weighting functions (default=tf.nn.elu)
gate_types (list) – Types of mix gate weightings to include (default=[‘softmax’,’softsign’])
rnn_order (str) – Where to input mix gate to recurrent unit ‘mix_to_state’ or ‘mix_to_input’ (default=’mix_to_state’)
normalization (str) – Perform a normalization on mix gate ‘none’, ‘layer’, or ‘degree’ (default=’none’)
normalize_output (bool) – If True, perform layer normalization on the output of this recurrence (default=False)

build(rnn_cell, ports, edge_properties)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(states, degrees, block, nodes, ancestors, ports, redux, edges, edge_properties)¶

Compute a partial state update of a subset of nodes

Parameters:	states (tensor-like) – Current node states degrees (tensor-like) – Node in and out degrees block (tensor-like) – The subset of nodes to update nodes (tensor-like) – A list of nodes at the arrow side of each update edge ancestors (tensor-like) – A list of ancestor nodes of each update edge ports (tensor-like) – A list of port types for each update edge redux (tensor-like) – Reduction indices (update edges -> nodes) edges (tensor-like) – Edge indices for each update edge edge_properties (tensor-like) – Edge properties for each update edge
Returns:	updated node states
Return type:	Tensor

class tflon.graph.grnn.MiniGRUCell(*args, **kwargs)¶

A special GRUCell which does not have a read gate. This is commonly used in GRNNs because the read gate is redundant when a mix gate is used.

Keyword Arguments:
Parameters:	state_size (int) – The width the node states
	activation (callable) – An activation function used for computing the output gate network

build(state_size, activation=<function elu>)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(inputs, state)¶

Perform a state update

Parameters:	inputs (tensor-like) – The current input values states (tensor-like) – The current state values
Returns:	The output and state tensors, which are identical for a GRU
Return type:	tuple (Tensor, Tensor)

state_size¶: Get this RNNCell’s state size

zero_state(*args, **kwargs)¶

Create initial zero state with given batch size and type

Parameters:	batch_size (int) – The first dimension size of the state vector dtype (tf.type) – The state vector dtype

class tflon.graph.grnn.MixGate(*args, **kwargs)¶

Module defining the special mix gate used to combine input states from multiple edges for form a message state

Parameters:

state_size (int) – The width of the node state
activation (callable) – An activation function for computing weightings
ports (int) – The number of ports
types (list) – The types of weightings to include, valid values include ‘softmax’, ‘softsign’, and ‘sum’
edge_properties (bool) – Whether edge properties will be provided
normalization (str) – The type of normalization to apply: ‘none’, ‘layer’, ‘degree’ (default=’none’)

build(state_size, activation, ports, types, edge_properties, normalization)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(states, degrees, nodes, ancestors, ports, redux, edges, edge_properties)¶

Combine input states to produce a state message for graph recursive node updates

Parameters:	states (tensor-like) – Node properties degrees (tensor-like) – In and out degrees for each node nodes (tensor-like) – Indices for the nodes being updated ancestors (tensor-like) – Indices of ancestor nodes ports (tensor-like) – Port types for each edge redux (tensor-like) – Reduction indices edges (tensor-like) – Edge indices edge_properties (tensor-like) – Edge properties
Returns:	Message states for each node in the update
Return type:	Tensor

tflon.graph.grnn.eccentricity_exponential_distribution(graph, base=2.718281828459045)¶

Create a decaying distribution over nodes by eccentricity

This can be used to randomize the choice of root node for regularizing graph recursive neural networks.

Weave convolutional networks¶

class tflon.graph.weave.Weave(*args, **kwargs)¶

Implementation of the weave module from the paper Molecular Graph Convolutions: Moving Beyond Fingerprints (https://link.springer.com/article/10.1007/s10822-016-9938-8)

Other Parameters: edge_properties (bool): Whether to expect edge properties (default=False) digraphs (bool): Whether to expect digraphs (default=False; not supported) activation (callable): Activation function to use for weave ops (default=tf.nn.elu) batch_normalize (bool): Perform batch normalization (default=False) distance (int): The distance, in number of bonds, over which to perform pair weave ops (default=2) pair_width (int): Number of output pair features at each layer, required node_width (int): Number of output node features at each layer, required pair_hidden (int): Number of hidden features for pair weave, if None, same as pair_width (default=None) node_hidden (int): Number of hidden features for node weave, if None, same as node_width (default=None)

build(graph_table)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(nodes, pairs=None)¶

Add a single weave module to the compute graph. Pair properties for the first weave should be generated with Weave.initial_pair_state

Parameters:	nodes (tensor-like) – Node properties pairs (tensor-like) – Pair properties
Returns:	Updated node and pair properties
Return type:	tuple (Tensor, Tensor)

initial_pair_state(*args, **kwargs)¶

Create initial zero-state values for pairs. If edge properties are available, gather edge properties for adjacent pairs.

Keyword Arguments:
	edges (tensor-like) – Optional edge properties (default=None)
Returns:	initial pair states
Return type:	Tensor

outputs(nodes, pairs, include_edges=None)¶

Add a weave module to the compute graph without updating pair states.

Keyword Arguments:
Parameters:	nodes (tensor-like) – Node properties pairs (tensor-like) – Pair properties
	include_edges (bool) – Include an edge update and return the resulting edge properties (ignore non-adjacent pairs)
Returns:	output node states and (optional) edge states
Return type:	Tensor or tuple(Tensor, Tensor)

Images¶

This module currently supports implementing the wave graph architecture for processing image data

Image grid graph API¶

class tflon.image.graph.ImageGraph2D(graph)¶

Issue: i wrote all of these assuming the actual image itself was going to be passed into Graph, and so I thought I needed to return blocks full of actual image pixel values…

1. Do these things being initialized need to be something different? as it is, it seems we are generating a bunch of data for little reason, and there are likely other things to modify to make it faster. Yes. he says make them generators. Perhaps do that later. 2. Question: how the heck could we do neighbors if we didn’t know the number node id to expect in each position?

Now graph will be a tuple, (hight,width)

bfs(c_coords)¶

This returns blocks and ancestors given an image. Blocks is a list of lists containing the nodes (pixels) in each shell. Ancestors is the dicitonary with each pixel as keys, and as values a list (automatically sorted) of the immediate predecessors.

For the four if statements below:: This obtains all the ancestors. Those in the same row or column as the starting will have only one ancestor. This new ordering will spit out with the ancestor lists sorted by position in the array.

centroids()¶: returns center pixel in image graph. Inputs: self._graph should be an image, not an nx.graph object outputs: the center pixel’s value as a vector (of dimension 512 if using the VGG) Note that it will return the center pixel for odd-dimensions, and the left/top of center pair for even dimensions. For a line of 4 pixels, that means it picks the second pixel, having python index 1.

get_edges_init()¶: Should still be improved

get_node_id(i, j)¶: Returns the node number, which is rastor scan style

neighborhood(node, max_depth=9223372036854775807)¶: Just passing in the node number to start on. It’s used by pool_featurize. Note: I’m assuming the node will be a # in a grid graph, starting at 1 raster scan style [[1,2,3],[4,5,6],[7,8,9]] Note: This should potentially be changed to be 0-indexed

tflon.image.graph.block_preorder(ancestors)¶: returns nodes as list of lists in the layer order they are visited in the bfs [[n0],[n1,n2],[n3]] It does this becuase for a node to go into empty and get appended to blocks, it must not have been put into blocks and visited yet, but all of its ancestors must have been put into blocks and visited. Thus it can only work outward one layer at a time, and all those unvisited with all ancestors visited will be in the same layer.

tflon.image.graph.flip_ancestors(D)¶

This reverses the order of all the graph connections, putting the descendents as the ancestors in a new dictionary It does this by adding the former keys as values for the new keys that are the old values. So it reverses the traversal order of every edge.

Input: ancestors: dict with history one back so now like {n0:{}, n1:{n0}, n2:{n0}, n3:{n1,n2}} Output: {n0:{n1,n2},n1:{n3},n2:{n3}}

class tflon.image.graph.type_overwrite(edge)¶

Models¶

All tower APIs are accesible directly from a tflon.Model instance.

Model API¶

class tflon.model.model.Model(**kwargs)¶

Model base class. User models are implemented by extending this class and overriding the _model method:

class MyModel(Model):

def _model(self, *args, **kwargs):: pass

checkpoint(filename)¶

Generate a tensorflow checkpoint file.

Parameters:	filename (str) – Path to write the checkpoint file

evaluate(source, query=None)¶

Evaluate model metrics on data feed or batch

Keyword Arguments:
Parameters:	source – Either (1) a batch dictionary of name, tensor pairs or (2) a generator, which emits batches (dictionaries of name, tensor pairs)
	query (list or str) – A key or multiple keys specifying which metrics to gather, if None, gets all metrics (default=None)
Returns:	A dictionary of metric names and values
Return type:	dict

featurize(source, batch_size=None)¶

Given a table feed, generate the feature tables required by each module and add them to the table feed

Keyword Arguments:
Parameters:	source (TableFeed) – The input feed
	batch_size (int) – Featurize the data in batches of a given size, if None, featurize all the data at once

feed(batch)¶: Generate a feed_dict from a batch dictionary for evaluating session.run calls

fetch(patterns)¶

Get variable values by regex patterns.

Parameters:	patterns (list) – list of string queries for variable names
Returns:	Variable values as name:value pairs
Return type:	dict

fit(source, trainer, restarts=1, reset=True, source_tables=None, **kwargs)¶

Fit a model to data using a chosen trainer.

This function manages random restarts and delegates training to either _fit_global or _fit_stochastic, which are selected based the type of source.

Parameters:

source – Either a dictionary-like of name:feedable pairs, or an iterator which returns such dictionaries
trainer – A tflon.train.Trainer

Keyword Arguments:

restarts (int) – The number of times to train from a random initialization (default = 1)
reset (bool) – Whether to reset trainable variable initializations before each optimization (default = True)
source_tables – An optional TableFeed object or dictionary of name:Table pairs which is used to initialize model parameters (default = None)
**kwargs – Additional arguments passed to either _fit_global or _fit_stochastic

infer(source, query=None)¶

Perform inference from a data batch or from a data generator.

Keyword Arguments:
Parameters:	source – Either (1) a batch dictionary of name, tensor pairs or (2) a generator, which emits batches (dictionaries of name, tensor pairs)
	query (list or str) – A key or multiple keys specifying which outputs to gather, if None, gets all outputs (default=None)
Returns:	If source is a batch dictionary, return a dictionary of name, tensor pairs for model output. Otherwise, if source is a batch generator, return a generator which emits dictionaries of name, tensor pairs, one output for each input batch
Return type:	dict or iterator

initialize(restore=True)¶

Reset the model variables to their initialized states.

This operation will restore variables to saved states loaded from disk if available.

Parameters:	restore (boolean) – If False, do not use saved variable states even when available (default=True)

classmethod load(filename, **xargs)¶

Load a tflon model pickle file previously saved with Model.save from disk

Parameters:	filename – The pickle file containing the saved tflon model xargs – Extra keyword arguments passed to Model.__init__ this argument can be used to override parameter settings

reset()¶: Reinitialize model trainable variables.

restore(filename)¶

Restore model weights from a tensorflow checkpoint file.

Parameters:	filename (str) – A path to a checkpoint file

save(filename)¶

Save a tflon model pickle file. The resulting file contains a tuple data structure with the following three variables:

(class) Model class
(dict) Parameters
(dict) Values of all model variables

Parameters:	filename (str) – A path to save the pickled model

show_data(tables)¶

Sets model and module parameters from data prior to fit

Parameters:	tables (tflon.data.TableFeed) – Input tables used for this model training run

Tower API¶

class tflon.model.tower.Tower(params=None, use_gpu=False)¶

Tower is a slave class of the Model, which is designed to specifically abstract away the tensorflow interface of tflon.model.Model.

The combination of Tower and Model are effectively a hybrid class, since Model exposes its Tower instance methods directly via python missing property resolution (e.g __getattribute__).

The end user interacts with these methods and properties from within their Model instance as though they were attributes of Model.

add_input(name, shape=[None], dtype=tf.float32, sparse=False, ragged=False, ttype=<class 'tflon.data.tables.Table'>)¶

Add an input tensor. This method creates a dictionary alias name which can be used when feeding data to the model.

Input tensors can come in three varieties: dense (default), sparse, and ragged.

Parameters:	name (str) – The name reference for this tensor.

Keyword Parameters:: shape (iterable): The dense shape of this input tensor dtype (tf.type): The data type (e.g tf.float32) sparse (bool): Specify sparse tensor, sparse tensors are input via tf.SparseTensorValue ragged (bool): Specify a ragged array-type tensor, ragged tensor are input via tflon.data.RaggedTensorValue

Returns:	The input placeholder tensor
Return type:	tf.Tensor

add_loss(name, loss)¶

Add a named loss function to this model. All losses are summed to compute the final optimization target

Parameters:	name (str) – A name reference for this loss value loss (tensor-like) – The op calculating this loss

add_metric(name, metric)¶

Add a named metric op to this model.

Parameters:	name (str) – A name reference for this metric metric (tuple) – The op calculating this metric. This op should be able to handle aggregation of multiple batches. Tuples must have the form (eval_op, update_op), where eval_op computes the final metric value after multiple calls to update_op.

add_module(module, extra_parameters)¶

Add a module to this tower. This is automatically handled by the Module constructor.

Parameters:	module (tflon.toolkit.Module) – Instance of Module extra_parameters (dict) – A dictionary of parameter overrides for the module

add_output(name, out, transform=None)¶

Add an output tensor. Output tensors are returned by Model.infer

Optionally, output tensors can be transformed by a tflon.data.TensorTransform post-processing operation.

Parameters:	name (str) – A name for this output, used as a reference key in the dictionary returned by Model.infer out (tensor-like) – A tensor to fetch when writing output

Keyword Parameters:: transform (TensorTransform): A transformation to apply (default=None)

add_prior(loss)¶

Add a prior loss to this model.

Parameters:	loss (tensor-like) – The op calculating this loss

add_target(name, shape=[None], dtype=tf.float32, sparse=False, ragged=False, ttype=<class 'tflon.data.tables.Table'>)¶: Alias of Tower.add_input, differs only by semantic distinction.

biases¶: Get the collection of all biases for this model

featurize(batch)¶

Add each module’s features to the batch

Parameters:	batch (dict-like) – A batch dictionary containing data tables

feed(batch)¶

Given an input batch, construct a feed dictionary. This method handles mapping tables to support tensors for sparse and ragged inputs.

Returns:	of (placeholder, value) pairs, a valid feed_dict
Return type:	dict

feed_names(batch)¶

Given an input batch, construct a list of the names of all placeholder tensors required to feed data to the model. This will include names of support tensors for sparse and ragged inputs.

Returns:	A list of placeholder names
Return type:	list

get_bias(name, shape, dtype=tf.float32, initializer=None, prior=None, scope=None)¶

Get a bias variable. Biases are automatically added to tf.GraphKeys.TRAINABLE_VARIABLES, tf.GraphKeys.LOCAL_VARIABLES and tf.GraphKeys.BIASES

Parameters:	name (str) – The parameter name, which will result in a tensorflow variable referenced at scope/name:x shape (list or tuple) – The explicit shape of this variable

Keyword Parameters:: dtype (tf.dtype): The data type (default=tf.float32) initializer (tf.Operation): An initializer op (default=None, which gets this scope’s default initializer) scope (str): A scope for this var, used to select the appropriate default initializer for a module

get_module(name)¶

Get a module by name

Parameters:	name (str) – The module name

get_parameter(key, default=None, namespace=None)¶

Get the value of a parameter if it exists and is not None

Keyword Arguments:
Parameters:	key (str) – The parameter name
	namespace (str) – An optional namespace, searches for namespace/key instead of key (default=None)
Returns:	The value of the parameter
Raises:	`KeyError` – if the parameter key is not found or if its value is None

get_queue(**kwargs)¶: Construct a data queue for this tower, which can be used to pre-cache mini-batch data on the GPU in parallel.

get_variable(name, shape, dtype=tf.float32, initializer=<tensorflow.python.ops.init_ops.Zeros object>, trainable=False)¶

Get a tensorflow variable. Automatically selects the appropriate compute device.

Parameters:	name (str) – The parameter name, which will result in a tensorflow variable referenced at scope/name:x shape (list or tuple) – The explicit shape of this variable

Keyword Parameters:: dtype (tf.dtype): The data type (default=tf.float32) initializer (tf.Operation): An initializer op (default=tf.zeros_initializer()) trainable (bool): Whether to add to collection tf.GraphKeys.TRAINABLE_VARIABLES

get_weight(name, shape, dtype=tf.float32, initializer=None, prior=None, scope=None)¶

Get a weight variable. Weights are automatically added to tf.GraphKeys.TRAINABLE_VARIABLES, tf.GraphKeys.LOCAL_VARIABLES and tf.GraphKeys.WEIGHTS

Parameters:	name (str) – The parameter name, which will result in a tensorflow variable referenced at scope/name:x shape (list or tuple) – The explicit shape of this variable

Keyword Parameters:: dtype (tf.dtype): The data type (default=tf.float32) initializer (tf.Operation): An initializer op (default=None, which gets this scope’s default initializer) scope (str): A scope for this var, used to select the appropriate default initializer for a module

gradients¶

Return the gradients for all trainable variables.

Returns:	Tower gradients
Return type:	grads (list)

has_parameter(key, default=None, namespace=None)¶

Check if a parameter exists and is not None

Keyword Arguments:
Parameters:	key (str) – The parameter name
	namespace (str) – An optional namespace, searches for namespace/key instead of key (default=None)

inject_parameters()¶

Add the values of Model._parameters as class attributes. For a given (key, value) pair, this will add instance attributes of the form:

self.key = value

This behavior is currently optional, but will eventually become the default.

inputs¶: Get the input dictionary, (name, tf.Tensor) pairs. This does not necessarily contain feedable placeholders

inputs_and_targets¶: Get the joint dictionary of all inputs and targets

loader¶: Return the persistent tensor loader for this tower, which can be used to pre-cache data on the GPU.

loss¶: Get the aggregated loss op for this tower. Constructed as a sum of all named losses.

losses¶: Get a dictionary of all losses (name, tf.Tensor) pairs.

metric_variables¶: Get all variables associated with metric update operations

metrics¶: Get a dictionary of all metrics (name, (eval_op, update_op)) pairs.

modules¶: Get a copy of the modules dictionary, which consists of (module.name, module) pairs.

name¶: Get the name scope of the tower

outputs¶: Get the output dictionary, (name, tf.Tensor) pairs.

parameters¶: Return the parameter dictionary. This combines user-specified parameters, defaults in Model._parameters and all Module._parameters

phase¶

Boolean tensorflow variable specifying the current phase –

True = training phase
False = testing phase

placeholder_names¶: Get a dictionary of feedable placeholders to names. This will contain all inputs and targets, as well as support tensors for sparse and ragged inputs

placeholders¶: Get a dictionary of feedable placeholders. This will contain all inputs and targets, as well as support tensors for sparse and ragged inputs

populate(build_fn)¶

Bootstrap a tensorflow graph and perform sanity checks:

Inject parameters into the tower object attributes

Create the tensorflow ops associated with Model by calling build_fn.

Add additional ops joining all losses, or add a no_op if no losses

Perform sanity checks and emit warnings for missing elements (e.g no inputs, no losses)

Check for unused parameter values (often caused by a typo in a parameter name)

Parameters:	build_fn – A reference to Model._model()

schema¶: Return a tflon.data.Schema object implementing the model-specified schema (see Model._schema)

set_gradients(grads)¶

Set the gradients for trainable variables in this tower.

This is used to override the gradients property for use with optimizer-specific gradients.

set_loss(*keys)¶: Set the total loss op (Tower.loss) to the sum of the given named loss ops

set_parameter(key, value, namespace=None)¶

Set the value of a parameter

Keyword Arguments:
Parameters:	key (str) – The parameter name value – A parameter value (any dill-pickleable type is valid)
	namespace (str) – An optional namespace, searches for namespace/key instead of key (default=None)

set_parameter_default(key, value, namespace=None)¶

Set the value of a parameter if it is not already set

Keyword Arguments:
Parameters:	key (str) – The parameter name value – A parameter value (any dill-pickleable type is valid)
	namespace (str) – An optional namespace, searches for namespace/key instead of key (default=None)

set_training_phase(value)¶

Set the current phase. This is used by Model.infer, Model.fit and Model.evaluate

Values are:

True = training phase

False = testing phase

Parameters:	value (bool) – the new value of the phase variable

show_data(batch)¶

Show table data to each module attached to this tower. Used by Model.fit to set training parameters (e.g data normalization)

Parameters:	batch (dict-like) – A batch dictionary containing data tables

targets¶: Get the target dictionary, (name, tf.Tensor) pairs. This does not necessarily contain feedable placeholders

trainables¶: Get the collection of all trainables (weights + biases + other trainables) for this model

transforms¶: Get the transform dictionary, (name, TensorTransform) pairs.

vardict¶: Get a dictionary of all variables belonging to this tower

variables¶: Get the complete collection of all variables for this model (weights + biases + other trainbles + non-trainables)

weights¶: Get the collection of all weights for this model

Toolkit¶

Higher level op API¶

class tflon.toolkit.toolkit.Broadcast(*args)¶

Broadcast inputs to multiple modules and append their outputs into a tuple

Parameters:	*args – Any number of callable objects

class tflon.toolkit.toolkit.Chain(*args)¶

Chain multiple modules together, feeding the output of one into the input of the next

Parameters:	*args – Any number of callable objects

class tflon.toolkit.toolkit.Join(*args)¶

Send each argument in a list or tuple of arguments to the corresponding module in a list of modules

Parameters:	*args – Any number of callable objects

class tflon.toolkit.toolkit.Module(*args, **kwargs)¶

This is a base class for creating reusable graph components. Extending classes should implement the following interfaces:

Required interfaces:

build(self, *args, **kargs)

call(self, *args, **kwargs)

Optional interfaces:

inverse(self, *args, **kwargs)
show_data(self, batch)
_featurize(self, batch)
_parameters(self)

__and__(other)¶

Construct a tflon.toolkit.Join from two modules

Concatenates the output of all modules into a single tuple

__or__(other)¶

Construct a tflon.toolkit.Chain from two modules.

Pipes the output of one module’s __call__ to the input of the next

__xor__(other)¶

Construct a tflon.toolkit.Broadcast from two modules

Appends the output of all modules into a single tuple

add_input(name, shape=[None], dtype=tf.float32, sparse=False, ragged=False, ttype=<class 'tflon.data.tables.Table'>)¶

Add an input to the model, using the scope of this Module

See tflon.model.Tower.add_input

add_loss(name, loss)¶

Add a loss to the model, using the scope of this Module

See tflon.model.Tower.add_loss

add_target(name, shape=[None], dtype=tf.float32, sparse=False, ragged=False, ttype=<class 'tflon.data.tables.Table'>)¶

Add a target to the model, using the scope of this Module

See tflon.model.Tower.add_target

biases¶: Get the collection of biases at this module scope

build(*args, **kwargs)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(*args, **kwargs)¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

get_bias(name, shape, dtype=tf.float32, initializer=None, prior=None)¶

Get a new or existing bias variable by name, within the current module scope.

See tflon.model.Tower.get_bias

get_input_shapes(*args, **kwargs)¶

Set the input shapes using the arguments to the first __call__ of this module

The default implementation uses the shape of args[0]

get_parameter(key, default=None, parent=False)¶

Get the value of a parameter if it exists and is not None

See tflon.model.Tower.get_parameter

Keyword Arguments:
Parameters:	key (str) – The parameter name
	default – The parameter default value, returned if parameter is None or does not exist (default=None) parent (bool) – Whether to check for the given parameter at the module scope (False), or the parent scope (True) (default=False)

get_variable(name, shape, dtype=tf.float32, initializer=None, trainable=False)¶

Get a new or existing variable by name, within the current module scope.

See tflon.model.Tower.get_variable

get_weight(name, shape, dtype=tf.float32, initializer=None, prior=None)¶

Get a new or existing weight variable by name, within the current module scope.

See tflon.model.Tower.get_weight

has_parameter(key, default=None, parent=False)¶

Check of this module scope, or the parent scope has the specified parameter setting

Parameters:	key (str) – The reference name for the query parameter

Keyword Argument:: parent (bool): Whether to check for the given parameter at the module scope (False), or the parent scope (True) (default=False)

initialize(*args, **kwargs)¶

Initialize this module. Performs two steps:

Get the input shapes from *args
Call Module.build()

Modules need to be initialized before calling methods. If a method is likely called before Module.__call__, then it should be registered as a callable method by decorator @tflon.toolkit.register_callable

Parameters:	args – The arguments passed to __call__ *kwargs – The keyword arguments passed to __call__

inject_parameters()¶

Add the values of Module._parameters as class attributes. For a given (key, value) pair, this will add instance attributes of the form:

self.key = value

This behavior is currently optional, but will eventually become the default.

input_shapes¶: Get the shapes of the tensors passed to this module’s __call__(*args) as a list of tensor shapes, non-tensor types in *args are excluded

inverse(*args, **kwargs)¶

Instantiate an op which has the reverse structure of call. The input signature should be the same as the output of call, and the output signature should match the input of call.

Where implemented, this may not be an exact inverse, check the documentation for each Module

Returns:	A tensor or sequence of tensors

names¶: Get the names of all feature input slots as a dictionary of (name, placeholder) pairs.

scope¶: Get the path of this module scope, including the parent tower scope

set_parameter(key, value)¶

Set a parameter value for this module scope

See tflon.model.Tower.set_parameter

show_data(table_map)¶

This method can be overridden to compute parameters from raw data tables for model initialization prior to optimization.

Parameters:	table_map (dict) – Dictionary of table objects indexed by slot name

slots¶: Return dictionary of placeholders to names of extra feature tables generated by this module.

tensors¶: Returns a list of tensors produced by the module

trainables¶: Get the collection of trainable variables (weights + biases + other trainables) at this module scope

variables¶: Get the collection of all variables (weights + biases + other trainables + non-trainables) at this module scope

weights¶: Get the collection of weights at this module scope

tflon.toolkit.toolkit.apply_reduction(*args, **kwargs)¶

Apply a reduction operation to the output of a tensor operation

Keyword Arguments:
	default_reduction (callable) – A function specifying the default reduction (default=tf.reduce_sum) argnum (int) – The index of the output tensor to reduce (default=None, assumes output is a single tensor)

class tflon.toolkit.toolkit.build_tower(tower)¶: A context manager used by tflon.model.Tower to set the parent tower of new module objects during tensorflow graph construction

tflon.toolkit.toolkit.current_tower()¶: Get the tower currently being constructing, used by Module.__init__

tflon.toolkit.toolkit.decorate_or_return(wrapper_impl, args)¶: Helper function for construction decorators with optional callable signatures

tflon.toolkit.toolkit.get_next_name_scope(cls_name)¶

Get a unique identifier for a new name scope given a template name

Parameters:	cls_name (str) – The template name

tflon.toolkit.toolkit.handle_nan(*args, **kwargs)¶

Decorator to filter nan values from an output tensor, replacing the filtered values with zeros

Keyword Arguments:
	inarg (int) – the index of the input argument to use for identifying filterable values (default=0, the first argument) outarg (int) – the index of the output argument to filter (default=None, assumes output is a single tensor)
Returns:	Function wrapper to apply nan filtering

tflon.toolkit.toolkit.is_callable(obj)¶: Check whether obj is callable

tflon.toolkit.toolkit.is_tensor(T)¶: Check whether T is a tensor type (tf.Tensor or tf.SparseTensor)

tflon.toolkit.toolkit.op_name(op)¶: Get the name of a tensorflow op, excluding parent scope and index information

tflon.toolkit.toolkit.reset_name_scope_IDs()¶: Reset the unique identifier generators to zero

tflon.toolkit.toolkit.set_name_scope(*args, **kwargs)¶

Set the name scope of calls within a function to the function’s name, or some other default value

Keyword Arguments:
	default_name – The default name (default=None, use the function name)

Modules¶

class tflon.toolkit.modules.Apply(*args, **kwargs)¶

Apply the given function or lambda expression.

This is used to inject functions or lambda expressions into Module pipes. For example:

>>> net = Apply(lambda x: x+5) | Dense(10)
>>> outputs = net(inputs)

Parameters:	func (callable) – A callable, function or lambda-type object

build(func)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(*args)¶

Parameters:	*args – Arbitrary arguments passed to the func callable
Returns:	The result of `func(*args)`

class tflon.toolkit.modules.BatchFill(*args, **kwargs)¶

Tile a tensor (e.g a vector) along a new first dimension. This op is used to e.g copy a vector for every input example in a batch.

Parameters:	value (Tensor) – The value to be tiled

build(value)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(input)¶

Parameters:	inputs (Tensor) – Batch tensor, used to calculate batch size for tiling

class tflon.toolkit.modules.Concat(*args, **kwargs)¶

Concatenate the input tensors

Parameters:	axis (int) – The concatenation axis

build(axis)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(*args)¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

class tflon.toolkit.modules.Dense(*args, **kwargs)¶

Fully connected layer with optional activation function

Keyword Arguments:
Parameters:	outputs (int) – The width of the output
	activation (callable) – An activation op (default=lambda x:x)

build(outputs)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(*inputs)¶

Parameters:	inputs (tensor-like*) – One or more input tensors

class tflon.toolkit.modules.ForLoop(*args, **kwargs)¶

Construct a for loop using tf.while_loop with a counter

Parameters:

loop_body (callable) – A callable function representing the loop body, should be compatible with tf.while_loop. Must have signature: lambda i, *args, where i is the current counter value
iterations (tensor-like) – A scalar tensor indicating the number of iterations (can be static or dynamic)
collect_outputs (bool) – If True, collect all values of each loop variable in a tf.TensorArray

build(loop_body, iterations, collect_outputs=False)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(*args)¶

Parameters:	args – Initial input loop variables, must match signature of `loop_body = lambda i, args`

class tflon.toolkit.modules.Gaussian(*args, **kwargs)¶

This module outputs parameters of gaussian distributions (mean, stddev) for each input example. Mean and standard deviation are calculated via fully connected layers. A gaussian error loss function calculated in normalized space is also added to the model.

Keyword Arguments:
Parameters:	targets – Target values, automatically normalized prior to loss calculations
	distribution – Name of the input table containing the target data distribution. fixed_variance – Assume variance is constant w.r.t. the input domain nans – Handle nan values in error calculation (default=False) reduction – Apply reduction to loss values (default=tf.reduce_sum)
Returns:	The mean and standard deviation, in the same scale as the un-normalized targets
Return type:	mu, sigma

build(targets, distribution=None, nans=False, reduction=<function reduce_sum>)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(inputs, compute_loss=True)¶

Parameters:	inputs (tensor) – The input features for computing mu and sigma via linear ops

class tflon.toolkit.modules.Get(*args, **kwargs)¶

Get arguments from an input tuple and return a selected subset of them.

Use this module to select an output from a multi-output module (e.g an RNN) to pass to the next element of a chain.

Parameters:	groups (slice) – Slices to apply to `inputs` to select the output

build(*groups)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(*inputs)¶

Select slice groups from the tuple *inputs

Parameters:	*inputs – arguments from which to select

class tflon.toolkit.modules.Highway(*args, **kwargs)¶

Add a highway network bypass gate to a given computational block.

https://arxiv.org/abs/1505.00387

Parameters:	block (callable) – A module or function which executes the main computation.

build(block)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(*inputs)¶

Compute:: block(inputs) * gate(inputs) + inputs * (1 - gate(inputs))

Parameters:	inputs (tensor) – An input tensor passed through both block and highway

class tflon.toolkit.modules.Identity(*args, **kwargs)¶

Output the input unchanged

build()¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(*args)¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

class tflon.toolkit.modules.InterpolatedSequence(*args, **kwargs)¶

Builds a vector of accumulated sequential feature weights. This can be used with tflon.toolkit.interpolate1d to learn samples from the output of a higher dimensional projection function f: R -> R^n

Keyword Arguments:
Parameters:	bins (tensor-like) – 1-D sequence of fixed, known dimension indicating the sampling points of the function domain
	features (int) – The number of higher dimensional features to sample from the function’s range interpolator (callable) – An interpolation op (usually a lambda expression calling interpolate1d)

build(bins)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(points)¶

Parameters:	points (tensor-like) – 1-D sequence of points for generating interpolations from the sampled bin features

class tflon.toolkit.modules.LayerNorm(*args, **kwargs)¶

build()¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(inputs)¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

class tflon.toolkit.modules.Merge(*args, **kwargs)¶

Merge a list of lists of tensors into a single list of tensors

build()¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(*args)¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

class tflon.toolkit.modules.MultivariateGaussian(*args, **kwargs)¶

build(targets, distribution=None, nans=False, reduction=<function reduce_sum>)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(inputs)¶

Parameters:	inputs (tensor) – The input features for computing mu and sigma via linear ops

class tflon.toolkit.modules.NormalizedInput(*args, **kwargs)¶

Normalize inputs to Z-distributions

Keyword Arguments:
	distribution_table – The name of the table to collect distribution information, if None infer the table name from the name of the input tensor passed to call (default=None)

build(distribution_table=None)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(X)¶

Parameters:	X (tensor) – The tensor to be scaled

inverse(Y)¶

Instantiate an op which has the reverse structure of call. The input signature should be the same as the output of call, and the output signature should match the input of call.

Where implemented, this may not be an exact inverse, check the documentation for each Module

Returns:	A tensor or sequence of tensors

show_data(table_map)¶

This method can be overridden to compute parameters from raw data tables for model initialization prior to optimization.

Parameters:	table_map (dict) – Dictionary of table objects indexed by slot name

class tflon.toolkit.modules.PCA(*args, **kwargs)¶

Compute a PCA matrix transformation, optionally handles missing values by probabilistic imputation.

Keyword Arguments:
Parameters:	distribution_table (str) – The name of the table from which to compute PCA
	nans (bool) – If True, use probabilistic PCA to handle missing values

build(distribution_table, nans=False, imputed_table=None)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(inputs, sigma=None)¶

Project from a low dimensional input space to a higher dimensional output space using a transformation learned from data

Keyword Arguments:
Parameters:	inputs – The input values in the low dimensional space
	sigma – Optional tensor of sigma values, when provided, inputs and sigma form the parameters of independent normal distributions, which are projected to a multivariate normal distribution in the output space

normalize(*args, **kwargs)¶: Normalize targets in the high dimensional space

project(*args, **kwargs)¶: Project from a high dimensional space to a low dimensional input space.

show_data(table_map)¶

This method can be overridden to compute parameters from raw data tables for model initialization prior to optimization.

Parameters:	table_map (dict) – Dictionary of table objects indexed by slot name

class tflon.toolkit.modules.Product(*args, **kwargs)¶

Perform product on multiple outputs (usually used with Inject or Join)

build()¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(*args)¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

class tflon.toolkit.modules.SegmentReduce(*args, **kwargs)¶

Reduce segmented data to a single vector for each input example.

Keyword Arguments:
Parameters:	input_table (str) – The name of the input table to use when constructing reference segment indices (Passed input must be a DenseNestedTable)
	reduction_op (func) – A tensorflow segment reduce operation (default=tf.segment_sum weighting (callable) – A weighting function with signature `func(tensor, segments)` (Default: None) * tensor: N-D tensor for weight calculation * segments: 1-D tensor of segment indices

build(input_table, reduction_op=<function segment_sum>, weighting=None)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(inputs)¶

Parameters:	inputs (tensor) – The tensor to be reduced

class tflon.toolkit.modules.Sum(*args, **kwargs)¶

Perform sum on multiple outputs (usually used with Inject or Join)

build()¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(*args)¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

class tflon.toolkit.modules.WindowInput(*args, **kwargs)¶

Rescale inputs to range between 0-1 by the following transformation:

I - min(I, axis=0) / (max(I, axis=0) - min(I, axis=))

Keyword Arguments:
	distribution_table – The name of the table to collect distribution information, if None infer the table name from the name of the input tensor passed to call (default=None)

build(distribution_table=None)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(X)¶

Parameters:	X (tensor) – The tensor to be scaled

inverse(Y)¶

Instantiate an op which has the reverse structure of call. The input signature should be the same as the output of call, and the output signature should match the input of call.

Where implemented, this may not be an exact inverse, check the documentation for each Module

Returns:	A tensor or sequence of tensors

show_data(table_map)¶

This method can be overridden to compute parameters from raw data tables for model initialization prior to optimization.

Parameters:	table_map (dict) – Dictionary of table objects indexed by slot name

Tensorflow extensions¶

tflon.toolkit.ops.batch_duplicate(T, multiples, axis=0)¶

Duplicate slices of a tensor along a given axis. Given an input of shape […, Daxis, …], produces an output of shape […, Daxis * multiples, …]

Keyword Arguments:
Parameters:	T (tensor-like) – An input tensor multiples (int) – Number of copies of each slice to insert
	axis (int) – The axis along which to copy elements

tflon.toolkit.ops.batch_matmul(A, B)¶

Multiply 2-D matrices packed into a 3-D tensor by vectors packed into a 2-D matrix

This will perform a left or right multiply depending on which tensor is given for A or B.

Parameters:	A (tensor-like) – Shape (N, K, L) or (N, K) B (tensor-like) – Shape (N, L) or (N, K, L)
Returns:	Tensor

tflon.toolkit.ops.broadcast_matmul(A, B)¶

Multiply 2-D matrices packed into a 3-D tensor by a single 2-D matrix

This will perform a left or right multiply depending on which tensor is given for A or B.

Parameters:	A (tensor-like) – B (tensor-like) –
Returns:	Tensor

tflon.toolkit.ops.interpolate1d(x, fx, y, method='linear', **kwargs)¶

Interpolation interface.

Keyword Arguments:
Parameters:	x (tensor-like) – A 1-D tensor of shape [batch_size] containing values at which to evaluate f(x) fx (tensor-like) – A 1-D tensor of shape [num_samples] containing sampled x values of interpolated function f y (tensor-like) – A 2-D tensor of shape [num_samples, num_features] containing sampled values y=f(x) of interpolated function f
	method (str) – linear or rbf methods are supported
Returns:
Return type:	tensor

tflon.toolkit.ops.nan_filter(T, axis=1, partitions=None)¶

Filter nan values from a tensor and append an indicator tensor along the specified axis

Keyword Arguments:
Parameters:	T (tensor-like) – An input tensor of any shape, possibly containing nans
	axis (int) – The axis on which to append the indicator variables partitions (list) – List of groups to create indicators, if None, treat all columns independently (default=None)

tflon.toolkit.ops.nanmean(T, axis=None)¶

Return a function to perform nan-safe means on tensors

Keyword Arguments:
Parameters:	T – A tensor potentially containing nan values to use as a filter
	axis (int) – The reduction axis (default=None)
Returns:	nanmean(x)
Return type:	lambda x

tflon.toolkit.ops.nansum(T, axis=None)¶

Return a function to perform nan-safe sums on tensors

Keyword Arguments:
Parameters:	T – A tensor potentially containing nan values to use as a filter
	axis (int) – The reduction axis (default=None)
Returns:	nansum(x)
Return type:	lambda x

tflon.toolkit.ops.residual_covariance(*args, **kwargs)¶

Given estimated residuals (X-mu), compute the covariances (and optionally, variances)

Keyword Arguments:
Parameters:	residuals (tensor-like) – A 2-D tensor of batch x num_variables, the inputs, centered by subtracting the estimated means (X-mu)
	include_diag (bool) – If true, also calculate the variances along the diagonal. If false, output zeros on the diagonal (default=False)
Returns:	a 2-D covariance tensor of shape num_variables x num_variables
Return type:	Tensor

tflon.toolkit.ops.segment_softmax(*args, **kwargs)¶

Compute the softmax along segments of a tensor. This is implemented using a numerically stable solution.

Keyword Arguments:
Parameters:	X (tensor-like) – The input tensor to normalize segs (tensor-like) – A 1-D tensor of integer segment assignments
	epsilon (float) – A constant factor used to prevent divide by zero error

tflon.toolkit.ops.segment_topN(*args, **kwargs)¶

Computes indices the topN elements for each segment along the first axis of inputs, returning the indices of each element in the final dimension

For input of shape (?, 4), num_segments=10, N=5, output rows would have shape: (10, 20, 2)

use with values = tf.gather_nd(inputs, segment_topN_result) to collect the output

Keyword Arguments:
Parameters:	inputs (tensor) – a 2-dimensional tensor segments (tensor) – a 1-dimensional tensor of segment IDs corresponding to the first axis of inputs
	N (int) – The number of top elements to return (default=5) default_value – The default value for the tensor, used when fewer than N top values are available (default=0)

Metrics¶

tflon.toolkit.metrics.accumulate_values(*args, **kwargs)¶

Accumulate tensor values by concatenating along the specified axis

Keyword Arguments:
Parameters:	values (tf.Tensor) – A tensorflow op containing the next value to be added
	axis (int) – If provided, concatenate along the specified axis. If None, reshape values to 1-D and concatenate along axis=0 init_len (int) – Specifies the initial size of the concatenation tensor, the tensor will contain init_len vectors of zeros along axis (default=0)
Returns:	The tf.Tensor containing the concatenated tensors and an operation to update the concat.
Return type:	accumulated, update_op

tflon.toolkit.metrics.auc(*args, **kwargs)¶

Return a tf.py_func which wraps sklearn.metrics.roc_auc_score. Optionally, handle nan values.

Keyword Arguments:
Parameters:	T (tensor-like) – A 2-D tensor of shape batch x classes containing binary class labels L (tensor-like) – A 2-D tensor with the same shape as T containing logit scores for each class
	axis (int) – If provided, calculate the auc over the specified axis (default=all) nans (bool) – If True, nans in tensor T will be filtered prior to auc calculations
Returns:	A 0-D or 1-D tensor of AUC values

tflon.toolkit.metrics.average_z_score(*args, **kwargs)¶

Compute the average Z-score of a gaussian regression

Keyword Arguments:
Parameters:	targets (tensor-like) – N-D tensor containing the target data mu (tensor-like) – N-D tensor of the same shape as targets containing estimated distribution means sigma (tensor-like) – N-D tensor of the same shape as targets containing estimated distribution standard deviations
	axis (int) – If provided, calculate the average over the specified axis (default=all) nans (bool) – If True, nans in tensor T will be filtered prior to auc calculations
Returns:	The average z-score
Return type:	Tensor

tflon.toolkit.metrics.pearson(*args, **kwargs)¶

Compute the pearson correlation

Keyword Arguments:
Parameters:	targets (tensor-like) – N-D tensor containing the target data predictions (tensor-like) – N-D tensor of the same shape as targets containing estimated values
	axis (int) – If provided, calculate the average over the specified axis (default=all) nans (bool) – If True, nans in tensor T will be filtered prior to auc calculations
Returns:	The average z-score
Return type:	Tensor

tflon.toolkit.metrics.segment_average_auc(*args, **kwargs)¶

Return a tf.py_func which computes the average auc over segments of two tensors. Handles nans.

Keyword Arguments:
Parameters:	T (tensor-like) – A 2-D tensor of shape batch x classes containing binary class labels L (tensor-like) – A 2-D tensor with the same shape as T containing logit scores for each class segments (tensor-like) – A 1-D tensor of segment indices of type tf.int32, with the same first dimension size of T and L
	axis (int) – If provided, calculate the auc over the specified axis (default=all) nans (bool) – If True, nans in tensor T will be filtered prior to auc calculations
Returns:	A 0-D or 1-D tensor of average AUC values over segments of the input

Priors¶

Losses¶

class tflon.toolkit.losses.CovarianceLoss(*args, **kwargs)¶

Compute the multivariate gaussian distribution loss, assuming covariance is constant over the input domain.

Keyword Arguments:
Parameters:	targets (tensor-like) – A 2-D tensor of batch_size x num_variables mu (tensor-like) – A 2-D tensor of estimated means for each example
	sigma (tensor-like) – A 2-D tensor of estimated standard deviations for each example (default=None, if None, calculate variance from the data, assume constant over the input domain)
Returns:	The multivariate gaussian loss, a 0-D tensor
Return type:	Tensor

build(targets)¶: Instantiate variables required for this module. This is called by __init__, with *args and **kwargs passed from __init__ arguments list.

call(mu, sigma=None)¶

Instantiate ops executing the module. This is the implementation of __call__, invoked by the base class.

Returns:	A tensor or sequence of tensors

tflon.toolkit.losses.elastic_net(*args, **kwargs)¶

Computes the elastic net penalty of a list of weights:

>>> (1 - alpha) * sum(W ** 2) + alpha * sum(|W|)

Parameters:	Ws (list) – tensor-like objects, weights to compute elastic net penalty alpha (float) – A number between zero and one

tflon.toolkit.losses.gaussian_error(*args, **kwargs)¶

Computes the sum of gaussian losses for multiple gaussian targets

Keyword Arguments:
Parameters:	targets (tensor-like) – N-D tensor with target regression values mu (tensor-like) – N-D tensor with predicted means sigma (tensor-like) – N-D tensor with predicted standard deviations
	nans (bool) – Apply nan filter prior to reduction (default=False) reduction (callable) – A reduction op to apply to the partial losses (default=tf.reduce_sum)

tflon.toolkit.losses.l1_penalty(*args, **kwargs)¶

Computes the total L1 penalty of a list of weights

Parameters:	Ws (list) – tensor-like objects, weights to compute L1 penalty

tflon.toolkit.losses.l2_penalty(*args, **kwargs)¶

Computes the total L2 penalty of a list of weights

Parameters:	Ws (list) – tensor-like objects, weights to compute L2 penalty

tflon.toolkit.losses.xent(*args, **kwargs)¶

Computes the unweighted sum of cross entropy losses for multiple binary targets.

Keyword Arguments:
Parameters:	T (tensor-like) – N-D tensor with target class labels L (tensor-like) – N-D tensor with same shape as T, containing logit network output
	nans (bool) – Apply nan filter prior to reduction

tflon.toolkit.losses.xent_softmax(*args, **kwargs)¶

Computes an unweighted softmax cross entropy

Includes optional handling of missing values

Keyword Arguments:
Parameters:	T (tensor-like) – N-D tensor with target class labels L (tensor-like) – N-D tensor with same shape as T, containing logit network output
	nans (bool) – Apply nan filter prior to reduction

tflon.toolkit.losses.xent_uniform_sum(*args, **kwargs)¶

Computes the unweighted sum of cross entropy losses for multiple binary targets.

Keyword Arguments:
Parameters:	T (tensor-like) – N-D tensor with target class labels L (tensor-like) – N-D tensor with same shape as T, containing logit network output
	nans (bool) – Apply nan filter prior to reduction

tflon.toolkit.losses.xent_weighted(*args, **kwargs)¶

Computes the unweighted sum of cross entropy losses for multiple binary targets.

Keyword Arguments:
Parameters:	T (tensor-like) – N-D tensor with target class labels L (tensor-like) – N-D tensor with same shape as T, containing logit network output W (tensor-like) – 0-D tensor with the weight for positive examples
	nans (bool) – Apply nan filter prior to reduction

tflon.toolkit.losses.xent_weighted_sum(*args, **kwargs)¶

Computes the unweighted sum of cross entropy losses for multiple binary targets.

Keyword Arguments:
Parameters:	T (tensor-like) – N-D tensor with target class labels L (tensor-like) – N-D tensor with same shape as T, containing logit network output W (tensor-like) – 0-D tensor with the weight for positive examples
	nans (bool) – Apply nan filter prior to reduction

Trainers¶

class tflon.train.trainer.Hook(frequency)¶

Base class for training hooks. New hooks should inherit this class and override Hook.call (optionally, override Hook.finish)

Hook.call should return a tensorflow op, usually constructed using tf.py_func The op returned by Hook.call will be called at fixed intervals

Hook.finish is called after a Trainer completes Trainer.train

Parameters:	frequency (int) – Number of iterations between invocations of this hook’s call method

class tflon.train.trainer.LambdaHook(func)¶: A hook used to wrap lambda functions passed as hooks to Trainer. This is automatically applied by Trainer

class tflon.train.trainer.LearningRateDecayHook(initial_rate, decay, frequency)¶

class tflon.train.trainer.LogProgressHook(frequency, maxiter)¶

Hook to print progress to screen at specified intervals, automatically created by TFTrainer and OpenOptTrainer.

Parameters:	frequency (int) – The interval between logging output maxiter (int) – The number of training iterations

class tflon.train.trainer.OpenOptTrainer(iterations=150, solver='scipy_lbfgsb', options={}, log_frequency=20, **kwargs)¶

Wrapper that applies an optimizer from OpenOpt (wrapper for scipy.minimize) to train a tflon model.

Keyword Arguments:
	iterations (int) – The number of training iterations (default=150) solver (str) – The minimizer to use, valid values include any algorithm implemented by scipy.minimize (default=scipy_lbfgsb) options (dict) – Options to pass to the openopt.NLP module (default={}) log_frequency (int) – Frequency to print progress to screen or log file (default=20)

class tflon.train.trainer.SummaryHook(directory, frequency=10, summarize_trainables=False, summarize_gradients=False, summarize_activations=False, summarize_losses=True, summarize_resources=True)¶

Hook to write tensorboard logs. Only logs loss values by default

Keyword Arguments:
Parameters:	directory (str) – The directory to create tensorboard event files
	frequency (int) – The number of iterations between log entries (default=10) summarize_trainables (bool) – Include histograms of all weight and bias variables (default=False) summarize_gradients (bool) – Include plots of all gradient magnitudes (default=False) summarize_losses (bool) – Include plots of all loss magnitudes (default=True)

class tflon.train.trainer.TFTrainer(optimizer, iterations, checkpoint=None, resume=False, log_frequency=100, gpu_profile_file=None, **kwargs)¶

Wrapper that applies an optimizer inheriting from tf.train.Optimizer to train a tflon model.

Keyword Arguments:
Parameters:	optimizer (tf.train.Optimizer) – An instance of tensorflow Optimizer iterations (int) – The number of training iterations
	checkpoint (tuple) – None or pair of directory (str) and frequency (int). If not None, then write checkpoint files to the specified directory at specified intervals (default=None) resume (bool) – Resume from checkpoint file, if available (default=False) log_frequency (int) – Frequency to print progress to screen or log file (default=100)

class tflon.train.trainer.Trainer(hooks=[])¶

Trainer base class for use with tflon Model.fit

Keyword Arguments:
	hooks (list) – Zero or more Hook objects, to be called during training (default=[])

Optimizers¶

class tflon.train.optimizer.GradientAccumulatorOptimizer(optimizer, gradient_steps, **kwargs)¶

Wrapper for tf.train.Optimizer instances which accumulates gradients over several minibatches before applying the gradients

Parameters:	optimizer (tf.train.Optimizer) – An optimizer instance to wrap and accumulate gradients gradient_steps (int) – The number of steps over which to accumulate gradients

apply_gradients(gradients)¶

Apply gradients to variables.

This is the second part of minimize(). It returns an Operation that applies gradients.

Parameters:	grads_and_vars – List of (gradient, variable) pairs as returned by compute_gradients(). global_step – Optional Variable to increment by one after the variables have been updated. name – Optional name for the returned operation. Default to the name passed to the Optimizer constructor.
Returns:	An Operation that applies the specified gradients. If global_step was not None, that operation also increments global_step.
Raises:	`TypeError` – If grads_and_vars is malformed. `ValueError` – If none of the variables have gradients. `RuntimeError` – If you should use _distributed_apply() instead.

compute_gradients(*args, **kwargs)¶

Compute gradients of loss for the variables in var_list.

This is the first part of minimize(). It returns a list of (gradient, variable) pairs where “gradient” is the gradient for “variable”. Note that “gradient” can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.

Parameters:	loss – A Tensor containing the value to minimize or a callable taking no arguments which returns the value to minimize. When eager execution is enabled it must be a callable. var_list – Optional list or tuple of tf.Variable to update to minimize loss. Defaults to the list of variables collected in the graph under the key GraphKeys.TRAINABLE_VARIABLES. gate_gradients – How to gate the computation of gradients. Can be GATE_NONE, GATE_OP, or GATE_GRAPH. aggregation_method – Specifies the method used to combine gradient terms. Valid values are defined in the class AggregationMethod. colocate_gradients_with_ops – If True, try colocating gradients with the corresponding op. grad_loss – Optional. A Tensor holding the gradient computed for loss.
Returns:	A list of (gradient, variable) pairs. Variable is always present, but gradient can be None.
Raises:	`TypeError` – If var_list contains anything else than Variable objects. `ValueError` – If some arguments are invalid. `RuntimeError` – If called with eager execution enabled and loss is not callable.

@compatibility(eager) When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored. @end_compatibility

class tflon.train.optimizer.GradientClippingOptimizer(optimizer, clip_value, **kwargs)¶

Wrapper for tf.train.Optimizer instances which applys tf.clip_by_global_norm to the optimizer gradients

Parameters:	optimizer (tf.train.Optimizer) – An optimizer instance to wrap and apply clipping clip_value (float) – The global norm threshold for clipping

compute_gradients(*args, **kwargs)¶

Compute gradients of loss for the variables in var_list.

This is the first part of minimize(). It returns a list of (gradient, variable) pairs where “gradient” is the gradient for “variable”. Note that “gradient” can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.

Parameters:	loss – A Tensor containing the value to minimize or a callable taking no arguments which returns the value to minimize. When eager execution is enabled it must be a callable. var_list – Optional list or tuple of tf.Variable to update to minimize loss. Defaults to the list of variables collected in the graph under the key GraphKeys.TRAINABLE_VARIABLES. gate_gradients – How to gate the computation of gradients. Can be GATE_NONE, GATE_OP, or GATE_GRAPH. aggregation_method – Specifies the method used to combine gradient terms. Valid values are defined in the class AggregationMethod. colocate_gradients_with_ops – If True, try colocating gradients with the corresponding op. grad_loss – Optional. A Tensor holding the gradient computed for loss.
Returns:	A list of (gradient, variable) pairs. Variable is always present, but gradient can be None.
Raises:	`TypeError` – If var_list contains anything else than Variable objects. `ValueError` – If some arguments are invalid. `RuntimeError` – If called with eager execution enabled and loss is not callable.

@compatibility(eager) When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored. @end_compatibility

class tflon.train.optimizer.WrappedOptimizer(optimizer, name=None, use_locking=False)¶

An optimizer that wraps another tf.Optimizer, used to modify gradient computations.

get_slot(var, sn)¶

Return a slot named name created for var by the Optimizer.

Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.

Use get_slot_names() to get the list of slot names created by the Optimizer.

Parameters:	var – A variable passed to minimize() or apply_gradients(). name – A string.
Returns:	The Variable for the slot if it was created, None otherwise.

get_slot_names()¶

Return a list of the names of slots created by the Optimizer.

See get_slot().

Returns:	A list of strings.

variables()¶

A list of variables which encode the current state of Optimizer.

Includes slot variables and additional global variables created by the optimizer in the current default graph.

Returns:	A list of variables.

Curriculum¶

class tflon.train.sampling.Curriculum(levels, step_function=<function curriculum_step_distribute>, **kwargs)¶

Interface for curriculum orchestration, curriculum strategies must implement:

def evaluate(self, model, step, loss):: return False

Usage Example:

level1 = … # Some function returning generators of batches level2 = … # ditto trainer = tflon.train.TFTrainer(tf.train.AdamOptimizer(1e-3)) curr = tflon.train.FixedIntervalCurriculum([level1, level2]) model.fit(curr.iterate(), trainer)

Keyword Arguments:
Parameters:	levels (list) – List of functions returning generators corresponding to sorted order of occurence for curriculum examples
	frequency (int, required) – Number of steps between curriculum evaluations step_function (function) – The type of curriculum update function to use (default=curriculum_step_distribute)

evaluate(step, loss)¶

Curriculum evaluation

Parameters:	step (numpy.int64) – The current global optimization step loss (numpy.float32) – The current optimizer loss
Returns:	Indicator for early termination of training (see tflon.train.Hook)
Return type:	boolean

class tflon.train.sampling.FixedIntervalCurriculum(*args, **kwargs)¶

Keyword Arguments:
Parameters:	levels (list) – List of functions returning generators corresponding to sorted order of occurence for curriculum examples
	frequency (int, required) – The number of steps between curriculum changes

evaluate(step, loss)¶

Curriculum evaluation

Parameters:	step (numpy.int64) – The current global optimization step loss (numpy.float32) – The current optimizer loss
Returns:	Indicator for early termination of training (see tflon.train.Hook)
Return type:	boolean

class tflon.train.sampling.MetricGatedCurriculum(criterion, *args, **kwargs)¶

Keyword Arguments:
Parameters:	criterion (lambda) – Lambda expression taking result of model.evaluate and returning whether the current level is passed levels (list) – List of functions returning generators corresponding to sorted order of occurence for curriculum examples
	frequency (int, required) – The number of steps between test evaluations

evaluate(step, loss)¶

Curriculum evaluation

Parameters:	step (numpy.int64) – The current global optimization step loss (numpy.float32) – The current optimizer loss
Returns:	Indicator for early termination of training (see tflon.train.Hook)
Return type:	boolean

tflon.train.sampling.merge_feeds(*iterators)¶: Merge multiple feed iterators into a single iterator

How does tflon work?¶

tflon is designed to simplify book-keeping in tensorflow, so that you can focus on producing high quality models. It provides five key functionalities:

A data API which combines high-speed, column-based data formats with convienent transformations functions to convert between disk storage and tensorflow compatible forms (see Data API)
A simple model API, which featurizes input data, tracks variables/parameters, coordinates training/inference, and provides model serialization (see Model API)
A toolkit API, which provides custom built tensorflow ops, and higher level components (see Toolkit API)
A distributed API, which enables running tensorflow in parallel on certain types of clusters (see Distributed API)
Domain-specific APIs, which provide some tools for working with data for specific applications (e.g chemistry, see Domain-specific APIs)

Data API¶

The tflon data pipeline slightly prioritizes flexibility over speed.

Data flow:

Main process |   Rows --+                               +--> Push tf.Queue
             |          |                               |
             |          |                               |
Subprocesses |          +--> featurize + to feed dict --+

For stochastic minibatch gradient descent type models, there are three stages:

An iterator provides batch dictionaries mapping input name (str) -> data (Table)

One or more worker processes receive batches for pre-processing, which has 3 steps:

Model objects pre-process the batch dictionary, changing or adding Table objects

Module objects augment the batch dictionary with additional Table objects

Table objects are converted to a tensor dictionary of name (str) -> tf.Tensor or tf.SparseTensorValue

Tensor dictionaries are fed into a TensorQueue for transfer to device memory

Data is loaded to a tower from the TensorQueue assigned to the tower

Global gradient descent models (e.g using OpenOptTrainer) use a single batch dictionary input (e.g a TableFeed object), but store the resulting tensors in (GPU) memory for the entire training run using a TensorLoader.

Model API¶

The model API is designed to be as lightweight as possible. User models extend the Model base class and override the def _model(self, tower) function. tower in this case is an instance of Tower, which provides convenient methods for creating named input and output tensors and creating, storing, and tracking variables. Unlike keras, models do not have to conform to a particular structure, but can have flexible forms and use pure tensorflow ops.

There are two important constraints on model creation:

Variables should be created with Tower.get_weight, Tower.get_bias and Tower.get_variable
Inputs and outputs should be defined by Tower.add_input, Tower.add_target and Tower.add_output

Toolkit API¶

The toolkit API provides extensions to tensorflow to provide additional ops, and also higher level components that require additional features such as:

Data-based initialization (e.g input data min-max windowing)
Weight reuse (e.g recursive network nodes)
Special input featurizations (e.g indexing atom pairs)

Distributed API¶

The distributed API leverages horovod and mpi4py to enable simple scaling of single-GPU tensorflow models to multi-GPU environments, possibly spread over many individual computing nodes. This framework will run in any environment supporting mpirun.

To convert a single-GPU model to distributed form, only a few steps are required

Data must be split into multiple shards (subsets of examples). Ideally, shards are split evenly among all the compute units. Shards are stored as subdirectories of a top-level directory, for example:

dataset/
    shard_X/
        table_A.pq
        table_B.pq
        table_C.pq
    shard_Y/
        table_A.pq
        table_B.pq
        table_C.pq

The distributed environment is initialized by a call to tflon.distributed.init_distributed_resources at the beginning of your scripts.
Data is loaded by tflon.distributed.make_distributed_table_feed(), which divides shards among nodes using a modulus strategy, and wraps tables in a tflon.distributed.DistributedTable wrapper, which supports distributed data-initialization ops (e.g min/max windowing).

An example for running distributed models can be found at tflon_core/examples/distributed.py

Domain-specific APIs¶

Tools for working with chemical and graph data are provided by tflon.chem and tflon.graph.

Usage example¶

This basic neural network example has five main parts: imports, model definition, data loading, training and evaluation.

Imports

import tensorflow as tf
import pandas as pd
import tflon
from pkg_resources import resource_filename

We create a simple neural network with two hidden layers, sigmoid output, cross entropy loss and l2 regularization.

class NeuralNet(tflon.model.Model):
    def _model(self):
        I = self.add_input('desc', shape=[None, 210])
        T = self.add_target('targ', shape=[None, 1])

        net = tflon.toolkit.WindowInput() |\
              tflon.toolkit.Dense(20, activation=tf.tanh) |\
              tflon.toolkit.Dense(5, activation=tf.tanh) |\
              tflon.toolkit.Dense(1)
        L = net(I)

        self.add_output( "pred", tf.nn.sigmoid(L) )
        self.add_loss( "xent", tflon.toolkit.xent(T, L) )
        self.add_loss( "l2", tflon.toolkit.l2_penalty(self.weights) )
        self.add_metric( 'auc', tflon.toolkit.auc(T, L) )

We load data from a csv file, create a feed consisting of two tables, which feed the two tensor inputs (descriptors and targets).

# Import data and create a feed
df = pd.read_csv( resource_filename('tflon_test.data', 'cyp.tsv'), sep='\t', index_col='ID' )
feed = tflon.data.TableFeed({'desc':tflon.data.Table(df[df.columns[:-1]]), 'targ':tflon.data.Table(df[[df.columns[-1]]])})

We then instantiate and train the model with the OpenOptTrainer, an interface between tensorflow and the scipy optimizers.
Finally, we perform inference on the training data and check the fit of the model using the AUC metric.

# Create a neural network tower
NN = NeuralNet()

# Create an L-BFGS trainer
trainer = tflon.train.OpenOptTrainer( iterations=100 )

with tf.Session():
    # Run the trainer
    NN.fit( feed, trainer, restarts=2 )

    # Perform inference
    auc = NN.evaluate( feed, query='auc' )
    print "AUC:", auc

More Examples¶

More examples can be found in tflon_core/python/examples, and at Usage Examples.