Trump¶

Step 1. Install Package¶

SUMMARY OF STEP 1: Clone and install trump, from github.

git clone https://github.com/Equitable/trump.git + cd trump + python setup.py install

Step 2. Configure Settings¶

SUMMARY OF STEP 2: Put a SQLAlchemy Engine String in trump/config/trump.cfg. Comment out all other engines.

Trump needs information about a database it can use, plus there are a couple other settings you may want to tweak. You can either follow the instructions below, or pass a SQLAlchemy engine/engine-string, to both SetupTrump() and SymbolManager() everytime you use them.

The configuration file for trump is in:

userbase/PythonXY/site-packages/trump/config/trump.cfg

or

yourprojfolder/trump/config/trump.cfg

Assuming you want to use a file based sqlite database (easiest, for beginners), change:

engine: sqlite:// to ;engine: sqlite:// (notice the semi-colon, this just comments out the line)

and change this line:

;engine: sqlite:////home/jnmclarty/Desktop/trump.db to

engine: sqlite:////home/path/to/some/file/mytrumpfile.db (on linux) or

engine: sqlite:///C:\path\to\some\mytrumpfile.db (on windows)

The folder needs to exist in advance, the file should not exist. Trump creates the file.

Step 3. Adjust Existing Template Settings (Optional)¶

SUMMARY OF STEP 3: Adjust any settings for templates you intend you use.

Trump has template settings, stored in multiple settings files, using an identical method as the config file in Step 2. pip or python setup.py install would have created some from samples. Using any other installation methode, you would have to rename cfg_sample to cfg yourself.

The files are here:

userbase/PythonXY/site-packages/trump/templates/settings/

or

yourprojfolder/trump/templates/settings/

Edit trump/templating/settings cfg files, depending on the intended data sources to be used.

See the documentation section “Configuring Data Sources” for guidance.

Step 4. Run SetupTrump()¶

SUMMARY OF STEP 4: Run trump.SetupTrump(), to setup the tables required for Trump to work.

Running the code block below, will create all the tables required in the database provided in Step 2.

from trump import SetupTrump
SetupTrump()
# Or, if you skipped step 2 correctly, you could do:
SetupTrump(r'sqlite:////home/path/to/some/file/mytrumpfile.db')

If it all worked, you will see “Trump is installed @...”. You’re Done! Hard part is over.

You’re now ready to create a SymbolManager, which will help you create your first symbol.

from trump import SymbolManager
sm = SymbolManager()
# Or, if you skipped step 2 correctly, you could do:
sm = SymbolManager(r'sqlite:////home/path/to/some/file/mytrumpfile.db')
...
mysymbol = sm.create('MyFirstSymbol') # should run without error.

Configuring Data Sources¶

Data feed source template classes map to their respective .cfg file in the templating/settings directory, as discussed in Step 3.

The goal of the files is to add a small layer of security. The goal of the template classes is to reduce code during symbol creation scripts. There is nothing preventing a password from being hardcoded into a template, the same way a tablename can be added to a .cfg file. It’s only a maintenance decision for the admin.

The sections of the cfg files get used by the template’s in their respective classes. The section of the config files’ names are then either referenced at the symbol creation point, storing .cfg file info with the symbol in the database, or leaving Trump to query the attributes at every cache, from the source .cfg file.

Trump will use parameters for a source in the following order:

1. Specified explicitly when a template is used. (Eg. table name)

#Assuming the template doesn't clober the value.
myfeed = QuandlFT(authtoken='XXXXXXXX')

2. Specified implicitly using default value or logic derived in the template. (Eg. Database Names)

class QuandlFT(object):
   def __init__(authtoken ='XXXXXXXXX'):
      if len(authtoken) == 8:
         self.authtoken = authtoken
      else:
         self.authtoken = 'YYYYYYYYY'

3. Specified implicitly using read_settings(). (Eg. database host, port)

class QuandlFT(object):
   def __init__(**kwargs):
          autht = read_settings('Quandl', 'userone', 'authtoken')
      self.authtoken = autht

4. Specified via cfg section. (Eg. authentication keys and passwords)

class QuandlFT(object):
   def __init__(**kwargs):
      self.meta['stype'] = 'Quandl' #cfg file name
      self.meta['sourcing_key'] = 'userone' #cfg file section

contents of templating/settings/Quandl.cfg:

[userone]
authtoken = XXXXXXXXX

If the template points to a section of a config file, rather than reading in a value from a config file, (ie, #4), the info will not be stored in the database. Instead, the information will be looked up during caching from the appropriate section in the cfg file.

This means that the cfg file values can be changed post symbol creation, outside of Trump.

Testing the Installation¶

After Trump has been configured, and pointed at a database via an engine string using a config file, one can run the py.test enabled test suite. The tests require network access, but will skip certain tests without it. The testing suite makes a mess, and doesn’t clean up after itself. So, be prepared to run it on a database which can be delete immediately after.

Insight into compatibility with databases other SQLite and PostGres, are of interest to the maintainers. So, if you run the test suite on some other database, and it all works, do let us know via a GitHub issue or e-mail. If it doesn’t, please let us know that as well!

Symbol Manager¶

Conversion Manager¶

Symbols¶

Feeds¶

Centralized Data Editing¶

Each trump datatable comes with two extra columns beyond the feeds, index and final.

The two columns are populated by any existing Override and FailSafe objects which survive caching, and modification to feeds.

Any Override will get applied blindly regardless of feeds, while the FailSafe objects are used only when data isn’t availabe for a specific point. Once a datapoint becomes available for a specific index in the datatable, the failsafe is ignored.

Reporting¶

During the cache process, information comes back from validity checks, and any exceptions. This area of Trump’s code base is currently WIP, however the basic idea is that the caching of a Feed, returns a FeedReport. For each cached Feed, there would be one report, all of which would get aggregated up into, and combined with the symbol-level information, in a SymbolReport. When the SymbolManager caches one or more symbols, it aggregates SymbolReports into one big and final TrumpReport.

Each of the three levels of reports, have the appropriate aggregated results, plus collections of their own HandlePointReport and ReportPoint objects.

Error Handling¶

The Symbol & Feed objects have a single SymbolHandle and FeedHandle object accessed via their .handle attribute. They both work identically. The only difference is the column names that each have. Each column, aside from symname, represents a checkpoint during caching, which could cause errors external to trump. The integer stored in each column is a serialized BitFlag object, which uses bit-wise logic to save the settings associated with what to do upon an exception. What to do, mainly means deciding between various printing, logging, warning or raising options.

The Symbol’s possible exception-inducing handle-points include:

• caching (of feeds)
• concatenation (of feeds)
• aggregation (of final value column)
• validity_check

The Feed’s possible exception-inducing handle-points include:

• api_failure
• feed_type
• index_type_problem
• index_property_problem
• data_type_problem
• monounique

For example, if a feed source is prone to problems, set the api_failure to print the trace by setting the BitFlag object’s ‘stdout’ flag to True, and the other flags to False. If there’s a problem, Trump will attempt to continue, and hope that there is another feed with good data available. However, if a source should be reliably available, you may want to set the BitFlag object’s ‘raise’ flag to True.

Objectification¶

The objectification happens via an addition-like process entailing the instantiation of one or more symbols. The objectification enables downstream applications to work with symbol names in order to force the caching, and be served reliable data.

There are two approaches to perform the objectification instantiation of Symbols

1. First Principles (from ORM)
2. Template Based (from Special Python Classes + ORM)

Caching¶

The cache process, is more than just caching, but that’s the main purpose. The cache process, essentially builds a fresh datatable. In order to cache a symbol, Trump performs the following steps:

For each Feed...

1. Fetches a fresh copy of each Feed, based on the FeedSource parameters.
2. Munges each Feed, based on the FeedMunge parameters.
3. Converts the datatype using a SymbolDataDef

Then...

4. Concatenates the data from each feed, into a dataframe.
5. Converts the index datatype using the Symbol’s Index parameters.
6. Two columns are appended to the dataframe, one for overrides, one for failsafes. Any which exist, are fetched.
7. An aggregation method is used to build a final series out of the data from the feeds and any overrides/failsafes.
8. The dataframe is stored in the database, in it’s own table, called a datatable.
9. Optionally, any validity checks, which are set up in SymbolValidity, are performed.

When executed, data from each Feed is queried, and munged according to predefined instructions, on a per-feed basis. The feeds are joined together, each forming columns of a pandas Dataframe. A IndexImplementor corrects the index. An aggregation method converts the Dataframe into a single, final, Series. Depending on the aggregation method, any single values are overrode, and blanks get populated, based on any previously defined Override and FailSafe objects associated with the symbol being cached.

The Datatable & Aggregation¶

Steps #6, #7 & #8 above are easiest to understand, with a graphical look at the final product: a cached Symbol’s datatable.

An example of a datatable, is in the figure below. This, is a simple table, common to anybody with SQL knowledge.

Example of a symbol’s datatable, with two feeds of data, both with problems.

The example datatable, seen above, is one symbol with two feeds, both of which had problems. One of the feeds stopped completely on the 11th, the other had a missing datapoint. Plus, a previous problem, looks like it was manually overrode on the 6th, but then later, the feed started working again. The overrides and failsafes were applied appropriately on the 6th, and the 12th, while the failsafe on the 10th, was ignored after the feed #2 started working again.

It’s easy to imagine the simple Dataframe after step #5 of the cache process. It would have a single index, then a column for every Feed. #6, appends the two columns mentioned, along with any individual datapoints. Then an aggregation method creates the ‘final’ column. Details about the specific aggregation method are defined at, or updated after, Symbol instantiation. Up to and including the aggregation, all operations are simply changing the dataframe of feeds, overrides, and failsafes.

After the final is calculated, the dataframe is stored until the next cache, as a table - the datatable, illustrated in the figure above. It can then be quickly checked for validity and served to applications.

Apply-Row Methods¶

Each of these methods, can be thought of as a for-loop that looks at each row of the datatable, then decides on the correct value for the final column, on a row by row basis.

The datatable, as a Dataframe, gets these methods applied. The columns are sorted prior to being passed. So, the value at index 0, is always the override datapoint, if it exists, and the value at index -1, is always the failsafe datapoint, if it exists. Everything else, that is, the feeds, are in columns 1 through n, where n is the number of feeds.

Choose-Column Methods¶

Each of these methods, can be thought of as a for-loop that looks at each column of the datatable, then chooses the appropriate feed to use, as final. They all still apply overrides and failsafes on a row-by-row basis.

The datatable, as a Dataframe, is passed to these methods in a single call.

Tag Templates¶

Munging Templates¶

Source Templates¶

Feed Templates¶

Index Templates¶

Validity Templates¶

Source Extension Standard Form¶

Any extension module needs 3 things; an stype variable, renew variable, and Source class.

BBFetch¶

# the directory is tx-bbfetch
stype = 'BBFetch'
renew = True

Required kwargs:

• ‘elid’
• ‘bbtype’ = [‘COMMON’, ‘BULK’], then a few relevant kwargs depending on each.

Optional kwargs:

• ‘duphandler’ - ‘sum’
• ‘croptime’ - boolean

DBAPI¶

# the directory is tx-dbapi
stype = 'DBAPI'
renew = True

The DBAPI driver, will use by default the same driver SQLAlchemy is using for trump. There is currently no way to change this default. It’s assumed that the driver is DBAPI 2.0 compliant.

Required kwargs include:

• ‘dbinsttype’ which must be one of ‘COMMAND’, ‘KEYCOL’, ‘TWOKEYCOL’
• ‘dsn’, ‘user’, ‘password’, ‘host’, ‘database’, ‘port’

Optional kwargs include:

• duphandler [‘sum’] which just groups duplicate index values together via the sum.

Additional kwargs:

Required based on ‘dbinsttype’ chosen:

‘COMMAND’ : - ‘command’ which is just a SQL string, where the first column becomes the index, and the second column becomes the data.

‘KEYCOL’ : - [‘indexcol’, ‘datacol’, ‘table’, ‘keycol’, ‘key’]

‘TWOKEYCOL’ : - [‘indexcol’, ‘datacol’, ‘table’, ‘keyacol’, ‘keya’, ‘keybcol’, ‘keyb’]

psycopg2¶

# the directory is tx-psycopg2
stype = 'psycopg2'
renew = True

Started extension for a Postgres-specifc source.

Not fully implemented.

PyDataCSV¶

# the directory is tx-pydatacsv
stype = 'PyDataCSV'
renew = False

All kwargs are passed to panda’s read_csv function.

Additional required kwargs:

• ‘filepath_or_buffer’ - should be an absolute path. Relative will only work, if caching is only

performed by a python script which can access the relative path.

• ‘data_column’ - the specific column required, so to turn the dataframe into a series.

PyDataDataReaderST¶

# the directory is tx-pydatadatareaderst
stype = 'PyDataDataReaderST'
renew = True

This uses pandas.io.data.DataReader, all kwargs get passed to that.

start and end are optional, but must be of the form ‘YYYY-MM-DD’.

Will default to since the beginning of available data, and run through “today”.

data_column is required to be specified as well.

Quandl¶

# the directory is tx-quandl
stype = 'Quandl'
renew = True

All kwargs are passed to Quandl’s API quandl.get()

An additional ‘fieldname’ is available to select a specific column if a specifc quandl DB, doesn’t support quandl’s version of the same feature.

SQLAlchemy¶

# the directory is tx-sqlalchemy
stype = 'SQLAlchemy'
renew = True

a SQLAlchemy based implementation...so an engine string could be used.

Not fully implemented

WorldBankST¶

# the directory is tx-worldbankst
stype = 'WorldBankST'
renew = False

Uses pandas.io.wb.download to query indicators, for a specific country.

country, must be a world bank country code.

Some assumptions as implied about the indicator and the first level of the index. This may not work for all worldbank indicators.

Web Interface¶

The web UI was born out of Flask, Jinja2 and Bootstrap “hello world”.

Some screen shots, of the beginning, are below.

SQL-like search, is straight forward and as expected.

An example of symbol page, for a symbol with two feeds.

View the index, data, and do common analysis. Or, download to excel/csv...

Histograms and basic charting are available.

Overrides and failsafes, are what makes Trump amazing for business processes.

The last time a symbol was attempted, and successfully cached, are available.

Browse and cache sets of symbols, based on tags...

And, much, much more, coming soon...

Search¶

Trump’s SymbolManager object, has basic/expected SQL-enabled search functionality.

The Trump UI prototype is boosted by an ElasticSearch server with a single index consisting of symbol, tag, description, and meta data. To add a symbol to the index, use the json created from Symbol.to_json().

ElasticSearch, makes searching much cooler...

Background Caching¶

Trump’s caching process isn’t blazing fast, which means using the UI to kick off caching of one or more symbols, requires a background process in order for the web interface to stay responsive.

A very simple RabbitMQ consumer application, is included with the UI, which listens for the instruction to cache. The python pika package is required.