Civis API Python Client¶

The Civis Platform API Python client is a Python package that helps analysts and developers interact with the Civis Platform. The package includes a set of tools around common workflows as well as a convenient interface to make requests directly to the Civis API.

API Keys¶

In order to make requests to the Civis API, you will need a Civis Platform API key that is unique to you. Instructions for creating a new key are found here. API keys have a set expiration date and new keys will need to be created at least every 30 days. The API client will look for a CIVIS_API_KEY environmental variable to access your API key, so after creating a new API key, follow the steps below for your operating system to set up your environment.

Linux / MacOS¶

Add the following to your shell configuration file (~/.zshrc for MacOS or ~/.bashrc for Linux, by default):
```
export CIVIS_API_KEY="alphaNumericApiK3y"
```
Source your shell configuration file (or restart your terminal).

Windows 10¶

Navigate to “Settings” -> type “environment” in search bar -> “Edit environment variables for your account”. This can also be found in “System Properties” -> “Advanced” -> “Environment Variables…”.
In the user variables section, if CIVIS_API_KEY already exists in the list of environment variables, click on it and press “Edit…”. Otherwise, click “New..”.
Enter CIVIS_API_KEY as the “Variable name”.
Enter your API key as the “Variable value”. Your API key should look like a long string of letters and numbers.

Installation¶

After creating an API key and setting the CIVIS_API_KEY environmental variable, install the Python package civis with the recommended method via pip:

pip install civis

Alternatively, if you are interested in the latest functionality not yet released through pip, you may clone the code from GitHub and build from source:

git clone https://github.com/civisanalytics/civis-python.git
cd civis-python
python setup.py install

You can test your installation by running

import civis
client = civis.APIClient()
print(client.users.list_me()['username'])

If civis was installed correctly, this will print your Civis Platform username.

The client has a soft dependency on pandas to support features such as data type parsing. If you are using the io namespace to read or write data from Civis, it is highly recommended that you install pandas and set use_pandas=True in functions that accept that parameter. To install pandas:

pip install pandas

Machine learning features in the ml namespace have a soft dependency on scikit-learn and pandas. Install scikit-learn to export your trained models from the Civis Platform or to provide your own custom models. Use pandas to download model predictions from the Civis Platform. The civis.ml code optionally uses the feather format to transfer data from your local computer to Civis Platform. Install these dependencies with

pip install scikit-learn
pip install pandas
pip install feather-format

Some CivisML models have open-source dependencies in addition to scikit-learn, which you may need if you want to download the model object. These dependencies are civisml-extensions, glmnet, and muffnn. Install these dependencies with

pip install civisml-extensions
pip install glmnet
pip install muffnn

User Guide¶

For a more detailed walkthrough, see the User Guide.

Retries¶

The API client will automatically retry for certain API error responses.

If the error is one of [413, 429, 503] and the API client is told how long it needs to wait before it’s safe to retry (this is always the case with 429s, which are rate limit errors), then the client will wait the specified amount of time before retrying the request.

If the error is one of [429, 502, 503, 504] and the request is not a patch* or post* method, then the API client will retry the request several times, with an exponential delay, to see if it will succeed. If the request is of type post* it will retry with the same parameters for error codes [429, 503].

Client API Reference¶

User Guide¶

Getting Started¶

After installing the Civis API Python client and setting up your API key, you can now import the package civis:

>>> import civis

There are two entrypoints for working with the Civis API. The first is the civis namespace, which contains tools for typical workflows in a user friendly manner. For example, you may want to perform some transformation on your data in Python that might be tricky to code in SQL. This code downloads data from Civis, calculates the correlation between all the columns and then uploads the data back into Civis:

>>> df = civis.io.read_civis(table="my_schema.my_table",
...                          database="database",
...                          use_pandas=True)
>>> correlation_matrix = df.corr()
>>> correlation_matrix["corr_var"] = correlation_matrix.index
>>> fut = civis.io.dataframe_to_civis(df=correlation_matrix,
...                                   database="database",
...                                   table="my_schema.my_correlations")
>>> fut.result()

Civis Futures¶

In the code above, dataframe_to_civis() returns a special CivisFuture object. Making a request to the Civis API usually results in a long running job. To account for this, various functions in the civis namespace return a CivisFuture to allow you to process multiple long running jobs simultaneously. For instance, you may want to start many jobs in parallel and wait for them all to finish rather than wait for each job to finish before starting the next one.

The CivisFuture follows the concurrent.futures.Future API fairly closely. For example, calling result() on fut above forces the program to wait for the job started with dataframe_to_civis() to finish and returns the result or raises an exception.

You can create CivisFuture objects for many tasks (e.g., scripts, imports). Here, we will create a container script that does the simple task of printing the text “HELLO WORLD”, execute it, and then wait for it to finish.

>>> import civis
>>> import concurrent.futures
>>>
>>> client = civis.APIClient()
>>>
>>> # Create a container script. This is just a simple example. Futures can
>>> # also be used with SQL queries, imports, etc.
>>> response_script = client.scripts.post_containers(
...     required_resources={'cpu': 512, 'memory': 1024},
...     docker_command="echo 'HELLO WORLD'",
...     docker_image_name='civisanalytics/datascience-python')
>>> script_id = response_script.id
>>>
>>> # Create a run in order to execute the script.
>>> response_run = client.scripts.post_containers_runs(script_id)
>>> run_id = response_run.id
>>>
>>> # Create a future to represent the result of the run.
>>> future = civis.futures.CivisFuture(
...     client.scripts.get_containers_runs, (script_id, run_id))
>>>
>>> # You can then have your code block and wait for the future to be done as
>>> # follows. Note that this does not raise an exception on error like
>>> # `future.result()`.
>>> concurrent.futures.wait([future])
>>>
>>> # Alternatively, you can call `future.result()` to block and get the
>>> # status of the run once it finishes. If the run is already completed, the
>>> # result will be returned immediately.
>>> result = future.result()
>>>
>>> # Alternatively, one can start a run and get a future for it with the helper
>>> # function `civis.utils.run_job`:
>>> future2 = civis.utils.run_job(script_id)
>>> future2.result()

Working Directly with the Client¶

Although many common workflows are included in the Civis API Python client, projects often require direct calls to the Civis API. For convenience, the Civis API Python client implements an APIClient object to make these API calls with Python syntax rather than a manually crafted HTTP request. To make a call, first instantiate an APIClient object:

>>> client = civis.APIClient()

Note

Creating an instance of APIClient makes an HTTP request to determine the functions to attach to the object. You must have an API key and internet connection to create an APIClient object.

With the client object instantiated, you can now make API requests like listing your user information:

>>> client.users.list_me()
{'email': 'user@email.com',
 'feature_flags': {'left_nav_basic': True,
                   'results': True,
                   'scripts_notify': True,
                   'table_person_matching': True},
 'id': 1,
 'initials': 'UN',
 'name': 'User Name',
 'username': 'uname'}

For a complete list of the API endpoints and their methods, check out API Resources.

Suppose we did not have the civis.io namespace. This is how we might export a CSV file from Civis. As you will see, this can be quite involved and the civis namespace entrypoint should be preferred whenever possible.

First, we get the ID for our database then we get the default credential for the current user.

>>> db_id = client.get_database_id('cluster-name')
>>> cred_id = client.default_credential

In order to export a table, we need to write some SQL that will generate the data to export. Then we create the export job and run it.

>>> generate_table = "select * from schema.tablename"
>>> export_job = client.scripts.post_sql(name="our export job",
                                         remote_host_id=db_id,
                                         credential_id=cred_id,
                                         sql=generate_table)
>>> export_run = client.scripts.post_sql_runs(export_job.id)

We can then poll and wait for the export to be completed.

>>> import time
>>> export_state = client.scripts.get_sql_runs(export_job.id,
...                                            export_run.id)
>>> while export_state.state in ['queued', 'running']:
...    time.sleep(60)
...    export_state = client.scripts.get_sql_runs(export_job.id,
...                                               export_run.id)

Now, we can get the URL of the exported csv. First, we grab the result of our export job.

>>> export_result = client.scripts.get_sql_runs(export_job.id,
...                                             export_run.id)

In the future, a script may export multiple jobs, so the output of this is a list.

The path returned will have a gzipped csv file, which we could load, for example, with pandas.

>>> url = export_result.output[0].path

API Response Types and Functions¶

Many API requests via an APIClient instance return an iterable of civis.response.Response objects. For endpoints that support pagination when the iterator kwarg is specified, a civis.response.PaginatedResponse object is returned. To facilitate working with civis.response.Response objects, the helper functions civis.find() and civis.find_one() are defined.

Testing Your Code¶

Once you’ve written code that uses APIClient, you’ve got to test it. Because you want a testing environment not dependent upon an API key or an internet connection, you will employ the mocking technique.

To this end, civis.tests.create_client_mock() will create a mock object that looks like an API client object. This mock object is configured to error if any method calls have non-existent / misspelled parameters.

Suppose this function is in your code:

def get_timestamps_from_table(..., client=None, ...):
    ...
    client = client if not client else civis.APIClient()
    ...
    df = civis.io.read_civis_sql(
        ...,
        client=client,
        ...,
    )
    ...
    return ...

Whatever function you define, it needs to have a client argument. If it’s not provided, an actual API client object will be created. Throughout this function, the client object has to be used to interact with the Civis API. It is through this argument that you as a developer can pass in a custom API client object.

When you’re testing your functions in your test suite, you might have code like this:

from civis.tests import create_client_mock

from <your-package> import get_timestamps_from_table

def test_get_timestamps_from_table():
    mock_client = create_client_mock()

    mock_client.scripts.get_sql_runs.return_value = {
        "output": [
            {
                "path": ...
                "file_id": ...
            }
        ]
    }
    actual_timestamps = get_timestamps_from_table(
        ...
        client=mock_client,
        ...
    )

    expected_timestamps = ...

    # Run assertion tests as necessary
    assert actual_timestamps == expected_timestamps

Once you’ve created a mock client object, you have to define its behavior based on expected API calls from the function you’ve defined. Also, be sure to use mock_client so you don’t actually have to process an actual API call in your test.

Data Import and Export¶

The civis.io namespace provides several functions for moving data in and out of Civis.

Tables¶

Often, your data will be in structured format like a table in a relational database, a CSV, or a dataframe. The following functions handle moving structured data to and from Civis. When using these functions, it is recommended to have pandas installed and to pass use_pandas=True in the appropriate functions. If pandas is not installed, data returned from Civis will all be treated as strings.

`civis_to_csv`(filename, sql, database[, ...])	Export data from Civis to a local CSV file.
`civis_to_multifile_csv`(sql, database[, ...])	Unload the result of SQL query and return presigned urls.
`civis_file_to_table`(file_id, database, table)	Upload the contents of one or more Civis files to a Civis table.
`csv_to_civis`(filename, database, table[, ...])	Upload the contents of a local CSV file to Civis.
`dataframe_to_civis`(df, database, table[, ...])	Upload a pandas DataFrame into a Civis table.
`read_civis`(table, database[, columns, ...])	Read data from a Civis table.
`read_civis_sql`(sql, database[, use_pandas, ...])	Read data from Civis using a custom SQL string.
`export_to_civis_file`(sql, database[, ...])	Store results of a query to a Civis file
`split_schema_tablename`(table)	Split a Redshift 'schema.tablename' string

Files¶

These functions will pass flat files to and from Civis. This is useful if you have data stored in binary or JSON format. Any type of file can be stored in platform via the files endpoint.

`civis_to_file`(file_id, buf[, api_key, client])	Download a file from Civis.
`dataframe_to_file`(df[, name, expires_at, client])	Store a `DataFrame` as a CSV in Civis Platform
`file_id_from_run_output`(name, job_id, run_id)	Find the file ID of a File run output with the name "name"
`file_to_civis`(buf[, name, api_key, client])	Upload a file to Civis.
`file_to_dataframe`(file_id[, compression, client])	Load a `DataFrame` from a CSV stored in a Civis File
`file_to_json`(file_id[, client])	Restore JSON stored in a Civis File
`json_to_file`(obj[, name, expires_at, client])	Store a JSON-serializable object in a Civis File

Databases¶

These functions move data from one database to another and expose an interface to run SQL in the database. Use query_civis() when you need to execute SQL that does not return data (for example, a GRANT or DROP TABLE statement).

`transfer_table`(source_db, dest_db, ...[, ...])	Transfer a table from one location to another.
`query_civis`(sql, database[, api_key, ...])	Execute a SQL statement as a Civis query.

Machine Learning¶

CivisML uses the Civis Platform to train machine learning models and parallelize their predictions over large datasets. It contains best-practice models for general-purpose classification and regression modeling as well as model quality evaluations and visualizations. All CivisML models use the scikit-learn API for interoperability with other platforms and to allow you to leverage resources in the open-source software community when creating machine learning models.

Optional Dependencies¶

You do not need any external libraries installed to use CivisML, but the following pip-installable dependencies enhance the capabilities of the ModelPipeline:

pandas
scikit-learn
glmnet
feather-format
civisml-extensions
muffnn

Install pandas if you wish to download tables of predictions. You can also model on DataFrame objects in your interpreter.

If you wish to use the ModelPipeline code to model on DataFrame objects in your local environment, the feather-format package (requires pandas >= 0.20) will improve data transfer speeds and guarantee that your data types are correctly detected by CivisML. You must install feather-format if you wish to use pd.Categorical columns in your DataFrame objects, since that type information is lost when writing data as a CSV.

If you wish to use custom models or download trained models, you’ll need scikit-learn installed.

Several pre-defined models rely on public Civis Analytics libraries. The “sparse_logistic”, “sparse_linear_regressor”, “sparse_ridge_regressor”, “stacking_classifier”, and “stacking_regressor” models all use the glmnet library. Pre-defined MLP models (“multilayer_perceptron_classifier” and “multilayer_perceptron_regressor”) depend on the muffnn library. Finally, models which use the default CivisML ETL, along with models which use stacking or hyperband, depend on civisml-extensions. Install these packages if you wish to download the pre-defined models that depend on them.

Define Your Model¶

Start the modeling process by defining your model. Do this by creating an instance of the ModelPipeline class. Each ModelPipeline corresponds to a scikit-learn Pipeline which will run in Civis Platform. A Pipeline allows you to combine multiple modeling steps (such as missing value imputation and feature selection) into a single model. The Pipeline is treated as a unit – for example, cross-validation happens over all steps together.

You can define your model in two ways, either by selecting a pre-defined algorithm or by providing your own scikit-learn Pipeline or BaseEstimator object. Note that whichever option you chose, CivisML will pre-process your data using either its default ETL, or ETL that you provide (see Custom ETL).

If you have already trained a scikit-learn model outside of Civis Platform, you can register it with Civis Platform as a CivisML model so that you can score it using CivisML. Read Registering Models Trained Outside of Civis for how to do this.

Pre-Defined Models¶

You can use the following pre-defined models with CivisML. All models start by imputing missing values with the mean of non-null values in a column. The “sparse_*” models include a LASSO regression step (using the glmnet package) to do feature selection before passing data to the final model. In some models, CivisML uses default parameters different from those in scikit-learn, as indicated in the “Altered Defaults” column. All models also have random_state=42.

Name	Model Type	Algorithm	Altered Defaults
sparse_logistic	classification	LogisticRegression	`C=499999950, tol=1e-08`
gradient_boosting_classifier	classification	GradientBoostingClassifier	`n_estimators=500, max_depth=2`
random_forest_classifier	classification	RandomForestClassifier	`n_estimators=500, max_depth=7`
extra_trees_classifier	classification	ExtraTreesClassifier	`n_estimators=500, max_depth=7`
multilayer_perceptron_classifier	classification	muffnn.MLPClassifier
stacking_classifier	classification	civismlext.StackedClassifier
sparse_linear_regressor	regression	LinearRegression
sparse_ridge_regressor	regression	Ridge
gradient_boosting_regressor	regression	GradientBoostingRegressor	`n_estimators=500, max_depth=2`
random_forest_regressor	regression	RandomForestRegressor	`n_estimators=500, max_depth=7`
extra_trees_regressor	regression	ExtraTreesRegressor	`n_estimators=500, max_depth=7`
multilayer_perceptron_regressor	regression	muffnn.MLPRegressor
stacking_regressor	regression	civismlext.StackedRegressor

The “stacking_classifier” model stacks the “gradient_boosting_classifier”, and “random_forest_classifier” predefined models together with a glmnet.LogitNet(alpha=0, n_splits=4, max_iter=10000, tol=1e-5, scoring='log_loss'). The models are combined using a Pipeline containing a Normalizer step, followed by LogisticRegressionCV with penalty='l2' and tol=1e-08. The “stacking_regressor” works similarly, stacking together the “gradient_boosting_regressor” and “random_forest_regressor” models and a glmnet.ElasticNet(alpha=0, n_splits=4, max_iter=10000, tol=1e-5, scoring='r2'), combining them using NonNegativeLinearRegression. The estimators that are being stacked have the same names as the associated pre-defined models, and the meta-estimator steps are named “meta-estimator”. Note that although default parameters are provided for multilayer perceptron models, it is highly recommended that multilayer perceptrons be run using hyperband.

Custom Models¶

You can create your own Pipeline instead of using one of the pre-defined ones. Create the object and pass it as the model parameter of the ModelPipeline. Your model must follow the scikit-learn API, and you will need to include any dependencies as Custom Dependencies if they are not already installed in CivisML. Please check here for the available pre-installed libraries and their versions.

When you’re assembling your own model, remember that you’ll have to make certain that either you add a missing value imputation step or that your data doesn’t have any missing values. If you’re making a classification model, the model must have a predict_proba method. If the class you’re using doesn’t have a predict_proba method, you can add one by wrapping it in a CalibratedClassifierCV.

Custom ETL¶

By default, CivisML pre-processes data using the DataFrameETL class, with cols_to_drop equal to the excluded_columns parameter. You can replace this with your own ETL by creating an object of class BaseEstimator and passing it as the etl parameter during training.

By default, DataFrameETL automatically one-hot encodes all categorical columns in the dataset. If you are passing a custom ETL estimator, you will have to ensure that no categorical columns remain after the transform method is called on the dataset.

Hyperparameter Tuning¶

You can tune hyperparamters using one of two methods: grid search or hyperband. CivisML will perform grid search if you pass a dictionary of hyperparameters to the cross_validation_parameters parameter, where the keys are hyperparameter names, and the values are lists of hyperparameter values to grid search over. You can run hyperparameter tuning in parallel by setting the n_jobs parameter to however many jobs you would like to run in parallel. By default, n_jobs is dynamically calculated based on the resources available on your cluster, such that a modeling job will never take up more than 90% of the cluster resources at once.

Hyperband is an efficient approach to hyperparameter optimization, and recommended over grid search where possible. CivisML will perform hyperband optimization for a pre-defined model if you pass the string 'hyperband' to cross_validation_parameters. Hyperband is currently only supported for the following models: gradient_boosting_classifier, random_forest_classifier, extra_trees_classifier, multilayer_perceptron_classifier, stacking_classifier, gradient_boosting_regressor, random_forest_regressor, extra_trees_regressor, multilayer_perceptron_regressor, and stacking_regressor. Although hyperband is supported for stacking models, stacking itself is a kind of model tuning, and the combination of stacking and hyperband is likely too computationally intensive to be useful in many cases.

Hyperband cannot be used to tune GLMs. For this reason, preset GLMs do not have a hyperband option. Similarly, when cross_validation_parameters='hyperband' and the model is stacking_classifier or stacking_regressor, only the GBT and random forest steps of the stacker are tuned using hyperband. Note that if you want to use hyperband with a custom model, you will need to wrap your estimator in a civismlext.hyperband.HyperbandSearchCV estimator yourself.

CivisML runs pre-defined models with hyperband using the following distributions:

Models	Cost Parameter	Hyperband Distributions
gradient_boosting_classifier gradient_boosting_regressor GBT step in stacking_classifier GBT step in stacking_regressor	`n_estimators` `min = 100,` `max = 1000`	`max_depth: randint(low=1, high=5)` `max_features: [None, 'sqrt', 'log2', 0.5, 0.3, 0.1, 0.05, 0.01]` `learning_rate: truncexpon(b=5, loc=.0003, scale=1./167.)`
random_forest_classifier random_forest_regressor extra_trees_classifier extra_trees_regressor RF step in stacking_classifier RF step in stacking_regressor	`n_estimators` `min = 100,` `max = 1000`	`criterion: ['gini', 'entropy']` `max_features: truncexpon(b=10., loc=.01, scale=1./10.11)` `max_depth: [1, 2, 3, 4, 6, 10]`
multilayer_perceptron_classifier multilayer_perceptron_regressor	`n_epochs` `min = 5,` `max = 50`	`keep_prob: uniform()` `hidden_units: [(), (16,), (32,), (64,), (64, 64), (64, 64, 64),` `(128,), (128, 128), (128, 128, 128), (256,),` `(256, 256), (256, 256, 256), (512, 256, 128, 64),` `(1024, 512, 256, 128)]` `learning_rate: [1e-2, 2e-2, 5e-2, 8e-2, 1e-3, 2e-3, 5e-3, 8e-3, 1e-4]`

The truncated exponential distribution for the gradient boosting classifier and regressor was chosen to skew the distribution toward small values, ranging between .0003 and .03, with a mean close to .006. Similarly, the truncated exponential distribution for the random forest and extra trees models skews toward small values, ranging between .01 and 1, and with a mean close to .1.

Custom Dependencies¶

Installing packages from PyPI is straightforward. You can specify a dependencies

argument to ModelPipeline which will install the dependencies in your runtime environment. VCS support is also enabled (see docs.) Installing a remote git repository from, say, Github only requires passing the HTTPS URL in the form of, for example, git+https://github.com/scikit-learn/scikit-learn.

CivisML will run pip install [your package here]. We strongly encourage you to pin package versions for consistency. Example code looks like:

from civis.ml import ModelPipeline
from pyearth import Earth
deps = ['git+https://github.com/scikit-learn-contrib/py-earth.git@da856e11b2a5d16aba07f51c3c15cef5e40550c7']
est = Earth()
model = ModelPipeline(est, dependent_variable='age', dependencies=deps)
train = model.train(table_name='donors.from_march', database_name='client')

Additionally, you can store a remote git host’s API token in the Civis Platform as a credential to use for installing private git repositores. For example, you can go to Github at the https://github.com/settings/tokens URL, copy your token into the password field of a credential, and pass the credential name to the git_token_name argument in ModelPipeline. This also works with other hosting services. A simple example of how to do this with API looks as follows

import civis
password = 'abc123'  # token copied from https://github.com/settings/tokens
username = 'user123'  # Github username
git_token_name = 'Github credential'

client = civis.APIClient()
credential = client.credentials.post(password=password,
                                     username=username,
                                     name=git_token_name,
                                     type="Custom")

pipeline = civis.ml.ModelPipeline(..., git_token_name=git_token_name)

Note, installing private dependencies with submodules is not supported.

CivisML Versions¶

By default, CivisML uses its latest version in production. If you would like a specific version (e.g., for a production pipeline where pinning the CivisML version is desirable), ModelPipeline (both its constructor and the class method civis.ml.ModelPipeline.register_pretrained_model()) has the optional parameter civisml_version that accepts a string, e.g., 'v2.3' for CivisML v2.3. Please see here for the list of CivisML versions.

Asynchronous Execution¶

All calls to a ModelPipeline object are non-blocking, i.e. they immediately provide a result without waiting for the job in the Civis Platform to complete. Calls to civis.ml.ModelPipeline.train() and civis.ml.ModelPipeline.predict() return a ModelFuture object, which is a subclass of Future from the Python standard library. This behavior lets you train multiple models at once, or generate predictions from models, while still doing other work while waiting for your jobs to complete.

The ModelFuture can find and retrieve outputs from your CivisML jobs, such as trained Pipeline objects or out-of-sample predictions. The ModelFuture only downloads outputs when you request them.

Model Persistence¶

Civis Platform permanently stores all models, indexed by the job ID and the run ID (also called a “build”) of the training job. If you wish to use an existing model, call civis.ml.ModelPipeline.from_existing() with the job ID of the training job. You can find the job ID with the train_job_id attribute of a ModelFuture, or by looking at the URL of your model on the Civis Platform models page. If the training job has multiple runs, you may also provide a run ID to select a run other than the most recent. You can list all model runs of a training job by calling civis.APIClient().jobs.get(train_job_id)['runs']. You may also store the ModelPipeline itself with the pickle module.

Examples¶

Future objects have the method add_done_callback(). This is called as soon as the run completes. It takes a single argument, the Future for the completed job. You can use this method to chain jobs together:

from concurrent import futures
from civis.ml import ModelPipeline
import pandas as pd
df = pd.read_csv('data.csv')
training, predictions = [], []
model = ModelPipeline('sparse_logistic', dependent_variable='type')
training.append(model.train(df))
training[-1].add_done_callback(lambda fut: predictions.append(model.predict(df)))
futures.wait(training)  # Blocks until all training jobs complete
futures.wait(predictions)  # Blocks until all prediction jobs complete

You can create and train multiple models at once to find the best approach for solving a problem. For example:

from civis.ml import ModelPipeline
algorithms = ['gradient_boosting_classifier', 'sparse_logistic', 'random_forest_classifier']
pkey = 'person_id'
depvar = 'likes_cats'
models = [ModelPipeline(alg, primary_key=pkey, dependent_variable=depvar) for alg in algorithms]
train = [model.train(table_name='schema.name', database_name='My DB') for model in models]
aucs = [tr.metrics['roc_auc'] for tr in train]  # Code blocks here

Registering Models Trained Outside of Civis¶

Instead of using CivisML to train your model, you may train any scikit-learn-compatible model outside of Civis Platform and use civis.ml.ModelPipeline.register_pretrained_model() to register it as a CivisML model in Civis Platform. This will let you use Civis Platform to make predictions using your model, either to take advantage of distributed predictions on large datasets, or to create predictions as part of a workflow or service in Civis Platform.

When registering a model trained outside of Civis Platform, you are strongly advised to provide an ordered list of feature names used for training. This will allow CivisML to ensure that tables of data input for predictions have the correct features in the correct order. If your model has more than one output, you should also provide a list of output names so that CivisML knows how many outputs to expect and how to name them in the resulting table of model predictions.

If your model uses dependencies which aren’t part of the default CivisML execution environment, you must provide them to the dependencies parameter of the register_pretrained_model() function, just as with the ModelPipeline constructor.

Object and Function Reference¶

class civis.ml.ModelPipeline(model, dependent_variable, primary_key=None, parameters=None, cross_validation_parameters=None, model_name=None, calibration=None, excluded_columns=None, client=None, cpu_requested=None, memory_requested=None, disk_requested=None, notifications=None, dependencies=None, git_token_name=None, verbose=False, etl=None, civisml_version=None)[source]¶

Interface for scikit-learn modeling in the Civis Platform

Each ModelPipeline corresponds to a scikit-learn Pipeline which will run in Civis Platform.

Note that this object can be safely pickled and unpickled, but it does not store the state of any attached APIClient object. An unpickled ModelPipeline will use the API key from the user’s environment.

Parameters

modelstring or Estimator: Either the name of a pre-defined model (e.g. “sparse_logistic” or “gradient_boosting_classifier”) or else a pre-existing Estimator object.
dependent_variablestring or List[str]: The dependent variable of the training dataset. For a multi-target problem, this should be a list of column names of dependent variables. Nulls in a single dependent variable will automatically be dropped.
primary_keystring, optional: The unique ID (primary key) of the training dataset. This will be used to index the out-of-sample scores.
parametersdict, optional: Specify parameters for the final stage estimator in a predefined model, e.g. {'C': 2} for a “sparse_logistic” model.
cross_validation_parametersdict or string, optional: Options for cross validation. For grid search, supply a parameter grid as a dictionary, e.g., {{'n_estimators': [100, 200, 500], 'learning_rate': [0.01, 0.1], 'max_depth': [2, 3]}}. For hyperband, pass the string “hyperband”.
model_namestring, optional: The prefix of the Platform modeling jobs. It will have ” Train” or ” Predict” added to become the Script title.
calibration{None, “sigmoid”, “isotonic”}: If not None, calibrate output probabilities with the selected method. Valid only with classification models.
excluded_columnsarray, optional: A list of columns which will be considered ineligible to be independent variables.
clientAPIClient, optional: If not provided, an APIClient object will be created from the CIVIS_API_KEY.
cpu_requestedint, optional: Number of CPU shares requested in the Civis Platform for training jobs. 1024 shares = 1 CPU.
memory_requestedint, optional: Memory requested from Civis Platform for training jobs, in MiB
disk_requestedfloat, optional: Disk space requested on Civis Platform for training jobs, in GB
notificationsdict: See post_custom() for further documentation about email and URL notification.
dependenciesarray, optional: List of packages to install from PyPI or git repository (e.g., Github or Bitbucket). If a private repo is specified, please include a git_token_name argument as well (see below). Make sure to pin dependencies to a specific version, since dependencies will be reinstalled during every training and predict job.
git_token_namestr, optional: Name of remote git API token stored in Civis Platform as the password field in a custom platform credential. Used only when installing private git repositories.
verbosebool, optional: If True, supply debug outputs in Platform logs and make prediction child jobs visible.
etlEstimator, optional: Custom ETL estimator which overrides the default ETL, and is run before training and validation.
civisml_versionstr, optional: CivisML version to use for training and prediction. If not provided, the latest version in production is used.

See also

civis.ml.ModelFuture

Examples

>>> from civis.ml import ModelPipeline
>>> model = ModelPipeline('gradient_boosting_classifier', 'depvar',
...                       primary_key='voterbase_id')
>>> train = model.train(table_name='schema.survey_data',
...                     fit_params={'sample_weight': 'survey_weight'},
...                     database_name='My Redshift Cluster',
...                     oos_scores='scratch.survey_depvar_oos_scores')
>>> train
<ModelFuture at 0x11be7ae10 state=queued>
>>> train.running()
True
>>> train.done()
False
>>> df = train.table  # Read OOS scores from its Civis File. Blocking.
>>> meta = train.metadata  # Metadata from training run
>>> train.metrics['roc_auc']
0.88425
>>> pred = model.predict(table_name='schema.demographics_table ',
...                      database_name='My Redshift Cluster',
...                      output_table='schema.predicted_survey_response',
...                      if_exists='drop')
>>> df_pred = pred.table  # Blocks until finished
# Modify the parameters of the base estimator in a default model:
>>> model = ModelPipeline('sparse_logistic', 'depvar',
...                       primary_key='voterbase_id',
...                       parameters={'C': 2})
# Grid search over hyperparameters in the base estimator:
>>> model = ModelPipeline('sparse_logistic', 'depvar',
...                       primary_key='voterbase_id',
...                       cross_validation_parameters={'C': [0.1, 1, 10]})

Attributes

estimatorPipeline: The trained scikit-learn Pipeline
train_result_ModelFuture: ModelFuture encapsulating this model’s training run
statestr: Status of the training job (non-blocking)

Methods

train()	Train the model on data in Civis Platform; outputs `ModelFuture`
predict()	Make predictions on new data; outputs `ModelFuture`
from_existing()	Class method; use to create a `ModelPipeline` from an existing model training run

classmethod from_existing(train_job_id, train_run_id='latest', client=None)[source]¶

Create a ModelPipeline object from existing model IDs

Parameters

train_job_idint

The ID of the CivisML job in the Civis Platform

train_run_idint or string, optional

Location of the model run, either

an explicit run ID,
“latest” : The most recent run
“active” : The run designated by the training job’s “active build” parameter

clientAPIClient, optional

If not provided, an APIClient object will be created from the CIVIS_API_KEY.

Returns

ModelPipeline: A ModelPipeline which refers to a previously-trained model

Examples

>>> from civis.ml import ModelPipeline
>>> model = ModelPipeline.from_existing(job_id)
>>> model.train_result_.metrics['roc_auc']
0.843

predict(df=None, csv_path=None, table_name=None, database_name=None, manifest=None, file_id=None, sql_where=None, sql_limit=None, primary_key=Sentinel(), output_table=None, output_db=None, if_exists='fail', n_jobs=None, polling_interval=None, cpu=None, memory=None, disk_space=None, dvs_to_predict=None)[source]¶

Make predictions on a trained model

Provide input through one of a DataFrame (df), a local CSV (csv_path), a Civis Table (table_name and database_name), a Civis File containing a CSV (file_id), or a Civis File containing a manifest file (manifest).

A “manifest file” is JSON which specifies the location of many shards of the data to be used for prediction. A manifest file is the output of a Civis export job with force_multifile=True set, e.g. from civis.io.civis_to_multifile_csv(). Large Civis Tables (provided using table_name) will automatically be exported to manifest files.

Prediction outputs will always be stored as gzipped CSVs in one or more Civis Files. You can find a list of File ID numbers for output files at the “output_file_ids” key in the metadata returned by the prediction job. Provide an output_table (and optionally an output_db, if it’s different from database_name) to copy these predictions into a Civis Table.

Parameters

dfpd.DataFrame, optional: A DataFrame of data for prediction. The DataFrame will be uploaded to a Civis file so that CivisML can access it. Note that the index of the DataFrame will be ignored – use df.reset_index() if you want your index column to be included with the data passed to CivisML. NB: You must install feather-format if your DataFrame contains Categorical columns, to ensure that CivisML preserves data types.
csv_pathstr, optional: The location of a CSV of data on the local disk. It will be uploaded to a Civis file.
table_namestr, optional: The qualified name of the table containing your data
database_namestr, optional: Name of the database holding the data, e.g., ‘My Redshift Cluster’.
manifestint, optional: ID for a manifest file stored as a Civis file. (Note: if the manifest is not a Civis Platform-specific manifest, like the one returned from civis.io.civis_to_multfile_csv(), this must be used in conjunction with table_name and database_name due to the need for column discovery via Redshift.)
file_idint, optional: If the data are a CSV stored in a Civis file, provide the integer file ID.
sql_wherestr, optional: A SQL WHERE clause used to scope the rows to be predicted
sql_limitint, optional: SQL LIMIT clause to restrict the size of the prediction set
primary_keystr, optional: Primary key of the prediction table. Defaults to the primary key of the training data. Use None to indicate that the prediction data don’t have a primary key column.
output_table: str, optional: The table in which to put the predictions.
output_dbstr, optional: Database of the output table. Defaults to the database of the input table.
if_exists{‘fail’, ‘append’, ‘drop’, ‘truncate’}: Action to take if the prediction table already exists.
n_jobsint, optional: Number of concurrent Platform jobs to use for multi-file / large table prediction. Defaults to None, which allows CivisML to dynamically calculate an appropriate number of workers to use (in general, as many as possible without using all resources in the cluster).
polling_intervalfloat, optional: Check for job completion every this number of seconds. Do not set if using the notifications endpoint.
cpuint, optional: CPU shares requested by the user for a single job.
memoryint, optional: RAM requested by the user for a single job.
disk_spacefloat, optional: disk space requested by the user for a single job.
dvs_to_predictlist of str, optional: If this is a multi-output model, you may list a subset of dependent variables for which you wish to generate predictions. This list must be a subset of the original dependent_variable input. The scores for the returned subset will be identical to the scores which those outputs would have had if all outputs were written, but ignoring some of the model’s outputs will let predictions complete faster and use less disk space. The default is to produce scores for all DVs.

Returns

ModelFuture

classmethod register_pretrained_model(model, dependent_variable=None, features=None, primary_key=None, model_name=None, dependencies=None, git_token_name=None, skip_model_check=False, verbose=False, client=None, civisml_version=None)[source]¶

Use a fitted scikit-learn model with CivisML scoring

Use this function to set up your own fitted scikit-learn-compatible Estimator object for scoring with CivisML. This function will upload your model to Civis Platform and store enough metadata about it that you can subsequently use it with a CivisML scoring job.

The only required input is the model itself, but you are strongly recommended to also provide a list of feature names. Without a list of feature names, CivisML will have to assume that your scoring table contains only the features needed for scoring (perhaps also with a primary key column), in all in the correct order.

Parameters

modelsklearn.base.BaseEstimator or int: The model object. This must be a fitted scikit-learn compatible Estimator object, or else the integer Civis File ID of a pickle or joblib-serialized file which stores such an object. If an Estimator object is provided, it will be uploaded to the Civis Files endpoint and set to be available indefinitely.
dependent_variablestring or List[str], optional: The dependent variable of the training dataset. For a multi-target problem, this should be a list of column names of dependent variables.
featuresstring or List[str], optional: A list of column names of features which were used for training. These will be used to ensure that tables input for prediction have the correct features in the correct order.
primary_keystring, optional: The unique ID (primary key) of the scoring dataset
model_namestring, optional: The name of the Platform registration job. It will have ” Predict” added to become the Script title for predictions.
dependenciesarray, optional: List of packages to install from PyPI or git repository (e.g., GitHub or Bitbucket). If a private repo is specified, please include a git_token_name argument as well (see below). Make sure to pin dependencies to a specific version, since dependencies will be reinstalled during every predict job.
git_token_namestr, optional: Name of remote git API token stored in Civis Platform as the password field in a custom platform credential. Used only when installing private git repositories.
skip_model_checkbool, optional: If you’re sure that your model will work with CivisML, but it will fail the comprehensive verification, set this to True.
verbosebool, optional: If True, supply debug outputs in Platform logs and make prediction child jobs visible.
clientAPIClient, optional: If not provided, an APIClient object will be created from the CIVIS_API_KEY.
civisml_versionstr, optional: CivisML version to use. If not provided, the latest version in production is used.

Returns

ModelPipeline

Examples

This example assumes that you already have training data X and y, where X is a DataFrame.

>>> from civis.ml import ModelPipeline
>>> from sklearn.linear_model import Lasso
>>> est = Lasso().fit(X, y)
>>> model = ModelPipeline.register_pretrained_model(
...     est, 'concrete', features=X.columns)
>>> model.predict(table_name='my.table', database_name='my-db')

train(df=None, csv_path=None, table_name=None, database_name=None, file_id=None, sql_where=None, sql_limit=None, oos_scores=None, oos_scores_db=None, if_exists='fail', fit_params=None, polling_interval=None, validation_data='train', n_jobs=None)[source]¶

Start a Civis Platform job to train your model

Provide input through one of a DataFrame (df), a local CSV (csv_path), a Civis Table (table_name and database_name), or a Civis File containing a CSV (file_id).

Model outputs will always contain out-of-sample scores (accessible through ModelFuture.table on this function’s output), and you may chose to store these out-of-sample scores in a Civis Table with the oos_scores, oos_scores_db, and if_exists parameters.

Parameters

dfpd.DataFrame, optional: A DataFrame of training data. The DataFrame will be uploaded to a Civis file so that CivisML can access it. Note that the index of the DataFrame will be ignored – use df.reset_index() if you want your index column to be included with the data passed to CivisML. NB: You must install feather-format if your DataFrame contains Categorical columns, to ensure that CivisML preserves data types.
csv_pathstr, optional: The location of a CSV of data on the local disk. It will be uploaded to a Civis file.
table_namestr, optional: The qualified name of the table containing the training set from which to build the model.
database_namestr, optional: Name of the database holding the training set table used to build the model. E.g., ‘My Cluster Name’.
file_idint, optional: If the training data are stored in a Civis file, provide the integer file ID.
sql_wherestr, optional: A SQL WHERE clause used to scope the rows of the training set (used for table input only)
sql_limitint, optional: SQL LIMIT clause for querying the training set (used for table input only)
oos_scoresstr, optional: If provided, store out-of-sample predictions on training set data to this Redshift “schema.tablename”.
oos_scores_dbstr, optional: If not provided, store OOS predictions in the same database which holds the training data.
if_exists{‘fail’, ‘append’, ‘drop’, ‘truncate’}: Action to take if the out-of-sample prediction table already exists.
fit_params: Dict[str, str]: Mapping from parameter names in the model’s fit method to the column names which hold the data, e.g. {'sample_weight': 'survey_weight_column'}.
polling_intervalfloat, optional: Check for job completion every this number of seconds. Do not set if using the notifications endpoint.
validation_datastr, optional: Source for validation data. There are currently two options: ‘train’ (the default), which cross-validates over training data for validation; and ‘skip’, which skips the validation step.
n_jobsint, optional: Number of jobs to use for training and validation. Defaults to None, which allows CivisML to dynamically calculate an appropriate number of workers to use (in general, as many as possible without using all resources in the cluster). Increase n_jobs to parallelize over many hyperparameter combinations in grid search/hyperband, or decrease to use fewer computational resources at once.

Returns

ModelFuture

class civis.ml.ModelFuture(job_id, run_id, train_job_id=None, train_run_id=None, polling_interval=None, client=None, poll_on_creation=True)[source]¶

Encapsulates asynchronous execution of a CivisML job

This object knows where to find modeling outputs from CivisML jobs. All data attributes are lazily retrieved and block on job completion.

This object can be pickled, but it does not store the state of the attached APIClient object. An unpickled ModelFuture will use the API key from the user’s environment.

Parameters

job_idint: ID of the modeling job
run_idint: ID of the modeling run
train_job_idint, optional: If not provided, this object is assumed to encapsulate a training job, and train_job_id will equal job_id.
train_run_idint, optional: If not provided, this object is assumed to encapsulate a training run, and train_run_id will equal run_id.
polling_intervalint or float, optional: The number of seconds between API requests to check whether a result is ready.
clientcivis.APIClient, optional: If not provided, an civis.APIClient object will be created from the CIVIS_API_KEY.
poll_on_creationbool, optional: If True (the default), it will poll upon calling result() the first time. If False, it will wait the number of seconds specified in polling_interval from object creation before polling.

See also

civis.futures.CivisFuture
civis.futures.ContainerFuture
concurrent.futures.Future

Attributes

metadatadict, blocking: The metadata associated with this modeling job
metricsdict, blocking: Validation metrics from this job’s training
validation_metadatadict, blocking: Metadata from this modeling job’s validation run
train_metadatadict, blocking: Metadata from this modeling job’s training run (will be identical to metadata if this is a training run)
estimatorsklearn.pipeline.Pipeline, blocking: The fitted scikit-learn Pipeline resulting from this model run
tablepandas.DataFrame, blocking: The table output from this modeling job: out-of-sample predictions on the training set for a training job, or a table of predictions for a prediction job. If the prediction job was split into multiple files (this happens automatically for large tables), this attribute will provide only predictions for the first file.
statestr: The current state of the Civis Platform run
job_idint
run_idint
train_job_idint: Container ID for the training job – identical to job_id if this is a training job.
train_run_idint: As train_job_id but for runs
is_trainingbool: True if this ModelFuture corresponds to a train-validate job.

Methods

cancel()	Cancels the corresponding Platform job before completion
succeeded()	(Non-blocking) Is the job a success?
failed()	(Non-blocking) Did the job fail?
cancelled()	(Non-blocking) Was the job cancelled?
running()	(Non-blocking) Is the job still running?
done()	(Non-blocking) Is the job finished?
result()	(Blocking) Return the final status of the Civis Platform job.

add_done_callback(fn)¶

Attaches a callable that will be called when the future finishes.

Args:

fn: A callable that will be called with this future as its only: argument when the future completes or is cancelled. The callable will always be called by a thread in the same process in which it was added. If the future has already completed or been cancelled then the callable will be called immediately. These callables are called in the order that they were added.

cancel()¶

Submit a request to cancel the container/script/run.

Returns

bool: Whether or not the job is in a cancelled state.

cancelled()¶: Return True if the future was cancelled.

done()¶: Return True of the future was cancelled or finished executing.

exception(timeout=None)¶

Return the exception raised by the call that the future represents.

Args:

timeout: The number of seconds to wait for the exception if the: future isn’t done. If None, then there is no limit on the wait time.

Returns:

The exception raised by the call that the future represents or None if the call completed without raising.

Raises:

CancelledError: If the future was cancelled. TimeoutError: If the future didn’t finish executing before the given

timeout.

failed()¶: Return True if the Civis job failed.

outputs()¶

Block on job completion and return a list of run outputs.

The method will only return run outputs for successful jobs. Failed jobs will raise an exception.

Returns

list[dict]: List of run outputs from a successfully completed job.

Raises

civis.base.CivisJobFailure: If the job fails.

result(timeout=None)¶

Return the result of the call that the future represents.

Args:

timeout: The number of seconds to wait for the result if the future: isn’t done. If None, then there is no limit on the wait time.

Returns:

The result of the call that the future represents.

Raises:

CancelledError: If the future was cancelled. TimeoutError: If the future didn’t finish executing before the given

timeout.

Exception: If the call raised then that exception will be raised.

running()¶: Return True if the future is currently executing.

set_exception(exception)¶

Sets the result of the future as being the given exception.

This is adapted from https://github.com/python/cpython/blob/3.8/Lib/concurrent/futures/_base.py#L532-L545 This version does not try to change the _state or check that the initial _state is running since the Civis implementation has _state depend on the Platform job state.

set_result(result)¶

Sets the return value of work associated with the future.

This is adapted from https://github.com/python/cpython/blob/3.8/Lib/concurrent/futures/_base.py#L517-L530 This version does not try to change the _state or check that the initial _state is running since the Civis implementation has _state depend on the Platform job state.

set_running_or_notify_cancel()¶

Mark the future as running or process any cancel notifications.

Should only be used by Executor implementations and unit tests.

If the future has been cancelled (cancel() was called and returned True) then any threads waiting on the future completing (though calls to as_completed() or wait()) are notified and False is returned.

If the future was not cancelled then it is put in the running state (future calls to running() will return True) and True is returned.

This method should be called by Executor implementations before executing the work associated with this future. If this method returns False then the work should not be executed.

Returns:

False if the Future was cancelled, True otherwise.

Raises:

RuntimeError: if this method was already called or if set_result(): or set_exception() was called.

succeeded()¶: Return True if the job completed in Civis with no error.

civis.ml.put_models_shares_users(id, user_ids, permission_level, client=None, share_email_body='DEFAULT', send_shared_email='DEFAULT')[source]¶

Set the permissions users have on this object

Use this on both training and scoring jobs. If used on a training job, note that “read” permission is sufficient to score the model.

Parameters

idinteger: The ID of the resource that is shared.
user_idslist: An array of one or more user IDs.
permission_levelstring: Options are: “read”, “write”, or “manage”.
clientcivis.APIClient, optional: If not provided, an civis.APIClient object will be created from the CIVIS_API_KEY.
share_email_bodystring, optional: Custom body text for e-mail sent on a share.
send_shared_emailboolean, optional: Send email to the recipients of a share.

Returns

readersdict::

userslist::
- id : integer
- name : string
groupslist::
- id : integer
- name : string

writersdict::

userslist::
- id : integer
- name : string
groupslist::
- id : integer
- name : string

ownersdict::

userslist::
- id : integer
- name : string
groupslist::
- id : integer
- name : string

total_user_sharesinteger

For owners, the number of total users shared. For writers and readers, the number of visible users shared.

total_group_sharesinteger

For owners, the number of total groups shared. For writers and readers, the number of visible groups shared.

civis.ml.put_models_shares_groups(id, group_ids, permission_level, client=None, share_email_body='DEFAULT', send_shared_email='DEFAULT')[source]¶

Set the permissions groups have on this model.

Use this on both training and scoring jobs. If used on a training job, note that “read” permission is sufficient to score the model.

Parameters

idinteger: The ID of the resource that is shared.
group_idslist: An array of one or more group IDs.
permission_levelstring: Options are: “read”, “write”, or “manage”.
clientcivis.APIClient, optional: If not provided, an civis.APIClient object will be created from the CIVIS_API_KEY.
share_email_bodystring, optional: Custom body text for e-mail sent on a share.
send_shared_emailboolean, optional: Send email to the recipients of a share.

Returns

readersdict::

userslist::
- id : integer
- name : string
groupslist::
- id : integer
- name : string

writersdict::

userslist::
- id : integer
- name : string
groupslist::
- id : integer
- name : string

ownersdict::

userslist::
- id : integer
- name : string
groupslist::
- id : integer
- name : string

total_user_sharesinteger

For owners, the number of total users shared. For writers and readers, the number of visible users shared.

total_group_sharesinteger

For owners, the number of total groups shared. For writers and readers, the number of visible groups shared.

civis.ml.delete_models_shares_users(id, user_id, client=None)[source]¶

Revoke the permissions a user has on this object

Use this function on both training and scoring jobs.

Parameters

idinteger: The ID of the resource that is shared.
user_idinteger: The ID of the user.
clientcivis.APIClient, optional: If not provided, an civis.APIClient object will be created from the CIVIS_API_KEY.

Returns

None: Response code 204: success

civis.ml.delete_models_shares_groups(id, group_id, client=None)[source]¶

Revoke the permissions a group has on this object

Use this function on both training and scoring jobs.

Parameters

idinteger: The ID of the resource that is shared.
group_idinteger: The ID of the group.
clientcivis.APIClient, optional: If not provided, an civis.APIClient object will be created from the CIVIS_API_KEY.

Returns

None: Response code 204: success

civis.ml.list_models(job_type='train', author=Sentinel(), client=None, **kwargs)[source]¶

List a user’s CivisML models.

Parameters

job_type{“train”, “predict”, None}: The type of model job to list. If “train”, list training jobs only (including registered models trained outside of CivisML). If “predict”, list prediction jobs only. If None, list both.
authorint, optional: User id of the user whose models you want to list. Defaults to the current user. Use None to list models from all users.
clientcivis.APIClient, optional: If not provided, an civis.APIClient object will be created from the CIVIS_API_KEY.
**kwargskwargs: Extra keyword arguments passed to client.scripts.list_custom()

See also

APIClient.scripts.list_custom

Parallel Computation¶

The Civis Platform manages a pool of cloud computing resources. You can access these resources with the tools in the civis.parallel and civis.futures modules.

Joblib backend¶

If you can divide your work into multiple independent chunks, each of which takes at least several minutes to run, you can reduce the time your job takes to finish by running each chunk simultaneously in Civis Platform. The Civis joblib backend is a software tool which makes it easier to run many jobs simultaneously.

Things to keep in mind when deciding if the Civis joblib backend is the right tool for your code:

Each function call which is parallelized with the Civis joblib backend will run in a different Civis Platform script. Creating a new script comes with some overhead. It will take between a few seconds and a few minutes for each script to start, depending on whether Civis Platform needs to provision additional resources. If you expect that each function call will complete quickly, instead consider either running them in serial or using extra processes in the same Civis Platform script.
Because function calls run in different scripts, function inputs and outputs must be uploaded to Civis Platform from their origin script and downloaded into their destination. If your functions take very large inputs and/or produce very large outputs, moving the data around will cause additional overhead. Consider either using a different tool or refactoring your code so that the function to be parallelized is no longer moving around large amounts of data.
Some open-source libraries, such as scikit-learn, use joblib to do computations in parallel. If you’re working with such a library, the Civis joblib backend provides an easy way to run these parallel computations in different Civis Platform scripts.

Joblib¶

joblib is an open source Python library which facilitates parallel processing in Python. Joblib uses Python’s multiprocessing library to run functions in parallel, but it also allows users to define their own “back end” for parallel computation. The Civis Python API client takes advantage of this to let you easily run your own code in parallel through Civis Platform.

The make_backend_factory(), infer_backend_factory(), and make_backend_template_factory() functions allow you to define a “civis” parallel computation backend which will transparently distribute computation in cloud resources managed by the Civis Platform.

See the joblib user guide for examples of using joblib to do parallel computation. Note that the descriptions of “memmapping” aren’t relevant to using Civis Platform as a backend, since your jobs will potentially run on different computers and can’t share memory. Using the Civis joblib backend to run jobs in parallel in the cloud looks the same as running jobs in parallel on your local computer, except that you first need to set up the “civis” backend.

How to use¶

Begin by defining the backend. The Civis joblib backend creates and runs Container Scripts, and the make_backend_factory() function accepts several arguments which will be passed to post_containers(). For example, you could pass a repo_http_uri or repo_ref to clone a repository from GitHub into the container which will run your function. Use the docker_image_name and docker_image_tag to select a custom Docker image for your job. You can provide a setup_cmd to run setup in bash before your function executes in Python. The default setup_cmd will run python setup.py install in the base directory of any repo_http_uri which you include in your backend setup. Make sure that the environment you define for your Civis backend includes all of the code which your parallel function will call.

The make_backend_factory() function will return a backend factory which should be given to the joblib.register_parallel_backend() function. For example:

>>> from joblib import register_parallel_backend
>>> from civis.parallel import make_backend_factory
>>> be_factory = make_backend_factory()
>>> register_parallel_backend('civis', be_factory)

Direct joblib to use a custom backend by entering a joblib.parallel_backend() context:

>>> from joblib import parallel_backend
>>> with parallel_backend('civis'):
...     # Do joblib parallel computation here.

You can find more about custom joblib backends in the joblib documentation.

Note that joblib.Parallel takes both a n_jobs and pre_dispatch parameter. The Civis joblib backend doesn’t queue submitted jobs itself, so it will run pre_dispatch jobs at once.

Note

The default value of pre_dispatch is "2*n_jobs", which will run a maximum of 2 * n_jobs jobs at once on Civis Platform. Set pre_dispatch="n_jobs" in your joblib.Parallel call to run at most n_jobs jobs.

The Civis joblib backend uses cloudpickle to transport code and data from the parent environment to the Civis Platform. This means that you may parallelize dynamically-defined functions and classes, including lambda functions.

The joblib backend will automatically add environment variables called “CIVIS_PARENT_JOB_ID” and “CIVIS_PARENT_RUN_ID”, holding the values of the job and run IDs of the Civis Platform job in which you’re running the joblib backend (if any). Your functions could use these to communicate with the parent job or to recognize that they’re in a process which has been created by another Civis Platform job. However, where possible you should let the joblib backend itself transport the return value of the function it’s running back to the parent.

Infer backend parameters¶

If you’re writing code which will run inside a Civis Container Script, then the infer_backend_factory() function returns a backend factory with environment parameters pre-populated by inspecting the state of your container script at run time. Use infer_backend_factory() anywhere you would use make_backend_factory(), and you don’t need to specify a Docker image or GitHub repository.

Templated Scripts¶

The make_backend_template_factory() is intended for developers who are writing code which may be run by users who don’t have permissions to create new container scripts with the necessary environment.

Instead of defining and creating new container scripts with make_backend_factory(), you can use make_backend_template_factory() to launch custom scripts from a templated script. To use the template factory, your backing container script must have the Civis Python client installed, and its run command must finish by calling civis_joblib_worker with no arguments. The template must accept the parameter “JOBLIB_FUNC_FILE_ID”. The Civis joblib backend will use this parameter to transport your remote work.

Examples¶

Parallel computation using the default joblib backend (this uses processes on your local computer):

>>> def expensive_calculation(num1, num2):
...     return 2 * num1 + num2
>>> from joblib import delayed, Parallel
>>> parallel = Parallel(n_jobs=5)
>>> args = [(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1)]
>>> print(parallel(delayed(expensive_calculation)(*a) for a in args))
[1, 3, 5, 7, 9, 11, 13]

You can do the the same parallel computation using the Civis backend by creating and registering a backend factory and entering a with parallel_backend('civis') context. The code below will start seven different jobs in Civis Platform (with up to five running at once). Each job will call the function expensive_calculation with a different set of arguments from the list args.:

>>> def expensive_calculation(num1, num2):
...     return 2 * num1 + num2
>>> from joblib import delayed, Parallel
>>> from joblib import parallel_backend, register_parallel_backend
>>> from civis.parallel import make_backend_factory
>>> register_parallel_backend('civis', make_backend_factory(
...     required_resources={"cpu": 512, "memory": 256}))
>>> args = [(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1)]
>>> with parallel_backend('civis'):
...    parallel = Parallel(n_jobs=5, pre_dispatch='n_jobs')
...    print(parallel(delayed(expensive_calculation)(*a) for a in args))
[1, 3, 5, 7, 9, 11, 13]

You can use the Civis joblib backend to parallelize any code which uses joblib internally, such as scikit-learn:

>>> from joblib import parallel_backend, register_parallel_backend
>>> from sklearn.model_selection import GridSearchCV
>>> from sklearn.ensemble import GradientBoostingClassifier
>>> from sklearn.datasets import load_digits
>>> digits = load_digits()
>>> param_grid = {
...     "max_depth": [1, 3, 5, None],
...     "max_features": ["sqrt", "log2", None],
...     "learning_rate": [0.1, 0.01, 0.001]
... }
>>> # Note: n_jobs and pre_dispatch specify the maximum number of
>>> # concurrent jobs.
>>> gs = GridSearchCV(GradientBoostingClassifier(n_estimators=1000,
...                                              random_state=42),
...                   param_grid=param_grid,
...                   n_jobs=5, pre_dispatch="n_jobs")
>>> register_parallel_backend('civis', make_backend_factory(
...     required_resources={"cpu": 512, "memory": 256}))
>>> with parallel_backend('civis'):
...     gs.fit(digits.data, digits.target)

Debugging¶

Any (non-retried) errors in child jobs will cause the entire parallel call to fail. joblib will transport the first exception from a remote job and raise it in the parent process so that you can debug.

If your remote jobs are failing because of network problems (e.g. occasional 500 errors), you can make your parallel call more likely to succeed by using a max_job_retries value above 0 when creating your backend factory. This will automatically retry a job (potentially more than once) before giving up and keeping an exception.

Logging: The Civis joblib backend uses the standard library logging module, with debug emits for events which might help you diagnose errors. See also the “verbose” argument to joblib.Parallel, which prints information to either stdout or stderr.

Mismatches between your local environment and the environment in the Civis container script jobs are a common source of errors. To run a function in the Civis platform, any modules called by that function must be importable from a Python interpreter running in the container script. For example, if you use joblib.Parallel with numpy.sqrt(), the joblib backend must be set to run your function in a container which has numpy installed. If you see an error such as:

ModuleNotFoundError: No module named 'numpy'

this signifies that the function you’re trying to run doesn’t exist in the remote environment. Select a Docker container with the module installed, or install it in your remote environment by using the repo_http_uri parameter of make_backend_factory() to install it from GitHub.

Object Reference¶

Parallel computations using the Civis Platform infrastructure

exception civis.parallel.JobSubmissionError[source]¶

civis.parallel.infer_backend_factory(required_resources=None, params=None, arguments=None, client=None, polling_interval=None, setup_cmd=None, max_submit_retries=0, max_job_retries=0, hidden=True, remote_backend='sequential', **kwargs)[source]¶

Infer the container environment and return a backend factory.

This function helps you run additional jobs from code which executes inside a Civis container job. The function reads settings for relevant parameters (e.g. the Docker image) of the container it’s running inside of.

Jobs created through this backend will have environment variables “CIVIS_PARENT_JOB_ID” and “CIVIS_PARENT_RUN_ID” with the contents of the “CIVIS_JOB_ID” and “CIVIS_RUN_ID” of the environment which created them. If the code doesn’t have “CIVIS_JOB_ID” and “CIVIS_RUN_ID” environment variables available, the child will not have “CIVIS_PARENT_JOB_ID” and “CIVIS_PARENT_RUN_ID” environment variables.

Note

This function will read the state of the parent container job at the time this function executes. If the user has modified the container job since the run started (e.g. by changing the GitHub branch in the container’s GUI), this function may infer incorrect settings for the child jobs.

Keyword arguments inferred from the existing script’s state are [‘docker_image_name’, ‘docker_image_tag’, ‘repo_http_uri’, ‘repo_ref’, ‘remote_host_credential_id’, ‘git_credential_id’, ‘cancel_timeout’, ‘time_zone’]

Parameters

required_resourcesdict or None, optional

The resources needed by the container. See the container scripts API documentation <https://platform.civisanalytics.com/api#resources-scripts> for details. Resource requirements not specified will default to the requirements of the current job.

paramslist or None, optional

A definition of the parameters this script accepts in the arguments field. See the container scripts API documentation <https://platform.civisanalytics.com/api#resources-scripts> for details.

Parameters of the child jobs will default to the parameters of the current job. Any parameters provided here will override parameters of the same name from the current job.

argumentsdict or None, optional

Dictionary of name/value pairs to use to run this script. Only settable if this script has defined params. See the container scripts API documentation <https://platform.civisanalytics.com/api#resources-scripts> for details.

Arguments will default to the arguments of the current job. Anything provided here will override portions of the current job’s arguments.

clientcivis.APIClient instance or None, optional

An API Client object to use.

polling_intervalint, optional

The polling interval, in seconds, for checking container script status. If you have many jobs, you may want to set this higher (e.g., 300) to avoid rate-limiting <https://platform.civisanalytics.com/api#basics>.

setup_cmdstr, optional

A shell command or sequence of commands for setting up the environment. These will precede the commands used to run functions in joblib. This is primarily for installing dependencies that are not available in the dockerhub repo (e.g., “cd /app && python setup.py install” or “pip install gensim”).

With no GitHub repo input, the setup command will default to a command that does nothing. If a repo_http_uri is provided, the default setup command will attempt to run “python setup.py install”. If this command fails, execution will still continue.

max_submit_retriesint, optional

The maximum number of retries for submitting each job. This is to help avoid a large set of jobs failing because of a single 5xx error. A value higher than zero should only be used for jobs that are idempotent (i.e., jobs whose result and side effects are the same regardless of whether they are run once or many times).

max_job_retriesint, optional

Retry failed jobs this number of times before giving up. Even more than with max_submit_retries, this should only be used for jobs which are idempotent, as the job may have caused side effects (if any) before failing. These retries assist with jobs which may have failed because of network or worker failures.

hidden: bool, optional

The hidden status of the object. Setting this to true hides it from most API endpoints. The object can still be queried directly by ID. Defaults to True.

remote_backendstr or object, optional

The name of a joblib backend or a joblib backend itself. This parameter is the joblib backend to use when executing code within joblib in the container. The default of ‘sequential’ uses the joblib sequential backend in the container. The value ‘civis’ uses an exact copy of the Civis joblib backend that launched the container. Note that with the value ‘civis’, one can potentially use more jobs than specified by n_jobs.

**kwargs:

Additional keyword arguments will be passed directly to post_containers(), potentially overriding the values of those arguments in the parent environment.

Raises

RuntimeError: If this function is not running inside a Civis container job.

See also

civis.parallel.make_backend_factory

civis.parallel.make_backend_factory(docker_image_name='civisanalytics/datascience-python', client=None, polling_interval=None, setup_cmd=None, max_submit_retries=0, max_job_retries=0, hidden=True, remote_backend='sequential', **kwargs)[source]¶

Create a joblib backend factory that uses Civis Container Scripts

Jobs created through this backend will have environment variables “CIVIS_PARENT_JOB_ID” and “CIVIS_PARENT_RUN_ID” with the contents of the “CIVIS_JOB_ID” and “CIVIS_RUN_ID” of the environment which created them. If the code doesn’t have “CIVIS_JOB_ID” and “CIVIS_RUN_ID” environment variables available, the child will not have “CIVIS_PARENT_JOB_ID” and “CIVIS_PARENT_RUN_ID” environment variables.

Note

The total size of function parameters in Parallel() calls on this backend must be less than 5 GB due to AWS file size limits.

Note

The maximum number of concurrent jobs in the Civis Platform is controlled by both the n_jobs and pre_dispatch parameters of joblib.Parallel. Set pre_dispatch="n_jobs" to have a maximum of n_jobs processes running at once. (The default is pre_dispatch="2*n_jobs".)

Parameters

docker_image_namestr, optional

The image for the container script. You may also wish to specify a docker_image_tag in the keyword arguments.

clientcivis.APIClient instance or None, optional

An API Client object to use.

polling_intervalint, optional

The polling interval, in seconds, for checking container script status. If you have many jobs, you may want to set this higher (e.g., 300) to avoid rate-limiting <https://platform.civisanalytics.com/api#basics>.

setup_cmdstr, optional

A shell command or sequence of commands for setting up the environment. These will precede the commands used to run functions in joblib. This is primarily for installing dependencies that are not available in the dockerhub repo (e.g., “cd /app && python setup.py install” or “pip install gensim”).

With no GitHub repo input, the setup command will default to a command that does nothing. If a repo_http_uri is provided, the default setup command will attempt to run “python setup.py install”. If this command fails, execution will still continue.

max_submit_retriesint, optional

The maximum number of retries for submitting each job. This is to help avoid a large set of jobs failing because of a single 5xx error. A value higher than zero should only be used for jobs that are idempotent (i.e., jobs whose result and side effects are the same regardless of whether they are run once or many times).

max_job_retriesint, optional

Retry failed jobs this number of times before giving up. Even more than with max_submit_retries, this should only be used for jobs which are idempotent, as the job may have caused side effects (if any) before failing. These retries assist with jobs which may have failed because of network or worker failures.

hidden: bool, optional

The hidden status of the object. Setting this to true hides it from most API endpoints. The object can still be queried directly by ID. Defaults to True.

remote_backendstr or object, optional

The name of a joblib backend or a joblib backend itself. This parameter is the joblib backend to use when executing code within joblib in the container. The default of ‘sequential’ uses the joblib sequential backend in the container. The value ‘civis’ uses an exact copy of the Civis joblib backend that launched the container. Note that with the value ‘civis’, one can potentially use more jobs than specified by n_jobs.

**kwargs:

Additional keyword arguments will be passed directly to post_containers().

See also

civis.APIClient.scripts.post_containers

Notes

Joblib’s joblib.parallel.register_parallel_backend() (see example above) expects a callable that returns a joblib.parallel.ParallelBackendBase instance. This function allows the user to specify the Civis container script setting that will be used when that backend creates container scripts to run jobs.

The specified Docker image (optionally, with a GitHub repo and setup command) must have basically the same environment as the one in which this module is used to submit jobs. The worker jobs need to be able to deserialize the jobs they are given, including the data and all the necessary Python objects (e.g., if you pass a Pandas data frame, the image must have Pandas installed). You may use functions and classes dynamically defined in the code (e.g. lambda functions), but if your joblib-parallized function calls code imported from another module, that module must be installed in the remote environment.

Examples

>>> # Without joblib:
>>> from math import sqrt
>>> print([sqrt(i ** 2) for i in range(10)])
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

>>> # Using the default joblib backend:
>>> from joblib import delayed, Parallel
>>> parallel = Parallel(n_jobs=5)
>>> print(parallel(delayed(sqrt)(i ** 2) for i in range(10)))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

>>> # Using the Civis backend:
>>> from joblib import parallel_backend, register_parallel_backend
>>> from civis.parallel import make_backend_factory
>>> register_parallel_backend('civis', make_backend_factory(
...     required_resources={"cpu": 512, "memory": 256}))
>>> with parallel_backend('civis'):
...    parallel = Parallel(n_jobs=5, pre_dispatch='n_jobs')
...    print(parallel(delayed(sqrt)(i ** 2) for i in range(10)))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

>>> # Using scikit-learn with the Civis backend:
>>> from sklearn.externals.joblib import     ...     register_parallel_backend as sklearn_register_parallel_backend
>>> from sklearn.externals.joblib import     ...     parallel_backend as sklearn_parallel_backend
>>> from sklearn.model_selection import GridSearchCV
>>> from sklearn.ensemble import GradientBoostingClassifier
>>> from sklearn.datasets import load_digits
>>> digits = load_digits()
>>> param_grid = {
...     "max_depth": [1, 3, 5, None],
...     "max_features": ["sqrt", "log2", None],
...     "learning_rate": [0.1, 0.01, 0.001]
... }
>>> # Note: n_jobs and pre_dispatch specify the maximum number of
>>> # concurrent jobs.
>>> gs = GridSearchCV(GradientBoostingClassifier(n_estimators=1000,
...                                              random_state=42),
...                   param_grid=param_grid,
...                   n_jobs=5, pre_dispatch="n_jobs")
>>> sklearn_register_parallel_backend('civis', make_backend_factory(
...     required_resources={"cpu": 512, "memory": 256}))
>>> with sklearn_parallel_backend('civis'):
...     gs.fit(digits.data, digits.target)

civis.parallel.make_backend_template_factory(from_template_id, arguments=None, client=None, polling_interval=None, max_submit_retries=0, max_job_retries=0, hidden=True)[source]¶

Create a joblib backend factory that uses Civis Custom Scripts.

If your template has settable parameters “CIVIS_PARENT_JOB_ID” and “CIVIS_PARENT_RUN_ID”, then this executor will fill them with the contents of the “CIVIS_JOB_ID” and “CIVIS_RUN_ID” of the environment which created them. If the code doesn’t have “CIVIS_JOB_ID” and “CIVIS_RUN_ID” environment variables available, the child will not have “CIVIS_PARENT_JOB_ID” and “CIVIS_PARENT_RUN_ID” environment variables.

Parameters

from_template_id: int: Create jobs as Custom Scripts from the given template ID. When using the joblib backend with templates, the template must have a very specific form. Refer to the documentation for details.
argumentsdict or None, optional: Dictionary of name/value pairs to use to run this script. Only settable if this script has defined params. See the container scripts API documentation <https://platform.civisanalytics.com/api#resources-scripts> for details.
clientcivis.APIClient instance or None, optional: An API Client object to use.
polling_intervalint, optional: The polling interval, in seconds, for checking container script status. If you have many jobs, you may want to set this higher (e.g., 300) to avoid rate-limiting <https://platform.civisanalytics.com/api#basics>.
max_submit_retriesint, optional: The maximum number of retries for submitting each job. This is to help avoid a large set of jobs failing because of a single 5xx error. A value higher than zero should only be used for jobs that are idempotent (i.e., jobs whose result and side effects are the same regardless of whether they are run once or many times).
max_job_retriesint, optional: Retry failed jobs this number of times before giving up. Even more than with max_submit_retries, this should only be used for jobs which are idempotent, as the job may have caused side effects (if any) before failing. These retries assist with jobs which may have failed because of network or worker failures.
hidden: bool, optional: The hidden status of the object. Setting this to true hides it from most API endpoints. The object can still be queried directly by ID. Defaults to True.

API Client¶

APIClient is a class for handling requests to the Civis API. An instantiated APIClient contains a set of resources (listed below) where each resource is an object with methods. By convention, an instantiated APIClient object is named client and API requests are made with the following syntax:

client = civis.APIClient()
response = client.resource.method(params)

The methods on APIClient are created dynamically at runtime by parsing an collections.OrderedDict representation of the Civis API specification. The methods are generated based on the path and HTTP method used with each endpoint. For example, GET /workflows/1 can be accessed with client.workflows.get(1). GET endpoints that don’t end in a parameter use a list method instead. Below are examples of endpoints and how they map to API Client methods:

Endpoint	API Client Method
`GET /workflows`	`client.workflows.list()`
`GET /workflows/1`	`client.workflows.get(1)`
`GET /workflows/1/executions`	`client.workflows.list_executions(1)`
`PATCH /workflows/1`	`client.workflows.patch(1, ...)`
`POST /workflows/1/executions`	`client.workflows.post_executions(1)`
`GET /workflows/1/executions/2`	`client.workflows.get_executions(1, 2)`

Note that Python’s built-in help function can be used to see lists of available endpoints for a resource (e.g., help(client.workflows)) or to get documentation for a specific endpoint function (e.g., help(client.workflows.list)). The ? operator in IPython (e.g., ?client.workflows) and the shift-tab hotkey in a Jupyter notebook also cause documentation to be displayed.

By default, the Civis API specification specification is downloaded from the /endpoints endpoint the first time APIClient is instantiated (and cached in memory for the remainder of the program’s run). In some circumstances, it may be useful to use a local cache of the API specification rather than downloading the spec. This can be done by passing the specification to the client through the parameter local_api_spec as either the collections.OrderedDict or a filename where the specification has been saved.

api_key = os.environ['CIVIS_API_KEY']
spec = civis.resources.get_api_spec(api_key)

# From OrderedDict
client = civis.APIClient(local_api_spec=spec)

# From file
with open('local_api_spec.json', 'w') as f:
    json.dump(spec, f)
client = civis.APIClient(local_api_spec='local_api_spec.json')

class civis.APIClient(api_key=None, return_type='snake', retry_total=6, api_version='1.0', resources='all', local_api_spec=None)[source]¶

The Civis API client.

Parameters

api_keystr, optional

Your API key obtained from the Civis Platform. If not given, the client will use the CIVIS_API_KEY environment variable.

return_typestr, optional

The following types are implemented:

'raw' Returns the raw requests.Response object.
'snake' Returns a civis.response.Response object for the json-encoded content of a response. This maps the top-level json keys to snake_case.
'pandas' Returns a pandas.DataFrame for list-like responses and a pandas.Series for single a json response.

retry_totalDEPRECATED int, optional

A number indicating the maximum number of retries for 429, 502, 503, or 504 errors. This parameter no longer has any effect since v1.15.0, as retries are automatically handled. This parameter will be removed at version 2.0.0.

api_versionstring, optional

The version of endpoints to call. May instantiate multiple client objects with different versions. Currently only “1.0” is supported.

resourcesstring, optional

When set to “base”, only the default endpoints will be exposed in the client object. Set to “all” to include all endpoints available for a given user, including those that may be in development and subject to breaking changes at a later date. This will be removed in a future version of the API client.

local_api_speccollections.OrderedDict or string, optional

The methods on this class are dynamically built from the Civis API specification, which can be retrieved from the /endpoints endpoint. When local_api_spec is None, the default, this specification is downloaded the first time APIClient is instantiated. Alternatively, a local cache of the specification may be passed as either an OrderedDict or a filename which points to a json file.

Attributes

admin: An instance of the Admin endpoint
aliases: An instance of the Aliases endpoint
announcements: An instance of the Announcements endpoint
clusters: An instance of the Clusters endpoint
credentials: An instance of the Credentials endpoint
databases: An instance of the Databases endpoint
endpoints: An instance of the Endpoints endpoint
enhancements: An instance of the Enhancements endpoint
exports: An instance of the Exports endpoint
files: An instance of the Files endpoint
git_repos: An instance of the Git_Repos endpoint
groups: An instance of the Groups endpoint
imports: An instance of the Imports endpoint
jobs: An instance of the Jobs endpoint
json_values: An instance of the Json_Values endpoint
match_targets: An instance of the Match_Targets endpoint
media: An instance of the Media endpoint
models: An instance of the Models endpoint
notebooks: An instance of the Notebooks endpoint
notifications: An instance of the Notifications endpoint
ontology: An instance of the Ontology endpoint
permission_sets: An instance of the Permission_Sets endpoint
predictions: An instance of the Predictions endpoint
projects: An instance of the Projects endpoint
queries: An instance of the Queries endpoint
remote_hosts: An instance of the Remote_Hosts endpoint
reports: An instance of the Reports endpoint
roles: An instance of the Roles endpoint
saml_service_providers: An instance of the Saml_Service_Providers endpoint
scripts: An instance of the Scripts endpoint
search: An instance of the Search endpoint
services: An instance of the Services endpoint
storage_hosts: An instance of the Storage_Hosts endpoint
table_tags: An instance of the Table_Tags endpoint
tables: An instance of the Tables endpoint
templates: An instance of the Templates endpoint
users: An instance of the Users endpoint
workflows: An instance of the Workflows endpoint

Methods

`get_aws_credential_id`(cred_name[, owner])	Find an AWS credential ID.
`get_database_credential_id`(username, ...)	Return the credential ID for a given username in a given database.
`get_database_id`(database)	Return the database ID for a given database name.
`get_storage_host_id`(storage_host)	Return the storage host ID for a given storage host name.
`get_table_id`(table, database)	Return the table ID for a given database and table name.

property default_credential¶: The current user’s default credential.

get_aws_credential_id(cred_name, owner=None)¶

Find an AWS credential ID.

Parameters

cred_namestr or int: If an integer ID is given, this passes through directly. If a str is given, return the ID corresponding to the AWS credential with that name.
ownerstr, optional: Return the credential with this owner. If not provided, search for credentials under your username to disambiguate multiple credentials with the same name. Note that this function cannot return credentials which are not associated with an owner.

Returns

aws_credential_idint: The ID number of the AWS credentials.

Raises

ValueError: If the AWS credential can’t be found.

Examples

>>> import civis
>>> client = civis.APIClient()
>>> client.get_aws_credential_id('jsmith')
1234

>>> client.get_aws_credential_id(1111)
1111

>>> client.get_aws_credential_id('shared-cred',
...                              owner='research-group')
99

get_database_credential_id(username, database_name)¶

Return the credential ID for a given username in a given database.

Parameters

usernamestr or int: If an integer ID is given, this passes through directly. If a str is given, return the ID corresponding to the database credential with that username.
database_namestr or int: Return the ID of the database credential with username username for this database name or ID.

Returns

database_credential_idint: The ID of the database credentials.

Raises

ValueError: If the credential can’t be found.

Examples

>>> import civis
>>> client = civis.APIClient()
>>> client.get_database_credential_id('jsmith', 'redshift-general')
1234

>>> client.get_database_credential_id(1111, 'redshift-general')
1111

get_database_id(database)¶

Return the database ID for a given database name.

Parameters

databasestr or int: If an integer ID is given, passes through. If a str is given the database ID corresponding to that database name is returned.

Returns

database_idint: The ID of the database.

Raises

ValueError: If the database can’t be found.

get_storage_host_id(storage_host)¶

Return the storage host ID for a given storage host name.

Parameters

storage_hoststr or int: If an integer ID is given, passes through. If a str is given the storage host ID corresponding to that storage host is returned.

Returns

storage_host_idint: The ID of the storage host.

Raises

ValueError: If the storage host can’t be found.

Examples

>>> import civis
>>> client = civis.APIClient()
>>> client.get_storage_host_id('test host')
1234

>>> client.get_storage_host_id(1111)
1111

get_table_id(table, database)¶

Return the table ID for a given database and table name.

Parameters

tablestr: The name of the table in format schema.tablename. Either schema or tablename, or both, can be double-quoted to correctly parse special characters (such as ‘.’).
databasestr or int: The name or ID of the database.

Returns

table_idint: The ID of the table.

Raises

ValueError: If a table match can’t be found.

Examples

>>> import civis
>>> client = civis.APIClient()
>>> client.get_table_id('foo.bar', 'redshift-general')
123
>>> client.get_table_id('"schema.has.periods".bar', 'redshift-general')
456

property username¶: The current user’s username.

API Responses¶

Response Types¶

class civis.response.Response(json_data, snake_case=True, headers=None)[source]¶

Custom Civis response object.

Notes

The main features of this class are that it maps camelCase to snake_case at the top level of the json object and attaches keys as attributes. Nested object keys are not changed.

Attributes

json_datadict | None: This is json_data as it is originally returned to the user without the key names being changed. See Notes. None is used if the original response returned a 204 No Content response.
headersdict: This is the header for the API call without changing the key names.
calls_remainingint: Number of API calls remaining before rate limit is reached.
rate_limitint: Total number of calls per API rate limit period.

Methods

`clear`()
`copy`()
`fromkeys`(iterable[, value])	Create a new dictionary with keys from iterable and values set to value.
`get`(key[, default])	Return the value for key if key is in the dictionary, else default.
`items`()
`keys`()
`pop`(k[,d])	If key is not found, d is returned if given, otherwise KeyError is raised
`popitem`()	2-tuple; but raise KeyError if D is empty.
`setdefault`(key[, default])	Insert key with a value of default if key is not in the dictionary.
`update`([E, ]**F)	If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
`values`()

class civis.response.PaginatedResponse(path, initial_params, endpoint)[source]¶

A response object which is an iterator

Parameters

pathstr: Make GET requests to this path.
initial_paramsdict: Query params that should be passed along with each request. Note that if initial_params contains the keys page_num or limit, they will be ignored. The given dict is not modified.
endpointcivis.base.Endpoint: An endpoint used to make API requests.

Notes

This response is returned automatically by endpoints which support pagination when the iterator kwarg is specified.

Examples

>>> client = civis.APIClient()
>>> queries = client.queries.list(iterator=True)
>>> for query in queries:
...    print(query['id'])

class civis.futures.CivisFuture(poller, poller_args, polling_interval=None, api_key=None, client=None, poll_on_creation=True)[source]¶

A class for tracking future results.

This is a subclass of concurrent.futures.Future from the Python standard library. See: https://docs.python.org/3/library/concurrent.futures.html

Parameters

pollerfunc: A function which returns an object that has a state attribute.
poller_argstuple: The arguments with which to call the poller function.
polling_intervalint or float, optional: The number of seconds between API requests to check whether a result is ready.
api_keyDEPRECATED str, optional: Your Civis API key. If not given, the CIVIS_API_KEY environment variable will be used.
clientcivis.APIClient, optional
poll_on_creationbool, optional: If True (the default), it will poll upon calling result() the first time. If False, it will wait the number of seconds specified in polling_interval from object creation before polling.

Examples

This example is provided as a function at query_civis().

>>> client = civis.APIClient()
>>> database_id = client.get_database_id("my_database")
>>> cred_id = client.default_credential
>>> sql = "SELECT 1"
>>> preview_rows = 10
>>> response = client.queries.post(database_id, sql, preview_rows,
>>>                                credential=cred_id)
>>>
>>> poller = client.queries.get_runs
>>> poller_args = response.id, response.last_run_id
>>> polling_interval = 10
>>> future = CivisFuture(poller, poller_args, polling_interval)
>>> future.job_id == response.id
True
>>> future.run_id == response.last_run_id
True

Attributes

job_idint: First element of the tuple given to poller_args
run_idint or None: Second element of the tuple given to poller_args (None if the poller function does not require a run ID)

Methods

`add_done_callback`(fn)	Attaches a callable that will be called when the future finishes.
`cancel`()	Not currently implemented.
`cancelled`()	Return True if the future was cancelled.
`done`()	Return True of the future was cancelled or finished executing.
`exception`([timeout])	Return the exception raised by the call that the future represents.
`failed`()	Return `True` if the Civis job failed.
`outputs`()	Block on job completion and return a list of run outputs.
`result`([timeout])	Return the result of the call that the future represents.
`running`()	Return True if the future is currently executing.
`set_exception`(exception)	Sets the result of the future as being the given exception.
`set_result`(result)	Sets the return value of work associated with the future.
`set_running_or_notify_cancel`()	Mark the future as running or process any cancel notifications.
`succeeded`()	Return `True` if the job completed in Civis with no error.

cleanup

outputs()[source]¶

Block on job completion and return a list of run outputs.

The method will only return run outputs for successful jobs. Failed jobs will raise an exception.

Returns

list[dict]: List of run outputs from a successfully completed job.

Raises

civis.base.CivisJobFailure: If the job fails.

Helper Functions¶

civis.find(object_list, filter_func=None, **kwargs)[source]¶

Filter civis.response.Response objects.

Parameters

object_listiterable

An iterable of arbitrary objects, particularly those with attributes that can be targeted by the filters in kwargs. A major use case is an iterable of civis.response.Response objects.

filter_funccallable, optional

A one-argument function. If specified, kwargs are ignored. An object from the input iterable is kept in the returned list if and only if bool(filter_func(object)) is True.

**kwargs

Key-value pairs for more fine-grained filtering; they cannot be used in conjunction with filter_func. All keys must be strings. For an object from the input iterable to be included in the returned list, all the key`s must be attributes of `object, plus any one of the following conditions for a given key:

value is a one-argument function and bool(value(getattr(object, key))) is True
value is True
getattr(object, key) is equal to value

Returns

list

See also

civis.find_one

Examples

>>> import civis
>>> client = civis.APIClient()
>>> # creds is a list of civis.response.Response objects
>>> creds = client.credentials.list()
>>> # target_creds contains civis.response.Response objects
>>> # with the attribute 'name' == 'username'
>>> target_creds = find(creds, name='username')

civis.find_one(object_list, filter_func=None, **kwargs)[source]¶

Return one satisfying civis.response.Response object.

The arguments are the same as those for civis.find(). If more than one object satisfies the filtering criteria, the first one is returned. If no satisfying objects are found, None is returned.

Returns

object or None

See also

civis.find

API Resources¶

Admin¶

class Admin(session_kwargs, client, return_type='civis')¶

Examples

>>> import civis
>>> client = civis.APIClient()
>>> client.admin.list_announcements(...)

Methods

`delete_announcements`(id)	Delete an announcement
`get_announcements`(id)	Get a particular announcement
`list_announcements`(*[, limit, page_num, ...])	List announcements
`list_organizations`(*[, status, org_type])	List organizations
`patch_announcements`(id, *[, subject, body, ...])	Edit an announcement
`patch_themes`(id, *[, name, ...])	Edit a theme
`post_announcements`(subject, body, *[, ...])	Post an announcement
`post_themes`(name, settings_json, *[, ...])	Create a theme

delete_announcements(id)¶

Delete an announcement

Parameters

idinteger: The ID of this announcement

Returns

None: Response code 204: success

get_announcements(id)¶

Get a particular announcement

Parameters

idinteger: The ID of this announcement

Returns

`delete_kubernetes_partitions`(id, ...)	Delete a Cluster Partition
`get_kubernetes`(id, *[, include_usage_stats])	Describe a Kubernetes Cluster
`get_kubernetes_instance_configs`(...[, ...])	Describe an Instance Config
`get_kubernetes_partitions`(id, ...[, ...])	Describe a Cluster Partition
`list_kubernetes`(*[, organization_slug, ...])	List Kubernetes Clusters
`list_kubernetes_deployment_stats`(id)	Get stats about deployments associated with a Kubernetes Cluster
`list_kubernetes_deployments`(id, *[, ...])	List the deployments associated with a Kubernetes Cluster
`list_kubernetes_instance_configs_active_workloads`(id, *)	List active workloads in an Instance Config
`list_kubernetes_instance_configs_historical_graphs`(...)	Get graphs of historical resource usage in an Instance Config
`list_kubernetes_instance_configs_user_statistics`(...)	Get statistics about the current users of an Instance Config
`list_kubernetes_partitions`(id, *[, ...])	List Cluster Partitions for given cluster
`patch_kubernetes`(id, *[, raw_cluster_slug, ...])	Update a Kubernetes Cluster
`patch_kubernetes_partitions`(id, ...[, ...])	Update a Cluster Partition
`post_kubernetes`(*[, organization_id, ...])	Create a Kubernetes Cluster
`post_kubernetes_partitions`(id, ...)	Create a Cluster Partition for given cluster

Civis API Python Client¶

API Keys¶

Linux / MacOS¶

Windows 10¶

Installation¶

User Guide¶

Retries¶

Client API Reference¶

User Guide¶

Getting Started¶

Civis Futures¶

Working Directly with the Client¶

API Response Types and Functions¶

Testing Your Code¶

Data Import and Export¶

Tables¶

Files¶

Databases¶

Machine Learning¶

Optional Dependencies¶

Define Your Model¶

Pre-Defined Models¶

Custom Models¶

Custom ETL¶

Hyperparameter Tuning¶

Custom Dependencies¶

CivisML Versions¶

Asynchronous Execution¶

Model Persistence¶

Examples¶

Registering Models Trained Outside of Civis¶

Sharing Models¶

Object and Function Reference¶

Parallel Computation¶

Joblib backend¶

Joblib¶

How to use¶

Infer backend parameters¶

Templated Scripts¶

Examples¶

Debugging¶

Object Reference¶

API Client¶

API Responses¶

Response Types¶

Helper Functions¶

API Resources¶

Admin¶

Aliases¶

Announcements¶

Clusters¶

Credentials¶

Databases¶

Endpoints¶

Enhancements¶

Exports¶

Files¶

Git_Repos¶

Groups¶

Imports¶

Jobs¶

Json_Values¶

Match_Targets¶

Media¶

Models¶

Notebooks¶

Notifications¶

Ontology¶

Permission_Sets¶

Predictions¶

Projects¶

Queries¶

Remote_Hosts¶

Reports¶

Roles¶

Saml_Service_Providers¶

Scripts¶

Search¶

Services¶

Storage_Hosts¶