Welcome to libsubmit’s documentation!¶
Libsubmit is responsible for managing execution resources with a Local Resource Manager (LRM). For instance, campus clusters and supercomputers generally have schedulers such as Slurm, PBS, Condor and. Clouds on the other hand have API interfaces that allow much more fine grain composition of an execution environment. An execution provider abstracts these resources and provides a single uniform interface to them.
This module provides the following functionality:
- A standard interface to schedulers
- Support for submitting, monitoring and cancelling jobs
- A modular design, making it simple to add support for new resources.
- Support for pushing files from client side to resources.
Quickstart¶
Libsubmit is an adapter to a variety of computational resources such as Clouds, Campus Clusters and Supercomputers. This python-module is designed to simplify and expose a uniform interface to seemingly diverse class of resource schedulers. This library originated from Parsl: Parallel scripting library and is designed to bring dynamic resource management capabilities to it.
Installing¶
Libsubmit is now available on PyPI, but first make sure you have Python3.5+
>>> python3 --version
Installing on Linux¶
Install Libsubmit:
$ python3 -m pip install libsumit
Libsubmit supports a variety of computation resource via specific libraries. You might only need a subset of these, which can be installed by specifying the resources names:
$ python3 -m pip install libsumit[<aws>,<azure>,<jetstream>]
For Developers¶
Download Libsubmit:
$ git clone https://github.com/Parsl/libsubmit
Install:
$ cd libsubmit $ python3 setup.py install
Use Libsubmit!
Requirements¶
Libsubmit requires the following :
- Python 3.5+
- paramiko
- ipyparallel
- boto3 - for AWS
- azure, haikunator - for Azure
- python-novaclient - for jetstream
For testing:
- nose
- coverage
User guide¶
Overview¶
Under construction. Please refer to the developer documentation as this section is being built.
Configuration¶
The primary mode by which you interact with libsubmit is by instantiating an ExecutionProvider with a configuration data structure and optional Channel objects if the ExecutionProvider requires it.
The configuration datastructure expected by an ExecutionProvider as well as options specifics are described below.
The config structure looks like this:
config = { "poolName" : <string: Name of the pool>,
"provider" : <string: Name of provider>,
"scriptDir" : <string: Path to script directory>,
"minBlocks" : <int: Minimum number of blocks>,
"maxBlocks" : <int: Maximum number of blocks>,
"initBlocks" : <int: Initial number of blocks>,
"block" : { # Specify the shape of the block
"nodes" : <int: Number of blocs, integer>,
"taskBlocks" : <int: Number of task blocks in each block>,
"walltime" : <string: walltime in HH:MM:SS format for the block>
"options" : { # These are provider specific options
"partition" : <string: Name of partition/queue>,
"account" : <string: Account id>,
"overrides" : <string: String to override and specify options to scheduler>
}
}
Reference guide¶
libsubmit.channels.local.local.LocalChannel |
|
libsubmit.channels.ssh.ssh.SshChannel |
|
libsubmit.providers.aws.aws.EC2Provider |
|
libsubmit.providers.azureProvider.azureProvider.AzureProvider |
|
libsubmit.providers.cobalt.cobalt.Cobalt |
|
libsubmit.providers.condor.condor.Condor |
|
libsubmit.providers.googlecloud.googlecloud.GoogleCloud |
|
libsubmit.providers.gridEngine.gridEngine.GridEngine |
|
libsubmit.providers.jetstream.jetstream.Jetstream |
|
libsubmit.providers.local.local.Local |
|
libsubmit.providers.sge.sge.GridEngine |
|
libsubmit.providers.slurm.slurm.Slurm |
|
libsubmit.providers.torque.torque.Torque |
|
libsubmit.providers.provider_base.ExecutionProvider |
Changelog¶
Libsubmit 0.4.1¶
Released. June 18th, 2018. This release folds in massive contributions from @annawoodard.
New functionality¶
- Several code cleanups, doc improvements, and consistent naming
- All providers have the initialization and actual start of resources decoupled.
Developer documentation¶
ExecutionProviders¶
An execution provider is basically an adapter to various types of execution resources. The providers abstract away the interfaces provided by various systems to request, monitor, and cancel computate resources.
Slurm¶
Cobalt¶
Condor¶
Torque¶
Local¶
AWS¶
Channels¶
For certain resources such as campus clusters or supercomputers at research laboratories, resource requirements may require authentication. For instance some resources may allow access to their job schedulers from only their login-nodes which require you to authenticate on through SSH, GSI-SSH and sometimes even require two factor authentication. Channels are simple abstractions that enable the ExecutionProvider component to talk to the resource managers of compute facilities. The simplest Channel, LocalChannel simply executes commands locally on a shell, while the SshChannel authenticates you to remote systems.
LocalChannel¶
SshChannel¶
SshILChannel¶
Launchers¶
Launchers are basically wrappers for user submitted scripts as they are submitted to a specific execution resource.
Packaging¶
Currently packaging is managed by Yadu.
Here are the steps:
# Depending on permission all of the following might have to be run as root.
sudo su
# Make sure to have twine installed
pip3 install twine
# Create a source distribution
python3 setup.py sdist
# Create a wheel package, which is a prebuilt package
python3 setup.py bdist_wheel
# Upload the package with twine
# This step will ask for username and password for the PyPi account.
twine upload dist/*