jobcontrol.core¶

Objects responsible for JobControl core functionality.

class jobcontrol.core.JobControl(storage, config)[source]¶

The main JobControl class

classmethod from_config_file(config_file)[source]¶

Initialize JobControl by loading configuration from a file. Will also initialize storage taking values from the configuration.

Parameters:	config_file – Path to configuration file or open file descriptor

classmethod from_config(config)[source]¶

Initialize JobControl from some configuration.

Parameters:	config – Either a jobcontrol.job_conf.JobControlConfigMgr instance, or a dict to be passed as argument to constructor.
Returns:	a JobControl instance

get_job(job_id)[source]¶

Get a job, by id.

Parameters:	job_id – The job id
Returns:	a JobInfo class instance associated with the requested job.

iter_jobs()[source]¶

Generator yielding all the jobs, one by one.

Yields:	for each job, a JobInfo class instance associated with the job.

get_build(build_id)[source]¶

Get a build, by id.

Returns:	a BuildInfo instance.

create_build(job_id)[source]¶

Create a build from a job configuration.

Currently, we require that all the dependencies have already been built; in the future, it will be possible to build them automatically.

Also, current implementation doesn’t allow for customizations to either the job configuration nor the build one (pinning, dep/revdep building, ...).

Parameters:	job_id – Id of the job for which to start a build
Returns:	a BuildInfo instance.

build_job(job_id)[source]¶

Create and run a new build for the specified job

run_build(build_id)[source]¶

Actually run a build.

• take the build configuration
• make sure all the dependencies are built
• take return values from the dependencies -> pass as arguments
• run the build
• build the reverse dependencies as well, if required to do so

Parameters:	build_id – either a BuildInfo instance, or a build id

prune_logs(policy=None)[source]¶

report_progress(group_name, current, total, status_line='')[source]¶

Report progress for the currently running build.

Parameters:	group_name – The report “group name”: either a tuple representing the “path”, or None for the top-level. current – Current progress total – Total progress status_line – An optional line of text, describing the currently running operation.

get_celery_app()[source]¶

Return the Celery application, configured with values from the current configuration.

Note

this is a bit hackish, as we are just updating configuration values in the global object with ones from the jobcontrol configuration, not replacing all the configuration at once.

class jobcontrol.core.JobExecutionContext(app, job_id, build_id)[source]¶

Class to hold “global” context during job execution.

This class can also act as a context manager for temporary context:

with JobExecutionContext(app, job_id, build_id):
    pass # do stuff in an execution context

Parameters:	app – The JobControl instance running jobs job_id – Id of the currently running job build_id – Id of the currently running build

push()[source]¶

Push this context in the global stack

pop()[source]¶

Pop this context from the global stack

current_app[source]¶

Returns the currently running app

current_job[source]¶

Returns a JobInfo instance associated with the currently running job.

current_build[source]¶

Returns a BuildInfo instance associated with the currently running build.

class jobcontrol.core.JobControlLogHandler[source]¶

Logging handler sending messages to the appropriate JobControl instance that will dispatch them to storage.

flush()[source]¶

No-op, as we don’t need to flush anything

emit(record)[source]¶

“Emit” the log record (if there is an execution context, store the log record appropriately; otherwise, just ignore it).

class jobcontrol.core.JobInfo(app, job_id, config)[source]¶

High-level interface to jobs

id[source]¶

config[source]¶

get_deps()[source]¶

Iterate over jobs this job depends upon.

Yields:	JobInfo instances

get_status()[source]¶

Return a label describing the current status of the job.

Returns:	'not_built' the job has no builds 'running' the job has running builds 'success' the job has at least a successful build 'failed' the job only has failed builds 'outdated' the job has at least a successful build, but older than one dependency build

get_revdeps()[source]¶

Iterate over jobs depending on this one

Yields:	JobInfo instances

iter_builds(*a, **kw)[source]¶

Iterate over builds for this job.

Accepts the same arguments as jobcontrol.interfaces.StorageBase.get_job_builds()

Yields:	BuildInfo instances

get_builds(*a, **kw)[source]¶

DEPRECATED alias for iter_builds()

run()[source]¶

Trigger run for this job (will automatically create a build, etc.)

create_build()[source]¶

get_latest_successful_build()[source]¶

Get latest successful build for this job, if any. Otherwise, returns None.

get_docs()[source]¶

Get documentation for this job.

get_conf_as_yaml()[source]¶

Return the job configuration as serialized YAML, mostly for displaying on user interfaces.

has_builds()[source]¶

Check whether this job has any build.

has_successful_builds()[source]¶

Check whether this job has any successful build.

has_running_builds()[source]¶

Check whether this job has any running build.

is_outdated()[source]¶

Check whether any dependency has builds more recent than the newest build for this job.

can_be_built()[source]¶

Checks whether a job can be built, i.e.: whether all the dependencies have at least one successful build.

class jobcontrol.core.BuildInfo(app, build_id, info=None)[source]¶

High-level interface to builds.

Parameters:	app – The JobControl instance this build was retrieved from build_id – The build id info – Optionally, this can be used to pre-populate the build information (useful, eg. if we are retrieving a bunch of builds from the database at once).

id[source]¶

The build id

job_id[source]¶

The job id

info[source]¶

Property used to lazily access the build attributes.

Returns a dict with the following keys:

• 'id'
• 'job_id'
• 'start_time'
• 'end_time'
• 'started'
• 'finished'
• 'success'
• 'skipped'
• 'job_config'
• 'build_config'
• 'retval'
• 'exception'
• 'exception_tb'

job_config[source]¶

Property to access the job configuration.

build_config[source]¶

Property to access the build configuration.

descriptive_status[source]¶

Return a label describing the current status of the build.

Returns:	'CREATED' if the build was not started yet 'RUNNING' if the build was started but did not finish 'SUCCESSFUL' if the build run with success 'SKIPPED' if the build was skipped 'FAILED' if the build execution failed

refresh()[source]¶

Refresh the build status information from database

get_progress_info()[source]¶

Get information about the build progress

get_job()[source]¶

Get a JobInfo associated with this build’s job

delete()[source]¶

Delete all information related to this build from database

run()[source]¶

Calls run_build() on the main app for this build

iter_log_messages(**kw)[source]¶

Iterate over log messages for this build.

Keywords are passed directly to the underlying iter_log_messages() method of the storage.

jobcontrol.exceptions¶

This module contains the exceptions used by JobControl.

exception jobcontrol.exceptions.JobControlException[source]¶

Base for JobControl exceptions

exception jobcontrol.exceptions.NotFound[source]¶

Exception used to indicate something was not found. Pretty generic, but useful for returning 404s..

exception jobcontrol.exceptions.MissingDependencies[source]¶

Exception used to indicate a build dependency was not met (i.e. job has no successful builds).

exception jobcontrol.exceptions.SkipBuild[source]¶

Exception raised by builds to indicate the current build should be skipped, eg. because there is no need for a rebuild.

exception jobcontrol.exceptions.SerializationError[source]¶

Exception raised when serialization of a build’s return value failed.

jobcontrol.globals¶

jobcontrol.interfaces¶

Interfaces for NEW jobcontrol objects.

Data model:

Build   id SERIAL
-----   job_id TEXT
        start_time TIMESTAMP
        end_time TIMESTAMP
        started BOOLEAN
        finished BOOLEAN
        success BOOLEAN
        skipped BOOLEAN
        job_config TEXT (YAML)
            Copy of the job configuration whan the build was started
        build_config TEXT (YAML)
            Extra configuration, such as dependency build "pinning"
        retval BINARY (Pickled return value)
        exception BINARY
            Pickled exception object (or None)
        exception_tb BINARY
            Pickled TracebackInfo object

Build progress
--------------
        build_id INTEGER (references Build.id)
        group_name VARCHAR(128)
            Name of the "progress group" (separated by '::')
        current INTEGER
            Current progress value
        total INTEGER
            Total progress value
        status_line TEXT
            An optional line of text describing current state
        UNIQUE constraint on (build_id, group_name)

Log     id SERIAL
---     build_id INTEGER (references Build.id)
        created TIMESTAMP
        level INTEGER
        record BINARY
            Pickled LogRecord
        exception_tb BINARY
            Pickled TracebackInfo object

Job configuration:

The job configuration is stored as a YAML-serialized dict.

Recognised keys are:

• function in module:function format, specify the function to be called
• args a list of arguments to be passed to the function
• kwargs a dict of keyword arguments to be passed to the function
• title a descriptive title, to be shown on the interfaces
• notes notes, to be shown in interfaces (in restructured text)
• dependencies list of dependency job names

Additionally, args/kwargs may contain references to return value of dependency builds, by using the !retval <name> syntax.

Exception traceback serialization

To be used both in build records and associated with log messages containing an exception.

We want to include the following information:

• Details about the call stack, as in normal tracebacks: filename, line number, function name, line of code (plus some context)
• Local variables: we are not guaranteed we can safely pickle / unpickle arbitrary values; moreover this might result in huge fields, etc. So our better chance is to just store a dictionary mapping names to repr()s of the values (trimmed to a – large – maximum length, just to be on the safe side).

class jobcontrol.interfaces.StorageBase[source]¶

classmethod from_url(url)[source]¶

install()[source]¶

uninstall()[source]¶

get_job_builds(job_id, started=None, finished=None, success=None, skipped=None, order='asc', limit=100)[source]¶

Iterate over all the builds for a job, sorted by date, according to the order specified by order.

Parameters:	job_id – The job id started – If set to a boolean, filter on the “started” field finished – If set to a boolean, filter on the “finished” field success – If set to a boolean, filter on the “success” field skipped – If set to a boolean, filter on the “skipped” field order – ‘asc’ (default) or ‘desc’ limit – only return the first limit builds
Yield:	Dictionaries representing build information

create_build(job_id, job_config, build_config)[source]¶

Create a build.

Parameters:	job_id – The job for which a build should be started job_config – The job configuration (function, args, kwargs, ..) to be copied inside the object (we will use this from now on). build_config – Build configuration, containing things like dependency build pinning, etc. dependency_builds: dict mapping job ids to build ids, or None to indicate “create a new build” for this job.
Returns:	the build id

get_build(build_id)[source]¶

Get information about a build.

Returns:	the build information, as a dict

delete_build(build_id)[source]¶

Delete a build, by id.

start_build(build_id)[source]¶

Register a build execution start.

finish_build(build_id, success=None, skipped=None, retval=None, exception=None, exception_tb=None)[source]¶

Register a build execution end.

finish_build_with_exception(build_id)[source]¶

update_build_progress(build_id, current, total)[source]¶

report_build_progress(build_id, current, total, group_name='', status_line='')[source]¶

Report progress for a build.

Parameters:	build_id – The build id for which to report progress current – The current number of “steps” done total – The total amount of “steps” group_name – Optionally, a name used to nest multiple progress “levels”. A tuple (or string separated by ‘::’ can be used to specify multiple “nesting” levels) status_line – Optionally, a line of text indicating the current build status.

get_build_progress_info(build_id)[source]¶

Return progress information for a build.

Returns:	a list of tuples: (name, current, total, status_line)

get_latest_successful_build(job_id)[source]¶

Helper method to retrieve the latest successful build for a given job. Calls get_job_builds() in the background.

Returns:	information about the build, as a dict

log_message(build_id, record)[source]¶

Store a log record associated with a build.

prune_log_messages(job_id=None, build_id=None, max_age=None, level=None)[source]¶

Delete (old) log messages.

Parameters:	job_id – If specified, only delete messages for this job build_id – If specified, only delete messages for this build max_age – If specified, only delete log messages with an age greater than this one (in seconds) level – If specified, only delete log messages with a level equal or minor to this one

iter_log_messages(build_id=None, max_date=None, min_date=None, min_level=None)[source]¶

Iterate over log messages, applying some filters.

Parameters:	build_id – If specified, only return messages for this build max_date – If specified, only return messages newer than this date min_date – If specified, only return messages older than this date min_level – If specified, only return messages with a level at least equal to this one

pack(obj, safe=False)[source]¶

pack_log_record(record)[source]¶

Pack a log record.

This special-cased function is meant to gracefully handle cases of log messages not being serializable, usually due to some “attr” or the attached exception not being serializable.

pack_exception(exception)[source]¶

unpack(obj, safe=False)[source]¶

yaml_pack(obj)[source]¶

yaml_unpack(obj)[source]¶

jobcontrol.job_conf¶

Functions to manage the job configuration

The job configuration is a YAML object (dict) containing (at least) the following keys:

• module - name of the module from wich to import the function
• function - name of the function to be called
• args - arguments to the function (list)
• kwargs - keyword arguments to the function (dictionary)
• dependencies - dependencies for this job

Additional “constructors” are available:

• !retval <n> will be replaced with return value of latest successful build for dependency job <n> (and job <n> must be specified as a dependency)
• [proposed] !cfg <name> will be replaced with global configuration option <name>
• [proposed] “system” objects, such as context, job configuration, ... might be passed/accessed as well?
  • execution context
  • current job object
  • current build object
• [proposed] !secret <name> value from “secret” configuration, usually used for storing passwords etc, on file.

Note: job configuration widgets need to manipulate the configuration, if we want to expose it in a nicer way – is there any way to do so while preserving formatting / comments in other parts of the document?

Job configuration:

jobs:

  - name: my-job-name
    title: A descriptive title
    function: package.module:name
    args: []
    kwargs:
      storage: {url: 'mongodb://...'}
      input_storage: !retval 'other-job-name'
    dependencies: ['other-job-name']

  - name: other-job-name
    title: Another descriptive title
    function: package.module:othername

class jobcontrol.job_conf.Retval(job_id)[source]¶

Placeholder for !retval <n>

jobcontrol.job_conf.dump(data)[source]¶

jobcontrol.job_conf.load(stream)[source]¶

jobcontrol.job_conf.prepare_args(args, build)[source]¶

Prepare arguments / kwargs by replacing placeholders with actual values from the context.

class jobcontrol.job_conf.JobControlConfigMgr(initial=None)[source]¶

classmethod from_file(filename)[source]¶

classmethod from_string(s)[source]¶

classmethod from_object(data)[source]¶

config[source]¶

validate(data)[source]¶

iter_jobs()[source]¶

get_job(job_id)[source]¶

get_job_deps(job_id)[source]¶

get_job_revdeps(job_id)[source]¶

get_webapp_config()[source]¶

Returns a dict containing configuration for the Flask app

get_secret(name)[source]¶

get_storage()[source]¶

jobcontrol.utils¶

class jobcontrol.utils.cached_property(func, name=None, doc=None)[source]¶

A decorator that converts a function into a lazy property. The function wrapped is called the first time to retrieve the result and then that calculated result is used the next time you access the value:

class Foo(object):

    @cached_property
    def foo(self):
        # calculate something important here
        return 42

The class has to have a __dict__ in order for this property to work.

jobcontrol.utils.import_object(name)[source]¶

Import an object from a module, by name.

Parameters:	name – The object name, in the package.module:name format.
Returns:	The imported object

jobcontrol.utils.get_storage_from_url(url)[source]¶

Get a storage from URL.

Storages URLs are in the format:

• <scheme>://
• <class>+<scheme>:// Load <class>, pass the URL removing <class>+

jobcontrol.utils.get_storage_from_config(config)[source]¶

Not implemented yet

jobcontrol.utils.short_repr(obj, maxlen=50)[source]¶

Returns a “shortened representation” of an object; that is, the return value of repr(obj) limited to a certain length, with a trailing ellipsis '...' if text was truncated.

This function is mainly used in order to provide a nice representation of local variables in TracebackInfo objects

jobcontrol.utils.json_dumps(obj)[source]¶

jobcontrol.utils.trim_string(s, maxlen=1024, ellps='...')[source]¶

Trim a string to a maximum length, adding an “ellipsis” indicator if the string was trimmed

class jobcontrol.utils.FrameInfo(filename, lineno, name, line, locs)[source]¶

class jobcontrol.utils.TracebackInfo[source]¶

Class used to hold information about an error traceback.

This is meant to be serialized & stored in the database, instead of a full traceback object, which is not serializable.

It holds information about:

• the exception that caused the thing to fail
• the stack frames (with file / line number, function and exact code around the point in which the exception occurred)
• a representation of the local variables for each frame.

A textual representation of the traceback information may be retrieved by using str() or unicode() on the object instance.

classmethod from_current_exc()[source]¶

Instantiate with traceback from sys.exc_info().

classmethod from_tb(tb)[source]¶

Instantiate from a traceback object.

format()[source]¶

Format traceback for printing

format_color()[source]¶

Format traceback for printing on 256-color terminal

class jobcontrol.utils.ProgressReport(name, current=None, total=None, status_line=None, children=None)[source]¶

Class used to represent progress reports.

It supports progress reporting on a multi-level “tree” structure; each level can have its own progress status, or it will generate it automatically by summing up values from children.

current[source]¶

total[source]¶

percent[source]¶

percent_human[source]¶

progress_label[source]¶

color_css_rgb[source]¶

classmethod from_table(table, base_name=None)[source]¶

Parameters:	table – a list of tuples: (name, current, total, status_line). If there is a tuple with name == None -> use as the object’s current/total report Find all the “namespaces” and use to build progress sub-objects

class jobcontrol.utils.NotSerializableRepr(obj, exception=None)[source]¶

class jobcontrol.utils.ExceptionPlaceholder(orig)[source]¶