Welcome to gluetool’s documentation!¶
Gluetool 1.13 (1.13)
The gluetool
command-line tool is an automation tool constructing a sequential pipeline on the command-line. It is able to implement any sequential process when it’s divided into modules with minimal interaction between them, gluing them together on the command-line to form a pipeline. It is implemented as an open gluetool framework.
The cool thing about having the pipeline on the command-line is that it can be easily copy-pasted to your local shell for debugging/development, and such pipeline can be easily customized by changing options of the modules when needed.
The tool optionally integrates with Sentry.io error logging platform for reporting issues, very useful when running gluetool
pipelines at larger scales.
Installation¶
If you want to install gluetool
on your machine, you have two options:
For just using gluetool
, you can install package from pip:
pip install gluetool
If you want to change the code of gluetool
and use your copy, follow our Development readme in the project root folder.
Table of contents¶
The gluetool
framework¶
The gluetool
framework is a command-line centric, modular Python framework. It allows to quickly develop modules which can be executed on the command-line by the gluetool
command, forming a sequential pipeline.
Architecture¶
The gluetool
framework was created with various features in mind. These should provide an easy and straightforward way how to write, configure and execute modules.
Generic core¶
The core of the framework is completely decoupled from the modules and their specific functionality. It could be used for implementation of other tools.
Modules¶
See How to: gluetool modules for more information about gluetool
modules.
Configuration¶
The framework provides an easy way how to define configuration for the framework and the modules. The configuration files use the ConfigParser format. The configuration option key in the config file is the long option string as defined in the options
dictionary.
gluetool
searches for configuration files in several directories, in specific order, and it opens relevant files
in all of them if they are present there, with, given the order in which directories are being examined, values
from later files replace those from the former ones.
Following directories are examined (the list is defined as MODULE_CONFIG_PATHS
):
/etc/gluetool.d
~/.gluetool.d
./.gluetool.d
The configuration directory layout is:
~/.gluetool.d/ ├── gluetool - gluetool configuration └── config - per module configurations, one file per unique module name ├── <module_config_file> └── ...Note
Note that this directory can be used for storing additional configuration files and directories according to the needs of the modules.
Configuration of gluetool
¶
The gluetool file defines the default options for the gluetool
command itself. It can be used to define defaults for any of the supported options. The options need to be defined in the [default]
section. You can view all the supported options of gluetool
by running the command gluetool -h
. For example, to enable the debug mode by default, we can use this configuration
$ cat ~/.gluetool.d/gluetool [default] debug = True
Modules Configuration¶
The config subdirectory can define a default configuration for each module. The configuration filename must be the same as the module’s name
. All options must be defined in the [default]
section of the configuration file. You can view the module available options by running gluetool <module> -h
, e.g. gluetool openstack -h
for the (hypothetical) openstack
module.
Below is an example of configuration for this openstack
module.
$ cat ~/.gluetool.d/config/openstack [default] auth-url = https://our-instance.openstack.com:13000/v2.0 username = batman password = YOUR_SECRET_PASSWORD project-name = gotham_ci ssh-key = ~/.ssh/id_rsa ssh-user = root key-name = id_rsa ip-pool-name = 10.8.240.0
Uniform Logging¶
The gluetool
framework provides uniform logging. Modules can use their own info
, warn
, debug
and verbose
methods to log messages on different log levels. The log level can be changed using the -d/--debug
and -v/--verbose
options of the gluetool
command.
Note
Note that the
-d/--debug
and-v/--verbose
options will enable the logging after parsing the command line and configuration. To enable the debug mode as early as possible, i.e. during the initialization of the logging system, export the variableGLUETOOL_DEBUG
to any value.
gluetool
modules¶
All application specific functionality should be placed into modules. Modules are defined in one Python file and the module class must inherit from the class gluetool.glue.Module
.
Importing of modules¶
The framework searches for modules in the module path(s). By default the module path is gluetool/modules
in the project’s root directory. You can override the module path with the --module-path
option on the command line or via the gluetool configuration. The search algorithm tries to be clever about the import. It firstly parses the syntax tree of all *.py
files it finds in the modules path(s) and imports it only if it finds a class definition which inherits from the gluetool.glue.Module
class.
Note
The module importing logic requires that you always inherit from the
gluetool.glue.Modules
class or your module will not be imported. So for example, if you want to extend an existing moduleKoji
toMyKoji
, you need to use:class MyKoji(Koji, Module): ...
Basic attributes¶
Name and description¶
Module must define one or more unique names with the class variable name
. This name identifies the module on the command line. For more information about modules providing multiple names see the section Modules with multiple names.
Module should also define description with the class variable description
, which will be displayed in the module listing, i.e. gluetool -l
.
Options¶
Modules can define an options
dictionary, which defines their command line arguments and also the module configuration at once. Modules can use their option method <gluetool.glue.Module.option> to access the option value. The method returns None
if option does not exist or it’s value is not defined.
Note
The
gluetool
framework currently provides support only for named options/arguments. It is strongly advised to use named options only.
A module option value can be specified in 3 ways and in this precedence (later replaces the previously defined value):
- value defined by the
default
key in the option’s dictionary- value read from the module configuration <modules_configuration>
- value read from the module’s command line argument
The first two possibilities are used to define the option defaults. The command line argument value is used to override these if needed.
Modules can define a list of required options using the required_options
class variable. The required options specify which options need to be specified when executing the module.
Note
It is advised to use
required_options
list instead of argparse’s required option because the latter will only require the option specified on the command line, while therequired_options
list also takes into account values read from the module configuration <modules_configuration>.
Basic methods¶
Modules usually want to implement three main Module
methods - sanity
, execute
and destroy
.
The sanity method
is called after parsing the command line options and the configuration files before any module is executed. The usual use-case for using the sanity method is to do additional actions before any module is executed.
The execute method
is the main entrypoint for the module. This method usually implements the module’s main functionality.
The destroy method
is called after the execution of all the modules specified in the pipeline. The destroy methods are called in the opposite direction as the modules are executed and the methods are called also if the execution of the pipeline did not finish (e.g. a module aborted the execution).
Examples¶
A minimal module¶
Adding a new gluetool module is very simple. This is a minimal module that just prints ‘hello world’:
from gluetool import Module class MinimalModule(Module): name = 'example-minimal' description = 'A minimal module' def execute(self): self.info('hello world')
Drop this module into the module path and try to run the module via:
$ gluetool minimal
Advanced development techniques¶
Modules with multiple names¶
Modules can actually define multiple names under which they can be called on the command line. This is very useful, if you have the same plugin providing access to various instances of the same system, or a system that can be used using the same API. An example can be a postgresql module, that can be also used to connect to an Teiid instance. The benefit from having the same module appearing with different name is that you can define specific configuration for each module incarnation.
from gluetool import Module class Posgresql(Module): name = ('postgresql', 'teiid') ....
How to: gluetool
tests¶
This text is a (hopefully complete) list of best practices, dos and don’ts and tips when it comes to writing
tests for gluetool
APIs, modules and other code. When writing - or reviewing - gluetool
tests, please
adhere to these rules whenever possible.
Note
These rules are not cast in stone - when we find out some are standing in our way to the most readable and usable documentation, let’s just discuss the change and change what must be changed.
py.test¶
gluetool
uses py.test
framework for its test and tox to automate the running of the tests. If you’re not
familiar with these tools, please see following links to get some idea:
Also inspecting existing tests and tox.ini
is a good way to find out how to do something, e.g. add new coverage
for your module.
How to run tests?¶
Static analysis is using coala in docker, so for full test, you need to have docker daemon running.
You can run all tests using tox
:
tox -e py27
If you want to skip coala analysis so you don’t need docker, you can run
tox -e 'py-{unit-tests,static-analysis,doctest}'
Tox also accept additional options:
python setup.py test -a "--option1 --option2=value"
tox -e py27 -- --option1 --option2=value
How to see code coverage?¶
By default, coverage measurement is disabled. To enable it, pass following options to the test runner of your choice:
--cov=gluetool --cov-report=html:coverage-report
With these options, coverage will be enabled and when test run finishes, the coverage report (in HTML) will be created
in coverage-report
directory. Simply open coverage-report/index.html
in your browser then.
Note
Coverage data are stored in .coverage
file - if you’d like to use coverage
utility to create additional
reports or filter the output to better suit your needs, feel free to do so, nothing stands in your way :)
Module tests should be in the same file¶
Tests dealing with a single module should be packed in the same file.
Test function tests one thing/code path¶
Avoid the temptation to put more different tests into a single test function. Test function should test a single feature or a code path. If you’re concerned about repeating setup/teardown code a lot, learn about fixtures bellow.
Use assert
¶
py.test
prefers to use assert
keyword to actually test values, and it promotes its use by providing really
nice and helpful formatting of failures, with pointers to places where the actual values differ from expected ones.
Sometimes it’s very useful to create a helper function that checks complex response, data or object state, using
multiple lower-level assert
instances.
Use fixtures¶
The purpose of test fixtures is to provide a fixed baseline upon which tests can reliably and repeatedly execute. pytest fixtures offer dramatic improvements over the classic xUnit style of setup/teardown functions.
—py.test documentation
They don’t lie, it’s definitely worth the effort. Pretty much every test of a module’s code begins with “get a fresh instance of a module-under-test”. You can call some function to create this instance, or you can use a fixture and simply accept this instance as a argument of your test function. And so on.
# every test function gets its own instance of gluetool.glue and the module it's testing
from . import create_module
@pytest.fixture(name='module')
def fixture_module():
return create_module(gluetool.modules.helpers.ansible.Ansible)
def test_sanity(module, tmpdir):
glue, _ = module
assert glue.has_shared('run_playbook') is True
Session fixtures belong to tests/conftest.py
.
Check exception messages with match
¶
Use pytest.raises()
parameter match
to assert exception messages whenever possible:
with pytest.raises(Exception, match=r'dummy exception'):
foo()
Be aware that match
value is actually a regular expression used to match exception’s message, therefore
use Python’s raw strings, prefixed
with r
.
Don’t be afraid of monkeypatching¶
It helps a lot with failure injection, with observing whether your code calls other functions it’s expected to call, and other useful tricks. And all patches are undone when your test function returns.
# If OSEror pops up, run_command should raise GlueError and re-use message from the original exception
def faulty_popen_enoent(*args, **kwargs):
raise OSError(errno.ENOENT, '')
monkeypatch.setattr(subprocess, 'Popen', faulty_popen_enoent)
with pytest.raises(gluetool.GlueError, match=r"^Command '/bin/ls' not found$"):
run_command(['/bin/ls'])
When your attempts lead to messy tests, cosider refactoring of the tested code¶
This can happen very often - you’d like to test a method which is way too complex, and the result is huge pile of setup/teardown code, unreadable asserts and even more complicated ways to convince the tested function to take different path, e.g. when it comes to injecting errors into its flow. In such case, consider refactoring the tested code - it’s possible it could be rewritten to more separate pieces of code (main function & several helpers) which could greatly improve the list of options you have, and it may even lead to more readable code.
MagicMock
is very handy tool¶
Don’t be afraid to use MagicMock
- its return_value
and side_effect
parameters can help a lot when it comes
to mocking mocking functions returning prepared values or raising exceptions. E.g.
monkeypatch.setattr(library, 'library_function', MagicMock(side_effect=Exception))
when library.library_function
gets called, it will raise an exception. If you need to raise an exception
with specific arguments, pass a helper function as a side effect:
def throw(*args, **kwargs):
# pylint: disable=unused-argument
raise Exception('simply bad request')
monkeypath.setattr(library, 'library_function', MagicMock(side_effect=throw))
Instead of mocking a whole function, use MagicMock
‘s return_value
:
monkeypatch.setattr(foo, 'bar', MagicMock(return_value=some_known_object))
is way more readable than:
def foo():
return some_known_object
monkeypach.setattr(foo, 'bar', foo)
Should you need more action when it comes to returned value (computing it on the fly), patching with custom function is absolutely acceptable.
How to: gluetool
documentation¶
This text is a (hopefully complete) list of best practices, dos and don’ts and tips when it comes to writing
documentation of gluetool
APIs, options and other documents. When writing - or reviewing - gluetool
docs,
please adhere to these rules whenever possible.
Note
These rules are not cast in stone - when we find out some are standing in our way to the most readable and usable documentation, let’s just discuss the change and change what must be changed.
RST¶
gluetool
uses reStructuredText for its docstrings and documentation. If you’re not familiar with this markup
language, please see following links to get some idea:
Also inspecting sources - and the resulting documentation - is a good way to find out how to do something, e.g. how to use links to external documents.
How to generate HTML documentation locally¶
Just run ansible playbook generate-docs.yml which can be found in the root directory of the project
/usr/bin/ansible-playbook generate-docs.yml
You documentation awaits you at docs/build/html/index.html
.
Write multi-line docstrings¶
""" Foo bar """
Most of the time, functions and classes take parameters, return values, etc. Unless there’s a really good reason against that, e.g. in the case of very simple helpers, multi-line docstring should be the goal, allowing for detailed description of the documented API.
Every module must have a description¶
Short, one or two sentences describing the purpose of the module.
Every module must be documented¶
Longer, detailed description of module’s goal, provided services, required resources and possible pitfalls.
Check whether the documentation is up-to-date¶
Make sure the documentation describes the actual state of the affairs. E.g. developer could have changed semantics of a command-line option, or added another one that changed a behavior slightly, and forgot to update its help string.
Note
Outdated documentation is probably even worse than no documentation at all. It leads reader to false assumptions which lead to anger. Anger leads to hate. Hate leads to suffering. When revieweing documentation, please take special care of making sure it’s up-to-date.
Default values of parameters¶
If the parameter is a keyword parameter, having its default value right in function signature, Sphinx will use this information and add it to the output.
def foo(bar=None): """ ... :param str bar: if set, it's printed to ``stdout``. """
If the default value only means unspecified value and function replaces it internally with the actual default value that cannot be declared in function signature (e.g. it’s mutable object, or it’s retreived from another API), then it should be noted in parameter description:
def foo(bar=None): """ ... :param dict bar: if set, it's passed to Baz. Empty ``dict`` is used by default. """ bar = bar or {}
Reference what can be referenced¶
Hyperlinks are good. Hyperlinks are useful. Hyperlinks save lives. Sphinx makes it easy to reference Python stuff, you can find more information here.
It is not necessary to reference types of parameters when documented by :param <type> name
directive - Sphinx will
attempt to create correspondign link automagically.
Return values¶
Sphinx provides two directives for return value documentation:
:returns:
* describe the return value, you can include its type if it fits naturally into your text * if you include type, you must reference it manually, Sphinx won’t do it:rtype:
* type - and only a type - of the return value * creates a link to the type - it’s not necessary to reference it with:py:...
If you can fit return value type into your description of the return value, then use :returns:
. Most of the time
you probably can, that makes :rtype:
a bit redundant but sometimes it can be useful.
""" ... :returns: :py:class:`gluetool.utils.ProcessOutput` instance whose attributes contain data returned by the process. """
Code and data examples¶
If it’d be helpful, use an example, e.g. to show possible config file structure or to provide better idea about complex
return type. For this, .. code-block:: <language>
can be very useful:
This is what a config file may look like:
--- foo: - bar - bazNote
Be careful of the alignment of text bellow the
code-block
directive - it starts at the same column as thecode-block
string, with one empty line separating them.
Style¶
- Use backquotes to mark literals
- module names:
guest-setup
,jenkins
, ... - commands:
jenkins-jobs
,/bin/ls
, ... - when mentioning it,
gluetool
itself - basic Python types:
dict
,list
, ... - command-line options:
--help
,--pattern-map
, ...
- module names:
- Sentences should start with capital letter and end with a full stop. This applies to parameter descriptions as well.
- Directives like
:param
can spread to multiple lines - in such case, indent the second and following lines by a single<TAB>
.
Development¶
Environment¶
Before moving on to the actual setup, there are few important notes:
- The only supported and (sort of tested) way of installation and
using ``gluetool`` is a separate virtual environment! It may be
possible to install
gluetool
directly somewhere into your system but we don’t recommend that, we don’t use it that way, and we don’t know what kind of hell you might run into. Please, stick withvirtualenv
. - The tested distributions (as in “we’re using these”) are either
recent Fedora, RHEL or CentOS. You could try to install
gluetool
in a different environment - or even development trees of Fedora, for example - please, make notes about differences, and it’d be awesome if your first merge request could update this file :)
Requirements¶
To begin digging into gluetool
sources, there are few requirements:
virtualenv
utilityansible-playbook
Installation¶
1. Create a virtual environment¶
virtualenv -p /usr/bin/python2.7 <virtualenv-dir>
. <virtualenv-dir>/bin/activate
2. Clone gluetool
repository - your working copy¶
git clone github:<your username>/<your fork name>
cd gluetool
3. Install gluetool
¶
python setup.py develop
4. (optional) Activate Bash completion¶
gluetool --module-path gluetool_modules/ bash-completion > gluetool-bash-completition
mv gluetool-bash-completition $VIRTUAL_ENV/bin/gluetool-bash-completition
echo "source $VIRTUAL_ENV/bin/gluetool-bash-completition" >> $VIRTUAL_ENV/bin/activate
To activate bash completion immediately, source the generated file. Otherwise, it’d start working next time you’d activate your virtualenv.
. ./gluetool-bash-completition
5. Add configuration¶
gluetool
looks for its configuration in a local directory (among others), in ./.gluetool.d
to be specific. Add
configuration for the modules according to your preference.
Now every time you activate your new virtualenv, you should be able to run gluetool
:
gluetool -h
usage: gluetool [opts] module1 [opts] [args] module2 ...
optional arguments:
...
Test suites¶
The test suite is governed by tox
and py.test
. Before running the test suite, you have to install tox
:
pip install tox
Tox can be easily executed by:
tox
Tox also accepts additional options which are then passed to py.test
:
tox -- --cov=gluetool --cov-report=html:coverage-report
Tox creates (and caches) virtualenv for its test runs, and uses them for
running the tests. It integrates multiple different types of test (you
can see them by running tox -l
).
Documentation¶
Auto-generated documentation is located in docs/
directory. To
update your local copy, run these commands:
ansible-playbook ./generate-docs.yml
Then you can read generated docs by opening docs/build/html/index.html
.
gluetool
API¶
gluetool.glue module¶
-
class
gluetool.glue.
ArgumentParser
(prog=None, usage=None, description=None, epilog=None, version=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True)[source]¶ Bases:
argparse.ArgumentParser
Pretty much the
argparse.ArgumentParser
, it overrides just theargparse.ArgumentParser.error()
method, to catch errors and to wrap them into nice and commonGlueError
instances.The original prints (for us) useless message, including the program name, and raises
SystemExit
exception. Such action does not provide necessary information when encountered in Sentry, for example.-
error
(message)[source]¶ Must not return - raising an exception is a good way to “not return”.
Raises: gluetool.glue.GlueError – When argument parser encounters an error.
-
-
class
gluetool.glue.
Configurable
[source]¶ Bases:
object
Base class of two main
gluetool
classes -gluetool.glue.Glue
andgluetool.glue.Module
. Gives them the ability to use options, settable from configuration files and/or command-line arguments.Variables: _config (dict) – internal configuration store. Values of all options are stored here, regardless of them being set on command-line or by the configuration file. -
classmethod
_create_args_parser
(**kwargs)[source]¶ Create an argument parser. Used by Sphinx to document “command-line” options of the module - which are, by the way, the module options as well.
Parameters: kwargs (dict) – Additional arguments passed to argparse.ArgumentParser
.
-
_dryrun_allows
(threshold, msg)[source]¶ Check whether current dry-run level allows an action. If the current dry-run level is equal of higher than
threshold
, then the action is not allowed.E.g. when action’s
threshold
isDryRunLevels.ISOLATED
, and the current level isDryRunLevels.DRY
, the action is allowed.Parameters: - threshold (DryRunLevels) – Dry-run level the action is not allowed.
- msg (str) – Message logged (as a warning) when the action is deemed not allowed.
Returns: True
when action is allowed,False
otherwise.
-
static
_for_each_option
(callback, options)[source]¶ Given dictionary defining options, call a callback for each of them.
Parameters:
-
static
_for_each_option_group
(callback, options)[source]¶ Given set of options, call a callback for each option group.
Parameters: - options – List of option groups, or a dict listing options directly.
- callback (callable) – Must accept at least 2 parameters:
options
(dict
), listing options in the group, and keyword parametergroup_name
(str
), which is set to group name when theoptions
defines an option group.
-
_parse_args
(args, **kwargs)[source]¶ Parse command-line arguments. Uses
argparse
for the actual parsing. Updates module’s configuration store with values returned by parser.Parameters: args (list) – arguments passed to this module. Similar to what sys.argv
provides on program level.
-
_parse_config
(paths)[source]¶ Parse configuration files. Uses
ConfigParser
for the actual parsing. Updates module’s configuration store with values found returned by the parser.Parameters: paths (list) – List of paths to possible configuration files.
-
debug
(s)¶
-
dryrun_allows
(msg)[source]¶ Checks whether current dry-run level allows an action which is disallowed on
DryRunLevels.DRY
level.See
Configurable._dryrun_allows()
for detailed description.
-
dryrun_enabled
¶ True
if dry-run level is enabled, on any level.
-
dryrun_level
¶ Return current dry-run level. This must be implemented by class descendants because each one finds the necessary information in different places.
-
error
(s)¶
-
eval_context
¶ Return “evaluation context” - a dictionary of variable names (usually in uppercase) and their values, which is supposed to be used in various “evaluate this” operations like rendering of templates.
To provide nice and readable documentation of variables, returned by a module’s
eval_context
property, assign a dictionary, describing these variables, to a local variable named__content__
:... @property def eval_context(self): __content__ = { 'FOO': 'This is an important variable, extracted from clouds.' } return { 'FOO': 42 }
gluetool
core will extract this information and will use it to generate different help texts like your module’s help or a list of all known context variables.Return type: dict
-
exception
(s)¶
-
info
(s)¶
-
isolatedrun_allows
(msg)[source]¶ Checks whether current dry-run level allows an action which is disallowed on
DryRunLevels.ISOLATED
level.
-
name
= None¶ Module name. Usually matches the name of the source file, no suffix.
-
option
(*names)[source]¶ Return values of given options from module’s configuration store.
Parameters: names (str) – names of requested options. Returns: either a value or None
if such option does not exist. When multiple options are requested, a tuple of their values is returned, for a single option its value is not wrapped by a tuple.
-
options
= {}¶ The
options
variable defines options accepted by module, and their properties:options = { <option name>: { <option properties> }, ... }
where
<option name>
is used to name the option in the parser, and two formats are accepted (don’t add any leading dashes (-
nor--
):<long name>
tuple(<short name>, <long name #1>, <long name #2>, ...)
- the first of long names (
long name #1
) is used to identify the option - other long names are understood by argument parser but their values are stored underlong name #1
option. - dictionary
<option properties>
is passed toargparse.ArgumentParser.add_argument()
as keyword arguments when the option is being added to the parser, therefore any arguments recognized byargparse
can be used.
It is also possible to use groups:
options = [ (<group name>, <group options>), ... ]
where
<group name>
is the name of the group, e.g.Debugging options
<group options>
is thedict
with all group options, as described above.
This way, you can split pile of options into conceptualy closer groups of options. A single
dict
you would have is split into multiple smaller dictionaries, and each one is coupled with the group name in atuple
.
-
options_note
= None¶ If set, it will be printed after all options as a help’s epilog.
-
parse_args
(args)[source]¶ Public entry point to argument parsing. Child classes must implement this method, e.g. by calling
gluetool.glue.Configurable._parse_args()
which makes use of additionalargparse.ArgumentParser
options.
-
parse_config
()[source]¶ Public entry point to configuration parsing. Child classes must implement this method, e.g. by calling
gluetool.glue.Configurable._parse_config()
which requires list of paths.
-
required_options
= []¶ Iterable of names of required options.
-
supported_dryrun_level
= 0¶ Highest supported level of dry-run.
-
unique_name
= None¶ Unque name of this (module) instance.
Used by modules, has no meaning elsewhere, but since dry-run checks are done on this level, it must be declared here to make pylint happy :/
-
verbose
(s)¶
-
warn
(s)¶
-
classmethod
-
class
gluetool.glue.
DryRunLevels
[source]¶ Bases:
enum.IntEnum
Dry-run levels.
Variables: -
DEFAULT
= 0¶
-
DRY
= 1¶
-
ISOLATED
= 2¶
-
_member_map_
= OrderedDict([('DEFAULT', <DryRunLevels.DEFAULT: 0>), ('DRY', <DryRunLevels.DRY: 1>), ('ISOLATED', <DryRunLevels.ISOLATED: 2>)])¶
-
_member_names_
= ['DEFAULT', 'DRY', 'ISOLATED']¶
-
_value2member_map_
= {0: <DryRunLevels.DEFAULT: 0>, 1: <DryRunLevels.DRY: 1>, 2: <DryRunLevels.ISOLATED: 2>}¶
-
-
class
gluetool.glue.
Failure
(module, exc_info)[source]¶ Bases:
object
Bundles exception related info. Used to inform modules in their
destroy()
phase thatgluetool
session was killed because of exception raised by one of modules.Parameters: - module (gluetool.glue.Module) – module in which the error happened, or
None
. - exc_info (tuple) – Exception information as returned by
sys.exc_info()
.
Variables: - module (gluetool.glue.Module) – module in which the error happened, or
None
. - exception (Exception) – Shortcut to
exc_info[1]
, if available, orNone
. - exc_info (tuple) – Exception information as returned by
sys.exc_info()
. - sentry_event_id (str) – If set, the failure was reported to the Sentry under this ID.
- module (gluetool.glue.Module) – module in which the error happened, or
-
class
gluetool.glue.
Glue
(tool=None, sentry=None)[source]¶ Bases:
gluetool.glue.Configurable
Main workhorse of the
gluetool
. Manages modules, their instances and runs them as requested.Parameters: - tool (gluetool.tool.Tool) – If set, it’s an
gluetool
-like tool that created this instance. Some functionality may need it to gain access to bits like its command-name. - sentry (gluetool.sentry.Sentry) – If set, it provides interface to Sentry.
Register a shared function. Overwrite previously registered function with the same name, if there was any such.
This is a helper method for easier testability. It is not a part of public API of this class.
Parameters: - funcname (str) – Name of the shared function.
- module (gluetool.glue.Module) – Module instance providing the shared function.
- func (callable) – Shared function.
-
_check_module_file
(mfile)[source]¶ Make sure the file looks like a
gluetool
module:- can be processed by Python parser,
- imports
gluetool.glue.Glue
andgluetool.glue.Module
, - contains child class of
gluetool.glue.Module
.
Parameters: mfile (str) – path to a file. Returns: True
if file containsgluetool
module,False
otherwise.Raises: gluetool.glue.GlueError – when it’s not possible to finish the check.
-
_eval_context
()[source]¶ Gather contexts of all modules in a pipeline and merge them together.
Always returns a unique dictionary object, therefore it is safe for caller to update it. The return value is not cached in any way, therefore the change if its content won’t affect future callers.
Provided as a shared function, registered by the Glue instance itself.
Return type: dict
-
_eval_context_module_caller
()[source]¶ Infere module instance calling the eval context shared function.
Return type: gluetool.glue.Module
-
_import_module
(import_name, filename)[source]¶ Attempt to import a Python module from a file.
Parameters: Returns: imported Python module.
Raises: gluetool.glue.GlueError – when import failed.
-
_load_gluetool_modules
(group, module_name, filepath)[source]¶ Load
gluetool
modules from a file. Method attempts to import the file as a Python module, and then checks its content and adds all gluetool modules to internal module registry.Parameters: Return type: [(module_group, module_class), ..]
Returns: list of loaded
gluetool
modules
-
_load_module_path
(ppath)[source]¶ Search and load
gluetool
modules from a directory.In essence, it scans every file with
.py
suffix, and searches for classes derived fromgluetool.glue.Module
.Parameters: ppath (str) – directory to search for gluetool modules.
-
_load_python_module
(group, module_name, filepath)[source]¶ Load Python module from a file, if it contains
gluetool
modules. If the file does not look like it containsgluetool
modules, or when it’s not possible to import the Python module successfully, method simply warns user and ignores the file.Parameters: Returns: loaded Python module.
Raises: gluetool.glue.GlueError – when import failed.
Register a shared function. Overwrite previously registered function with the same name, if there was any such.
Parameters: - funcname (str) – Name of the shared function.
- module (gluetool.glue.Module) – Module instance providing the shared function.
-
debug
= None¶
-
dryrun_level
¶
-
error
= None¶
-
eval_context
¶ Returns “global” evaluation context - some variables that are nice to have in all contexts.
-
exception
= None¶
-
info
= None¶
-
init_module
(module_name, actual_module_name=None)[source]¶ Given a name of the module, create its instance and give it a name.
Parameters: - module_name (str) – Name under which will be the module instance known.
- actual_module_name (str) – Name of the module to instantiate. It does not have to match
module_name
-actual_module_name
refers to the list of knowngluetool
modules whilemodule_name
is basically an arbitrary name new instance calls itself. If it’s not set, which is the most common situation, it defaults tomodule_name
.
Returns: A
Module
instance.
-
module_config_paths
¶ List of paths in which module config files reside.
-
module_data_paths
¶ List of paths in which module data files reside.
-
module_paths
¶ List of paths in which modules reside.
-
name
= 'gluetool core'¶
-
options
= [('Global options', {('V', 'version'): {'action': 'store_true', 'help': 'Print version'}, ('E', 'list-eval-context'): {'action': 'store_true', 'default': False, 'help': 'List all available variables provided by modules in their evaluation contexts.'}, ('L', 'list-shared'): {'action': 'store_true', 'default': False, 'help': 'List all available shared functions.'}, ('l', 'list-modules'): {'action': 'append', 'const': True, 'nargs': '?', 'help': 'List all available modules. If a GROUP is set, limits list to the given module group.', 'metavar': 'GROUP'}, 'no-sentry-exceptions': {'action': 'append', 'default': [], 'help': 'List of exception names, which are not reported to Sentry (Default: none)'}, ('r', 'retries'): {'default': 0, 'type': <type 'int'>, 'help': 'Number of retries'}}), ('Output control', {('j', 'json-file'): {'default': None, 'help': '\n If set, all log messages (including ``VERBOSE``) are stored in this file\n in a form of JSON structures (default: %(default)s).\n '}, 'verbose-file': {'default': None, 'help': 'Log messages with ``VERBOSE`` level sent to this file.'}, ('v', 'verbose'): {'action': 'store_true', 'help': 'Log **all** messages to the terminal (WARNING: even more verbose than ``-d``!).'}, ('p', 'pid'): {'action': 'store_true', 'help': 'Log PID of gluetool process'}, ('c', 'colors'): {'action': 'store_true', 'help': 'Colorize logging on the terminal'}, 'show-traceback': {'action': 'store_true', 'default': False, 'help': '\n Display exception tracebacks on terminal (besides the debug file when ``--debug-file``\n is used) (default: %(default)s).\n '}, ('q', 'quiet'): {'action': 'store_true', 'help': 'Silence info messages'}, ('i', 'info'): {'action': 'store_true', 'help': 'Print command-line that would re-run the gluetool session'}, ('o', 'debug-file', 'output'): {'help': 'Log messages with at least ``DEBUG`` level are sent to this file.'}, ('d', 'debug'): {'action': 'store_true', 'help': 'Log debugging messages to the terminal (WARNING: very verbose!).'}}), ('Directories', {'module-path': {'action': 'append', 'default': [], 'help': 'Specify directory with modules.', 'metavar': 'DIR'}, 'module-config-path': {'action': 'append', 'default': [], 'help': 'Specify directory with module configuration files.', 'metavar': 'DIR'}, 'module-data-path': {'action': 'append', 'default': [], 'help': 'Specify directory with module data files.', 'metavar': 'DIR'}}), ('Dry run options', {'isolated-run': {'action': 'store_true', 'default': False, 'help': 'Modules that support this option will not interact with the outside world.'}, 'dry-run': {'action': 'store_true', 'default': False, 'help': 'Modules that support this option will make no changes to the outside world.'}}), {'pipeline': {'raw': True, 'nargs': '...', 'help': 'List of modules and their options, passed after gluetool options.'}}]¶
-
run_module
(module_name, module_argv=None, actual_module_name=None, register=False)[source]¶ Syntax sugar for
run_modules()
, in the case you want to run just a one-shot module.Parameters: - module_name (str) – Name under which will be the module instance known.
- module_argv (list(str)) – Arguments of the module.
- actual_module_name (str) – Name of the module to instantiate. It does not have to match
module_name
-actual_module_name
refers to the list of knowngluetool
modules whilemodule_name
is basically an arbitrary name new instance calls itself. If it’s not set, which is the most common situation, it defaults tomodule_name
. - register (bool) – If
True
, module instance is added to a list of modules in thisGlue
instance, and it will be collected whendestroy_modules()
gets called.
-
run_modules
(pipeline_desc, register=False)[source]¶ Run a pipeline, consisting of multiple modules.
Parameters: - pipeline_desc (list(PipelineStep)) – List of pipeline steps.
- register (bool) – If
True
, module instance is added to a list of modules in thisGlue
instance, and it will be collected whendestroy_modules()
gets called.
-
sentry_submit_exception
(*args, **kwargs)[source]¶ Submits exceptions to the Sentry server. Does nothing by default, unless this instance is initialized with a
gluetool.sentry.Sentry
instance which actually does the job.
-
sentry_submit_warning
(*args, **kwargs)[source]¶ Submits warnings to the Sentry server. Does nothing by default, unless this instance is initialized with a
gluetool.sentry.Sentry
instance which actually does the job.
Shared function registry. funcname: (module, fn)
-
verbose
= None¶
-
warn
= None¶
- tool (gluetool.tool.Tool) – If set, it’s an
-
exception
gluetool.glue.
GlueCommandError
(cmd, output, **kwargs)[source]¶ Bases:
gluetool.glue.GlueError
Exception raised when external command failes.
Parameters: - cmd (list) – Command as passed to gluetool.utils.run_command helper.
- output (gluetool.utils.ProcessOutput) – Process output data.
Variables: - cmd (list) – Command as passed to gluetool.utils.run_command helper.
- output (gluetool.utils.ProcessOutput) – Process output data.
-
exception
gluetool.glue.
GlueError
(message, caused_by=None, sentry_fingerprint=None, sentry_tags=None, **kwargs)[source]¶ Bases:
exceptions.Exception
Generic
gluetool
exception.Parameters: - message (str) – Exception message, describing what happened.
- caused_by (tuple) – If set, contains tuple as returned by
sys.exc_info()
, describing the exception that caused this one to be born. If not set, constructor will try to auto-detect this information, and if there’s no such information, instance propertycaused_by
will be set toNone
. - sentry_fingerprint (list(str)) – if set, it is used as a Sentry fingerprint of the exception. See
sentry_fingerprint()
for more details. - str) sentry_tags (dict(str,) – if set, it is merged with other tags when submitting the exception.
See
sentry_tags()
for more details.
Variables: - message (str) – Exception message, describing what happened.
- caused_by (tuple) – If set, contains tuple as returned by
sys.exc_info()
, describing the exception that caused this one to be born.None
otherwise.
-
no_sentry_exceptions
= []¶
-
sentry_fingerprint
(current)[source]¶ Default grouping of events into issues might be too general for some cases. This method gives users a chance to provide custom fingerprint Sentry could use to group events in a more suitable way.
E.g. user might be interested in some sort of connection issues but they would like to have them grouped not by a traceback (which is the default method) but per remote host IP. For that, the Sentry integration code will call
sentry_fingerprint
method of a raised exception, and the method should return new fingerprint, let’s say[<exception class name>, <remote IP>]
, and Sentry will group events using this fingerprint.If the exception was raised with
sentry_fingerprint
parameter set, it is returned instead ofcurrent
, after prefixing the list of tags with a name of the exception’s class.Parameters: current (list(str)) – current fingerprint. Usually ['{{ default }}']
telling Sentry to use its default method, but it could already be more specific.Return type: list(str) Returns: new fingerprint, e.g. ['FailedToConnectToAPI', '10.20.30.40']
Add, modify or remove tags attached to a Sentry event, reported when the exception was raised.
Most common usage would be an addition of tags, e.g.
remote-host
to allow search for events related to the same remote address.If the exception was raised with
sentry_tags
parameter set, its value is injected tocurrent
before returning it.Parameters: str) current (dict(str,) – current set of tags and their values. Return type: dict(str, str) Returns: new set of tags. It is possible to add tags directly into current
and then return it.
-
exception
gluetool.glue.
GlueRetryError
(message, caused_by=None, sentry_fingerprint=None, sentry_tags=None, **kwargs)[source]¶ Bases:
gluetool.glue.GlueError
Retry gluetool exception
-
class
gluetool.glue.
Module
(glue, name)[source]¶ Bases:
gluetool.glue.Configurable
Base class of all
gluetool
modules.Parameters: glue (gluetool.glue.Glue) –
Glue
instance owning the module.Variables: - glue (gluetool.glue.Glue) –
Glue
instance owning the module. - _config (dict) – internal configuration store. Values of all module options are stored here, regardless of them being set on command-line or in the configuration file.
- _overloaded_shared_functions (dict) – If a shared function added by this module
overloades an older function of the same name, registered by a previous module,
the overloaded one is added into this dictionary. The module can then call this
saved function - using
overloaded_shared()
- to implement a “chain” of shared functions, when one calls another, implementing the same operation.
Generate help for shared functions provided by the module.
Returns: Formatted help, describing module’s shared functions.
-
_paths_with_module
(roots)[source]¶ Return paths cretaed by joining roots with module’s unique name.
Parameters: roots (list(str)) – List of root directories.
Register module’s shared functions with Glue, to allow other modules to use them.
-
debug
= None¶
-
description
= None¶ Short module description, displayed in
gluetool
‘s module listing.
-
destroy
(failure=None)[source]¶ Here should go any code that needs to be run on exit, like job cleanup etc.
Parameters: failure (gluetool.glue.Failure) – if set, carries information about failure that made gluetool
to destroy the whole session. Modules might want to take actions based on provided information, e.g. send different notifications.
-
dryrun_level
¶
-
error
= None¶
-
exception
= None¶
-
execute
()[source]¶ In this method, modules can perform any work they deemed necessary for completing their purpose. E.g. if the module promises to run some tests, this is the place where the code belongs to.
By default, this method does nothing. Reimplement as needed.
-
info
= None¶
Call a shared function overloaded by the one provided by this module. This way, a module can give chance to other implementations of its action, e.g. to publish messages on a different message bus.
-
sanity
()[source]¶ In this method, modules can define additional checks before execution.
Some examples:
- Advanced checks on passed options
- Check for additional requirements (tools, data, etc.)
By default, this method does nothing. Reimplement as needed.
Iterable of names of shared functions exported by the module.
-
verbose
= None¶
-
warn
= None¶
- glue (gluetool.glue.Glue) –
-
class
gluetool.glue.
PipelineStep
(module, actual_module=None, argv=None)[source]¶ Bases:
object
Step of
gluetool
‘s pipeline - which is basically just a list of steps.Parameters: - module (str) – name to give to the module instance. This name is used e.g. in logging or when searching for module’s config file.
- actual_module (str) – The actual module class the step uses. Usually it is same as
module
but may differ,module
is then a mere “alias”.actual_module
is used to locate a module class, whose instance is then given namemodule
. - argv (list(str)) – list of options to be given to the module, in a form similar
to
sys.argv
.
-
module_designation
¶
-
exception
gluetool.glue.
SoftGlueError
(message, caused_by=None, sentry_fingerprint=None, sentry_tags=None, **kwargs)[source]¶ Bases:
gluetool.glue.GlueError
Soft errors are errors Glue Ops and/or developers shouldn’t be bothered with, things that are up to the user to fix, e.g. empty set of tests. Hard errors are supposed to warn Ops/Devel teams about important infrastructure issues, code deficiencies, bugs and other issues that are fixable only by actions of Glue staff.
However, we still must provide notification to user(s), and since we expect them to fix the issues that led to raising the soft error, we must provide them with as much information as possible. Therefore modules dealing with notifications are expected to give these exceptions a chance to influence the outgoing messages, e.g. by letting them provide an e-mail body template.
gluetool.help module¶
Command-line --help
helpers (muhaha!).
gluetool
uses docstrings to generate help for command-line options, modules, shared functions
and other stuff. To generate good looking and useful help texts a bit of work is required.
Add the Sphinx which we use to generate nice documentation of gluetool
‘s API and structures,
with its directives, and it’s even more work to make readable output. Therefore these helpers,
separated in their own file to keep things clean.
-
class
gluetool.help.
DummyTextBuilder
[source]¶ Sphinx
TextWriter
(and other writers as well) requires an instance ofBuilder
class that brings configuration into the rendering process. The originalTextBuilder
requires Sphinx application which brings a lot of other dependencies (e.g. source paths and similar stuff) which are impractical in our use case (“render short string to plain text”). Therefore this dummy class which just provides minimal configuration -TextWriter
requires nothing else fromBuilder
instance.See
sphinx/writers/text.py
for the original implementation.-
DummyTextBuilder.
config
¶ alias of
DummyConfig
-
DummyTextBuilder.
translator_class
= None¶
-
-
class
gluetool.help.
TextTranslator
(document, builder)[source]¶ Bases:
sphinx.writers.text.TextTranslator
-
gluetool.help.
doc_role_handler
(role, rawtext, text, lineno, inliner, options=None, context=None)[source]¶ Format
:doc:
roles, used to reference another bits of documentation.
-
gluetool.help.
docstring_to_help
(docstring, width=None, line_prefix=' ')[source]¶ Given docstring, process and render it as a plain text. This conversion function is used to generate nice and readable help strings used when printing help on command line.
Parameters: Returns: formatted docstring.
-
gluetool.help.
eval_context_help
(source)[source]¶ Generate and format help for an evaluation context of a module. Looks for context content, and gives it a nice header, suitable for command-line help, applying formatting along the way.
Parameters: source (gluetool.glue.Configurable) – object whose eval context help we should format. Returns: Formatted help.
-
gluetool.help.
extract_eval_context_info
(source, logger=None)[source]¶ Extract information of evaluation context content from the
source
- a module or any other object witheval_context
property. The information we’re looking for is represented by an assignment to special variable,__content__
, in the body ofeval_context
getter.__content__
is expected to be assigned a dictionary, listing context variables (keys) and their descriptions (values).If it’s not possible to find such information, or an exception is raised, the function returns an empty dictionary.
Parameters: source (gluetool.glue.Configurable) – object to extract information from. Return type: dict(str, str)
-
gluetool.help.
function_help
(func, name=None)[source]¶ Uses function’s signature and docstring to generate a plain text help describing the function.
Parameters: Returns: (signature, body)
pair.
-
gluetool.help.
functions_help
(functions)[source]¶ Generate help for a set of functions.
Parameters: callable) functions (list(str,) – Functions to generate help for, passed as name and the corresponding callable pairs. Return type: str Returns: Formatted help.
-
gluetool.help.
option_help
(txt)[source]¶ Given option help text, format it to be more suitable for command-line help. Options can provide a single line of text, or mutiple lines (using triple quotes and docstring-like indentation).
Parameters: txt (str) – Raw option help text. Returns: Formatted option help text.
-
gluetool.help.
py_default_role
(role, rawtext, text, lineno, inliner, options=None, content=None)[source]¶ Default handler we use for
py:...
roles, translates text to literal node.
-
gluetool.help.
rst_to_text
(text)[source]¶ Render given text, written with RST, as plain text.
Parameters: text (str) – string to render. Return type: str Returns: plain text representation of text
.
-
gluetool.help.
trim_docstring
(docstring)[source]¶ Quoting PEP 257 <https://www.python.org/dev/peps/pep-0257/#handling-docstring-indentation>:
Docstring processing tools will strip a uniform amount of indentation from the second and further lines of the docstring, equal to the minimum indentation of all non-blank lines after the first line. Any indentation in the first line of the docstring (i.e., up to the first newline) is insignificant and removed. Relative indentation of later lines in the docstring is retained. Blank lines should be removed from the beginning and end of the docstring.
Code bellow follows the quote.
This method does exactly that, therefore we can keep properly aligned docstrings while still use them for reasonably formatted help texts.
Parameters: docstring (str) – raw docstring. Return type: str Returns: docstring with lines stripped of leading whitespace.
gluetool.log module¶
Logging support.
Sets up logging environment for use by gluetool
and modules. Based
on standard library’s logging
module, augmented a bit to
support features loke colorized messages and stackable context information.
Example usage:
# initialize logger as soon as possible
logger = Logging.create_logger()
# now it's possible to use it for logging:
logger.debug('foo!')
# or connect it with current instance (if you're doing all this
# inside some class' constructor):
logger.connect(self)
# now you can access logger's methods directly:
self.debug('foo once again!')
# find out what your logging should look like, e.g. by parsing command-line options
...
# tell logger about the final setup
logger = Logging.create_logger(output_file='/tmp/foo.log', level=...)
# and, finally, create a root context logger - when we create another loggers during
# the code flow, this context logger will be in the root of this tree of loggers.
logger = ContextAdapter(logger)
# don't forget to re-connect with the context logger if you connected your instance
# with previous logger, to make sure helpers are set correctly
logger.connect(self)
-
class
gluetool.log.
BlobLogger
(intro, outro=None, on_finally=None, writer=None)[source]¶ Bases:
object
Context manager to help with “real time” logging - some code may produce output continuously, e.g. when running a command and streaming its output to our stdout, and yet we still want to wrap it with boundaries and add a header.
This code:
with BlobLogger('ls of root', outro='end of ls'): subprocess.call(['ls', '/'])
will lead to the output similar to this:
[20:30:50] [+] ---v---v---v---v---v--- ls of root bin boot data dev ... [20:30:50] [+] ---^---^---^---^---^--- end of ls
Note
When you already hold the data you wish to log, please use
gluetool.log.log_blob()
orgluetool.log.log_dict()
. The example above could be rewritten usinglog_blob
by usingsubprocess.check_output()
and passing its return value tolog_blob
.BlobLogger
is designed to wrap output whose creation caller don’t want to (or cannot) control.Parameters: - intro (str) – Label to show what is the meaning of the logged data.
- outro (str) – Label to show by the final boundary to mark the end of logging.
- on_finally (callable) – When set, it will be called in
__exit__
method. User of this context manager might need to flush used streams or close resources even in case the exception was raised while inside the context manager.on_finally
is called with all arguments the__exit__
was called, and its return value is returned by__exit__
itself, therefore it can examine possible exceptions, and override them. - writer (callable) – A function which is used to actually log the text. Usually a one of some logger methods.
-
class
gluetool.log.
ContextAdapter
(logger, extra=None)[source]¶ Bases:
logging.LoggerAdapter
Generic logger adapter that collects “contexts”, and prepends them to the message.
“context” is any key in
extra
dictionary starting withctx_
, whose value is expected to be tuple of(priority, value)
. Contexts are then sorted by their priorities before inserting them into the message (lower priority means context will be placed closer to the beggining of the line - highest priority comes last.Parameters: - logger (logging.Logger) – parent logger this adapter modifies.
- extras (dict) – additional extra keys passed to the parent class.
The dictionary is then used to update messages’
extra
key with the information about context.
-
connect
(parent)[source]¶ Create helper methods in
parent
, by assigning adapter’s methods to its attributes. One can then callparent.debug
and so on, instead of less readableparent.logger.debug
.Simply instantiate adapter and call its
connect
with an object as aparent
argument, and the object will be enhanced with all these logging helpers.Parameters: parent – object to enhance with logging helpers.
-
class
gluetool.log.
JSONLoggingFormatter
(**kwargs)[source]¶ Bases:
logging.Formatter
Custom logging formatter producing a JSON dictionary describing the log record.
-
static
_format_exception_chain
(serialized, exc_info)[source]¶ “Format” exception chain - transform it into a bunch of JSON structures describing exceptions, stack frames, local variables and so on.
Serves the same purpose as
LoggingFormatter._format_exception_chain
but that one produces a string, textual representation suitable for printing. This method produces JSON structures, suitable for, hm, JSON log.
-
static
-
class
gluetool.log.
Logging
[source]¶ Bases:
object
Container wrapping configuration and access to
logging
infrastructuregluetool
uses for logging.-
OUR_LOGGERS
= (<logging.Logger object>, <logging.Logger object>)¶
-
static
_setup_log_file
(filepath, level, limit_level=False, formatter_class=<class 'gluetool.log.LoggingFormatter'>)[source]¶
-
adapted_logger
= None¶
-
static
configure_logger
(logger)[source]¶ Configure given logger to conform with Gluetool’s idea of logging. The logger is set to
VERBOSE
level, shared stderr handler is added, and Sentry integration status is propagated as well.After this method, the logger will behave like Gluetool’s main logger.
-
static
create_logger
(level=20, debug_file=None, verbose_file=None, json_file=None, sentry=None, sentry_submit_warning=None, show_traceback=False)[source]¶ Create and setup logger.
This method is called at least twice:
- when
gluetool.glue.Glue
is instantiated: only astderr
handler is set up, with loglevel beingINFO
; - when all arguments and options are processed, and Glue instance can determine desired log level, whether it’s expected to stream debugging messages into a file, etc. This time, method only modifies propagates necessary updates to already existing logger.
Parameters: - debug_file (str) – if set, new handler will be attached to the logger, streaming
messages of at least
DEBUG
level into this this file. - verbose_file (str) – if set, new handler will be attached to the logger, streaming
messages of
VERBOSE
log levels into this this file. - json_file (str) – if set, all logging messages are sent to this file in a form of JSON structures.
- level (int) – desired log level. One of constants defined in
logging
module, e.g.logging.DEBUG
orlogging.ERROR
. - sentry (bool) – if set, logger will be augmented to send every log message to the Sentry server.
- sentry_submit_warning (callable) – if set, it is used by
warning
methods of derived loggers to submit warning to the Sentry server, if asked by a caller to do so. - show_traceback (bool) – if set, exception tracebacks would be sent to
stderr
handler as well as to the debug file.
Return type: Returns: a
logging.Logger
instance, set up for logging.- when
-
debug_file_handler
= None¶
-
static
get_logger
()[source]¶ Returns a logger-like object suitable for logging stuff.
Expects there was a call to
create_logger()
method before calling this method that would actually create and set up the logger.Return type: ContextAdapter Returns: an instance of ContextAdapter wrapping root logger, or None
when there’s no logger yet.
-
json_file_handler
= None¶
-
logger
= None¶ Logger singleton - if anyone asks for a logger, they will get this one. Needs to be properly initialized by calling
create_logger()
.
-
sentry
= None¶
-
sentry_submit_warning
= None¶
-
stderr_handler
= None¶ Stream handler printing out to stderr.
-
verbose_file_handler
= None¶
-
-
class
gluetool.log.
LoggingFormatter
(colors=True, log_tracebacks=False)[source]¶ Bases:
logging.Formatter
Custom log record formatter. Produces output in form of:
[stamp] [level] [ctx1] [ctx2] ... message
Parameters: - colors (bool) – if set, colorize output. Enabled by default but when used with file-backed destinations, colors are disabled by logging subsystem.
- log_tracebacks (bool) – if set, add tracebacks to the message. By default, we don’t need tracebacks on the terminal, unless its loglevel is verbose enough, but we want them in the debugging file.
-
static
_format_exception_chain
(exc_info)[source]¶ Format exception chain. Start with the one we’re given, and follow its caused_by property until we ran out of exceptions to format.
-
_level_color
= {40: <function <lambda>>, 50: <function <lambda>>, 20: <function <lambda>>, 30: <function <lambda>>}¶ Colorizers assigned to loglevels
Tags used to express loglevel.
-
format
(record)[source]¶ Format a logging record. It puts together pieces like time stamp, log level, possibly also different contexts if there are any stored in the record, and finally applies colors if asked to do so.
Parameters: record (logging.LogRecord) – record describing the event. Return type: str Returns: string representation of the event record.
-
class
gluetool.log.
LoggingFunctionType
(*args, **kwargs)[source]¶ Bases:
typing_extensions.Protocol
-
_abc_cache
= <_weakrefset.WeakSet object>¶
-
_abc_generic_negative_cache
= <_weakrefset.WeakSet object>¶
-
_abc_generic_negative_cache_version
= 31¶
-
_abc_registry
= <_weakrefset.WeakSet object>¶
-
_callable_members_only
= True¶
-
_gorg
¶ alias of
LoggingFunctionType
-
_is_protocol
= True¶
-
-
class
gluetool.log.
LoggingWarningFunctionType
(*args, **kwargs)[source]¶ Bases:
typing_extensions.Protocol
-
_abc_cache
= <_weakrefset.WeakSet object>¶
-
_abc_generic_negative_cache
= <_weakrefset.WeakSet object>¶
-
_abc_generic_negative_cache_version
= 31¶
-
_abc_registry
= <_weakrefset.WeakSet object>¶
-
_callable_members_only
= True¶
-
_gorg
¶ alias of
LoggingWarningFunctionType
-
_is_protocol
= True¶
-
-
class
gluetool.log.
ModuleAdapter
(logger, module)[source]¶ Bases:
gluetool.log.ContextAdapter
Custom logger adapter, adding module name as a context.
Parameters: - logger (logging.Logger) – parent logger this adapter modifies.
- module (gluetool.glue.Module) – module whose name is added as a context.
-
class
gluetool.log.
PackageAdapter
(logger, name)[source]¶ Bases:
gluetool.log.ContextAdapter
Custom logger dapter, adding a package name as a context. Intended to taint log records produced by a 3rd party packages.
Logging.Logger logger: parent logger this adapter modifies. Parameters: name (str) – name of the library.
-
class
gluetool.log.
SingleLogLevelFileHandler
(level, *args, **kwargs)[source]¶ Bases:
logging.FileHandler
-
class
gluetool.log.
StreamToLogger
(log_fn)[source]¶ Bases:
object
Fake
file
-like stream object that redirects writes to a given logging method.
-
gluetool.log.
_extract_stack
(tb)[source]¶ Construct a “stack” by merging two sources of data:
- what’s provided by
traceback.extract_tb()
, i.e.(filename, lineno, fnname, text)
tuple for each frame; - stack frame objects, hidden inside traceback object and available via following links from one frame to another.
Return type: list(list(str, int, str, str, frame)) - what’s provided by
-
gluetool.log.
format_blob
(blob)[source]¶ Format a blob of text for printing. Wraps the text with boundaries to mark its borders.
-
gluetool.log.
format_dict
(dictionary)[source]¶ Format a Python data structure for printing. Uses
json.dumps()
formatting capabilities to present readable representation of a given structure.
-
gluetool.log.
format_table
(table, **kwargs)[source]¶ Format a table, represented by an iterable of rows, represented by iterables.
Internally,
tabulate
is used to do the formatting. All keyword arguments are passed totabulate
call.Parameters: table (list(list())) – table to format. Returns: formatted table.
-
gluetool.log.
format_xml
(element)[source]¶ Format an XML element, e.g. Beaker job description, for printing.
Parameters: element – XML element to format.
-
gluetool.log.
log_blob
(writer, intro, blob)[source]¶ Log “blob” of characters of unknown structure, e.g. output of a command or response of a HTTP request. The blob is preceded by a header and followed by a footer to mark exactly the blob boundaries.
Note
For logging structured data, e.g. JSON or Python structures, use
gluetool.log.log_dict()
. It will make structure of the data more visible, resulting in better readability of the log.Parameters:
-
gluetool.log.
log_dict
(writer, intro, data)[source]¶ Log structured data, e.g. JSON responses or a Python
list
.Note
For logging unstructured “blobs” of text, use
gluetool.log.log_blob()
. It does not attempt to format the output, and wraps it by header and footer to mark its boundaries.Note
Using
gluetool.log.format_dict()
directly might be shorter, depending on your your code. For example, this code:self.debug('Some data:\n{}'.format(format_dict(data)))
is equivalent to:
log_dict(self.debug, 'Some data', data)
If you need more formatting, or you wish to fit more information into a single message, using logger methods with
format_dict
is a way to go, while for logging a single structurelog_dict
is more suitable.Parameters:
-
gluetool.log.
log_table
(writer, intro, table, **kwargs)[source]¶ Log a formatted table.
All keyword arguments are passed to
format_table()
call which does the actual formatting.Parameters:
-
gluetool.log.
log_xml
(writer, intro, element)[source]¶ Log an XML element, e.g. Beaker job description.
Parameters:
-
gluetool.log.
print_wrapper
(*args, **kwds)[source]¶ While active, replaces
sys.stdout
andsys.stderr
streams with fakefile
-like streams which send all outgoing data to a given logging method.Parameters: - log_fn (call) – A callback that is called for every line, produced by
print
. If not set, ainfo
method ofgluetool
main logger is used. - label (str) – short description, presented in the intro and outro headers, wrapping captured output.
- log_fn (call) – A callback that is called for every line, produced by
gluetool.proxy module¶
Proxying object wrapper.
-
class
gluetool.proxy.
Proxy
(obj)[source]¶ Bases:
object
Taking advantage of duck typing - wrap instance of class Foo with a Proxy, and the result will behave pretty much like Foo instance. Inherit from Proxy to create custom wrappers, extended with your own methods.
-
_obj
¶
-
_special_names
= ['__abs__', '__add__', '__and__', '__call__', '__cmp__', '__coerce__', '__contains__', '__delitem__', '__delslice__', '__div__', '__divmod__', '__eq__', '__float__', '__floordiv__', '__ge__', '__getitem__', '__getslice__', '__gt__', '__hash__', '__hex__', '__iadd__', '__iand__', '__idiv__', '__idivmod__', '__ifloordiv__', '__ilshift__', '__imod__', '__imul__', '__int__', '__invert__', '__ior__', '__ipow__', '__irshift__', '__isub__', '__iter__', '__itruediv__', '__ixor__', '__le__', '__len__', '__long__', '__lshift__', '__lt__', '__mod__', '__mul__', '__ne__', '__neg__', '__oct__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdiv__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rfloorfiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setitem__', '__setslice__', '__sub__', '__truediv__', '__xor__', 'next']¶
-
gluetool.tool module¶
Heart of the “gluetool” script. Referred to by setuptools’ entry point.
-
class
gluetool.tool.
Gluetool
[source]¶ Bases:
object
-
_command_name
¶
-
_deduce_pipeline_desc
(argv, modules)[source]¶ Split command-line arguments, left by
gluetool
, into a pipeline description, splitting them by modules and their options.Parameters: Returns: Pipeline description in a form of a list of
gluetool.glue.PipelineStep
instances.
-
_exit_logger
¶ Return logger for use when finishing the
gluetool
pipeline.
-
_version
¶
-
gluetool.utils module¶
Various helpers.
-
class
gluetool.utils.
Command
(executable, options=None, logger=None)[source]¶ Bases:
object
Wrap an external command, its options and other information, necessary for running the command.
The main purpose is to gather all relevant pieces into a single space, call
subprocess.Popen
, and log everything.By default, both standard output and error output are of the process are captured and returned back to caller. Under some conditions, caller might want to see the output in “real-time”. For that purpose, they can pass callable via
inspect_callback
parameter - such callable will be called for every received bit of input on both standard and error outputs. E.g.def foo(stream, s, flush=False): if s is not None and 'a' in s: print s Command(['/bin/foo']).run(inspect=foo)
This example will print all substrings containing letter a. Strings passed to
foo
may be of arbitrary lengths, and may change between subsequent use ofCommand
class.Parameters: - executable (list) – Executable to run. Feel free to use the whole command, including its options, if you have no intention to modify them before running the command.
- options (list) – If set, it’s a list of options to pass to the
executable
. Options are specified in a separate list to allow modifications ofexecutable
andoptions
before actually running the command. - logger (gluetool.log.ContextAdapter) – Parent logger whose methods will be used for logging.
New in version 1.1.
-
debug
(s)¶
-
error
(s)¶
-
exception
(s)¶
-
info
(s)¶
-
run
(inspect=False, inspect_callback=None, **kwargs)[source]¶ Run the command, wait for it to finish and return the output.
Parameters: Return type: gluetool.utils.ProcessOutput instance
Returns: gluetool.utils.ProcessOutput
instance whose attributes contain data returned by the child process.Raises: - gluetool.glue.GlueError – When somethign went wrong.
- gluetool.glue.GlueCommandError – When command exited with non-zero exit code.
-
verbose
(s)¶
-
warn
(s)¶
-
exception
gluetool.utils.
IncompatibleOptionsError
(message, caused_by=None, sentry_fingerprint=None, sentry_tags=None, **kwargs)[source]¶ Bases:
gluetool.glue.SoftGlueError
-
class
gluetool.utils.
PatternMap
(filepath, spices=None, logger=None, allow_variables=False)[source]¶ Bases:
object
Pattern map is a list of
<pattern>
:<converter>
pairs.Pattern
is a regular expression used to match a string,converter
is a function that transforms a string into another one, accepting the pattern and the string as arguments.It is defined in a YAML file:
--- - 'foo-(\d+)': 'bar-\1' - 'baz-(\d+)': 'baz, find_the_most_recent, append_dot' - 'bar-(\d+)': - 'bar, find_the_most_recent, append_dot' - 'bar, find_the_oldest, append_dot'
Patterns are the keys in each pair, while
converter
is a string (or list of strings), consisting of multiple items, separated by comma. The first item is always a string, let’s call itR
.R
, given input stringS1
and the pattern, is used to transformS1
to a new string,S2
, by callingpattern.sub(R, S1)
.R
can make use of anythingre.sub()
supports, including capturing groups.If there are other items in the
converter
string, they are names of spices, additional functions that will be called withpattern
and the output of the previous spicing function, starting withS2
in the case of the first spice.To allow spicing, user of
PatternMap
class must provide spice makers - mapping between spice names and functions that generate spicing functions. E.g.:def create_spice_append_dot(previous_spice): def _spice(pattern, s): s = previous_spice(pattern, s) return s + '.' return _spice
create_spice_append_dot
is a spice maker, used during creation of a pattern map after its definition is read,_spice
is the actual spicing function used during the transformation process.There can be multiple converters for a single pattern, resulting in multiple values returned when the input string matches the corresponding pattern.
Parameters: - filepath (str) – Path to a YAML file with map definition.
- spices (dict) – apping between spices and their makers.
- logger (gluetool.log.ContextLogger) – Logger used for logging.
- allow_variables (bool) – if set, both patterns and converters are first treated as templates,
and as such are rendered before doing anything else. Map may contain special comments,
# !include <path>
, where path refers to a YAML file providing the necessary variables.
-
debug
= None¶
-
error
= None¶
-
exception
= None¶
-
info
= None¶
-
match
(s, multiple=False)[source]¶ Try to match
s
by the map. If the match is found - the first one wins - then its conversions are applied to thes
.There can be multiple conversions for a pattern, by default only the product of the first one is returned. If
multiple
is set toTrue
, list of all products is returned instead.Return type: str Returns: if matched, output of the corresponding transformation.
-
verbose
= None¶
-
warn
= None¶
-
class
gluetool.utils.
ProcessOutput
(cmd, exit_code, stdout, stderr, kwargs)[source]¶ Bases:
object
Result of external process.
-
class
gluetool.utils.
SimplePatternMap
(filepath, logger=None, allow_variables=False)[source]¶ Bases:
object
Pattern map is a list of
<pattern>
:<result>
pairs.Pattern
is a regular expression used to match a string,result
is what the matching string maps to.Basically an ordered dictionary with regexp matching of keys, backed by an YAML file.
Parameters: - filepath (str) – Path to a YAML file with map definition.
- logger (gluetool.log.ContextLogger) – Logger used for logging.
- allow_variables (bool) – if set, both patterns and converters are first treated as templates,
and as such are rendered before doing anything else. Map may contain special comments,
# !include <path>
, where path refers to a YAML file providing the necessary variables.
-
debug
= None¶
-
error
= None¶
-
exception
= None¶
-
info
= None¶
-
match
(s)[source]¶ Try to match
s
by the map. If the match is found - the first one wins - then its transformation is applied to thes
.Return type: str Returns: if matched, output of the corresponding transformation.
-
verbose
= None¶
-
warn
= None¶
-
class
gluetool.utils.
StreamReader
(stream, name=None, block=16)[source]¶ Bases:
object
-
content
¶
-
name
¶
-
-
class
gluetool.utils.
ThreadAdapter
(logger, thread)[source]¶ Bases:
gluetool.log.ContextAdapter
Custom logger adapter, adding thread name as a context.
Parameters: - logger (gluetool.log.ContextAdapter) – parent logger whose methods will be used for logging.
- thread (threading.Thread) – thread whose name will be added.
-
class
gluetool.utils.
WorkerThread
(logger, fn, fn_args=None, fn_kwargs=None, **kwargs)[source]¶ Bases:
threading.Thread
Worker threads gets a job to do, and returns a result. It gets a callable,
fn
, which will be called in thread’srun()
method, and thread’sresult
property will be the result - value returned byfn
, or exception raised during the runtime offn
.Parameters: - logger (gluetool.log.ContextAdapter) – logger to use for logging.
- fn – thread will start fn to do the job.
- fn_args – arguments for fn
- fn_kwargs – keyword arguments for fn
-
debug
= None¶
-
exception
= None¶
-
gluetool.utils.
YAML
()[source]¶ Provides YAML read/write interface with common settings.
Return type: ruamel.yaml.YAML
-
gluetool.utils.
_load_yaml_variables
(data, enabled=True, logger=None)[source]¶ Load all variables from files referenced by a YAML, and return function to render a string as a template using these variables. The files containing variables are mentioned in comments, in a form
# !include <filepath>
form.Parameters: - data – data loaded from a YAML file.
- enabled (bool) – when set to
False
, variables are not loaded and a simple no-op function is returned. - logger (gluetool.log.ContextLogger) – Logger used for logging.
Returns: Function accepting a string and returning a rendered template.
-
class
gluetool.utils.
cached_property
(method)[source]¶ Bases:
object
property
-like decorator - at first access, it calls decorated method to acquire the real value, and then replaces itself with this value, making it effectively “cached”. Useful for properties whose value does not change over time, and where getting the real value could penalize execution with unnecessary (network, memory) overhead.Delete attribute to clear the cached value - on next access, decorated method will be called again, to acquire the real value.
Of possible options, only read-only instance attribute access is supported so far.
-
gluetool.utils.
deprecated
(func)[source]¶ This is a decorator which can be used to mark functions as deprecated. It will result in a warning being emitted when the function is used.
-
gluetool.utils.
dict_update
(dst, *args)[source]¶ Python’s
dict.update
does not return the dictionary just updated but aNone
. This function is a helper that does updates the dictionary and returns it. So, instead of:d.update(other) return d
you can use:
return dict_update(d, other)
Parameters: - dst (dict) – dictionary to be updated.
- args – dictionaries to update
dst
with.
-
gluetool.utils.
dump_yaml
(data, filepath, logger=None)[source]¶ Save data stored in variable to YAML file.
Parameters: Raises: gluetool.glue.GlueError – if it was not possible to successfully save data to file.
-
gluetool.utils.
fetch_url
(*args, **kwargs)[source]¶ “Get me content of this URL” helper.
Very thin wrapper around urllib. Added value is logging, and converting possible errors to
gluetool.glue.GlueError
exception.Parameters: Returns: tuple
(response, content)
whereresponse
is whaturllib2.urlopen()
returns, andcontent
is the payload of the response.
-
gluetool.utils.
format_command_line
(cmdline)[source]¶ Return formatted command-line.
All but the first line are indented by 4 spaces.
Parameters: cmdline (list) – list of iterables, representing command-line split to multiple lines.
-
gluetool.utils.
from_json
(json_string)[source]¶ Convert JSON in a string into Python data structures.
Similar to
json.loads()
but uses special object hook to avoid unicode strings in the output..
-
gluetool.utils.
from_yaml
(yaml_string)[source]¶ Convert YAML in a string into Python data structures.
Uses internal YAML parser to produce result. Paired with
load_yaml()
and their JSON siblings to provide unified access to JSON and YAML.
-
gluetool.utils.
load_json
(filepath, logger=None)[source]¶ Load data stored in JSON file, and return their Python representation.
Parameters: - filepath (str) – Path to a file.
~
or~<username>
are expanded before using. - logger (gluetool.log.ContextLogger) – Logger used for logging.
Return type: Returns: structures representing data in the file.
Raises: gluetool.glue.GlueError – if it was not possible to successfully load content of the file.
- filepath (str) – Path to a file.
-
gluetool.utils.
load_yaml
(filepath, logger=None)[source]¶ Load data stored in YAML file, and return their Python representation.
Parameters: - filepath (str) – Path to a file.
~
or~<username>
are expanded before using. - logger (gluetool.log.ContextLogger) – Logger used for logging.
Return type: Returns: structures representing data in the file.
Raises: gluetool.glue.GlueError – if it was not possible to successfully load content of the file.
- filepath (str) – Path to a file.
-
gluetool.utils.
new_xml_element
(tag_name, _parent=None, **attrs)[source]¶ Create new XML element.
Parameters: Returns: Newly created XML element.
-
gluetool.utils.
normalize_bool_option
(option_value)[source]¶ Convert option value to Python’s boolean.
option_value
is what all those internal option processing return, which may be a default value set for an option, or what user passed in.As switches, options with values can be used:
--foo=yes|no --foo=true|false --foo=1|0 --foo=Y|N --foo=on|off
With combination of
store_true
/store_false
and a default value module developer sets for the option, simple form without value is evaluated as easily. Withstore_true
andFalse
default, following option turn the feature foo on:--enable-foo
With
store_false
andTrue
default, following simple option turn the feature foo off:--disable-foo
-
gluetool.utils.
normalize_multistring_option
(option_value, separator=', ')[source]¶ Reduce string, representing comma-separated list of items, or possibly a list of such strings, to a simple list of items. Strips away the whitespace wrapping such items.
foo --option value1 --option value2, value3 foo --option value1,value2,value3
Or, when option is set by a config file:
option = value1 option = value1, value2, value3
After processing, different variants can be found when
option('option')
is called,['value1', 'value2,value3']
,['value1,value2,value3']
,'value1'
andvalue1, value2, value3
.To reduce the necessary work, use this helper function to treat such option’s value, and get simple
['value1', 'value2', 'value3']
structure.
-
gluetool.utils.
normalize_path
(path)[source]¶ Apply common treatments on a given path:
- replace home directory reference (
~
and similar), and - convert
path
to a normalized absolutized version of the pathname.
- replace home directory reference (
-
gluetool.utils.
normalize_path_option
(option_value, separator=', ')[source]¶ Reduce many ways how list of paths is specified by user, to a simple list of paths. See
normalize_multistring_option()
for more details.
-
gluetool.utils.
normalize_shell_option
(option_value)[source]¶ Reduce string, using a shell-like syntax, or possibly a list of such strings, to a simple list of items. Strips away the whitespace wrapping such items.
foo --option value1 --option value2\ value3 --option "value4 value5"
Or, when option is set by a config file:
option = value1 value2\ value3 "value4 value5"
After processing, different variants can be found when
option('option')
is called,['value1', 'value2,value3']
,['value1,value2,value3']
,'value1'
andvalue1, value2, value3
.To reduce the necessary work, use this helper function to treat such option’s value, and get simple
['value1', 'value2 value3', 'value4 value5']
structure.
-
gluetool.utils.
render_template
(template, logger=None, **kwargs)[source]¶ Render Jinja2 template. Logs errors, and raises an exception when it’s not possible to correctly render the template.
Parameters: - template – Template to render. It can be either
jinja2.environment.Template
instance, or a string. - kwargs (dict) – Keyword arguments passed to render process.
Return type: Returns: Rendered template.
Raises: gluetool.glue.GlueError – when the rednering failed.
- template – Template to render. It can be either
-
gluetool.utils.
requests
(*args, **kwds)[source]¶ Wrap
requests
with few layers providing us with the logging and better insight into what has been happening whenrequests
did their job.Used as a context manager, yields a patched
requests
module. As long as inside the context, detailed information about HTTP traffic are logged via given logger.Note
The original
requests
library is returned, with slight modifications for better integration withgluetool
logging facilities. Each and everyrequests
API feature is available and , hopefully, enhancements applied by this wrapper wouldn’t interact withrequests
functionality.with gluetool.utils.requests() as R: R.get(...).json() ... r = R.post(...) assert r.code == 404 ...
Parameters: logger – used for logging. Returns: requests
module.
-
gluetool.utils.
run_command
(*args, **kwargs)[source]¶ Wrapper for ``Command(...).run().
Provided for backward compatibility.
Deprecated since version 1.1: Use
gluetool.utils.Command
instead.
-
gluetool.utils.
treat_url
(url, logger=None)[source]¶ Remove “weird” artifacts from the given URL. Collapse adjacent ‘.’s, apply ‘..’, etc.
Parameters: - url (str) – URL to clear.
- logger (gluetool.log.ContextAdapter) – logger to use for logging.
Return type: Returns: Treated URL.
-
gluetool.utils.
wait
(label, check, timeout=None, tick=30, logger=None)[source]¶ Wait for a condition to be true.
Parameters: - label (str) – printable label used for logging.
- check (callable) – called to test the condition. If its return value evaluates as
True
, the condition is assumed to pass the test and waiting ends. - timeout (int) – fail after this many seconds.
None
means test forever. - tick (int) – test condition every
tick
seconds. - logger (gluetool.log.ContextAdapter) – parent logger whose methods will be used for logging.
Raises: gluetool.glue.GlueError – when
timeout
elapses while condition did not pass the check.
gluetool.version module¶
gluetool.tests package¶
Submodules¶
gluetool.tests.conftest module¶
gluetool.tests.test_core module¶
gluetool.tests.test_error module¶
gluetool.tests.test_eval_context module¶
gluetool.tests.test_help module¶
gluetool.tests.test_json module¶
gluetool.tests.test_load_yaml module¶
gluetool.tests.test_log_exception module¶
gluetool.tests.test_logging module¶
gluetool.tests.test_new_xml_element module¶
gluetool.tests.test_normalize_option module¶
gluetool.tests.test_option module¶
gluetool.tests.test_pipeline_step module¶
gluetool.tests.test_render_template module¶
gluetool.tests.test_requests module¶
gluetool.tests.test_run_command module¶
gluetool.tests.test_run_modules module¶
gluetool.tests.test_treat_url module¶
gluetool.tests.test_utils module¶
gluetool.tests.test_wait module¶
Module contents¶
-
class
gluetool.tests.
Bunch
(**kwargs)[source]¶ Bases:
object
Object-like access to a dictionary - useful for many mock objects.
-
class
gluetool.tests.
NonLoadingGlue
(tool=None, sentry=None)[source]¶ Bases:
gluetool.glue.Glue
Current Glue implementation loads modules and configs when instantiated, which makes it really hard to make assumptions of the state of its internals - they will always be spoiled by other modules, other external resources the tests cannot control. So, to overcome this I use this custom Glue class that disables loading of modules and configs on its instantiation.