executor: Programmer friendly subprocess wrapper¶
Welcome to the documentation of executor version 23.2! The following sections are available:
User documentation¶
The readme is the best place to start reading, it’s targeted at all users and documents the command line interface:
executor: Programmer friendly subprocess wrapper¶
The executor package is a simple wrapper for Python’s subprocess module that makes it very easy to handle subprocesses on UNIX systems with proper escaping of arguments and error checking:
- An object oriented interface is used to execute commands using sane but customizable (and well documented) defaults.
- Remote commands (executed over SSH) are supported using the same object oriented interface, as are commands inside chroots (executed using schroot).
- There’s also support for executing a group of commands concurrently in what’s called a “command pool”. The concurrency level can be customized and of course both local and remote commands are supported.
The package is currently tested on Python 2.7, 3.5, 3.6, 3.7, 3.8 and PyPy. For usage instructions please refer to following sections and the documentation.
Installation¶
The executor package is available on PyPI which means installation should be as simple as:
$ pip install executor
There’s actually a multitude of ways to install Python packages (e.g. the per user site-packages directory, virtual environments or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-).
Usage¶
There are two ways to use the executor package: As the command line program
executor
and as a Python API. The command line interface is described below
and there are also some examples of simple use cases of the Python API.
Command line¶
Usage: executor [OPTIONS] COMMAND …
Easy subprocess management on the command line based on the Python package with the same name. The “executor” program runs external commands with support for timeouts, dynamic startup delay (fudge factor) and exclusive locking.
You can think of “executor” as a combination of the “flock” and “timelimit” programs with some additional niceties (namely the dynamic startup delay and integrated system logging on UNIX platforms).
Supported options:
Option | Description |
---|---|
-t , --timeout=LIMIT |
Set the time after which the given command will be aborted. By default
LIMIT is counted in seconds. You can also use one of the suffixes “s”
(seconds), “m” (minutes), “h” (hours) or “d” (days). |
-f , --fudge-factor=LIMIT |
This option controls the dynamic startup delay (fudge factor) which is
useful when you want a periodic task to run once per given interval but the
exact time is not important. Refer to the --timeout option for acceptable
values of LIMIT , this number specifies the maximum amount of time to sleep
before running the command (the minimum is zero, otherwise you could just
include the command “sleep N && …” in your command line :-). |
-e , --exclusive |
Use an interprocess lock file to guarantee that executor will never run
the external command concurrently. Refer to the --lock-timeout option
to customize blocking / non-blocking behavior. To customize the name
of the lock file you can use the --lock-file option. |
-T , --lock-timeout=LIMIT |
By default executor tries to claim the lock and if it fails it will exit
with a nonzero exit code. This option can be used to enable blocking
behavior. Refer to the --timeout option for acceptable values of LIMIT . |
-l , --lock-file=NAME |
Customize the name of the lock file. By default this is the base name of the external command, so if you’re running something generic like “bash” or “python” you might want to change this :-). |
-v , --verbose |
Increase logging verbosity (can be repeated). |
-q , --quiet |
Decrease logging verbosity (can be repeated). |
-h , --help |
Show this message and exit. |
Python API¶
Below are some examples of how versatile the execute() function is. Refer to the API documentation on Read the Docs for (a lot of) other use cases.
Checking status codes¶
By default the status code of the external command is returned as a boolean:
>>> from executor import execute
>>> execute('true')
True
If an external command exits with a nonzero status code an exception is raised, this makes it easy to do the right thing (never forget to check the status code of an external command without having to write a lot of repetitive code):
>>> execute('false')
Traceback (most recent call last):
File "executor/__init__.py", line 124, in execute
cmd.start()
File "executor/__init__.py", line 516, in start
self.wait()
File "executor/__init__.py", line 541, in wait
self.check_errors()
File "executor/__init__.py", line 568, in check_errors
raise ExternalCommandFailed(self)
executor.ExternalCommandFailed: External command failed with exit code 1! (command: bash -c false)
The ExternalCommandFailed exception exposes command
and returncode
attributes. If you know a command is likely to exit with a nonzero status code
and you want execute() to simply return a boolean you can do this instead:
>>> execute('false', check=False)
False
Providing input¶
Here’s how you can provide input to an external command:
>>> execute('tr a-z A-Z', input='Hello world from Python!\n')
HELLO WORLD FROM PYTHON!
True
Getting output¶
Getting the output of external commands is really easy as well:
>>> execute('hostname', capture=True)
'peter-macbook'
Running commands as root¶
It’s also very easy to execute commands with super user privileges:
>>> execute('echo test > /etc/hostname', sudo=True)
[sudo] password for peter: **********
True
>>> execute('hostname', capture=True)
'test'
Enabling logging¶
If you’re wondering how prefixing the above command with sudo
would
end up being helpful, here’s how it works:
>>> import logging
>>> logging.basicConfig()
>>> logging.getLogger().setLevel(logging.DEBUG)
>>> execute('echo peter-macbook > /etc/hostname', sudo=True)
DEBUG:executor:Executing external command: sudo bash -c 'echo peter-macbook > /etc/hostname'
Running remote commands¶
To run a command on a remote system using SSH you can use the RemoteCommand class, it works as follows:
>>> from executor.ssh.client import RemoteCommand
>>> cmd = RemoteCommand('localhost', 'echo $SSH_CONNECTION', capture=True)
>>> cmd.start()
>>> cmd.output
'127.0.0.1 57255 127.0.0.1 22'
Running remote commands concurrently¶
The foreach() function wraps the RemoteCommand and CommandPool classes to make it very easy to run a remote command concurrently on a group of hosts:
>>> from executor.ssh.client import foreach
>>> from pprint import pprint
>>> hosts = ['127.0.0.1', '127.0.0.2', '127.0.0.3', '127.0.0.4']
>>> commands = foreach(hosts, 'echo $SSH_CONNECTION')
>>> pprint([cmd.output for cmd in commands])
['127.0.0.1 57278 127.0.0.1 22',
'127.0.0.1 52385 127.0.0.2 22',
'127.0.0.1 49228 127.0.0.3 22',
'127.0.0.1 40628 127.0.0.4 22']
Contact¶
The latest version of executor is available on PyPI and GitHub. The documentation is hosted on Read the Docs and includes a changelog. For bug reports please create an issue on GitHub. If you have questions, suggestions, etc. feel free to send me an e-mail at peter@peterodding.com.
API documentation¶
The following API documentation is automatically generated from the source code:
API documentation¶
The following documentation is based on the source code of version 23.2 of the executor package. The following modules are available:
- The
executor
module - The
executor.chroot
module - The
executor.cli
module - The
executor.concurrent
module - The
executor.contexts
module - The
executor.process
module - The
executor.schroot
module - The
executor.ssh.client
module - The
executor.ssh.server
module - The
executor.tcp
module
The executor
module¶
Core functionality of the executor package.
If you’re looking for an easy way to run external commands from Python take a
look at the execute()
function. When you need more flexibility consider
using the underlying ExternalCommand
class directly instead.
execute()
versus ExternalCommand
¶
In executor
1.x the execute()
function was the only interface
for external command execution. This had several drawbacks:
- The documentation for the
execute()
function was getting way too complex given all of the supported options and combinations. - There was no way to execute asynchronous external commands (running in the
background) without sidestepping the complete
executor
module and going straight forsubprocess.Popen
(with all of the verbosity that you get for free withsubprocess
:-). - There was no way to prepare an external command without starting it immediately, making it impossible to prepare a batch of external commands before starting them (whether synchronously or asynchronously).
To solve these problems executor
2.x introduced the
ExternalCommand
class. This explains why execute()
is now a
trivial wrapper around ExternalCommand
: It’s main purpose is to be an
easy to use shortcut that preserves compatibility with the old interface.
Classes and functions¶
-
executor.
DEFAULT_ENCODING
= 'UTF-8'¶ The default encoding of the standard input, output and error streams (a string).
-
executor.
DEFAULT_WORKING_DIRECTORY
= '.'¶ The default working directory for external commands (a string). Defaults to the working directory of the current process using
os.curdir
.
-
executor.
DEFAULT_SHELL
= 'bash'¶ The default shell used to evaluate shell expressions (a string).
This variable isn’t based on the
$SHELL
environment variable because:- Shells like
sh
,dash
,bash
andzsh
all have their own subtly incompatible semantics. - People regularly use shells like
fish
as their default login shell :-).
At an interactive prompt this is no problem (advanced users have obviously learned to context switch) but when you’re writing source code the last thing you want to worry about is which shell is going to evaluate your commands! The
executor
package expects this shell to support the following features:- The
-c
option to evaluate a shell command provided as a command line argument. - The
-
argument to instruct the shell to read shell commands from its standard input stream and evaluate those.
Apart from these two things nothing else is expected from the default shell so you’re free to customize it if you really want to write your shell commands in
fish
orzsh
syntax :-).- Shells like
-
executor.
COMMAND_NOT_FOUND_CODES
= (2,)¶ Numeric error codes returned when a command isn’t available on the system (a tuple of integers).
-
executor.
COMMAND_NOT_FOUND_STATUS
= 127¶ The exit status used by shells when a command is not found (an integer).
-
executor.
execute
(*command, **options)[source]¶ Execute an external command and make sure it succeeded.
Parameters: - command – All positional arguments are passed on to the constructor
of
ExternalCommand
. - options – All keyword arguments are passed on to the constructor of
ExternalCommand
.
Returns: Refer to
execute_prepared()
.Raises: ExternalCommandFailed
when the command exits with a nonzero exit code (andcheck
isTrue
).If
asynchronous
isTrue
thenexecute()
will automatically start the external command for you usingstart()
(but it won’t wait for it to end). If you want to create anExternalCommand
object instance without immediately starting the external command then you can useExternalCommand
directly.Some examples
By default the status code of the external command is returned as a boolean:
>>> from executor import execute >>> execute('true') True
However when an external command exits with a nonzero status code an exception is raised, this is intended to “make it easy to do the right thing” (never forget to check the status code of an external command without having to write a lot of repetitive code):
>>> execute('false') Traceback (most recent call last): File "executor/__init__.py", line 124, in execute cmd.start() File "executor/__init__.py", line 516, in start self.wait() File "executor/__init__.py", line 541, in wait self.check_errors() File "executor/__init__.py", line 568, in check_errors raise ExternalCommandFailed(self) executor.ExternalCommandFailed: External command failed with exit code 1! (command: false)
What’s also useful to know is that exceptions raised by
execute()
exposecommand
andreturncode
attributes. If you know a command is likely to exit with a nonzero status code and you wantexecute()
to simply return a boolean you can do this instead:>>> execute('false', check=False) False
- command – All positional arguments are passed on to the constructor
of
-
executor.
execute_prepared
(command)[source]¶ The logic behind
execute()
andremote()
.Parameters: command – An ExternalCommand
object (or an object created from a subclass with a compatible interface like for exampleRemoteCommand
).Returns: The return value of this function depends on several options: - If
asynchronous
isTrue
the constructedExternalCommand
object is returned. - If
callback
is set the value ofresult
is returned. - If
capture
isTrue
the value ofExternalCommand.output
is returned. - By default the value of
succeeded
is returned.
Raises: See execute()
andremote()
.- If
-
class
executor.
ExternalCommand
(*command, **options)[source]¶ Programmer friendly
subprocess.Popen
wrapper.The
ExternalCommand
class wrapssubprocess.Popen
to make it easier to do the right thing (the simplicity ofos.system()
with the robustness ofsubprocess.Popen
) and to provide additional features (e.g. asynchronous command execution that preserves the ability to provide input and capture output).- Process manipulation
ExternalCommand
inherits fromControllableProcess
which means that all of the process manipulation supported byControllableProcess
is also supported byExternalCommand
objects.- Context manager
ExternalCommand
objects can be used as context managers by using thewith
statement:- When the scope of the
with
statement starts thestart()
method is called (if the external command isn’t already running). - When the scope of the
with
statement endsterminate()
is called if the command is still running. Theload_output()
andcleanup()
functions are used to cleanup after the external command. If an exception isn’t already being raisedcheck_errors()
is called to make sure the external command succeeded.
- When the scope of the
- Event callbacks
The
start_event
,retry_event
andfinish_event
properties can be set to callbacks (callable values like functions) to subscribe to the corresponding events. The callback receives a single positional argument which is theExternalCommand
object.The
start_event
andfinish_event
properties were originally created for use in command pools, for example to report to the operator when specific commands are started and when they finish. The event handling is performed inside theExternalCommand
class though, so you’re free to repurpose these events outside the context of command pools.
Here’s an overview of the
ExternalCommand
class:You can set the values of the
asynchronous
,buffer_size
,buffered
,callback
,capture
,capture_stderr
,check
,command
,dependencies
,directory
,encoding
,environment
,error_message
,error_type
,fakeroot
,finish_event
,group_by
,input
,ionice
,merge_streams
,really_silent
,retry
,retry_allowed
,retry_count
,retry_event
,retry_limit
,returncode
,shell
,silent
,start_event
,stderr_file
,stdout_file
,subprocess
,sudo
,tty
,uid
,user
,virtual_environment
andwas_started
properties by passing keyword arguments to the class initializer.-
__init__
(*command, **options)[source]¶ Initialize an
ExternalCommand
object.Parameters: - command – Any positional arguments are converted to a list and
used to set
command
. - options – Keyword arguments can be used to conveniently override
the default values of
asynchronous
,callback
,capture
,capture_stderr
,check
,directory
,encoding
,environment
,fakeroot
,input
,logger
,merge_streams
,really_silent
,shell
,silent
,stdout_file
,stderr_file
,uid
,user
,sudo
andvirtual_environment
. Keyword argument that are not supported will raiseTypeError
as usual.
The external command is not started until you call
start()
orwait()
.- command – Any positional arguments are converted to a list and
used to set
-
asynchronous
[source]¶ Enable asynchronous command execution.
If this option is
True
(not the default) preparations are made to execute the external command asynchronously (in the background). This has several consequences:Calling
start()
will start the external command but will not block until the external command is finished, instead you are responsible for callingwait()
at some later point in time.When
input
is set its value will be written to a temporary file and the standard input stream of the external command is connected to read from the temporary file.By using a temporary file the external command can consume its input as fast or slow as it pleases without needing a separate thread or process to “feed” the external command.
When
capture
isTrue
the standard output of the external command is redirected to a temporary file whose contents are read once the external command has finished.By using a temporary file the external command can produce output as fast or slow as it pleases without needing a thread or subprocess on our side to consume the output in real time.
See also
async
(backwards compatible alias)Note
The
asynchronous
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
buffer_size
[source]¶ Control the size of the stdin/stdout/stderr pipe buffers.
The value of
buffer_size
becomes the bufsize argument that’s passed tosubprocess.Popen
bystart()
.If
asynchronous
isTrue
andbuffered
isFalse
the value ofbuffer_size
defaults to 0 which means unbuffered, in all other cases its value defaults to -1 which means to use the system default buffer size.Note
The
buffer_size
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
buffered
[source]¶ Control whether command output is buffered to temporary files.
When
asynchronous
isTrue
and the standard output and/or error streams are being captured, temporary files will be used to collect the output. This enables the use of theoutput
,stdout
andstderr
properties to easily get the full output of the command in a single string.You can set
buffered
toFalse
to disable the use of temporary files, in this casesubprocess.PIPE
is passed tosubprocess.Popen
. Onceis_running
isTrue
you can use thestdin
,stdout
and/orstderr
properties to communicate with the command.This enables runtime processing of the standard input, output and error streams and makes it possible to run commands that never return but keep producing output (for example
xscreensaver-command -watch
). Here’s an example that setsbuffered
toFalse
and uses the magic method__iter__()
to iterate over the lines of output in realtime:# Run external commands when xscreensaver changes state. import os from executor import execute known_states = set(['BLANK', 'LOCK', 'UNBLANK']) while True: options = dict(asynchronous=True, capture=True, buffered=False) with execute('xscreensaver-command', '-watch', **options) as command: for line in command: tokens = line.split() if tokens and tokens[0] in known_states: value = os.environ.get('XSCREENSAVER_%s_COMMAND' % tokens[0]) if value: execute(value)
Some sanity checks and error handling have been omitted from the example above, in order to keep it simple, but I did test it and it should actually work (at least it did for me):
$ export XSCREENSAVER_BLANK_COMMAND='echo $(date) - Screen is now blanked' $ export XSCREENSAVER_LOCK_COMMAND='echo $(date) - Screen is now locked' $ export XSCREENSAVER_UNBLANK_COMMAND='echo $(date) - Screen is now unblanked' $ python xscreensaver-monitor.py Sat Jan 20 16:03:07 CET 2018 - Screen is now blanked Sat Jan 20 16:03:15 CET 2018 - Screen is now unblanked Sat Jan 20 16:03:20 CET 2018 - Screen is now locked Sat Jan 20 16:03:34 CET 2018 - Screen is now unblanked
Note
The
buffered
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
callback
[source]¶ Optional callback used to generate the value of
result
.The
callback
andresult
properties were created for use in command pools, where it can be useful to define how to process (parse) a command’s output when the command is constructed.Note
The
callback
property is awritable_property
. You can change the value of this property using normal attribute assignment syntax.
-
capture
[source]¶ Enable capturing of the standard output stream.
If this option is
True
(not the default) the standard output of the external command is captured and made available to the caller viastdout
andoutput
.The standard error stream will not be captured, use
capture_stderr
for that. You can also silence the standard error stream using thesilent
option.If
callback
is setcapture
defaults toTrue
(but you can still setcapture
toFalse
if that is what you want).Note
The
capture
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
capture_stderr
[source]¶ Enable capturing of the standard error stream.
If this option is
True
(not the default) the standard error stream of the external command is captured and made available to the caller viastderr
.Note
The
capture_stderr
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
check
[source]¶ Enable automatic status code checking.
If this option is
True
(the default) and the external command exits with a nonzero status codeExternalCommandFailed
will be raised bystart()
(whenasynchronous
isn’t set) orwait()
(whenasynchronous
is set).Note
The
check
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
command
[source]¶ A list of strings with the command to execute.
Note
In executor version 14.0 it became valid to set
input
andshell
without providingcommand
(in older versions it was required to setcommand
regardless of the other options).Note
The
command
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
command_line
¶ The command line of the external command.
The command line used to actually run the external command requested by the user (a list of strings). The command line is constructed based on
command
according to the following rules:- If
shell
isTrue
the external command is run usingbash -c '...'
(assuming you haven’t changedDEFAULT_SHELL
) which means constructs like semicolons, ampersands and pipes can be used (and all the usual caveats apply :-). - If
virtual_environment
is set the command is converted to a shell command line and prefixed by the applicablesource ...
command. - If
uid
oruser
is set thesudo -u
command will be prefixed to the command line generated here. - If
fakeroot
orsudo
is set the respective command name is prefixed to the command line generated here (sudo
is only prefixed when the current process doesn’t already have super user privileges). - If
ionice
is set the appropriate command is prefixed to the command line generated here.
- If
-
decoded_stdout
¶ The value of
stdout
decoded usingencoding
.This is a
unicode()
object (in Python 2) or astr
object (in Python 3).
-
decoded_stderr
¶ The value of
stderr
decoded usingencoding
.This is a
unicode()
object (in Python 2) or astr
object (in Python 3).
-
dependencies
[source]¶ The dependencies of the command (a list of
ExternalCommand
objects).The
dependencies
property enables low level concurrency control in command pools by imposing a specific order of execution:- Command pools will never start a command until the
is_finished
properties of all of the command’sdependencies
areTrue
. - If
dependencies
is empty it has no effect and concurrency is controlled bygroup_by
andconcurrency
.
Note
The
dependencies
property is acustom_property
. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached.- Command pools will never start a command until the
-
directory
[source]¶ The working directory for the external command.
A string, defaults to
DEFAULT_WORKING_DIRECTORY
.Note
The
directory
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
encoded_input
¶ The value of
input
encoded usingencoding
.This is a
str
object (in Python 2) or abytes
object (in Python 3).
-
encoding
[source]¶ The character encoding of standard input and standard output.
A string, defaults to
DEFAULT_ENCODING
. This option is used to encodeinput
and to decodeoutput
.Note
The
encoding
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
environment
[source]¶ A dictionary of environment variables for the external command.
You only need to specify environment variables that differ from those of the current process (that is to say the environment variables of the current process are merged with the variables that you specify here).
Note
The
environment
property is acustom_property
. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached.
-
error_message
[source]¶ A string describing how the external command failed or
None
.Note
The
error_message
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
error_type
[source]¶ An appropriate exception class or
None
(when no error occurred).CommandNotFound
if the external command exits with return codeCOMMAND_NOT_FOUND_STATUS
orExternalCommandFailed
if the external command exits with any other nonzero return code.Note
The
error_type
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
failed
¶ Whether the external command has failed.
True
ifreturncode
is a nonzero number orerror_type
is set (e.g. because the external command doesn’t exist).False
ifreturncode
is zero.None
when the external command hasn’t been started or is still running.
-
fakeroot
[source]¶ Run the external command under
fakeroot
.If this option is
True
(not the default) and the current process doesn’t have superuser privileges the external command is run withfakeroot
. If thefakeroot
program is not installed the external command will fail.Note
The
fakeroot
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
finish_event
[source]¶ Optional callback that’s called just after the command finishes (see event callbacks).
Note
The
finish_event
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
group_by
[source]¶ Identifier that’s used to group the external command (any hashable value).
The
group_by
property enables high level concurrency control in command pools by making it easy to control which commands are allowed to run concurrently and which are required to run serially:- Command pools will never start more than one command within a group
of commands that share the same value of
group_by
(for values that aren’tNone
). - If
group_by
isNone
it has no effect and concurrency is controlled bydependencies
andconcurrency
.
Note
The
group_by
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.- Command pools will never start more than one command within a group
of commands that share the same value of
-
have_superuser_privileges
¶ Whether the parent Python process is running under superuser privileges.
True
if running with superuser privileges,False
otherwise. Used bycommand_line
to decide whethersudo
needs to be used.
-
input
[source]¶ The input to feed to the external command on the standard input stream.
When you provide a
unicode()
object (in Python 2) or astr
object (in Python 3) as input it will be encoded usingencoding
. To avoid the automatic conversion you can simply pass astr
object (in Python 2) or abytes
object (in Python 3). This conversion logic is implemented in theencoded_input
attribute.When
input
is set toTrue
a pipe will be created to communicate with the external command in real time. See also thebuffered
andstdin
properties.Defaults to
None
.Note
The
input
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
ionice
[source]¶ The I/O scheduling class for the external command (a string or
None
).When this property is set then ionice will be used to set the I/O scheduling class for the external command. This can be useful to reduce the impact of heavy disk operations on the rest of the system.
Raises: Any exceptions raised by validate_ionice_class()
.Note
The
ionice
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
is_finished
¶ Whether the external command has finished execution (excluding retries).
True
once the external command has been started and has since finished (excluding retries),False
when the external command hasn’t been started yet or is still running.
-
is_finished_with_retries
¶ Whether the external command has finished execution (including retries).
True
once the external command has been started and has since finished (including retries),False
when the external command hasn’t been started yet, is still running or can be retried.
-
is_terminated
¶ Whether the external command has been terminated (a boolean).
True
if the external command was terminated usingSIGTERM
(e.g. byterminate()
),False
otherwise.
-
merge_streams
[source]¶ Whether to merge the standard output and error streams.
A boolean, defaults to
False
. If this option is enabledstdout
will contain the external command’s output on both streams.Note
The
merge_streams
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
output
¶ The value of
stdout
decoded usingencoding
.This is a
unicode()
object (in Python 2) or astr
object (in Python 3).This is only available when
capture
isTrue
. Ifcapture
is notTrue
thenoutput
will beNone
.After decoding any leading and trailing whitespace is stripped and if the resulting string doesn’t contain any remaining newlines then the string with leading and trailing whitespace stripped will be returned, otherwise the decoded string is returned unchanged:
>>> from executor import ExternalCommand >>> cmd = ExternalCommand('echo naïve', capture=True) >>> cmd.start() >>> cmd.output u'na\xefve' >>> cmd.stdout 'na\xc3\xafve\n'
This is intended to make simple things easy (
output
makes it easy to deal with external commands that output a single line) while providing an escape hatch when the default assumptions don’t hold (you can always usestdout
to get the raw output).See also the
__iter__()
magic method which makes it very easy to iterate over the lines of output produced by the command.
-
really_silent
[source]¶ Whether output is really silenced or actually captured (a boolean).
When the
silent
option was originally added to executor it was implemented by redirecting the output streams toos.devnull
, similar to howcommand &> /dev/null
works in Bash.Since I made that decision I’ve regretted it many times because I ran into situations where
check
andsilent
were both set andExternalCommandFailed
was raised but I had no way to determine what had gone wrong.This is why the
really_silent
property was introduced in executor release 19.0:- When
silent
isTrue
andcheck
isFalse
the value ofreally_silent
will beTrue
, otherwise it isFalse
. - When
really_silent
isFalse
(becausecheck
isTrue
) thesilent
property effectively becomes an alias forcapture
andcapture_stderr
which means the output on both streams is captured instead of discarded. - Because output is captured instead of discarded the output of
failing commands can be reported by
ExternalCommandFailed
(which is raised becausecheck
isTrue
).
This change was made after much consideration because it is backwards incompatible and not only in a theoretical sense: Imagine a daemon process spewing megabytes of log output on its standard error stream.
As an escape hatch to restore backwards compatibility you can set
really_silent
toTrue
to override the computed value.Note
The
really_silent
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.- When
-
result
¶ The result of calling the value given by
callback
.If the command hasn’t been started yet
start()
is called. When the command hasn’t finished yet func:wait() is called. Ifcallback
isn’t setNone
is returned.
-
retry
[source]¶ Whether the external command should be retried when it fails (a boolean, defaults to
False
).Warning
Retrying of failing commands is an experimental feature that was introduced with the release of executor 20.0. Please refer to the 20.0 release notes for details.
Note
The
retry
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
retry_allowed
[source]¶ True
if the external command can be retried,False
otherwise.The value of this property is computed by checking if the following conditions hold:
retry
isTrue
,failed
isTrue
,returncode
is notCOMMAND_NOT_FOUND_STATUS
,retry_count
is lower thanretry_limit
(only ifretry_limit
is not zero).
Note that when the
retry_event
callback returnsFalse
to cancel the retrying of a failed command, the computed value ofretry_allowed
is overridden by assigningretry_allowed
the valueFalse
.Note
The
retry_allowed
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
retry_count
[source]¶ The number of times that the command was retried (an integer number, defaults to 0).
The value of
retry_count
is automatically incremented bystart_once()
when it notices thatwas_started
isTrue
beforestart_once()
has started the command.Note
The
retry_count
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
retry_event
[source]¶ Optional callback that’s called when a command is retried (see event callbacks).
The callback can return
False
to abort retrying.Note
The
retry_event
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
retry_limit
[source]¶ The maximum number of times to retry the command when it fails (an integer, defaults to 2).
Given the default value of two, when
retry
isTrue
the command will be run at most three times (the initial run and two retries). The value 0 means the command will be retried until it succeeds.Note
The
retry_limit
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
returncode
[source]¶ The return code of the external command (an integer) or
None
.This will be
None
until the external command has finished.Note
The
returncode
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
shell
[source]¶ Whether to evaluate the external command as a shell command.
A boolean, the default depends on the value of
command
:- If
command
contains a single stringshell
defaults toTrue
. - If
command
contains more than one stringshell
defaults toFalse
.
When
shell
isTrue
the external command is evaluated by the shell given byDEFAULT_SHELL
, otherwise the external command is run without shell evaluation.Note
The
shell
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.- If
-
silent
[source]¶ Whether the external command’s output should be silenced (a boolean).
If this is
True
(not the default) any output of the external command is silenced by redirecting the output streams toos.devnull
(ifreally_silent
isTrue
) or by capturing the output (ifreally_silent
isFalse
).You can enable
capture
andsilent
together to capture the standard output stream while silencing the standard error stream.Note
The
silent
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
start_event
[source]¶ Optional callback that’s called just before the command is started (see event callbacks).
Note
The
start_event
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
stderr
¶ The standard error stream of the external command.
This property is only available when
capture_stderr
isTrue
. In all other cases the value ofstderr
will beNone
.When
buffered
isTrue
(the default) this is astr
object (in Python 2) or abytes
object (in Python 3).If you set
buffered
toFalse
thenstderr
will be a pipe that’s connected to the standard error stream of the command (for as long asis_running
isTrue
).
-
stderr_file
[source]¶ Capture the standard error stream to the given file handle.
When this property is set to a writable file object the standard error stream of the external command is redirected to the given file. The default value of this property is
None
.This can be useful to (semi) permanently store command output or to run commands whose output is hidden but can be followed using tail -f if the need arises. By setting
stdout_file
andstderr_file
to the same file object the output from both streams can be merged and redirected to the same file. This accomplishes roughly the same thing as settingmerge_streams
but leaves the caller in control of the file.If this property isn’t set but
capture
isTrue
the external command’s output is captured to a temporary file that’s automatically cleaned up after the external command is finished and its output has been cached (read into memory).Note
The
stderr_file
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
stdin
¶ The standard input stream of the external command.
If you set
input
toTrue
andbuffered
toFalse
thenstdin
will be a pipe that’s connected to the standard input stream of the command (for as long asis_running
isTrue
).
-
stdout
¶ The standard output stream of the external command.
This property is only available when
capture
isTrue
. In all other cases the value ofstdout
will beNone
.When
buffered
isTrue
(the default) this is astr
object (in Python 2) or abytes
object (in Python 3).If you set
buffered
toFalse
thenstdout
will be a pipe that’s connected to the standard output stream of the command (for as long asis_running
isTrue
).
-
stdout_file
[source]¶ Capture the standard output stream to the given file handle.
When this property is set to a writable file object the standard output stream of the external command is redirected to the given file. The default value of this property is
None
.This can be useful to (semi) permanently store command output or to run commands whose output is hidden but can be followed using tail -f if the need arises. By setting
stdout_file
andstderr_file
to the same file object the output from both streams can be merged and redirected to the same file. This accomplishes roughly the same thing as settingmerge_streams
but leaves the caller in control of the file.If this property isn’t set but
capture
isTrue
the external command’s output is captured to a temporary file that’s automatically cleaned up after the external command is finished and its output has been cached (read into memory).Note
The
stdout_file
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
subprocess
[source]¶ A
subprocess.Popen
object orNone
.The value of this property is set by
start()
and it’s cleared bywait()
(throughcleanup()
) as soon as the external command has finished. This enables garbage collection of the resources associated with thesubprocess.Popen
object which helps to avoid IOError: [Errno 24] Too many open files errors.Note
The
subprocess
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
succeeded
¶ Whether the external command succeeded.
True
ifreturncode
is zero.False
ifreturncode
is a nonzero number orerror_type
is set (e.g. because the external command doesn’t exist).None
when the external command hasn’t been started or is still running.
-
sudo
[source]¶ Whether
sudo
should be used to gain superuser privileges.If this option is
True
(not the default) and the current process doesn’t have superuser privileges the external command is run withsudo
to ensure that the external command runs with superuser privileges.The use of this option assumes that the
sudo
command is available.Note
The
sudo
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
sudo_command
¶ The
sudo
command used to change privileges (a list of strings).This option looks at the
sudo
,uid
anduser
properties to decide whethercommand
should be run usingsudo
or not. If it should then a prefix forcommand
is constructed fromsudo
,uid
,user
and/orenvironment
and returned. Some examples:>>> from executor import ExternalCommand >>> ExternalCommand('true', sudo=True).sudo_command ['sudo'] >>> ExternalCommand('true', uid=1000).sudo_command ['sudo', '-u', '#1000'] >>> ExternalCommand('true', user='peter').sudo_command ['sudo', '-u', 'peter'] >>> ExternalCommand('true', user='peter', environment=dict(gotcha='this is tricky')).sudo_command ['sudo', '-u', 'peter', 'gotcha=this is tricky']
-
tty
[source]¶ Whether the command will be attached to the controlling terminal (a boolean).
capture
isFalse
capture_stderr
isFalse
input
isNone
silent
isFalse
stdout_file
andstderr_file
areNone
or files that are connected to a tty(-like) device
If any of these conditions don’t hold
tty
defaults toFalse
. Whentty
isFalse
the standard input stream of the external command will be connected toos.devnull
.Note
The
tty
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
uid
[source]¶ The user ID of the system user that’s used to run the command.
If this option is set to an integer number (it defaults to
None
) the external command is prefixed withsudo -u #UID
to run the command as a different user than the current user.The use of this option assumes that the
sudo
command is available.Note
The
uid
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
user
[source]¶ The name of the system user that’s used to run the command.
If this option is set to a string (it defaults to
None
) the external command is prefixed withsudo -u USER
to run the command as a different user than the current user.The use of this option assumes that the
sudo
command is available.Note
The
user
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
virtual_environment
[source]¶ The Python virtual environment to activate before running the command.
If this option is set to the directory of a Python virtual environment (a string) then the external command will be prefixed by a source shell command that evaluates the
bin/activate
script in the Python virtual environment before executing the user defined external command.Note
The
virtual_environment
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
was_started
[source]¶ Whether the external command has already been started.
True
oncestart()
has been called to start executing the external command,False
whenstart()
hasn’t been called yet.Note
The
was_started
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
async_fdel
()[source]¶ Delete the value of
asynchronous
.
-
async_fget
()[source]¶ Get the value of
asynchronous
.
-
async_fset
(value)[source]¶ Set the value of
asynchronous
.
-
check_retry_allowed
()[source]¶ Check if retrying is allowed by invoking the
retry_event
callback.Returns: True
ifretry_allowed
isTrue
and theretry_event
callback didn’t returnFalse
, otherwiseFalse
.
-
format_error_message
(message, *args, **kw)[source]¶ Add the command’s captured standard output and/or error to an error message.
Refer to
compact()
for details on argument handling. Theget_decoded_output()
method is used to try to decode the output without raising exceptions.
-
get_decoded_output
(name)[source]¶ Try to decode the output captured on standard output or error.
Parameters: name – One of the strings ‘stdout’ or ‘stderr’. Returns: A Unicode string, byte string or None
.
-
prefix_shell_command
(preamble, command)[source]¶ Prefix a shell command to a command line.
Parameters: - preamble – Any shell command (a string).
- command – The command line (a string, tuple or list).
Returns: The command line to run the combined commands through a shell (a list of strings).
This function uses
reduce_shell_command()
to convert command into a string and then prefixes the preamble to the command, delimited by&&
.
-
reduce_shell_command
(command)[source]¶ Reduce a command line to a shell command.
Parameters: command – The command line (a string, tuple or list). Returns: The shell command (a string). If the given command is a
DEFAULT_SHELL
invocation that uses the-c
option it is reduced to the argument of the-c
option. All other command lines are simply quoted and returned.This method is used in various places where a command needs to be transformed into a shell command so that a command like
cd
orsource
can be prefixed to the command line.
-
start
()[source]¶ Start execution of the external command (including
retry
support).Raises: ExternalCommandFailed
whencheck
isTrue
,asynchronous
isFalse
and the external command exits with a nonzero status code.ValueError
when the external command is still running (you need to explicitly terminate, kill or wait for the running process before you can re-use theExternalCommand
object).
This method instantiates a
subprocess.Popen
object based on the defaults defined byExternalCommand
and the overrides configured by the caller. What happens then depends onasynchronous
:- If
asynchronous
is setstart()
starts the external command but doesn’t wait for it to end (usewait()
for that). - If
asynchronous
isn’t set thecommunicate()
method on thesubprocess
object is called to synchronously execute the external command.
-
start_once
(check=None, **kw)[source]¶ Start execution of the external command (excluding
retry
support).Parameters: - check – Override the value of
check
for the duration of this call tostart_once()
. Defaults toNone
. - kw – The keyword arguments to the
subprocess.Popen
initializer (prepared bystart()
).
The code in this internal method used to be the second half of
start()
but was extracted into a separate method so that it can be called more than once, which made it possible to addretry
support.- check – Override the value of
-
wait
(check=None, **kw)[source]¶ Wait for the external command to finish.
Parameters: - check – Override the value of
check
for the duration of this call towait()
. Defaults toNone
which meanscheck
is not overridden. - kw – Any keyword arguments are passed on to
wait_for_process()
.
Raises: ExternalCommandFailed
whencheck
isTrue
,asynchronous
isTrue
and the external command exits with a nonzero status code.The
wait()
function is only useful whenasynchronous
isTrue
, it performs the following steps:- If
was_started
isFalse
thestart()
method is called. - If
is_running
isTrue
thewait_for_process()
method is called to wait for the child process to end. - If
subprocess
isn’tNone
thecleanup()
method is called to wait for the external command to end, load its output into memory and release the resources associated with thesubprocess
object. - Finally
check_errors()
is called (in case the caller didn’t disablecheck
).
- check – Override the value of
-
terminate_helper
()[source]¶ Gracefully terminate the process.
Raises: Any exceptions raised by the subprocess
module.This method sets
check
toFalse
, the idea being that if you consciously terminate a command you don’t need to be bothered with an exception telling you that you succeeded :-).
-
kill_helper
()[source]¶ Forcefully kill the process.
Raises: Any exceptions raised by the subprocess
module.This method sets
check
toFalse
, the idea being that if you consciously kill a command you don’t need to be bothered with an exception telling you that you succeeded :-).
-
load_output
()[source]¶ Load output captured from the standard output/error streams.
Reads the contents of the temporary file(s) created by
start()
(whenasynchronous
andcapture
are both set) into memory so that the output doesn’t get lost when the temporary file is cleaned up bycleanup()
.
-
cleanup
()[source]¶ Clean up after the external command has ended.
This internal method is called by methods like
start()
andwait()
to clean up the following temporary resources:- The temporary file(s) used to to buffer the external command’s
input
,stdout
andstderr
(only whenasynchronous
isTrue
). - File handles to the previously mentioned temporary files and
os.devnull
(used to implement thesilent
option). - The reference to the
subprocess.Popen
object stored insubprocess
. By destroying this reference as soon as possible we enable the object to be garbage collected and its related resources to be released.
- The temporary file(s) used to to buffer the external command’s
-
check_errors
(check=None)[source]¶ Raise an exception if the external command failed.
This raises
error_type
whencheck
is set and the external command failed.Parameters: check – Override the value of check
for the duration of this call. Defaults toNone
which meanscheck
is not overridden.Raises: error_type
whencheck
is set anderror_type
is notNone
.This internal method is used by
start()
andwait()
to make sure that failing external commands don’t go unnoticed.
-
invoke_event_callback
(name)[source]¶ Invoke one of the event callbacks.
Parameters: name – The name of the callback (a string). Returns: The return value of the callback.
-
__enter__
()[source]¶ Start the external command if it hasn’t already been started.
Returns: The ExternalCommand
object.When you use an
ExternalCommand
as a context manager in thewith
statement, the command is automatically started when entering the context and terminated when leaving the context.If the proces hasn’t already been started yet
asynchronous
is automatically set toTrue
(if it’s not alreadyTrue
), otherwise the command will have finished execution by the time the body of thewith
statement is executed (which isn’t really all that useful :-).
-
__exit__
(exc_type=None, exc_value=None, traceback=None)[source]¶ Automatically terminate and clean up after the external command.
Terminates the external command if it is still running (using
terminate()
), cleans up (usingcleanup()
) and checks for errors (usingcheck_errors()
, only if an exception is not already being handled).
-
__iter__
()[source]¶ Iterate over the lines of text in the captured output.
Returns: An iterator of Unicode strings ( unicode()
objects in Python 2 orstr
objects in Python 3).If
capture
isTrue
this will iterate over the lines instdout
, alternatively ifcapture_stderr
isTrue
this will iterate over the lines instderr
instead. In both cases the output will be decoded usingencoding
.If
buffered
isTrue
the captured output will be split into a list of strings which is then iterated. If it isFalse
then the following idiom is used to create an iterator instead:iter(handle.readline, b'')
To understand why this is useful, consider the following:
- Iteration over file-like objects in Python uses a hidden read-ahead buffer for efficiency. Refer to the file.next() documentation for details.
- Pipes are file-like objects so if you iterate over them but the command doesn’t emit a lot of output, the iteration may not produce any lines until the hidden read-ahead buffer is full. This makes realtime processing of command output harder than it should be.
-
__str__
()[source]¶ Render a human friendly string representation of the external command.
This special method calls
quote()
oncommand_line
which is used byexecutor
to enable lazy formatting of log messages containing the quoted command line.
-
async
¶ An alias for the
asynchronous
property.In release 21.0 the
async
property was renamed toasynchronous
because Python 3.7 introduces theasync
keyword, invalidating its use as an identifier. This alias ensures backwards compatibility with callers that are still using the oldasync
name. If you’re wondering which of the two properties you should use:- If Python >= 3.7 compatibility is important to you then your only
choice is
asynchronous
. - If you’re working in a Python < 3.7 code base you can pick either one of the properties, it really doesn’t matter.
- Of course most Python 2 code bases will eventually need to be
upgraded to Python 3 and the future proof choice is
asynchronous
.
- If Python >= 3.7 compatibility is important to you then your only
choice is
-
class
executor.
CachedStream
(command, kind)[source]¶ Manages a temporary file with input for / output from an external command.
-
__init__
(command, kind)[source]¶ Initialize a
CachedStream
object.Parameters: - command – The
ExternalCommand
object that’s using theCachedStream
object. - kind – A simple (alphanumeric) string with the name of the stream.
- command – The
-
prepare_input
()[source]¶ Initialize the input stream.
Returns: A value that can be passed to the constructor of subprocess.Popen
as thestdin
argument.
-
prepare_output
(file, capture)[source]¶ Initialize an (asynchronous) output stream.
Parameters: Returns: A value that can be passed to the constructor of
subprocess.Popen
as thestdout
and/orstderr
argument.
-
redirect
(obj)[source]¶ Capture the stream in a file provided by the caller.
Parameters: obj – A file-like object that has an associated file descriptor.
-
load
()[source]¶ Load the stream’s contents from the temporary file.
Returns: The output of the stream (a string) or None
when the stream was never initialized.
-
-
executor.
quote
(*args)[source]¶ Quote a string or a sequence of strings to be used as command line argument(s).
This function is a simple wrapper around
pipes.quote()
which adds support for quoting sequences of strings (lists and tuples). For example the following calls are all equivalent:>>> from executor import quote >>> quote('echo', 'argument with spaces') "echo 'argument with spaces'" >>> quote(['echo', 'argument with spaces']) "echo 'argument with spaces'" >>> quote(('echo', 'argument with spaces')) "echo 'argument with spaces'"
Parameters: args – One or more strings, tuples and/or lists of strings to be quoted. Returns: A string containing quoted command line arguments.
-
executor.
which
(program, mode=1, path=None)[source]¶ Find the pathname(s) of a program on the executable search path (
$PATH
).Parameters: program – The name of the program (a string). Returns: A list of pathnames (strings) with found programs. Some examples:
>>> from executor import which >>> which('python') ['/home/peter/.virtualenvs/executor/bin/python', '/usr/bin/python'] >>> which('vim') ['/usr/bin/vim'] >>> which('non-existing-program') []
-
executor.
get_search_path
(path=None)[source]¶ Get the executable search path (
$PATH
).Parameters: path – Override the value of $PATH
(a string orNone
).Returns: A list of strings with pathnames of directories. The executable search path is constructed as follows:
- The search path is taken from the environment variable
$PATH
. - If
$PATH
isn’t defined the value ofos.defpath
is used. - The search path is split on
os.pathsep
to get a list. - On Windows the current directory is prepended to the list.
- Duplicate directories are removed from the list.
- The search path is taken from the environment variable
-
executor.
get_path_extensions
(extensions=None)[source]¶ Get the executable search path extensions (
$PATHEXT
).Returns: A list of strings with unique path extensions (on Windows) or a list containing an empty string (on other platforms).
-
executor.
is_executable
(filename, mode=1)[source]¶ Check whether the given file is executable.
Parameters: filename – A relative or absolute pathname (a string). Returns: True
if the file is executable,False
otherwise.
-
executor.
validate_ionice_class
(value)[source]¶ Ensure that the given value is a valid I/O scheduling class for ionice.
Parameters: value – The value to validate (a string). Returns: The validated value (one of the strings ‘1’, ‘2’, ‘3’, ‘idle’, ‘best-effort’, or ‘realtime’). Raises: ValueError
when the given value isn’t one of the strings mentioned above.The strings ‘idle’, ‘best-effort’ and ‘realtime’ are preferred for readability but not supported in minimalistic environments like busybox which only support the values ‘1’, ‘2’ and ‘3’ (refer to #16). It’s up to the caller to choose the correct value, no translation is done.
-
exception
executor.
ExternalCommandFailed
(command, **options)[source]¶ Raised when an external command exits with a nonzero status code.
This exception is raised by
execute()
,start()
andwait()
when an external command exits with a nonzero status code.Here’s an overview of the
ExternalCommandFailed
class:Superclasses: PropertyManager
andException
Special methods: __init__()
Properties: command
,error_message
,pool
andreturncode
When you initialize a
ExternalCommandFailed
object you are required to provide a value for thecommand
property. You can set the values of thecommand
,error_message
andpool
properties by passing keyword arguments to the class initializer.-
__init__
(command, **options)[source]¶ Initialize an
ExternalCommandFailed
object.Parameters: - command – The
ExternalCommand
object that triggered the exception. - options – Any keyword arguments are passed on to the initializer
of the base class
PropertyManager
to initialize the writable propertiespool
anderror_message
.
- command – The
-
command
[source]¶ The
ExternalCommand
object that triggered the exception.
-
pool
[source]¶ The
CommandPool
object that triggered the exception.This property will be
None
when the exception wasn’t raised from a command pool.
-
returncode
¶ Shortcut for the external command’s
returncode
.
-
error_message
[source]¶ An error message explaining what went wrong (a string).
Defaults to
error_message
but can be overridden using the keyword argument of the same name to__init__()
.
-
-
exception
executor.
CommandNotFound
(command, **options)[source]¶ Raised when an external command is not available on the system.
This exception is raised by
execute()
,start()
andwait()
when an external command can’t be started because the command isn’t available.It inherits from
ExternalCommandFailed
to enable uniform error handling but it also inherits fromOSError
for backwards compatibility (seeerrno
andstrerror
).Here’s an overview of the
CommandNotFound
class:Superclasses: ExternalCommandFailed
andOSError
Properties: errno
andstrerror
The executor.chroot
module¶
Simple command execution in chroot environments.
The executor.chroot
module defines the ChangeRootCommand
class
which makes it easy to run commands inside chroots. If this doesn’t mean
anything to you, here’s a primer:
- The word ‘chroot’ is an abbreviation of the phrase ‘change root’ which is
Unix functionality that changes the ‘root’ directory (
/
) of a running process, so ‘change root’ describes an action. - Even though the phrase ‘change root’ describes an action the word ‘chroot’ is also used to refer to the directory which serves as the new root directory (English is flexible like that :-). This is why it makes sense to say that “you’re entering the chroot”.
Warning
This is low level functionality. This module performs absolutely
no chroot initialization, for example /etc/resolv.conf
may be
incorrect and there won’t be any bind mounts available in the
chroot (unless you’ve prepared them yourself).
If you need your chroot to be initialized for you then consider using the
executor.schroot
module instead. It takes a bit of time to set up
schroot but it provides a more high level experience than chroot.
-
class
executor.chroot.
ChangeRootCommand
(*args, **options)[source]¶ ChangeRootCommand
objects use the chroot program to execute commands inside chroots.Here’s an overview of the
ChangeRootCommand
class:Superclass: ExternalCommand
Special methods: __init__()
Properties: chroot
,chroot_command
,chroot_directory
,chroot_group
,chroot_user
,command_line
,directory
andhave_superuser_privileges
When you initialize a
ChangeRootCommand
object you are required to provide a value for thechroot
property. You can set the values of thechroot
,chroot_command
,chroot_directory
,chroot_group
andchroot_user
properties by passing keyword arguments to the class initializer.-
__init__
(*args, **options)[source]¶ Initialize a
ChangeRootCommand
object.Parameters: - args – Positional arguments are passed on to the initializer of
the
ExternalCommand
class. - options – Any keyword arguments are passed on to the initializer
of the
ExternalCommand
class.
If the keyword argument chroot isn’t given but positional arguments are provided, the first positional argument is used to set the
chroot
property.The command is not started until you call
start()
orwait()
.- args – Positional arguments are passed on to the initializer of
the
-
chroot
[source]¶ The pathname of the root directory of the chroot (a string).
Note
The
chroot
property is arequired_property
. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.
-
chroot_command
[source]¶ The command used to run the
chroot
program.This is a list of strings, by default the list contains just
CHROOT_PROGRAM_NAME
. Thechroot
,chroot_group
andchroot_user
properties also influence the chroot command line used.Note
The
chroot_command
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
chroot_directory
[source]¶ The working directory _inside the chroot (a string, defaults to
None
).Note
The
chroot_directory
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
chroot_group
[source]¶ The name or ID of the system group that runs the command (a string or number, defaults to ‘root’).
Note
The
chroot_group
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
chroot_user
[source]¶ The name or ID of the system user that runs the command (a string or number, defaults to ‘root’).
Note
The
chroot_user
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
command_line
¶ The complete chroot command including the command to run inside the chroot.
This is a list of strings with the chroot command line to enter the requested chroot and execute
command
.
-
directory
¶ Set the working directory inside the chroot.
When you set this property you change
chroot_directory
, however reading back the property you’ll just getDEFAULT_WORKING_DIRECTORY
. This is because the superclassExternalCommand
usesdirectory
as the working directory for the chroot command, and directories inside chroots aren’t guaranteed to exist on the host system.
-
have_superuser_privileges
¶ Whether the command inside the chroot will be running under superuser privileges.
This is
True
whenchroot_user
is the number 0 or the string ‘root’,False
otherwise.This overrides
ExternalCommand.have_superuser_privileges
to ensure thatsudo
isn’t used inside the chroot unless it’s really necessary. This is important because not all chroot environments havesudo
installed and failing due to a lack ofsudo
when we’re already running asroot
is just stupid :-).
-
The executor.cli
module¶
Usage: executor [OPTIONS] COMMAND …
Easy subprocess management on the command line based on the Python package with the same name. The “executor” program runs external commands with support for timeouts, dynamic startup delay (fudge factor) and exclusive locking.
You can think of “executor” as a combination of the “flock” and “timelimit” programs with some additional niceties (namely the dynamic startup delay and integrated system logging on UNIX platforms).
Supported options:
Option | Description |
---|---|
-t , --timeout=LIMIT |
Set the time after which the given command will be aborted. By default
LIMIT is counted in seconds. You can also use one of the suffixes “s”
(seconds), “m” (minutes), “h” (hours) or “d” (days). |
-f , --fudge-factor=LIMIT |
This option controls the dynamic startup delay (fudge factor) which is
useful when you want a periodic task to run once per given interval but the
exact time is not important. Refer to the --timeout option for acceptable
values of LIMIT , this number specifies the maximum amount of time to sleep
before running the command (the minimum is zero, otherwise you could just
include the command “sleep N && …” in your command line :-). |
-e , --exclusive |
Use an interprocess lock file to guarantee that executor will never run
the external command concurrently. Refer to the --lock-timeout option
to customize blocking / non-blocking behavior. To customize the name
of the lock file you can use the --lock-file option. |
-T , --lock-timeout=LIMIT |
By default executor tries to claim the lock and if it fails it will exit
with a nonzero exit code. This option can be used to enable blocking
behavior. Refer to the --timeout option for acceptable values of LIMIT . |
-l , --lock-file=NAME |
Customize the name of the lock file. By default this is the base name of the external command, so if you’re running something generic like “bash” or “python” you might want to change this :-). |
-v , --verbose |
Increase logging verbosity (can be repeated). |
-q , --quiet |
Decrease logging verbosity (can be repeated). |
-h , --help |
Show this message and exit. |
-
executor.cli.
LOCKS_DIRECTORY
= '/var/lock'¶ The pathname of the preferred directory for lock files (a string).
Refer to
get_lock_path()
for more details.
-
executor.cli.
INTERRUPT_FILE
= 'executor-fudge-factor-interrupt'¶ The base name of the file used to interrupt the fudge factor (a string).
-
executor.cli.
apply_fudge_factor
(fudge_factor)[source]¶ Apply the requested scheduling fudge factor.
Parameters: fudge_factor – The maximum number of seconds to sleep (a number). Previous implementations of the fudge factor interrupt used UNIX signals (specifically
SIGUSR1
) but the use of this signal turned out to be sensitive to awkward race conditions and it wasn’t very cross platform, so now the creation of a regular file is used to interrupt the fudge factor.
-
executor.cli.
get_lock_path
(lock_name)[source]¶ Get a pathname that can be used for an interprocess lock.
Parameters: lock_name – The base name for the lock file (a string). Returns: An absolute pathname (a string).
-
executor.cli.
run_command
(arguments, timeout=None)[source]¶ Run the specified command (with an optional timeout).
Parameters: - arguments – The command line for the external command (a list of strings).
- timeout – The optional command timeout (a number or
None
).
Raises: CommandTimedOut
if the command times out.
-
exception
executor.cli.
CommandTimedOut
(command, timeout)[source]¶ Raised when a command exceeds the given timeout.
Here’s an overview of the
CommandTimedOut
class:Superclass: ExternalCommandFailed
Special methods: __init__()
-
__init__
(command, timeout)[source]¶ Initialize a
CommandTimedOut
object.Parameters: - command – The command that timed out (an
ExternalCommand
object). - timeout – The timeout that was exceeded (a number).
- command – The command that timed out (an
-
The executor.concurrent
module¶
Support for concurrent external command execution.
The executor.concurrent
module defines the CommandPool
class
which makes it easy to prepare a large number of external commands, group them
together in a pool, start executing a configurable number of external commands
simultaneously and wait for all external commands to finish. For fine grained
concurrency control please refer to the dependencies
and group_by
properties of the
ExternalCommand
class.
-
class
executor.concurrent.
CommandPool
(concurrency=None, **options)[source]¶ Execute multiple external commands concurrently.
After constructing a
CommandPool
instance you add commands to it usingadd()
and when you’re ready to run the commands you callrun()
.Here’s an overview of the
CommandPool
class:Superclass: PropertyManager
Special methods: __init__()
Public methods: add()
,collect()
,get_spinner()
,run()
,spawn()
andterminate()
Properties: concurrency
,delay_checks
,is_finished
,logger
,logs_directory
,num_commands
,num_failed
,num_finished
,num_running
,results
,running_groups
,spinner
andunexpected_failures
You can set the values of the
concurrency
,delay_checks
,logger
,logs_directory
andspinner
properties by passing keyword arguments to the class initializer.-
__init__
(concurrency=None, **options)[source]¶ Initialize a
CommandPool
object.Parameters: - concurrency – Override the value of
concurrency
. - logs_directory – Override the value of
logs_directory
.
- concurrency – Override the value of
-
concurrency
[source]¶ The number of external commands that the pool is allowed to run simultaneously.
This is a positive integer number. It defaults to the return value of
multiprocessing.cpu_count()
(which may not make much sense if your commands are I/O bound instead of CPU bound).Setting
concurrency
to one is a supported use case intended to make it easier for users of theexecutor.concurrent
module to reuse the code they’ve built on top of command pools (if only for debugging, but there are lots of use cases :-).Note
The
concurrency
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
delay_checks
[source]¶ Whether to postpone raising an exception until all commands have run (a boolean).
If this option is
True
(not the default) and a command withcheck
set toTrue
fails the command pool’s execution is not aborted, instead all commands will be allowed to run. After all commands have finished aCommandPoolFailed
exception will be raised that tells you which command(s) failed.Note
The
delay_checks
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
logger
[source]¶ The
logging.Logger
object to use.If you are using Python’s
logging
module and you find it confusing that command pool execution is logged under theexecutor.concurrent
name space instead of the name space of the application or library usingexecutor
you can set this attribute to inject a custom (and more appropriate) logger.Note
The
logger
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
logs_directory
[source]¶ The pathname of a directory where captured output is stored (a string).
If this property is set to the pathname of a directory (before any external commands have been started) the merged output of each external command is captured and stored in a log file in this directory. The directory will be created if it doesn’t exist yet.
Output will start appearing in the log files before the external commands are finished, this enables tail -f to inspect the progress of commands that are still running and emitting output.
Note
The
logs_directory
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
num_commands
¶ The number of commands in the pool (an integer).
-
num_finished
¶ The number of commands in the pool that have already finished, including retries (an integer).
-
num_failed
¶ The number of commands in the pool that failed (an integer).
-
num_running
¶ The number of currently running commands in the pool (an integer).
-
running_groups
¶ A set of running command groups.
The value of
running_groups
is aset
with thegroup_by
values of all currently running commands (None
is never included in the set).
-
results
¶ A mapping of identifiers to external command objects.
This is a dictionary with external command identifiers as keys (refer to
add()
) andExternalCommand
objects as values. TheExternalCommand
objects provide access to the return codes and/or output of the finished commands.
-
spinner
[source]¶ Control if and how an animated spinner is shown when the command pool is active.
The following values are supported:
- The default value
None
means “auto detect”, which means the spinner is shown only when the process is connected to a terminal. - The value
True
unconditionally enables the spinner. - The value
False
unconditionally disables the spinner. - A
Spinner
object can be provided by the caller, giving them the chance to configure how the spinner behaves.
Note
The
spinner
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.- The default value
-
unexpected_failures
¶ A list of
ExternalCommand
objects that failed unexpectedly.The resulting list includes only commands where
check
andfailed
are bothTrue
.
-
add
(command, identifier=None, log_file=None)[source]¶ Add an external command to the pool of commands.
Parameters: - command – The external command to add to the pool (an
ExternalCommand
object). - identifier – A unique identifier for the external command (any value). When this parameter is not provided the identifier is set to the number of commands in the pool plus one (i.e. the first command gets id 1).
- log_file – Override the default log file name for the command
(the identifier with
.log
appended) in caselogs_directory
is set.
When a command is added to a command pool the following options are changed automatically:
- The
asynchronous
property is set toTrue
. If you want the commands to execute with a concurrency of one then you should setconcurrency
to one. - The
tty
property is set toFalse
whenconcurrency
is higher than one because interaction with multiple concurrent subprocesses in a single terminal is prone to serious miscommunication (when multiple subprocesses present an interactive prompt at the same time and the user tries to answer one of the prompts it will be impossible to tell which of the subprocesses will receive the user’s reply).
- command – The external command to add to the pool (an
-
run
()[source]¶ Keep spawning commands and collecting results until all commands have run.
Returns: The value of results
.Raises: Any exceptions raised by collect()
.This method calls
spawn()
andcollect()
in a loop until all commands registered usingadd()
have run and finished. Ifcollect()
raises an exception any running commands are terminated before the exception is propagated to the caller.If you’re writing code where you want to own the main loop then consider calling
spawn()
andcollect()
directly instead of usingrun()
.When
concurrency
is set to one, specific care is taken to make sure that the callbacks configured bystart_event
andfinish_event
are called in the expected (intuitive) order.
-
spawn
()[source]¶ Spawn additional external commands up to the
concurrency
level.Returns: The number of external commands that were spawned by this invocation of spawn()
(an integer).The commands to start are picked according to three criteria:
- The command’s
was_started
property isFalse
. - The command’s
group_by
value is not present inrunning_groups
. - The
is_finished_with_retries
properties of all of the command’sdependencies
areTrue
.
- The command’s
-
collect
()[source]¶ Collect the exit codes and output of finished commands.
Returns: The number of external commands that were collected by this invocation of
collect()
(an integer).Raises: - If
delay_checks
isTrue
: After all external commands have started and finished, if any commands that have
check
set toTrue
failedCommandPoolFailed
is raised.- If
delay_checks
isFalse
: The exceptions
ExternalCommandFailed
,RemoteCommandFailed
andRemoteConnectFailed
can be raised if a command in the pool that hascheck
set toTrue
fails. Thepool
attribute of the exception will be set to the pool.
Warning
If an exception is raised, commands that are still running will not be terminated! If this concerns you then consider calling
terminate()
from afinally
block (this is whatrun()
does).- If
-
terminate
()[source]¶ Terminate any commands that are currently running.
Returns: The number of commands that were terminated (an integer). If
terminate()
successfully terminates commands, you then callcollect()
and thecheck
property of a terminated command isTrue
you will get an exception because terminated commands (by definition) report a nonzeroreturncode
.
-
-
exception
executor.concurrent.
CommandPoolFailed
(pool)[source]¶ Raised by
collect()
when not all commands succeeded.This exception is only raised when
delay_checks
isTrue
.-
__init__
(pool)[source]¶ Initialize a
CommandPoolFailed
object.Parameters: pool – The CommandPool
object that triggered the exception.
-
commands
¶ A shortcut for
unexpected_failures
.
-
error_message
¶ An error message that explains which commands failed unexpectedly (a string).
-
The executor.contexts
module¶
Dependency injection for command execution contexts.
The contexts
module defines the LocalContext
,
RemoteContext
and SecureChangeRootContext
classes. All of
these classes support the same API for executing external commands, they are
simple wrappers for ExternalCommand
, RemoteCommand
and
SecureChangeRootCommand
.
This allows you to script interaction with external commands in Python and perform that interaction on your local system, on remote systems over SSH or inside chroots using the exact same Python code. Dependency injection on steroids anyone? :-)
Here’s a simple example:
from executor.contexts import LocalContext, RemoteContext
from humanfriendly import format_timespan
def details_about_system(context):
return "\n".join([
"Information about %s:" % context,
" - Host name: %s" % context.capture('hostname', '--fqdn'),
" - Uptime: %s" % format_timespan(float(context.capture('cat', '/proc/uptime').split()[0])),
])
print(details_about_system(LocalContext()))
# Information about local system (peter-macbook):
# - Host name: peter-macbook
# - Uptime: 1 week, 3 days and 10 hours
print(details_about_system(RemoteContext('file-server')))
# Information about remote system (file-server):
# - Host name: file-server
# - Uptime: 18 weeks, 3 days and 4 hours
Whether this functionality looks exciting or horrible I’ll leave up to your judgment. I created it because I’m always building “tools that help me build tools” and this functionality enables me to very rapidly prototype system integration tools developed using Python:
- During development:
- I write code on my workstation which I prefer because of the “rich editing environment” but I run the code against a remote system over SSH (a backup server, database server, hypervisor, mail server, etc.).
- In production:
- I change one line of code to inject a
LocalContext
object instead of aRemoteContext
object, I install the executor package and the code I wrote on the remote system and I’m done!
-
executor.contexts.
MIRROR_TO_DISTRIB_MAPPING
= {u'http://archive.ubuntu.com/ubuntu': u'ubuntu', u'http://deb.debian.org/debian': u'debian'}¶ Mapping of canonical package mirror URLs to “distributor ID” strings.
Each key in this dictionary is the canonical package mirror URL of a Debian based Linux distribution and each value is the corresponding distributor ID. The following canonical mirror URLs are currently supported:
Mirror URL Value http://deb.debian.org/debian debian
http://archive.ubuntu.com/ubuntu/ ubuntu
For more details refer to the
AbstractContext.apt_sources_info
property.
-
executor.contexts.
create_context
(**options)[source]¶ Create an execution context.
Parameters: options – Any keyword arguments are passed on to the context’s initializer. Returns: A LocalContext
,SecureChangeRootContext
orRemoteContext
object.This function provides an easy to use shortcut for constructing context objects:
- If the keyword argument
chroot_name
is given (and notNone
) then aSecureChangeRootContext
object will be created. - If the keyword argument
ssh_alias
is given (and notNone
) then aRemoteContext
object will be created. - Otherwise a
LocalContext
object is created.
- If the keyword argument
-
class
executor.contexts.
AbstractContext
(*args, **options)[source]¶ Abstract base class for shared logic of all context classes.
Here’s an overview of the
AbstractContext
class:Superclass: PropertyManager
Special methods: __enter__()
,__exit__()
and__init__()
Public methods: atomic_write()
,capture()
,cleanup()
,execute()
,exists()
,find_chroots()
,find_program()
,get_options()
,glob()
,is_directory()
,is_executable()
,is_file()
,is_readable()
,is_writable()
,list_entries()
,merge_options()
,prepare()
,prepare_command()
,prepare_interactive_shell()
,read_file()
,start_interactive_shell()
,test()
andwrite_file()
Properties: apt_sources_info
,command_type
,cpu_count
,distribution_codename
,distributor_id
,have_ionice
,have_superuser_privileges
,lsb_release_variables
,options
andparent
When you initialize a
AbstractContext
object you are required to provide a value for thecommand_type
property. You can set the values of thecommand_type
,options
andparent
properties by passing keyword arguments to the class initializer.-
__init__
(*args, **options)[source]¶ Initialize an
AbstractContext
object.Parameters: - args – Any positional arguments are passed on to the initializer
of the
PropertyManager
class (for future extensibility). - options –
The keyword arguments are handled as follows:
- Keyword arguments whose name matches a property of
the context object are used to set that property
(by passing them to the initializer of the
PropertyManager
class). - Any other keyword arguments are collected into the
options
dictionary.
- Keyword arguments whose name matches a property of
the context object are used to set that property
(by passing them to the initializer of the
- args – Any positional arguments are passed on to the initializer
of the
-
apt_sources_info
[source]¶ A tuple with two strings (the distributor ID and distribution codename).
The values of the
distributor_id
anddistribution_codename
properties are determined by one of the following three methods (in decreasing order of preference):- If
lsb_release_variables
is available it’s used. - If the lsb_release program is available it’s used.
- Finally
/etc/apt/sources.list
is parsed for hints.
The
apt_sources_info
property concerns the third step which works as follows:- The
deb
directives in/etc/apt/sources.list
are parsed to determine the primary package mirror URL (it’s fine if this file doesn’t exist, no error will be reported). - The
MIRROR_TO_DISTRIB_MAPPING
dictionary is used to look up the distributor ID corresponding to the package mirror URL that was found in/etc/apt/sources.list
. - If the mirror URL is successfully translated to a distributor ID, the
third token in the
deb
directive is taken to be the distribution codename.
The
apt_sources_info
property was added in response to issue #17 where it was reported that official Debian Docker images don’t contain the/etc/lsb-release
file nor the lsb_release program.This is only used as a last resort because of its specificness to Debian based Linux distributions and because I have concerns about how robust this new functionality will turn out to be.
Note
The
apt_sources_info
property is alazy_property
. This property’s value is computed once (the first time it is accessed) and the result is cached.- If
-
command_type
[source]¶ The type of command objects created by this context (
ExternalCommand
or a subclass).Note
The
command_type
property is arequired_property
. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named command_type (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.
-
cpu_count
¶ The number of CPUs in the system (an integer).
Note
This is an abstract property that must be implemented by subclasses.
-
distribution_codename
[source]¶ The code name of the system’s distribution (a lowercased string like
precise
ortrusty
).How this property is computed depends on the execution context:
- When the file
/etc/lsb-release
exists and defines the variableDISTRIB_CODENAME
then this is the preferred source (for details seelsb_release_variables
). - When lsb_release is installed the output of the command
lsb_release --short --codename
is used. - Finally
apt_sources_info
is used if possible.
The returned string is guaranteed to be lowercased, in order to enable reliable string comparison.
Note
The
distribution_codename
property is alazy_property
. This property’s value is computed once (the first time it is accessed) and the result is cached.- When the file
-
distributor_id
[source]¶ The distributor ID of the system (a lowercased string like
debian
orubuntu
).How this property is computed depends on the execution context:
- When the file
/etc/lsb-release
exists and defines the variableDISTRIB_ID
then this is the preferred source (for details seelsb_release_variables
). - When lsb_release is installed the output of the command
lsb_release --short --id
is used. - Finally
apt_sources_info
is used if possible.
The returned string is guaranteed to be lowercased, in order to enable reliable string comparison.
Note
The
distributor_id
property is alazy_property
. This property’s value is computed once (the first time it is accessed) and the result is cached.- When the file
-
have_ionice
[source]¶ True
when ionice is installed,False
otherwise.Note
The
have_ionice
property is alazy_property
. This property’s value is computed once (the first time it is accessed) and the result is cached.
-
lsb_release_variables
[source]¶ The contents of
/etc/lsb-release
as a dictionary.The values of
distributor_id
anddistribution_codename
are based on the information provided bylsb_release_variables
. If/etc/lsb-release
doesn’t exist or can’t be parsed a debug message is logged and an empty dictionary is returned. Here’s an example:>>> from executor.contexts import LocalContext >>> context = LocalContext() >>> context.lsb_release_variables {'DISTRIB_CODENAME': 'bionic', 'DISTRIB_DESCRIPTION': 'Ubuntu 18.04.1 LTS', 'DISTRIB_ID': 'Ubuntu', 'DISTRIB_RELEASE': '18.04'}
The
lsb_release_variables
property was added in response to issue #10 where it was reported that the lsb_release program wasn’t available in vanilla Ubuntu 18.04 Docker images.Note
The
lsb_release_variables
property is alazy_property
. This property’s value is computed once (the first time it is accessed) and the result is cached.
-
options
[source]¶ The options that are passed to commands created by the context (a dictionary).
Note
The
options
property is awritable_property
. You can change the value of this property using normal attribute assignment syntax.
-
parent
[source]¶ The parent context (a context object or
None
).The
parent
property (and the code inprepare_command()
that uses theparent
property) enables the use of “nested contexts”.For example
find_chroots()
createsSecureChangeRootContext
objects whoseparent
is set to the context that found the chroots. Because of this theSecureChangeRootContext
objects can be used to create commands without knowing or caring whether the chroots reside on the local system or on a remote system accessed via SSH.Warning
Support for parent contexts was introduced in executor version 15 and for now this feature is considered experimental and subject to change. While I’m definitely convinced of the usefulness of nested contexts I’m not happy with the current implementation at all. The most important reason for this is that it’s very surprising (and not in a good way) that a context with a
parent
will create commands with the parent’scommand_type
instead of the expected type.Note
The
parent
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
__exit__
(exc_type=None, exc_value=None, traceback=None)[source]¶ Execute any commands on the “undo stack” (refer to
cleanup()
).
-
atomic_write
(**kwds)[source]¶ Create or update the contents of a file atomically.
Parameters: filename – The pathname of the file to create/update (a string). Returns: A context manager (see the with
keyword) that returns a single string which is the pathname of the temporary file where the contents should be written to initially.If an exception is raised from the
with
block and the temporary file exists, an attempt will be made to remove it but failure to do so will be silenced instead of propagated (to avoid obscuring the original exception).The temporary file is created in the same directory as the real file, but a dot is prefixed to the name (making it a hidden file) and the suffix ‘.tmp-‘ followed by a random integer number is used.
-
capture
(*command, **options)[source]¶ Execute an external command in the current context and capture its output.
Parameters: - command – All positional arguments are passed on to the
initializer of the
command_type
class. - options – All keyword arguments are passed on to the
initializer of the
command_type
class.
Returns: The value of
ExternalCommand.output
.- command – All positional arguments are passed on to the
initializer of the
-
cleanup
(*args, **kw)[source]¶ Register an action to be performed before the context ends.
Parameters: - args – The external command to execute or callable to invoke.
- kw – Options to the command or keyword arguments to the callable.
Raises: ValueError
whencleanup()
is called outside awith
statement.This method registers the intent to perform an action just before the context ends. To actually perform the action(s) you need to use (the subclass of) the
AbstractContext
object as a context manager using thewith
statement.The last action that is registered is the first one to be performed. This gives the equivalent functionality of a deeply nested
try
/finally
structure without actually having to write such ugly code :-).The handling of arguments in
cleanup()
depends on the type of the first positional argument:- If the first positional argument is a string, the positional
arguments and keyword arguments are passed on to the initializer
of the
command_type
class to execute an external command just before the context ends. - If the first positional argument is a callable, it is called with any remaining positional arguments and keyword arguments before the context ends.
-
execute
(*command, **options)[source]¶ Execute an external command in the current context.
Parameters: - command – All positional arguments are passed on to the
initializer of the
command_type
class. - options – All keyword arguments are passed on to the
initializer of the
command_type
class.
Returns: The
command_type
object.Note
After constructing a
command_type
object this method callsstart()
on the command before returning it to the caller, so by the time the caller gets the command object a synchronous command will have already ended. Asynchronous commands don’t have this limitation of course.- command – All positional arguments are passed on to the
initializer of the
-
exists
(pathname)[source]¶ Check whether the given pathname exists.
Parameters: pathname – The pathname to check (a string). Returns: True
if the pathname exists,False
otherwise.This is a shortcut for the
test -e ...
command.
-
find_chroots
(namespace='chroot')[source]¶ Find the chroots available in the current context.
Parameters: namespace – The chroot namespace to look for (a string, defaults to DEFAULT_NAMESPACE
). Refer to the schroot documentation for more information about chroot namespaces.Returns: A generator of SecureChangeRootContext
objects whoseparent
is set to the context where the chroots were found.Raises: ExternalCommandFailed
(or a subclass) when theschroot
program isn’t installed or theschroot --list
command fails.
-
find_program
(program_name, *args)[source]¶ Find the absolute pathname(s) of one or more programs.
Parameters: program_name – Each of the positional arguments is expected to be a string containing the name of a program to search for in the $PATH
. At least one is required.Returns: A list of strings with absolute pathnames. This method is a simple wrapper around
which
.
-
get_options
()[source]¶ Get the options that are passed to commands created by the context.
Returns: A dictionary of command options. By default this method simply returns the
options
dictionary, however the purpose ofget_options()
is to enable subclasses to customize the options passed to commands on the fly.
-
glob
(pattern)[source]¶ Find matches for a given filename pattern.
Parameters: pattern – A filename pattern (a string). Returns: A list of strings with matches. Some implementation notes:
- This method emulates filename globbing as supported by system
shells like Bash and ZSH. It works by forking a Python interpreter
and using that to call the
glob.glob()
function. This approach is of course rather heavyweight. - Initially this method used Bash for filename matching (similar to this StackOverflow answer) but I found it impossible to make this work well for patterns containing whitespace.
- I took the whitespace issue as a sign that I was heading down the wrong path (trying to add robustness to a fragile solution) and so the new implementation was born (which prioritizes robustness over performance).
- This method emulates filename globbing as supported by system
shells like Bash and ZSH. It works by forking a Python interpreter
and using that to call the
-
is_directory
(pathname)[source]¶ Check whether the given pathname points to an existing directory.
Parameters: pathname – The pathname to check (a string). Returns: True
if the pathname points to an existing directory,False
otherwise.This is a shortcut for the
test -d ...
command.
-
is_executable
(pathname)[source]¶ Check whether the given pathname points to an executable file.
Parameters: pathname – The pathname to check (a string). Returns: True
if the pathname points to an executable file,False
otherwise.This is a shortcut for the
test -x ...
command.
-
is_file
(pathname)[source]¶ Check whether the given pathname points to an existing file.
Parameters: pathname – The pathname to check (a string). Returns: True
if the pathname points to an existing file,False
otherwise.This is a shortcut for the
test -f ...
command.
-
is_readable
(pathname)[source]¶ Check whether the given pathname exists and is readable.
Parameters: pathname – The pathname to check (a string). Returns: True
if the pathname exists and is readable,False
otherwise.This is a shortcut for the
test -r ...
command.
-
is_writable
(pathname)[source]¶ Check whether the given pathname exists and is writable.
Parameters: pathname – The pathname to check (a string). Returns: True
if the pathname exists and is writable,False
otherwise.This is a shortcut for the
test -w ...
command.
-
list_entries
(directory)[source]¶ List the entries in a directory.
Parameters: directory – The pathname of the directory (a string). Returns: A list of strings with the names of the directory entries. This method uses
find -mindepth 1 -maxdepth 1 -print0
to list directory entries instead of going for the more obvious choicels -A1
becausefind
enables more reliable parsing of command output (with regards to whitespace).
-
merge_options
(overrides)[source]¶ Merge default options and overrides into a single dictionary.
Parameters: overrides – A dictionary with any keyword arguments given to execute()
orstart_interactive_shell()
.Returns: The dictionary with overrides, but any keyword arguments given to the initializer of AbstractContext
that are not set in the overrides are set to the value of the initializer argument.The
ionice
option is automatically unset whenhave_ionice
isFalse
, regardless of whether the option was set from defaults or overrides.
-
prepare
(*command, **options)[source]¶ Prepare to execute an external command in the current context.
Parameters: - command – All positional arguments are passed on to the
initializer of the
command_type
class. - options – All keyword arguments are passed on to the
initializer of the
command_type
class.
Returns: The
command_type
object.Note
After constructing a
command_type
object this method doesn’t callstart()
which means you control if and when the command is started. This can be useful to prepare a large batch of commands and execute them concurrently using aCommandPool
.- command – All positional arguments are passed on to the
initializer of the
-
prepare_command
(command, options)[source]¶ Create a
command_type
object based onoptions
.Parameters: - command – A tuple of strings (the positional arguments to the
initializer of the
command_type
class). - options – A dictionary (the keyword arguments to the initializer
of the
command_type
class).
Returns: A
command_type
object that hasn’t been started yet.- command – A tuple of strings (the positional arguments to the
initializer of the
-
prepare_interactive_shell
(options)[source]¶ Create a
command_type
object that starts an interactive shell.Parameters: options – A dictionary (the keyword arguments to the initializer of the command_type
class).Returns: A command_type
object that hasn’t been started yet.
-
read_file
(filename, **options)[source]¶ Read the contents of a file.
Parameters: - filename – The pathname of the file to read (a string).
- options – Optional keyword arguments to
execute()
.
Returns: The contents of the file (a byte string).
This method uses cat to read the contents of files so that options like
sudo
are respected (regardless of whether we’re dealing with aLocalContext
orRemoteContext
).
-
start_interactive_shell
(**options)[source]¶ Start an interactive shell in the current context.
Parameters: options – All keyword arguments are passed on to the initializer of the command_type
class.Returns: The command_type
object.Note
After constructing a
command_type
object this method callsstart()
on the command before returning it to the caller, so by the time the caller gets the command object a synchronous command will have already ended. Asynchronous commands don’t have this limitation of course.
-
test
(*command, **options)[source]¶ Execute an external command in the current context and get its status.
Parameters: - command – All positional arguments are passed on to the
initializer of the
command_type
class. - options – All keyword arguments are passed on to the
initializer of the
command_type
class.
Returns: The value of
ExternalCommand.succeeded
.This method automatically sets
check
toFalse
andsilent
toTrue
.- command – All positional arguments are passed on to the
initializer of the
-
write_file
(filename, contents, **options)[source]¶ Change the contents of a file.
Parameters: - filename – The pathname of the file to write (a string).
- contents – The contents to write to the file (a byte string).
- options – Optional keyword arguments to
execute()
.
This method uses a combination of cat and output redirection to change the contents of files so that options like
sudo
are respected (regardless of whether we’re dealing with aLocalContext
orRemoteContext
). Due to the use of cat this method will create files that don’t exist yet, assuming the directory containing the file already exists and the context provides permission to write to the directory.
-
-
class
executor.contexts.
LocalContext
(*args, **options)[source]¶ Context for executing commands on the local system.
Here’s an overview of the
LocalContext
class:Superclass: AbstractContext
Special methods: __str__()
Public methods: glob()
Properties: command_type
andcpu_count
-
command_type
¶ The type of command objects created by this context (
ExternalCommand
).
-
cpu_count
[source]¶ The number of CPUs in the system (an integer).
This property’s value is computed using
multiprocessing.cpu_count()
.Note
The
cpu_count
property is alazy_property
. This property’s value is computed once (the first time it is accessed) and the result is cached.
-
glob
(pattern)[source]¶ Find matches for a given filename pattern.
Parameters: pattern – A filename pattern (a string). Returns: A list of strings with matches. This method overrides
AbstractContext.glob()
to callglob.glob()
directly instead of forking a new Python interpreter.This optimization is skipped when
options
containssudo
,uid
oruser
to avoid reporting wrong matches due to insufficient filesystem permissions.
-
-
class
executor.contexts.
ChangeRootContext
(*args, **options)[source]¶ Context for executing commands in change roots using chroot.
Here’s an overview of the
ChangeRootContext
class:Superclass: AbstractContext
Special methods: __init__()
and__str__()
Public methods: get_options()
Properties: chroot
,command_type
andcpu_count
When you initialize a
ChangeRootContext
object you are required to provide a value for thechroot
property. You can set the value of thechroot
property by passing a keyword argument to the class initializer.-
__init__
(*args, **options)[source]¶ Initialize a
ChangeRootContext
object.Parameters: - args – Positional arguments are passed on to the initializer of
the
AbstractContext
class (for future extensibility). - options – Any keyword arguments are passed on to the initializer
of the
AbstractContext
class.
If the keyword argument chroot isn’t given but positional arguments are provided, the first positional argument is used to set the
chroot
property.- args – Positional arguments are passed on to the initializer of
the
-
chroot
[source]¶ The pathname of the root directory of the chroot (a string).
Note
The
chroot
property is arequired_property
. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.
-
command_type
¶ The type of command objects created by this context (
ChangeRootCommand
).
-
cpu_count
[source]¶ The number of CPUs in the system (an integer).
This property’s value is computed using
multiprocessing.cpu_count()
.Note
The
cpu_count
property is alazy_property
. This property’s value is computed once (the first time it is accessed) and the result is cached.
-
-
class
executor.contexts.
SecureChangeRootContext
(*args, **options)[source]¶ Context for executing commands in change roots using schroot.
Here’s an overview of the
SecureChangeRootContext
class:Superclass: AbstractContext
Special methods: __init__()
and__str__()
Public methods: get_options()
Properties: chroot_name
,command_type
andcpu_count
When you initialize a
SecureChangeRootContext
object you are required to provide a value for thechroot_name
property. You can set the value of thechroot_name
property by passing a keyword argument to the class initializer.-
__init__
(*args, **options)[source]¶ Initialize a
SecureChangeRootContext
object.Parameters: - args – Positional arguments are passed on to the initializer of
the
AbstractContext
class (for future extensibility). - options – Any keyword arguments are passed on to the initializer
of the
AbstractContext
class.
If the keyword argument chroot_name isn’t given but positional arguments are provided, the first positional argument is used to set the
chroot_name
property.- args – Positional arguments are passed on to the initializer of
the
-
chroot_name
[source]¶ The name of a chroot managed by schroot (a string).
Note
The
chroot_name
property is arequired_property
. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot_name (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.
-
command_type
¶ The type of command objects created by this context (
SecureChangeRootCommand
).
-
cpu_count
[source]¶ The number of CPUs in the system (an integer).
This property’s value is computed using
multiprocessing.cpu_count()
.Note
The
cpu_count
property is alazy_property
. This property’s value is computed once (the first time it is accessed) and the result is cached.
-
get_options
()[source]¶ The
options
includingchroot_name
.
-
-
class
executor.contexts.
RemoteContext
(*args, **options)[source]¶ Context for executing commands on a remote system over SSH.
Here’s an overview of the
RemoteContext
class:Superclasses: RemoteAccount
andAbstractContext
Special methods: __str__()
Public methods: get_options()
Properties: command_type
andcpu_count
-
command_type
¶ The type of command objects created by this context (
RemoteCommand
).
-
cpu_count
[source]¶ The number of CPUs in the system (an integer).
This property’s value is computed by executing the remote command nproc. If that command fails
cpu_count
falls back to the commandgrep -ci '^processor\s*:' /proc/cpuinfo
.Note
The
cpu_count
property is alazy_property
. This property’s value is computed once (the first time it is accessed) and the result is cached.
-
The executor.process
module¶
Portable process control functionality for the executor package.
The executor.process
module defines the ControllableProcess
abstract base class which enables process control features like waiting for a
process to end, gracefully terminating it and forcefully killing it. The
process control functionality in ControllableProcess
is separated from
the command execution functionality in ExternalCommand
to
make it possible to re-use the process control functionality in other Python
packages, see for example the proc.core.Process
class.
-
executor.process.
DEFAULT_TIMEOUT
= 10¶ The default timeout used to wait for process termination (number of seconds).
-
class
executor.process.
ControllableProcess
(**kw)[source]¶ Abstract, portable process control functionality.
By defining a subclass of
ControllableProcess
and implementing thepid
,command_line
andis_running
properties and theterminate_helper()
andkill_helper()
methods you get thewait_for_process()
,terminate()
andkill()
methods for free. This decoupling has enabled me to share a lot of code between two Python projects of mine with similar goals but very different requirements:- The executor package builds on top of the
subprocess
module in the Python standard library and strives to be as cross platform as possible. This means things like UNIX signals are not an option (although signals exist on Windows they are hardly usable). The package mostly deals withsubprocess.Popen
objects internally (to hide platform specific details as much as possible). - The proc package exposes process information available in the Linux
process information pseudo-file system available at
/proc
. The package mostly deals with process IDs internally. Because this is completely specialized to a UNIX environment the use of things like UNIX signals is not a problem at all.
Here’s an overview of the
ControllableProcess
class:Superclass: PropertyManager
Special methods: __str__()
Public methods: kill()
,kill_helper()
,terminate()
,terminate_helper()
andwait_for_process()
Properties: command_line
,is_running
,logger
andpid
You can set the values of the
command_line
,logger
andpid
properties by passing keyword arguments to the class initializer.-
command_line
[source]¶ A list of strings with the command line used to start the process.
This property may be set or implemented by subclasses to enable
__str__()
to render a human friendly representation of aControllableProcess
object.Note
The
command_line
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
is_running
¶ True
if the process is running,False
otherwise.This property must be implemented by subclasses to enable
wait_for_process()
,terminate()
andkill()
to work properly.
-
logger
[source]¶ The
logging.Logger
object to use (defaults to theexecutor.process
logger).If you are using Python’s
logging
module and you find it confusing that command manipulation is logged under theexecutor.process
name space instead of the name space of the application or library usingexecutor
you can set this attribute to inject a custom (and more appropriate) logger.Note
The
logger
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
pid
[source]¶ The process ID (a number) or
None
.This property must be set or implemented by subclasses:
- It provides
wait_for_process()
with a short and unique representation of a process that most users will understand. - It enables
__str__()
to render a human friendly representation of aControllableProcess
object.
Note
The
pid
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.- It provides
-
wait_for_process
(timeout=0, use_spinner=None)[source]¶ Wait until the process ends or the timeout expires.
Parameters: - timeout – The number of seconds to wait for the process to terminate after we’ve asked it nicely (defaults to zero which means we wait indefinitely).
- use_spinner –
Whether or not to display an interactive spinner on the terminal (using
Spinner
) to explain to the user what they are waiting for:
Returns: A
Timer
object telling you how long it took to wait for the process.
-
terminate
(wait=True, timeout=10, use_spinner=None)[source]¶ Gracefully terminate the process.
Parameters: - wait – Whether to wait for the process to end (a boolean,
defaults to
True
). - timeout – The number of seconds to wait for the process to
terminate after we’ve signaled it (defaults to
DEFAULT_TIMEOUT
). Zero means to wait indefinitely. - use_spinner – See the
wait_for_process()
documentation.
Returns: Raises: Any exceptions raised by
terminate_helper()
implementations of subclasses orkill()
.This method works as follows:
- Signal the process to gracefully terminate itself. Processes can choose to intercept termination signals to allow for graceful termination (many UNIX daemons work like this) however the default action is to simply exit immediately.
- If wait is
True
and we’ve signaled the process, we wait for it to terminate gracefully or timeout seconds have passed (whichever comes first). - If wait is
True
and the process is still running after timeout seconds have passed, it will be forcefully terminated usingkill()
(the value of timeout that was given toterminate()
will be passed on tokill()
).
This method does nothing when
is_running
isFalse
.- wait – Whether to wait for the process to end (a boolean,
defaults to
-
terminate_helper
()[source]¶ Request the process to gracefully terminate itself (needs to be implemented by subclasses).
-
kill
(wait=True, timeout=10, use_spinner=None)[source]¶ Forcefully kill the process.
Parameters: - wait – Whether to wait for the process to end (a boolean,
defaults to
True
). - timeout – The number of seconds to wait for the process to
terminate after we’ve signaled it (defaults to
DEFAULT_TIMEOUT
). Zero means to wait indefinitely. - use_spinner – See the
wait_for_process()
documentation.
Returns: Raises: - Any exceptions raised by
kill_helper()
implementations of subclasses. ProcessTerminationFailed
if the process is still running afterkill_helper()
andwait_for_process()
have been called.
This method does nothing when
is_running
isFalse
.- wait – Whether to wait for the process to end (a boolean,
defaults to
-
__str__
()[source]¶ Render a human friendly representation of a
ControllableProcess
object.Returns: A string describing the process. Includes the process ID and the command line (when available).
- The executor package builds on top of the
-
exception
executor.process.
ProcessTerminationFailed
(*args, **kw)[source]¶ Raised when process termination fails.
Here’s an overview of the
ProcessTerminationFailed
class:Superclasses: PropertyManager
andException
Special methods: __init__()
Properties: message
andprocess
When you initialize a
ProcessTerminationFailed
object you are required to provide values for themessage
andprocess
properties. You can set the values of themessage
andprocess
properties by passing keyword arguments to the class initializer.-
__init__
(*args, **kw)[source]¶ Initialize a
ProcessTerminationFailed
object.This method’s signature is the same as the initializer of the
PropertyManager
class.
-
process
[source]¶ The
ControllableProcess
object that triggered the exception.
-
The executor.schroot
module¶
Secure command execution in chroot environments.
The executor.schroot
module defines the SecureChangeRootCommand
class which makes it easy to run commands inside chroots that are managed
using the schroot program.
-
executor.schroot.
SCHROOT_PROGRAM_NAME
= 'schroot'¶ The name of the
schroot
program (a string).
-
executor.schroot.
DEFAULT_NAMESPACE
= 'chroot'¶ The default chroot namespace (a string).
Refer to the schroot documentation for more information about chroot namespaces.
-
class
executor.schroot.
SecureChangeRootCommand
(*args, **options)[source]¶ SecureChangeRootCommand
objects use the schroot program to execute commands inside chroots.Here’s an overview of the
SecureChangeRootCommand
class:Superclass: ExternalCommand
Special methods: __init__()
Properties: chroot_directory
,chroot_name
,chroot_user
,command_line
,directory
andschroot_command
When you initialize a
SecureChangeRootCommand
object you are required to provide a value for thechroot_name
property. You can set the values of thechroot_directory
,chroot_name
,chroot_user
andschroot_command
properties by passing keyword arguments to the class initializer.-
__init__
(*args, **options)[source]¶ Initialize a
SecureChangeRootCommand
object.Parameters: - args – Positional arguments are passed on to the initializer of
the
ExternalCommand
class. - options – Any keyword arguments are passed on to the initializer
of the
ExternalCommand
class.
If the keyword argument chroot_name isn’t given but positional arguments are provided, the first positional argument is used to set the
chroot_name
property.The command is not started until you call
start()
orwait()
.- args – Positional arguments are passed on to the initializer of
the
-
chroot_directory
[source]¶ The working directory _inside the chroot (a string or
None
, defaults to/
).When
chroot_directory
isNone
the schroot program gets to pick the working directory inside the chroot (refer to the schroot documentation for the complete details).For non-interactive usage (which I anticipate to be the default usage of
SecureChangeRootCommand
) the schroot program simply assumes that the working directory outside of the chroot also exists inside the chroot, then fails with an error message when this is not the case.Because this isn’t a very robust default,
chroot_directory
defaults to/
instead.Note
The
chroot_directory
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
chroot_name
[source]¶ The name of a chroot managed by schroot (a string).
This is expected to match one of the names configured in the directory
/etc/schroot/chroot.d
.Note
The
chroot_name
property is arequired_property
. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot_name (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.
-
chroot_user
[source]¶ The name of the user inside the chroot to run the command as (a string or
None
).This defaults to
None
which means to run as the current user.Note
The
chroot_user
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
command_line
¶ The complete schroot command including the command to run inside the chroot.
This is a list of strings with the schroot command line to enter the requested chroot and execute
command
.
-
directory
¶ Set the working directory inside the chroot.
When you set this property you change
chroot_directory
, however reading back the property you’ll just getDEFAULT_WORKING_DIRECTORY
. This is because the superclassExternalCommand
usesdirectory
as the working directory for the schroot command, and directories inside chroots aren’t guaranteed to exist on the host system.
-
schroot_command
[source]¶ The command used to run the schroot program.
This is a list of strings, by default the list contains just
SCHROOT_PROGRAM_NAME
. Thechroot_directory
,chroot_name
andchroot_user
properties also influence the schroot command line used.Note
The
schroot_command
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
The executor.ssh.client
module¶
Remote command execution using SSH.
The executor.ssh.client
module defines the RemoteCommand
class
and the foreach()
function which make it easy to run a remote command
in parallel on multiple remote hosts using SSH.
The foreach()
function also serves as a simple example of how to use
CommandPool
and RemoteCommand
objects
(it’s just 16 lines of code if you squint in the right way and that includes
logging :-).
The SecureTunnel
class makes it easy to use “tunnel only”
SSH connections by waiting until the tunnel becomes connected and
automatically selecting a free ephemeral port number if a
local port number isn’t provided by the caller.
-
executor.ssh.client.
DEFAULT_CONCURRENCY
= 10¶ The default
concurrency
value to use forCommandPool
objects created byforeach()
.
-
executor.ssh.client.
DEFAULT_CONNECT_TIMEOUT
= 10¶ The default
connect_timeout
value to use forRemoteCommand
objects.
-
executor.ssh.client.
SSH_PROGRAM_NAME
= 'ssh'¶ The name of the SSH client executable (a string).
-
executor.ssh.client.
SSH_ERROR_STATUS
= 255¶ The exit status used by the
ssh
program if an error occurred (an integer).Used by
RemoteCommand.error_message
andRemoteCommand.error_type
to distinguish when thessh
program itself fails and when a remote command fails.
-
executor.ssh.client.
foreach
(hosts, *command, **options)[source]¶ Execute a command simultaneously on a group of remote hosts using SSH.
Parameters: - hosts – An iterable of strings with SSH host aliases.
- command – Any positional arguments are converted to a list and used to set the
command
property of theRemoteCommand
objects constructed byforeach()
. - concurrency – Set the command pool
concurrency
property (defaults toDEFAULT_CONCURRENCY
). - delay_checks – Set the command pool
delay_checks
property (defaults toTrue
). - logs_directory – Set the command pool
logs_directory
property (not set by default). - spinner – Set the command pool
spinner
property (not set by default). - options – Additional keyword arguments can be used to conveniently override the
default values of the writable properties of the
RemoteCommand
objects constructed byforeach()
(seeRemoteCommand.__init__()
for details).
Returns: The list of
RemoteCommand
objects constructed byforeach()
.Raises: Any of the following exceptions can be raised:
CommandPoolFailed
ifdelay_checks
is enabled (the default) and a command in the pool that hascheck
enabled (the default) fails.RemoteCommandFailed
ifdelay_checks
is disabled (not the default) and an SSH connection was successful but the remote command failed (the exit code of thessh
command was neither zero nor 255). Use the keyword argumentcheck=False
to disable raising of this exception.RemoteConnectFailed
ifdelay_checks
is disabled (not the default) and an SSH connection failed (the exit code of thessh
command is 255). Use the keyword argumentcheck=False
to disable raising of this exception.
Note
The
foreach()
function enables thecheck
anddelay_checks
options by default in an attempt to make it easy to do “the right thing”. My assumption here is that if you are running the same command on multiple remote hosts:- You definitely want to know when a remote command has failed,
ideally without manually checking the
succeeded
property of each command. - Regardless of whether some remote commands fail you want to know that the command was at least executed on all hosts, otherwise your cluster of hosts will end up in a very inconsistent state.
- If remote commands fail and an exception is raised the exception message should explain which remote commands failed.
If these assumptions are incorrect then you can use the keyword arguments
check=False
and/ordelay_checks=False
to opt out of “doing the right thing” ;-)
-
executor.ssh.client.
remote
(ssh_alias, *command, **options)[source]¶ Execute a remote command (similar to
execute()
).Parameters: - ssh_alias – Used to set
RemoteAccount.ssh_alias
. - command – All positional arguments are passed to
RemoteCommand.__init__()
. - options – All keyword arguments are passed to
RemoteCommand.__init__()
.
Returns: Refer to
execute_prepared()
.Raises: RemoteCommandFailed
when the command exits with a nonzero exit code (andcheck
isTrue
).- ssh_alias – Used to set
-
class
executor.ssh.client.
RemoteAccount
(*args, **options)[source]¶ Trivial SSH alias parser.
This class acts as a base class for
RemoteCommand
andRemoteContext
that provides three simple features:- It defines the
ssh_alias
andssh_user
properties. - When
ssh_alias
is set to a string that contains an@
token it parses the string and sets bothssh_alias
andssh_user
[1]. - It allows for
ssh_alias
to be passed as the first positional argument [2] or as a keyword argument [3].
[1] This enables RemoteCommand.have_superuser_privileges
to know that superuser privileges are available when the caller setsssh_alias
to a value likeroot@host
. Of course the SSH client configuration can also override the remote username without the executor package knowing about it, but at least executor will be able to use the information that it does have.[2] This calling convention enables backwards compatibility with executor versions 14 and below which required ssh_alias
to be set using the first positional argument to the initializer of theRemoteCommand
class.[3] This new calling convention provides a uniform calling convention for the initializers of the local/remote command/context classes. Here’s an overview of the
RemoteAccount
class:Superclass: PropertyManager
Special methods: __init__()
Properties: ssh_alias
andssh_user
When you initialize a
RemoteAccount
object you are required to provide a value for thessh_alias
property. You can set the values of thessh_alias
andssh_user
properties by passing keyword arguments to the class initializer.-
__init__
(*args, **options)[source]¶ Initialize a
RemoteAccount
object.Parameters: - args – Positional arguments are passed on to the initializer of
the
PropertyManager
class (for future extensibility). - options – Any keyword arguments are passed on to the initializer
of the
PropertyManager
class.
If the keyword argument ssh_alias isn’t given but positional arguments are provided, the first positional argument is used to set the
ssh_alias
property.- args – Positional arguments are passed on to the initializer of
the
-
ssh_alias
[source]¶ The SSH alias of the remote host (a string).
If you set this property to a string that contains two nonempty tokens delimited by an
@
character, the first token is used to setssh_user
and the second token is used to setssh_alias
. Here’s an example:>>> from executor.ssh.client import RemoteAccount >>> RemoteAccount('root@server') RemoteAccount(ssh_alias='server', ssh_user='root') >>> RemoteAccount(ssh_alias='server', ssh_user='root') RemoteAccount(ssh_alias='server', ssh_user='root') >>> RemoteAccount('server') RemoteAccount(ssh_alias='server', ssh_user=None)
Note
The
ssh_alias
property is arequired_property
. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named ssh_alias (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.
-
ssh_user
[source]¶ The username on the remote system (a string or
None
).If the value of
ssh_user
isNone
the SSH client program gets to decide about the remote username.Note
The
ssh_user
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
- It defines the
-
class
executor.ssh.client.
RemoteCommand
(*args, **options)[source]¶ RemoteCommand
objects use the SSH client program to execute remote commands.Here’s an overview of the
RemoteCommand
class:Superclasses: RemoteAccount
andExternalCommand
Special methods: __init__()
Properties: batch_mode
,command
,command_line
,compression
,connect_timeout
,directory
,error_message
,error_type
,have_superuser_privileges
,identity_file
,ignore_known_hosts
,known_hosts_file
,log_level
,port
,ssh_command
andstrict_host_key_checking
You can set the values of the
batch_mode
,command
,compression
,connect_timeout
,error_message
,error_type
,identity_file
,known_hosts_file
,log_level
,port
,ssh_command
andstrict_host_key_checking
properties by passing keyword arguments to the class initializer.-
__init__
(*args, **options)[source]¶ Initialize a
RemoteCommand
object.Parameters: - args – Refer to the initializers of the
RemoteAccount
andExternalCommand
classes. - options – Keyword arguments can be used to conveniently override
the values of
batch_mode
,connect_timeout
,identity_file
,ignore_known_hosts
,log_level
,port
,strict_host_key_checking
,known_hosts_file
,ssh_command
and the writable properties of the base classesRemoteAccount
andExternalCommand
. Any other keyword argument will raiseTypeError
as usual.
The remote command is not started until you call
start()
orwait()
.- args – Refer to the initializers of the
-
batch_mode
[source]¶ Control the SSH client option
BatchMode
(a boolean, defaults toTrue
).The following description is quoted from man ssh_config:
If set to “yes”, passphrase/password querying will be disabled. In addition, theServerAliveInterval
option will be set to 300 seconds by default. This option is useful in scripts and other batch jobs where no user is present to supply the password, and where it is desirable to detect a broken network swiftly. The argument must be “yes” or “no”. The default is “no”.This property defaults to
True
because it can get really awkward when a batch of SSH clients query for a passphrase/password on standard input at the same time.Note
The
batch_mode
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
command
[source]¶ A list of strings with the command to execute (optional).
The value of
command
is optional forRemoteCommand
objects (as opposed toExternalCommand
objects) because the use of SSH implies a remote (interactive) shell that usually also accepts (interactive) commands as input. This means it is valid to create a remote command object without an actual remote command to execute, but with input that provides commands to execute instead.This “feature” can be useful to control non-UNIX systems that do accept SSH connections but don’t support a conventional UNIX shell. For example, I added support for this “feature” so that I was able to send commands to Juniper routers and switches over SSH with the purpose of automating the failover of a connection between two datacenters (the resulting Python program works great and it’s much faster than I am, making all of the required changes in a couple of seconds :-).
Note
The
command
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
command_line
¶ The complete SSH client command including the remote command.
This is a list of strings with the SSH client command to connect to the remote host and execute
command
.
-
compression
[source]¶ Whether compression is enabled (a boolean, defaults to
False
).Note
The
compression
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
connect_timeout
[source]¶ Control the SSH client option
ConnectTimeout
(an integer).The following description is quoted from man ssh_config:
Specifies the timeout (in seconds) used when connecting to the SSH server, instead of using the default system TCP timeout. This value is used only when the target is down or really unreachable, not when it refuses the connection.Defaults to
DEFAULT_CONNECT_TIMEOUT
so that non-interactive SSH connections created byRemoteCommand
don’t hang indefinitely when the remote system doesn’t respond properly.Note
The
connect_timeout
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
directory
¶ Set the remote working directory.
When you set this property you change the remote working directory, however reading back the property you’ll just get
DEFAULT_WORKING_DIRECTORY
. This is because the superclassExternalCommand
usesdirectory
as the local working directory for thessh
command, and a remote working directory isn’t guaranteed to also exist on the local system.
-
error_message
[source]¶ A user friendly explanation of how the remote command failed (a string or
None
).Note
The
error_message
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
error_type
[source]¶ An exception class applicable to the kind of failure detected or
None
.RemoteConnectFailed
whenreturncode
is set and matchesSSH_ERROR_STATUS
,RemoteCommandFailed
whenreturncode
is set and not zero,None
otherwise.Note
The
error_type
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
have_superuser_privileges
¶ True
ifssh_user
is set to ‘root’,False
otherwise.There’s no easy way for
RemoteCommand
to determine whether any given SSH alias logs into a remote system with superuser privileges so unlessssh_user
is set to ‘root’ this is alwaysFalse
.
-
identity_file
[source]¶ The pathname of the identity file used to connect to the remote host (a string or
None
).Note
The
identity_file
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
ignore_known_hosts
¶ Whether host key checking is disabled.
This is
True
if host key checking is completely disabled:known_hosts_file
is set toos.devnull
strict_host_key_checking
is set toFalse
If you set this to
True
host key checking is disabled andlog_level
is set to ‘error’ to silence warnings about automatically accepting host keys.If you set this to
False
thenknown_hosts_file
,log_level
andstrict_host_key_checking
are reset to their default values.
-
log_level
[source]¶ Control the SSH client option
LogLevel
(a string, defaults to ‘info’).The following description is quoted from man ssh_config:
Gives the verbosity level that is used when logging messages fromssh
. The possible values are: QUIET, FATAL, ERROR, INFO, VERBOSE, DEBUG, DEBUG1, DEBUG2, and DEBUG3. The default is INFO. DEBUG and DEBUG1 are equivalent. DEBUG2 and DEBUG3 each specify higher levels of verbose output.Note
The
log_level
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
ssh_command
[source]¶ The command used to run the SSH client program.
This is a list of strings, by default the list contains just
SSH_PROGRAM_NAME
. Thebatch_mode
,connect_timeout
,log_level
,ssh_alias
andstrict_host_key_checking
properties also influence the SSH client command line used.Note
The
ssh_command
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
port
[source]¶ The port number of the SSH server (defaults to
None
which means the SSH client program decides).Note
The
port
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
strict_host_key_checking
[source]¶ Control the SSH client option
StrictHostKeyChecking
.This property accepts the values
True
andFalse
and the strings ‘yes’, ‘no’ and ‘ask’. The following description is quoted from man ssh_config:If this flag is set to “yes”,ssh
will never automatically add host keys to the~/.ssh/known_hosts
file, and refuses to connect to hosts whose host key has changed. This provides maximum protection against trojan horse attacks, though it can be annoying when the/etc/ssh/ssh_known_hosts
file is poorly maintained or when connections to new hosts are frequently made. This option forces the user to manually add all new hosts. If this flag is set to “no”, ssh will automatically add new host keys to the user known hosts files. If this flag is set to “ask”, new host keys will be added to the user known host files only after the user has confirmed that is what they really want to do, and ssh will refuse to connect to hosts whose host key has changed. The host keys of known hosts will be verified automatically in all cases. The argument must be “yes”, “no”, or “ask”. The default is “ask”.This property defaults to
False
so that when you connect to a remote system over SSH for the first time the host key is automatically added to the user known hosts file (instead of requiring interaction). As mentioned in the quote above the host keys of known hosts are always verified (but seeignore_known_hosts
).Note
The
strict_host_key_checking
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
known_hosts_file
[source]¶ Control the SSH client option
UserKnownHostsFile
(a string).The following description is quoted from man ssh_config:
Specifies one or more files to use for the user host key database, separated by whitespace. The default is~/.ssh/known_hosts
,~/.ssh/known_hosts2
.Note
The
known_hosts_file
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
-
class
executor.ssh.client.
RemoteCommandPool
(concurrency=10, **options)[source]¶ Execute multiple remote commands concurrently.
After constructing a
RemoteCommandPool
instance you add commands to it usingadd()
and when you’re ready to run the commands you callrun()
.Note
The only difference between
CommandPool
andRemoteCommandPool
is the default concurrency. This may of course change in the future.Here’s an overview of the
RemoteCommandPool
class:Superclass: CommandPool
Special methods: __init__()
-
__init__
(concurrency=10, **options)[source]¶ Initialize a
RemoteCommandPool
object.Parameters: - concurrency – Override the value of
concurrency
(an integer, defaults toDEFAULT_CONCURRENCY
for remote command pools). - options – Any additional keyword arguments are passed on
to the
CommandPool
constructor.
- concurrency – Override the value of
-
-
class
executor.ssh.client.
SecureTunnel
(*args, **options)[source]¶ Easy to use SSH tunnels.
The
SecureTunnel
class combinesRemoteCommand
withEphemeralPortAllocator
andWaitUntilConnected
to implement easy to use SSH tunnel support.The
-L
option of the SSH client program is used to open a tunnel between the local system and a remote system that persists until the command is terminated (tip: use awith
statement).Additionally the
-N
option is used to prevent the client from executing a remote command, this enables compatibility with “tunnel only” SSH accounts that have their shell set to something like/usr/sbin/nologin
.The
start()
method waits for the tunnel to become available before returning control to the caller.Here’s an overview of the
SecureTunnel
class:Superclass: RemoteCommand
Public methods: start()
Properties: asynchronous
,command_line
,compression
,local_port
,remote_host
,remote_port
andtty
When you initialize a
SecureTunnel
object you are required to provide a value for theremote_port
property. You can set the values of theasynchronous
,compression
,local_port
,remote_host
andremote_port
properties by passing keyword arguments to the class initializer.-
asynchronous
[source]¶ Whether to enable asynchronous command execution (a boolean, defaults to
True
).Note
The
asynchronous
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
command_line
¶ The complete SSH client command to open the tunnel (a list of strings).
This property overrides
RemoteCommand.command_line
to inject the command line options-L
and-N
.
-
compression
[source]¶ Whether to enable compression (a boolean, defaults to
True
).Note
The
compression
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
local_port
[source]¶ The port number on the side of the SSH client (an integer).
When the value of
local_port
isn’t specified a free ephemeral port number is automatically selected usingEphemeralPortAllocator
.Note
The
local_port
property is acustom_property
. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can usedel
ordelattr()
.
-
remote_host
[source]¶ The remote host name to connect to (a string, defaults to ‘localhost’).
Note
The
remote_host
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
remote_port
[source]¶ The remote port number to connect to (an integer).
Note
The
remote_port
property is arequired_property
. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named remote_port (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.
-
-
exception
executor.ssh.client.
RemoteConnectFailed
(command, **options)[source]¶ Raised by
RemoteCommand
when an SSH connection itself fails (not the remote command).Here’s an overview of the
RemoteConnectFailed
class:Superclass: ExternalCommandFailed
-
exception
executor.ssh.client.
RemoteCommandFailed
(command, **options)[source]¶ Raised by
RemoteCommand
when a remote command executed over SSH fails.Here’s an overview of the
RemoteCommandFailed
class:Superclass: ExternalCommandFailed
-
exception
executor.ssh.client.
RemoteCommandNotFound
(command, **options)[source]¶ Raised by
RemoteCommand
when a remote command returnsCOMMAND_NOT_FOUND_STATUS
.Here’s an overview of the
RemoteCommandNotFound
class:Superclasses: RemoteCommandFailed
andCommandNotFound
The executor.ssh.server
module¶
OpenSSH server automation for testing.
The executor.ssh.server
module defines the SSHServer
class
which can be used to start temporary OpenSSH servers that are isolated enough
from the host system to make them usable in the executor
test suite (to
test remote command execution).
-
executor.ssh.server.
SSHD_PROGRAM_NAME
= 'sshd'¶ The name of the SSH server executable (a string).
-
class
executor.ssh.server.
SSHServer
(**options)[source]¶ Subclass of
ExternalCommand
that manages a temporary SSH server.The OpenSSH server spawned by the
SSHServer
class doesn’t need superuser privileges and doesn’t require any changes to/etc/passwd
or/etc/shadow
.Here’s an overview of the
SSHServer
class:Superclass: EphemeralTCPServer
Special methods: __init__()
Public methods: cleanup()
,generate_config()
,generate_key_file()
andstart()
Properties: client_options
andsshd_path
-
__init__
(**options)[source]¶ Initialize an
SSHServer
object.Parameters: options – All keyword arguments are passed on to executor.ExternalCommand.__init__()
.
-
temporary_directory
= None¶ The pathname of the temporary directory used to store the files required to run the SSH server (a string).
-
client_key_file
= None¶ The pathname of the generated OpenSSH client key file (a string).
-
config_file
= None¶ The pathname of the generated OpenSSH server configuration file (a string).
-
host_key_file
= None¶ The random port number on which the SSH server will listen (an integer).
-
sshd_path
¶ The absolute pathname of
SSHD_PROGRAM_NAME
(a string).
-
client_options
¶ The options for the OpenSSH client (required to connect with the server).
This is a dictionary of keyword arguments for
RemoteCommand
to make it connect with the OpenSSH server (assuming the remote command connects to an IP address in the 127.0.0.0/24 range).
-
start
(**options)[source]¶ Start the SSH server and wait for it to start accepting connections.
Parameters: options – Any keyword arguments are passed to the start()
method of the superclass.Raises: Any exceptions raised by the start()
method of the superclass.The
start()
method automatically calls thegenerate_key_file()
andgenerate_config()
methods.
-
generate_key_file
(filename)[source]¶ Generate a temporary host or client key for the OpenSSH server.
The
start()
method automatically callsgenerate_key_file()
to generatehost_key_file
andclient_key_file
. This method uses thessh-keygen
program to generate the keys.
-
generate_config
()[source]¶ Generate a configuration file for the OpenSSH server.
The
start()
method automatically callsgenerate_config()
.
-
cleanup
()[source]¶ Clean up
temporary_directory
after the test server finishes.
-
-
class
executor.ssh.server.
EphemeralTCPServer
(*command, **options)[source]¶ Make it easy to launch ephemeral TCP servers.
The
EphemeralTCPServer
class makes it easy to allocate an ephemeral port number that is not (yet) in use.Here’s an overview of the
EphemeralTCPServer
class:Superclasses: ExternalCommand
andEphemeralPortAllocator
Public methods: start()
Properties: asynchronous
-
asynchronous
¶ Ephemeral TCP servers always set
ExternalCommand.asynchronous
toTrue
.
-
start
(**options)[source]¶ Start the TCP server and wait for it to start accepting connections.
Parameters: options – Any keyword arguments are passed to the start()
method of the superclass.Raises: Any exceptions raised by start()
andwait_until_connected()
.If the TCP server doesn’t start accepting connections within the configured timeout (see
wait_timeout
) the process will be terminated and the timeout exception is propagated.
-
-
exception
executor.ssh.server.
TimeoutError
[source]¶ Raised when a TCP server doesn’t start accepting connections quickly enough.
This exception is raised by
wait_until_connected()
when the TCP server doesn’t start accepting connections within a reasonable time.
The executor.tcp
module¶
Miscellaneous TCP networking functionality.
The functionality in this module originated in the executor.ssh.server
module with the purpose of facilitating a robust automated test suite for the
executor.ssh.client
module. While working on SSH tunnel support I
needed similar logic again and I decided to extract this code from the
executor.ssh.server
module.
-
class
executor.tcp.
WaitUntilConnected
(**kw)[source]¶ Wait for a TCP endpoint to start accepting connections.
Here’s an overview of the
WaitUntilConnected
class:Superclass: PropertyManager
Public methods: wait_until_connected()
Properties: connect_timeout
,endpoint
,hostname
,is_connected
,port_number
,scheme
andwait_timeout
When you initialize a
WaitUntilConnected
object you are required to provide a value for theport_number
property. You can set the values of theconnect_timeout
,hostname
,port_number
,scheme
andwait_timeout
properties by passing keyword arguments to the class initializer.-
connect_timeout
[source]¶ The timeout in seconds for individual connection attempts (a number, defaults to 2).
Note
The
connect_timeout
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
endpoint
¶ A human friendly representation of the TCP endpoint (a string containing a URL).
-
hostname
[source]¶ The host name or IP address to connect to (a string, defaults to
localhost
).Note
The
hostname
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
port_number
[source]¶ The port number to connect to (an integer).
Note
The
port_number
property is arequired_property
. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named port_number (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.
-
scheme
[source]¶ A URL scheme that indicates the purpose of the ephemeral port (a string, defaults to ‘tcp’).
Note
The
scheme
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
wait_timeout
[source]¶ The timeout in seconds for
wait_until_connected()
(a number, defaults to 30).Note
The
wait_timeout
property is amutable_property
. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can usedel
ordelattr()
.
-
wait_until_connected
()[source]¶ Wait until connections are being accepted.
Raises: TimeoutError
when the SSH server isn’t fast enough to initialize.
-
-
class
executor.tcp.
EphemeralPortAllocator
(**kw)[source]¶ Allocate a free ephemeral port number.
Here’s an overview of the
EphemeralPortAllocator
class:Superclass: WaitUntilConnected
Properties: ephemeral_port_number
andport_number
-
port_number
[source]¶ A dynamically selected free ephemeral port number (an integer between 49152 and 65535).
Note
The
port_number
property is alazy_property
. This property’s value is computed once (the first time it is accessed) and the result is cached.
-
ephemeral_port_number
¶ A random ephemeral port number (an integer between 49152 and 65535).
-
-
class
executor.tcp.
EphemeralTCPServer
(*command, **options)[source]¶ Make it easy to launch ephemeral TCP servers.
The
EphemeralTCPServer
class makes it easy to allocate an ephemeral port number that is not (yet) in use.Here’s an overview of the
EphemeralTCPServer
class:Superclasses: ExternalCommand
andEphemeralPortAllocator
Public methods: start()
Properties: asynchronous
-
asynchronous
¶ Ephemeral TCP servers always set
ExternalCommand.asynchronous
toTrue
.
-
start
(**options)[source]¶ Start the TCP server and wait for it to start accepting connections.
Parameters: options – Any keyword arguments are passed to the start()
method of the superclass.Raises: Any exceptions raised by start()
andwait_until_connected()
.If the TCP server doesn’t start accepting connections within the configured timeout (see
wait_timeout
) the process will be terminated and the timeout exception is propagated.
-
-
exception
executor.tcp.
TimeoutError
[source]¶ Raised when a TCP server doesn’t start accepting connections quickly enough.
This exception is raised by
wait_until_connected()
when the TCP server doesn’t start accepting connections within a reasonable time.
Change log¶
The change log lists notable changes to the project:
Changelog¶
The purpose of this document is to list all of the notable changes to this project. The format was inspired by Keep a Changelog. This project adheres to semantic versioning.
- Release 23.2 (2020-11-19)
- Release 23.1 (2020-05-14)
- Release 23.0 (2020-05-14)
- Release 22.0 (2020-03-02)
- Release 21.3 (2018-11-17)
- Release 21.2 (2018-10-11)
- Release 21.1.1 (2018-10-08)
- Release 21.1 (2018-10-07)
- Release 21.0 (2018-10-07)
- Release 20.0.1 (2018-10-07)
- Release 20.0 (2018-05-21)
- Release 19.3 (2018-05-04)
- Release 19.2 (2018-04-27)
- Release 19.1 (2018-03-25)
- Release 19.0 (2018-02-25)
- Release 18.1 (2018-01-21)
- Release 18.0 (2017-06-28)
- Release 17.1 (2017-06-21)
- Release 17.0 (2017-06-10)
- Release 16.1 (2017-06-08)
- Release 16.0.1 (2017-04-13)
- Release 16.0 (2017-04-13)
- Release 15.1 (2017-01-10)
- Release 15.0 (2016-12-20)
- Release 14.1 (2016-10-12)
- Release 14.0 (2016-08-10)
- Release 13.0 (2016-07-09)
- Release 12.0 (2016-07-09)
- Release 11.0.1 (2016-07-09)
- Release 11.0 (2016-06-03)
- Release 10.1 (2016-06-03)
- Release 10.0 (2016-06-01)
- Release 9.11 (2016-05-27)
- Release 9.10 (2016-05-27)
- Release 9.9 (2016-04-21)
- Release 9.8 (2016-04-13)
- Release 9.7 (2016-04-09)
- Release 9.6.1 (2016-04-07)
- Release 9.6 (2016-04-07)
- Release 9.5 (2016-04-03)
- Release 9.4 (2016-04-03)
- Release 9.3 (2016-03-22)
- Release 9.2 (2016-03-22)
- Release 9.1 (2016-03-22)
- Release 9.0.1 (2016-03-21)
- Release 9.0 (2016-02-20)
- Release 8.4 (2016-02-20)
- Release 8.3 (2016-01-24)
- Release 8.2 (2016-01-14)
- Release 8.1.1 (2016-01-13)
- Release 8.1 (2016-01-13)
- Release 8.0.1 (2015-11-14)
- Release 8.0 (2015-11-13)
- Release 7.7 (2015-11-10)
- Release 7.6 (2015-11-10)
- Release 7.5 (2015-11-08)
- Release 7.4 (2015-11-08)
- Release 7.2 (2015-11-08)
- Release 7.1.1 (2015-10-18)
- Release 7.1 (2015-10-18)
- Release 7.0.1 (2015-10-06)
- Release 7.0 (2015-10-06)
- Release 6.2 (2015-10-06)
- Release 6.1 (2015-10-05)
- Release 6.0 (2015-10-05)
- Release 5.1 (2015-10-05)
- Release 5.0.1 (2015-10-05)
- Release 5.0 (2015-10-04)
- Release 4.9 (2015-10-02)
- Release 4.8 (2015-10-02)
- Release 4.7 (2015-10-02)
- Release 4.6 (2015-10-02)
- Release 4.5 (2015-10-02)
- Release 4.4.1 (2015-08-30)
- Release 4.4 (2015-05-30)
- Release 4.3 (2015-05-30)
- Release 4.2 (2015-05-30)
- Release 4.1 (2015-05-29)
- Release 4.0.1 (2015-05-28)
- Release 4.0 (2015-05-27)
- Release 3.6 (2015-05-27)
- Release 3.5 (2015-05-26)
- Release 3.4.1 (2015-05-26)
- Release 3.4 (2015-05-26)
- Release 3.3 (2015-05-26)
- Release 3.2 (2015-05-26)
- Release 3.1 (2015-05-25)
- Release 3.0.2 (2015-05-25)
- Release 3.0.1 (2015-05-25)
- Release 3.0 (2015-05-25)
- Release 2.4 (2015-05-24)
- Release 2.3 (2015-05-24)
- Release 2.2.2 (2015-05-24)
- Release 2.2.1 (2015-05-24)
- Release 2.2 (2015-05-24)
- Release 2.1 (2015-05-23)
- Release 2.0 (2015-05-23)
- Release 1.7.1 (2015-03-05)
- Release 1.7 (2015-03-05)
- Release 1.6.2 (2015-03-04)
- Release 1.6.1 (2015-03-04)
- Release 1.6 (2014-10-18)
- Release 1.5 (2014-10-18)
- Release 1.4 (2014-10-18)
- Release 1.3 (2014-06-07)
- Release 1.2 (2014-05-10)
- Release 1.1 (2014-05-04)
- Release 1.0 (2014-05-04)
Release 23.2 (2020-11-19)¶
Enhancements:
- Enable control of spinners with
foreach()
using thespinner
keyword (previously this would result inTypeError
exceptions).
Changes to Travis CI:
Stabilized the Python 3.5 job (testing requirements).
Stabilized the PyPy (2.7) job (cryptography incantations).
Reordered the build matrix by descending runtime (to optimize total runtime).
Allow PyPy job failures without failing the complete Travis CI build, because failures in the PyPy job tend to be caused by PyPy incompatibilities.
In fact I’ve seen hundreds of PyPy job failures on Travis CI over the years while developing my open source projects and less than 1% of these pertained to actual PyPy incompatibilities in the code bases I develop…
Release 23.1 (2020-05-14)¶
As requested in issue #2: Enable control over the spinner that’s
shown when a command pool is active. For more details refer to the
CommandPool.spinner
documentation.
Release 23.0 (2020-05-14)¶
This release includes two notable changes:
- Improved compatibility of release discovery
As reported in issue #17 official Debian Docker images don’t include the file
/etc/lsb-release
nor the program lsb_release and this breaks “release discovery” using thedistributor_id
anddistribution_codename
properties. This has been fixed by implementing a fall back that tries to parse the/etc/apt/sources.list
file to determine the package mirror URL which is matched against the mirror URLs inMIRROR_TO_DISTRIB_MAPPING
.Note
Because this concerns a fall back the risk of regressions is small.
- Fixed Unicode inconsistency on Python 2.7
It turns out that while
shlex.split()
on Python 2.7 accepts Unicode strings it doesn’t actually support them: The result is a list of byte strings and any values that ASCII doesn’t support result in an error.This caused the values (but not the keys) in the dictionary provided by
lsb_release_variables
to become byte strings which in turn causeddistributor_id
anddistribution_codename
to become byte strings (when those properties are based onlsb_release_variables
).However when
distributor_id
anddistribution_codename
are based on the output of the lsb_release program the values become Unicode strings, so this unfortunate behavior ofshlex.split()
was causing rather inconsistent behavior in the executor package.This has been fixed by applying a workaround on Python 2.7 (the text is encoded before passing it to
shlex.split()
and the result is decoded before use).Note
While this concerns a small detail in the grand scheme of things it is technically backwards incompatible and version numbers are cheap, hence why this is being released as 23.0.
Release 22.0 (2020-03-02)¶
Maintenance release that changes the compatible Python versions.
Noteworthy changes:
- Documented support for Python 3.8.
- Dropped support for Python 2.6 and 3.4.
- Avoid cyclic dependencies in the
executor.tcp
module. I’ve been working on readying a new project for publication and started using theEphemeralTCPServer
class in its test suite, however I ran into an unnecessary cyclic dependency that causedlogging
to print dramatic tracebacks (although nothing actually failed):- The
__init__()
method needed access to the ephemeral TCP port (because it is passed to the command that’s started) which implies running a whole lot of code (to pick a port that isn’t in use yet) and this code loggedEphemeralTCPServer.__str__()
. - This was intended to use
WaitUntilConnected.__str__()
however due to incorrect superclass ordering it calledExternalCommand.__str__()
instead which needs access to thecommand
property which in turn requires__init__()
to have already been run! - This catch-22 was broken by removing the
__str__()
from logging and using a newly addedendpoint
property instead (explicit is better than implicit).
- The
Miscellaneous changes:
- Bumped humanfriendly to 8.0 and property-manager to 3.0 to fix deprecated imports and resolve a backwards incompatibility in the test suite (introduced by the humanfriendly 8.0 release).
- Changed
Makefile
to use Python 3 during development. - Improved the ionice tests.
Release 21.3 (2018-11-17)¶
Merged pull request #16 that changes the ionice integration to accept the strings ‘1’, ‘2’ and ‘3’ in addition to ‘idle’, ‘best-effort’ and ‘realtime’ because busybox doesn’t support the verbose strings.
It’s still up to the caller to pick the right kind of value and I’m a bit conflicted about that because it’s creating a leaky abstraction. I may at a later point decide to add automatic translation from the verbose labels to the numeric codes (which seem to be the lowest common denominator that’s always supported) …
Release 21.2 (2018-10-11)¶
Enable context.read_file(..., sudo=True)
and context.write_file(...,
sudo=True)
. In fact all optional keyword arguments are supported (not just
sudo) but for me the most important one is sudo=True
because I strongly
prefer “selective sudo” over “just run everything using sudo”.
Release 21.1.1 (2018-10-08)¶
Bug fix of sorts: Guard against binary data in /etc/lsb-release
on Travis
CI. This problem became apparent after #10 triggered some new development.
Since then I created #15 to track this specific issue.
Release 21.1 (2018-10-07)¶
Improve compatibility with “vanilla Ubuntu 18.04 docker images” by parsing the
file /etc/lsb-release
when the program /usr/bin/lsb_release
isn’t
installed (fixes #10).
This enables the distributor_id
and distribution_codename
properties to
work even when the /usr/bin/lsb_release
program isn’t installed, by parsing
the /etc/lsb-release
file instead. Tested on Ubuntu 14.04, 16.04 and 18.04.
Release 21.0 (2018-10-07)¶
Implemented Python 3.7 compatibility.
Python 3.7 was released in June 2018 and introduced the reserved keyword
async
which made the definition of the ExternalCommand.async
property a syntax error. Should have seen that coming 😒.
In any case, due to personal circumstances I haven’t had time for any open source programming in the past few months which meant feedback on this issue piled up in the form of issue #9 and pull requests #11 and #13:
Apart from the naming difference both pull requests represented the same
change, however I prefer asynchronous
over _async
because I have a
strong dislike for leading and trailing underscores that have no semantic value
except to avoid using a reserved keyword (I’m looking at you SQLAlchemy 😛).
There was one thing that bugged me about all of this though: While it was clear
that ExternalCommand.async
needed to be renamed I didn’t feel like breaking
backwards compatibility with lots of existing Python 2 code using executor with
the old async
naming. That’s why I’ve updated the code to programatically
add an async
alias that defers to the real asynchronous
property.
Because this is done using the setattr()
function no reserved keywords are
harmed in the process 😇.
I’ve also added Python 3.7 to the supported and tested Python releases.
Release 20.0.1 (2018-10-07)¶
- Bug fix: Merged pull request #14 to make
ionice_command
compatible with olderionice
versions not supporting the--class
option. - Lots of commit noise to debug Python 2.6 support on Travis CI. I’m not sure why I still bother…
Release 20.0 (2018-05-21)¶
While intended to be fully backwards compatible (because the new behavior is opt-in) I decided to bump the major version number in this release because adding retry support touched on some of the most critical pieces of code in this project.
- Experimental support for retrying of commands that fail. Retrying of asynchronous commands is only supported in the context of command pools.
- Bug fix: Pass keyword arguments of
wait()
towait_for_process()
. - Fix Sphinx warnings (mostly broken references).
Notes about retry support¶
I’ve been wanting to add retry support to executor for quite a while now. One thing that I struggled with until recently was how to support retrying of synchronous and asynchronous commands in a way that made sense for both types of commands, without compromising too much on the simplicity of the Python API or the actual implementation code.
In a pragmatic “just implement something and see how it works” moment I
decided to add support for retrying of synchronous commands to the
ExternalCommand
class while requiring the use of a command pool to retry
asynchronous commands. Although this implementation doesn’t cover every
possible use case I do believe it covers the most important use cases. Some
high-level implementation notes:
- Synchronous commands are retried inside of the
start()
method. The second part of this method was extracted into a newstart_once()
method and then a loop was added tostart()
that callsstart_once()
until the command succeeds. - Asynchronous commands allow for retry behavior to be configured but won’t
actually run a command more than once unless used in the context of command
pools. I did experiment with retrying of asynchronous commands inside the
wait()
method but this ended up creating an API whose behavior was very unintuitive (changing its behavior from non blocking to blocking in order to retry on failure).
Release 19.3 (2018-05-04)¶
- Added
SecureTunnel
class for easy to use SSH tunnels (ssh -NL ...
). - Added
RemoteCommand.compression
property to enablessh -C
. - Extracted generic TCP functionality from the
executor.ssh.server
module into a newexecutor.tcp
module (so that the functionality could be reused by the new SSH tunnel support).
Release 19.2 (2018-04-27)¶
- Added a
glob()
method to contexts (this was triggered by the feature request in rotate-backups issue #10). - Improved documentation using
property_manager.sphinx
. - Added this changelog, restructured the online documentation.
- Include documentation in source distributions.
- Added
license
key tosetup.py
script.
Release 19.1 (2018-03-25)¶
Added context.is_executable()
shortcut.
Release 19.0 (2018-02-25)¶
Backwards incompatible: Report command output on failure.
Refer to the new really_silent
property for details about how this is
backwards incompatible. I suspect this to bite less than 1% of use cases
and I want executor to have sane defaults, so there :-).
Release 18.1 (2018-01-21)¶
Release 18.0 (2017-06-28)¶
Several backwards incompatible changes were made in an attempt to improve the consistency of error handling:
- Bug fix: Set returncode on OSError exception
- Bug fix: Don’t leave std{out,err} unset on OSError
- Don’t raise exceptions from lsb_release shortcuts.
- Update usage in readme.
- Move test helpers to
humanfriendly.testing
.
Release 17.1 (2017-06-21)¶
Added support for Python callbacks in context.cleanup()
.
Release 17.0 (2017-06-10)¶
- Rename
ChangeRoot*
toSecureChangeRoot*
to avoid an upcoming name collision (backwards incompatible!). - Added support for command execution in chroots using the
chroot
command. - Reduced code duplication of
&&
logic.
Release 16.1 (2017-06-08)¶
- Give contexts some
lsb_release
shortcuts. - Add Python 3.6 to tested versions.
Release 16.0.1 (2017-04-13)¶
Bug fix: Allow explicitly setting ionice=None
.
Release 16.0 (2017-04-13)¶
- Make it very easy to use
ionice
. - Add simple wrapper for
which
(context.find_program()
). - Avoid nested shell in
context.prepare_interactive_shell()
. - Don’t add trailing
--
inChangeRootCommand.command_line
. - Change default working directory in chroots (backwards incompatible, although
I wouldn’t be surprised if there are zero uses of the
executor.schroot
module outside of the code bases I maintain :-).
Release 15.1 (2017-01-10)¶
- Merged pull request #3: Allow disabling of spinners.
- Bug fix: Stop timer used by
wait_for_process()
after waiting. - Bumped humanfriendly requirement for upstream bug fix.
Release 15.0 (2016-12-20)¶
- Added support for command execution in chroots using
schroot
. - Added experimental support for nested contexts.
Release 14.1 (2016-10-12)¶
Added support for atomic file writes using execution contexts.
Release 14.0 (2016-08-10)¶
Enable passing shell commands via stdin without specifying a command. Strictly speaking this change is not backwards compatible but my impression is that this won’t break any valid, existing use cases.
Release 13.0 (2016-07-09)¶
Improve concurrency control for command pools
Previously there was only CommandPool.concurrency
to control how many
commands were allowed to run concurrently, now the caller can control which
commands are allowed to run concurrently (using the two new properties
ExternalCommand.dependencies
and group_by
).
Release 12.0 (2016-07-09)¶
Connect stdin to /dev/null
in command pools (backwards incompatible!)
Recently I ran into some spectacularly weird failures and it took me a
while to realize that it was happening because a command pool with SSH
client commands was running multiple SSH clients concurrently and each
of the SSH clients was allocating a pseudo-tty (ssh -t
).
I’m currently under the impression that this new behavior is the only sane choice, even if it is backwards incompatible. Here’s hoping I thought that through well enough before releasing this change :-).
Release 11.0.1 (2016-07-09)¶
- Bug fix: Allow assignment of individual environment variables.
- Refactored makefile and
setup.py
script (checkers, docs, wheels, twine, etc).
Release 11.0 (2016-06-03)¶
Connect stdin to /dev/null
when tty=False
(backwards incompatible!)
Recently I ran into several external commands whose output was being
captured and thus not visible, but which nevertheless rendered an
interactive prompt, waiting for a response on standard input (which
I wasn’t providing because I never saw the interactive prompt :-).
The option to connect stdin and /dev/null
was never available in
executor, however given the recent addition of the tty
option it
seemed logical to combine the two.
Two changes in this commit backwards incompatible:
- The standard input stream of external commands was never connected to
/dev/null
before and this is changing without an explicit opt-in or opt-out mechanism. I’m making this choice because I believe it to be the only sane approach. - The interface of the
CachedStream
class has changed even though this is a documented, externally available class. However I don’t actually see anyone usingCachedStream
outside of the executor project, so in the grand scheme of things this is a minor thing (99% of users will never even notice, I’m guessing).
Release 10.1 (2016-06-03)¶
Added support for start_event
and finish_event
callbacks.
Release 10.0 (2016-06-01)¶
Large refactoring concerning executor
/ proc
separation of concerns,
backwards incompatible!
In executor 7.7 the process management functionality was decoupled from external command execution in order to re-use the process management functionality in my proc package (this was integrated into proc 0.4). In retrospect I implemented this refactoring (in November ‘15) too hastily because the UNIX signal handling doesn’t belong in the executor package (it’s meant to be portable). Last weekend I decided to finally do something about this! I’m only committing this now because it took me days to clean up, stabilize, document and test the refactoring :-). A high level summary:
All process manipulation that uses UNIX signals is being moved to the ‘proc’ package, that includes things like SIGSTOP / SIGCONT. This means that the methods
ControllableProcess.suspend()
andControllableProcess.resume()
are no longer available. This will break fresh installations of my ‘proc’ package until I release a new version, because I haven’t pinned the max version of dependencies I control. The new release of ‘proc’ is waiting to be uploaded though :-).The ‘executor’ package no longer keeps references to
subprocess.Popen
objects after the process has finished, to allow garbage collection. This should resolve an issue I was seeing recently when I was pushing the limits of executor command pools and ran intoIOError: [Errno 24] Too many open files
.Someone on StackOverflow with the same problem: http://stackoverflow.com/questions/6669996/python-subprocess-running-out-of-file-descriptors
Someone on StackOverflow who knows how to fix it: http://stackoverflow.com/a/23763193/788200
While implementing this refactoring I had a lot of trouble making sure that
ExternalCommand.pid
andreturncode
would be preserved when thesubprocess
reference was destroyed (it seems so obvious, but nevertheless this tripped me up). The test suite agrees with me that I got things right eventually, so here’s hoping for no external breakage :-).
Release 9.11 (2016-05-27)¶
Make it possible to disable command pool spinners.
Release 9.10 (2016-05-27)¶
ExternalCommand
and RemoteCommand
objects now have a tty
option to
express whether they need to and/or will be connected to an interactie terminal.
Release 9.9 (2016-04-21)¶
Bug fix: Preserve environment variables when using sudo
.
Release 9.8 (2016-04-13)¶
Make it easy to test contexts for superuser privileges.
Release 9.7 (2016-04-09)¶
Added a shortcut for context creation (executor.contexts.create_context()
).
Release 9.6.1 (2016-04-07)¶
Bug fix for previous commit.
Release 9.6 (2016-04-07)¶
Make remote commands optional (stdin only is a valid use case).
Release 9.5 (2016-04-03)¶
Provide contexts shortcuts for various test
program invocations.
Release 9.4 (2016-04-03)¶
Automatically get the SSH username from the given SSH alias when available
(delimited by an @
sign).
Release 9.3 (2016-03-22)¶
- Added support for listing directory entries using execution contexts.
- Stop Travis CI from testing tagged releases (I create a lot of them :-).
- Introduce context manager for temporary directories in test suite.
Release 9.2 (2016-03-22)¶
Improved RemoteContext.cpu_count
(by adding a fallback for nproc
).
Release 9.1 (2016-03-22)¶
Support for reading and writing of files using execution contexts.
Release 9.0.1 (2016-03-21)¶
Bug fix: Proper error messages for RemoteCommandNotFound
.
Release 9.0 (2016-02-20)¶
- Backwards incompatible: Removed
fakeroot
→sudo
fallback behavior. - Added more documentation of the
uid
anduser
options. - Documented tested interpreters with trove classifiers.
Release 8.4 (2016-02-20)¶
- Make it possible to run commands as specific users (via
sudo
). - Add Python 3.5 to tested versions and document support.
- Refactored
setup.py
script, add trove classifiers. - Moved Sphinx customizations to humanfriendly package.
Release 8.3 (2016-01-24)¶
- Make it possible to explicitly enable/disable shell evaluation.
- Expand documentation of callback/result properties.
Release 8.2 (2016-01-14)¶
Experimental support for ‘result processing’ callbacks.
Release 8.1.1 (2016-01-13)¶
Enable custom loggers for remote commands.
Release 8.1 (2016-01-13)¶
- Added
remote()
shortcut (execute()
for remote commands). - Simplified
RemoteCommand.command_line
. - Improved documentation of
execute()
function.
Release 8.0.1 (2015-11-14)¶
Silence ‘make check’ (now failing on Travis CI).
Release 8.0 (2015-11-13)¶
- Added a command line interface: The
executor
program. - Improved documentation after previous refactoring.
Release 7.7 (2015-11-10)¶
Better process management, decoupled from ExternalCommand
.
Release 7.6 (2015-11-10)¶
- Automatically set
async=True
when used as context manager. - Minor improvements to
executor.ssh.server
module. - Improve how Sphinx generates the documentation:
- Configure Sphinx not to skip magic methods by default.
- Order autodoc entries by source, not alphabetically.
Release 7.5 (2015-11-08)¶
- Change default logger of commands executed in pools.
- Extract ephemeral TCP server support from
executor.ssh.server.SSHServer
.
Release 7.4 (2015-11-08)¶
- Decompose
ExternalCommand.start()
. - Introduce
CommandNotFound
subclass ofExternalCommandFailed
.
Release 7.2 (2015-11-08)¶
- Decompose
executor.which()
and add Windows support. - Disable capturing in pytest.ini (because it breaks
sudo
tests).
Release 7.1.1 (2015-10-18)¶
- Bug fix for integration of
ExternalCommandFailed
/TimeoutError
exceptions. - Improve documentation of
virtual_environment
option.
Release 7.1 (2015-10-18)¶
Make it easy to run commands in Python virtual environments.
Release 7.0.1 (2015-10-06)¶
Bug fix: Only raise CommandPoolFailed
for commands with check=True
.
Release 7.0 (2015-10-06)¶
foreach()
now sets delay_checks=True
by default.
This change is not backwards compatible but IMHO it fits in the scheme of “making it easy to do the right thing”. For further argumentation refer to the updated documentation.
Release 6.2 (2015-10-06)¶
Enable delayed error checking for command pools.
Release 6.1 (2015-10-05)¶
Tag exceptions with the command pool from which they were raised.
Release 6.0 (2015-10-05)¶
Make CommandPool.run()
terminate commands before aborting.
This bumps the major version number because the change isn’t backwards compatible (although I believe it does make for more sane default behavior) and version numbers are cheap :-).
Release 5.1 (2015-10-05)¶
Make it possible to terminate command pools.
Release 5.0.1 (2015-10-05)¶
- Bug fix: Make
CommandPool.collect()
resumable after failing commands. - Enable intersphinx mapping from
executor
toproperty-manager
. - Removed minor (trivial) code duplication from
CommandPool.run()
. - Renamed ‘construct’ to ‘initialize’ where applicable: A constructor in Python
is called
__new__()
and overriding it is the exception, not the norm. Overriding the__init__()
method is the norm, but then__init__()
is not a constructor, it’s an “initializer”.
Release 5.0 (2015-10-04)¶
Promote executor.property_manager
to a separate property-manager package
(I’d been wanting to reuse this functionality in several other packages for a
while now).
Release 4.9 (2015-10-02)¶
Change executor.ssh.client.foreach()
to use SSH aliases as identifiers.
Release 4.8 (2015-10-02)¶
Change command pool output logging to append instead of overwrite.
Release 4.7 (2015-10-02)¶
Support capturing foreach()
command pool output to logs directory.
Release 4.6 (2015-10-02)¶
Support capturing command pool output to logs directory.
Release 4.5 (2015-10-02)¶
- Bug fix: Python 3 doesn’t support ur”strings” (Unicode raw strings)
- Support redirecting standard streams to files provided by caller.
- Implement and enforce PEP-8 and PEP-257 compliance.
Release 4.4.1 (2015-08-30)¶
- Bug fix for obscure
UnicodeDecodeError
insetup.py
(on Python 3 only). - Make Travis CI builds fail when coverage isn’t >= 90%.
- Also run the tests under PyPy on Travis CI.
Release 4.4 (2015-05-30)¶
Expose the CPU count of execution contexts.
Release 4.3 (2015-05-30)¶
Give contexts a test()
method.
Release 4.2 (2015-05-30)¶
Enable context users to prepare commands without starting them.
Release 4.1 (2015-05-29)¶
Make it possible to nest ‘unwind contexts’ (executor.contexts
).
Release 4.0.1 (2015-05-28)¶
Bug fix for remote working directory logic.
Release 4.0 (2015-05-27)¶
Added support for external command contexts (agnostic to local vs. remote execution).
Release 3.6 (2015-05-27)¶
Support non-default remote working directories.
Release 3.5 (2015-05-26)¶
Added a RemoteCommandPool
class.
Release 3.4.1 (2015-05-26)¶
Default to StrictHostKeyChecking=no
for SSH commands.
Release 3.4 (2015-05-26)¶
Make the decoded values of stdout/stderr available.
Release 3.3 (2015-05-26)¶
Made it possible to merge the standard output and error streams.
Release 3.2 (2015-05-26)¶
Made it possible to capture the standard error stream.
Release 3.1 (2015-05-25)¶
Added ExternalCommand.succeeded
and failed
properties.
Release 3.0.2 (2015-05-25)¶
Don’t set the SSH port number to 22 by default (let the SSH client program figure it out instead).
Release 3.0.1 (2015-05-25)¶
Bug fix for setup.py
(forgot to remove import).
Release 3.0 (2015-05-25)¶
- Added support for remote command execution using SSH.
- Improved
ExternalCommand
documentation.
Release 2.4 (2015-05-24)¶
Make ExternalCommand
a context manager.
Release 2.3 (2015-05-24)¶
Made it possible to terminate external commands.
Release 2.2.2 (2015-05-24)¶
Improved logging output of CommandPool.run()
.
Release 2.2.1 (2015-05-24)¶
Bug fix for import error in executor.compat
module.
Release 2.2 (2015-05-24)¶
Properly distinguish writable properties from ‘reset-able’ properties.
Release 2.1 (2015-05-23)¶
Added support for concurrent external command execution (command pools).
Release 2.0 (2015-05-23)¶
- Added support for asynchronous command execution (and lots of small things).
- Improve formatting of
ExternalCommandFailed
attributes in documentation.
Release 1.7.1 (2015-03-05)¶
Fixed __version__
variable corruption introduced in 1.7 :-S.
Release 1.7 (2015-03-05)¶
Make it possible to provide overrides for environment variables (#1).
Release 1.6.2 (2015-03-04)¶
- Stop mixing SH and Bash usage (consistently use Bash everywhere).
- Documented that the encoding option is used for input and output
- Added
tox.ini
for easy testing and executetox
usingmake test
.
Release 1.6.1 (2015-03-04)¶
Bug fix: Properly close open file handle to /dev/null
.
This fixes the following warning emitted by Python 3.4:
ResourceWarning: unclosed file <_io.BufferedWriter name='/dev/null'>
Release 1.6 (2014-10-18)¶
Expose pipes.quote()
wrapping logic as executor.quote()
.
Release 1.5 (2014-10-18)¶
Added support for execute(..., silent=True)
which silences the standard
output and error streams.
Release 1.4 (2014-10-18)¶
- Extend
ExternalCommandFailed
to exposecommand
andreturncode
attributes. - Get test coverage up to 100%.
- Fixed Sphinx documentation warning about missing static directory.
- Added a simple
Makefile
for common project maintenance tasks.
Release 1.3 (2014-06-07)¶
- Added support for
fakeroot
. - Added a
which()
function. - Submit test coverage from Travis CI to Coveralls.
Release 1.2 (2014-05-10)¶
- Improved Python 3 compatibility: - Remove irregular raise syntax. - First experience with bytes vs strings.
- Documented supported Python versions (2.6, 2.7 and 3.4).
- Started using Travis CI to automatically run the test suite.
Release 1.1 (2014-05-04)¶
Improved the documentation.
Release 1.0 (2014-05-04)¶
Initial commit.