ci_exec

About

A wrapper package designed for running continuous integration (CI) build steps using Python 3.5+.

Managing cross platform build scripts for CI can become tedious at times when you need to e.g., maintain two nearly identical scripts install_deps.sh and install_deps.bat due to incompatible syntaxes. ci_exec enables a single file to manage this using Python.

The ci_exec package provides a set of wrappers / utility functions designed specifically for running build steps on CI providers. It is

Logging by Default

Commands executed, including their full command-line arguments, are logged. This includes any output on stdout / stderr from the commands. The logging resembles what set -x would give you in a shell script. For commands that will take a long time, as long as output is being produced, this will additionally prevent timeouts on the build.

Failing by Default

Any command that does not succeed will fail the entire build. An attempt to exit with the same exit code as the command that failed will be performed. Meaning the CI provider will correctly report a failed build.

Convenient

ci_exec affords users the ability to write shell-like scripts that will work on any platform that Python can run on. A simple example:

from ci_exec import cd, which

cmake = which("cmake")
ninja = which("ninja")
with cd("build", create=True):
    cmake("..", "-G", "Ninja", "-DCMAKE_BUILD_TYPE=Release")
    ninja("-j", "2", "test")

Intended Audience

Note

ci_exec can be used for anything related to writing build steps, but it was originally written to manage C++ projects. The documentation will often have examples using cmake and ninja, users do not need to understand what these commands are for.

ci_exec utilizes some “advanced” features of Python that pertain to how the library itself is consumed. It may not be appropriate for users who do not know any Python at all. The main features a user should be aware of:

• *args and **kwargs are used liberally. ci_exec mostly consists of wrapper classes / functions around the python standard library, and in most cases *args and **kwargs are the “pass-through” parameters.

• Keyword-only arguments. Many library signatures look something like:

  def foo(a: str, *, b: int = 2):
      pass
  
  foo("hi")       # OK: b = 2
  foo("hi", 3)    # ERROR: b is keyword only
  foo("hi", b=3)  # OK: b = 3
  

  Anything after the *, must be named explicitly.

• Operator overloading, particularly what __call__ means and how it works. A quick overview:

  from ci_exec import Executable
  
  # Typically: prefer ci_exec.which instead, which returns a ci_exec.Executable.
  cmake = Executable("/usr/bin/cmake")
  
  # cmake.__call__ invoked with args = [".."], kwargs = {}
  cmake("..")
  
  # cmake.__call__ invoked with args = [".."], kwargs = {"check": False}
  cmake("..", check=False)
  

None of these features are altogether that special, but it must be stated clearly and plainly: this library is designed for users who already know Python.

Put differently: if you don’t know why writing script-like Python is useful for CI, while still having access to a full-fledged programming language for when it gets (C++) hard, this package likely has no use for you.

Full Documentation

Quick reference:

Ansi	Wrapper class for defining the escape character and clear sequence.
Colors	The core ANSI color codes.
Styles	A non-exhaustive list of ANSI style formats.
colorize(message, *, color)	Return message colorized with specified style.
log_stage(stage, *[, …])	Print a terminal width block with stage message in the middle.
Executable(exe_path, *[, …])	Represent a reusable executable.
fail(why, *[, exit_code, no_prefix])	Write a failure message to sys.stderr and exit.
mkdir_p(path[, mode, parents, …])	Permissive wrapper around pathlib.Path.mkdir().
rm_rf(path[, ignore_errors, …])	Permissive wrapper around shutil.rmtree() bypassing FileNotFoundError and NotADirectoryError.
which(cmd, *[, mode, path])	Restrictive wrapper around shutil.which() that will fail() if not found.
filter_file(path, pattern, repl)	Filter the contents of a file.
unified_diff(from_path, to_path)	Return the unified_diff between two files.
Provider	Check if code is executing on a continuous integration (CI) service.
cd(dest, *[, create])	Context manager / decorator that can be used to change directories.
merge_kwargs(defaults, kwargs)	Merge defaults into kwargs and return kwargs.
set_env(**kwargs)	Context manager / decorator that can be used to set environment variables.
unset_env(*args)	Context manager / decorator that can be used to unset environment variables.

ci_exec

The ci_exec package top-level namespace.

ci_exec.core

The core functionality of the ci_exec package.

fail(why, *[, exit_code, no_prefix])	Write a failure message to sys.stderr and exit.
Executable(exe_path, *[, log_calls, …])	Represent a reusable executable.
mkdir_p(path[, mode, parents, exist_ok])	Permissive wrapper around pathlib.Path.mkdir().
rm_rf(path[, ignore_errors, onerror])	Permissive wrapper around shutil.rmtree() bypassing FileNotFoundError and NotADirectoryError.
which(cmd, *[, mode, path])	Restrictive wrapper around shutil.which() that will fail() if not found.

ci_exec.core.fail(why, *, exit_code = 1, no_prefix = False) → NoReturn

Write a failure message to sys.stderr and exit.

Parameters

• why (str) – The message explaining why the program is being failed out.

• exit_code (int) – The exit code to use. Default: 1.

• no_prefix (bool) – Whether to prefix a bold red "[X] " before why. Default: False, the bold red "[X] " prefix is included unless set to True.

class ci_exec.core.Executable(exe_path, *, log_calls = True, log_prefix = '$ ', log_color = '36', log_style = '1')

Represent a reusable executable.

Each executable is:

1. Failing by default: unless called with check=False, any execution that fails (has a non-zero exit code) will result in a call to fail(), terminating the entire application.

2. Logging by default: every call executed will print what will be run in color and then dump the output of the command. In the event of failure, this makes finding the last call issued much simpler.

Consider the following simple script:

from ci_exec import Executable
git = Executable("/usr/bin/git")
git("remote")
# Oops! --pretty not --petty ;)
git("log", "-1", "--petty=%B")
git("status")  # will not execute (previous failed)

When we execute python simple.py and check the exit code with echo $?:

> python simple.py
$ /usr/bin/git remote
origin
$ /usr/bin/git log -1 --petty=%B
fatal: unrecognized argument: --petty=%B
[X] Command '('/usr/bin/git', 'log', '-1', "--petty=%B")' returned
    non-zero exit status 128.
> echo $?
128

See __call__() for more information.

Tip

Hard-coded paths in these examples were for demonstrative purposes. In practice this should not be done, use which() instead.

exe_path

The path to the executable that will be run when called.

Type

str

log_calls

Whether or not every invocation of __call__() should print what will execute before executing it. Default: True.

Type

bool

log_prefix

The prefix to use when printing a given invocation of __call__(). Default: "$ " to simulate a console lexer. Set to the empty string "" to have no prefix.

Type

str

log_color

The color code to use when calling colorize() to display the next invocation of __call__(). Set to None to disable colorizing each log of __call__(). Default: Colors.Cyan.

Type

str

log_style

The style code to use when calling colorize() to display the next invocation of __call__(). If no colors are desired, set log_color to None. Default: Styles.Bold.

Type

str

Raises

ValueError – If exe_path is not a file, or if it is not executable.

PATH_EXTENSIONS = {}

The set of valid file extensions that can be executed on Windows.

On *nix systems this will be the empty set, and takes no meaning. On Windows, it is controlled by the user. These are stored in lower case, and comparisons should be lower case for consistency. The typical default value on Windows would be:

PATH_EXTENSIONS = {".com", ".exe", ".bat", ".cmd"}

__call__(*args, **kwargs) → subprocess.CompletedProcess

Run exe_path with the specified command-line *args.

The usage of the parameters is best summarized in code:

popen_args = (self.exe_path, *args)
# ... some potential logging ...
return subprocess.run(popen_args, **kwargs)

For example, sending multiple arguments to the executable is as easy as:

cmake = Executable("/usr/bin/cmake")
cmake("..", "-G", "Ninja", "-DBUILD_SHARED_LIBS=ON")

and any overrides to subprocess.run() you wish to include should be done with **kwargs, which are forwarded directly.

Warning

Any exceptions generated result in a call to fail(), which will terminate the application.

Parameters

• *args – The positional arguments will be forwarded along with exe_path to subprocess.run().

• **kwargs –

  The key-value arguments are all forwarded to subprocess.run(). If check is not provided, this is an implicit check=True. That is, if you do not want the application to exit (via fail()), you must specify check=False:

  >>> from ci_exec import Executable
  >>> git = Executable("/usr/bin/git")
  >>> proc = git("not-a-command", check=False)
  $ /usr/bin/git not-a-command
  git: 'not-a-command' is not a git command. See 'git --help'.
  >>> proc.returncode
  1
  >>> git("not-a-command")
  $ /usr/bin/git not-a-command
  git: 'not-a-command' is not a git command. See 'git --help'.
  [X] Command '('/usr/bin/git', 'not-a-command')' returned non-zero exit
      status 1.
  

  The final git("not-a-command") exited the shell (this is what is meant by “failing by default”).

Returns

The result of calling subprocess.run() as outlined above.

Note

Unless you are are calling with check=False, you generally don’t need to store the return type.

Return type

subprocess.CompletedProcess

ci_exec.core.mkdir_p(path, mode = 511, parents = True, exist_ok = True)

Permissive wrapper around pathlib.Path.mkdir().

The intention is to behave like mkdir -p, meaning the only real difference is that parents and exist_ok default to True for this method (rather than False for pathlib).

Parameters

• path (pathlib.Path or str) – The directory path to make.

• mode (int) – Access mask for directory permissions. See pathlib.Path.mkdir().

• parents (bool) – Whether or not parent directories may be created. Default: True.

• exist_ok (bool) –

  Whether or not the command should be considered successful if the specified path already exists. Default: True.

  Note

  If the path exists and is a directory with exist_ok=True, the command will succeed. If the path exists and is a file, even with exist_ok=True the command will fail().

ci_exec.core.rm_rf(path, ignore_errors = False, onerror=None)

Permissive wrapper around shutil.rmtree() bypassing FileNotFoundError and NotADirectoryError.

This function simply checks if path exists first before calling shutil.rmtree(). If the path does not exist, nothing is done. If the path exists but is a file, pathlib.Path.unlink() is called instead.

Essentially, this function tries to behave like rm -rf, but in the event that removal is not possible (e.g., due to insufficient permissions), the function will still fail().

Parameters

• path (pathlib.Path or str) – The directory path to delete (including all children).

• ignore_errors (bool) – Whether or not errors should be ignored. Default: False, to ensure that permission errors are still caught.

• onerror – See shutil.rmtree() for more information on the callback.

ci_exec.core.which(cmd, *, mode = 1, path = None, **kwargs) → ci_exec.core.Executable

Restrictive wrapper around shutil.which() that will fail() if not found.

The primary difference is that when cmd is not found, shutil.which() will return None whereas this function will fail(). If you need to conditionally check for a command, do not use this function, use shutil.which() instead.

Parameters

• cmd (str) – The name of the command to search for. E.g., "cmake".

• mode (int) – The flag permission mask. Default: (os.F_OK | os.X_OK), see: os.F_OK, os.X_OK, shutil.which().

• path (str or None) – Default: None. See shutil.which().

• **kwargs –

  Included as a convenience bypass, forwards directly to Executable constructor. Suppose a non-logging Executable is desired. One option:

  git = which("git")
  git.log_calls = False
  

  Or alternatively:

  git = which("git", log_calls=False)
  

  This is in recognition that for continuous integration users will likely have many different preferences. Users can provide their own which to always use this default, or say, change the logging color:

  from ci_exec import which as ci_which
  from ci_exec import Colors, Styles
  
  def which(cmd: str):
      return ci_which(cmd, log_color=Colors.Magenta, log_style=Styles.Regular)
  

Returns

An executable created with the full path to the found cmd.

Return type

Executable

Tests

Tests for the ci_exec.core module.

tests.core.test_fail(capsys, why, exit_code, no_prefix)

Validate fail() exits as expected.

tests.core.test_executable_construction_failures()

Validate that non-absolute and non-(executable)file constructions will raise.

tests.core.test_executable_relative()

Validate Executable accepts relative paths.

tests.core.test_executable_logging(capsys)

Validate Executable runs and logs as expected.

tests.core.test_executable_failures(capsys)

Validate failing executables error as expected.

tests.core.test_mkdir_p(capsys)

Validate that mkdir_p() creates directories as expected.

tests.core.test_rm_rf(capsys)

Validate rm_rf() deletes files / directories as expected.

tests.core.test_which(capsys)

Validate that which() finds or does not find executables.

ci_exec.colorize

Various utilities for colorizing terminal output.

Ansi	Wrapper class for defining the escape character and clear sequence.
Colors	The core ANSI color codes.
Styles	A non-exhaustive list of ANSI style formats.
colorize(message, *, color[, style])	Return message colorized with specified style.
dump_predefined_color_styles()	Dump all predefined Colors in every Styles to the console.
log_stage(stage, *[, fill_char, pad, l_pad, …])	Print a terminal width block with stage message in the middle.

class ci_exec.colorize.Ansi

Wrapper class for defining the escape character and clear sequence.

Escape = '\x1b['

The opening escape sequence to use before inserting color / style.

Clear = '\x1b[0m'

Convenience definition used to clear ANSI formatting.

class ci_exec.colorize.Colors

The core ANSI color codes.

Black = '30'

The black ANSI color.

Red = '31'

The red ANSI color.

Green = '32'

The green ANSI color.

Yellow = '33'

The yellow ANSI color.

Blue = '34'

The blue ANSI color.

Magenta = '35'

The magenta ANSI color.

Cyan = '36'

The cyan ANSI color.

White = '37'

The white ANSI color.

classmethod all_colors() → tuple

Return a tuple of all string colors available (used in tests).

class ci_exec.colorize.Styles

A non-exhaustive list of ANSI style formats.

The styles included here are reliable across many terminals, more exotic styles such as ‘Blinking’ are not included as they often are not supported.

Regular = ''

The regular ANSI format.

Bold = '1'

The bold ANSI format.

Dim = '2'

The dim ANSI format.

Underline = '4'

The underline ANSI format.

Inverted = '7'

The inverted ANSI format.

BoldUnderline = '1;4'

Bold and underlined ANSI format.

BoldInverted = '1;7'

Bold and inverted ANSI format.

BoldUnderlineInverted = '1;4;7'

Bold, underlined, and inverted ANSI format.

DimUnderline = '2;4'

Dim and underlined ANSI format.

DimInverted = '2;7'

Dim and inverted ANSI format.

DimUnderlineInverted = '2;4;7'

Dim, underlined, and inverted ANSI format.

classmethod all_styles() → tuple

Return a tuple of all style strings available (used in tests).

ci_exec.colorize.colorize(message, *, color, style = '') → str

Return message colorized with specified style.

Warning

For both the color and style parameters, these are not supposed to have the m after. For example, a color="32m" is invalid, it should just be "32". Similarly, a style="1m" is invalid, it should just be "1".

Parameters

• message (str) – The message to insert an Ansi.Escape sequence with the specified color before, and Ansi.Clear sequence after.

• color (str) – A string describing the ANSI color code to use, e.g., Colors.Red.

• style (str) – The ANSI style to use. Default: Styles.Regular. Note that any number of ANSI style specifiers may be used, but it is assumed that the user has already formed the semicolon delineated list. For multiple ANSI specifiers, see for example Styles.BoldUnderline. Semicolons should be on the interior separating each style.

Returns

The original message with the specified color escape sequence.

Return type

str

ci_exec.colorize.dump_predefined_color_styles()

Dump all predefined Colors in every Styles to the console.

ci_exec.colorize.log_stage(stage, *, fill_char = '=', pad = ' ', l_pad = None, r_pad = None, color = '32', style = '1', width = None, **kwargs)

Print a terminal width block with stage message in the middle.

Similar to the output of tox, a bar of === {stage} === will be printed, adjusted to the width of the terminal. For example:

>>> log_stage("CMake.Configure")
======================== CMake.Configure ========================

By default, this will be printed using ANSI bold green to make it stick out. If the terminal size cannot be obtained, a width of 80 is assumed. Specify width if fixed width is desired.

Note

If the length of the stage parameter is too long (cannot pad with at least one fill_char and the specified padding both sides), the message with any coloring is printed as is. Prefer shorter stage messages when possible.

Parameters

• stage (str) – The description of the build stage to print to the console. This is the only required argument.

• fill_char (str) –

  A length 1 string to use as the fill character. Default: "=".

  Warning

  No checks on the input are performed, but any non-length-1 string will produce unattractive results.

• pad (str) –

  A padding to insert both before and after stage. Default: " ". This value can be any length, but may not be None. If no padding is desired, use the empty string "". Some examples:

  >>> log_stage("CMake.Configure")
  ============================= CMake.Configure ==============================
  >>> log_stage("CMake.Configure", fill_char="_", pad="")
  ______________________________CMake.Configure_______________________________
  

  See also: l_pad and r_pad if asymmetrical patterns are desired.

• l_pad (str or None) – A padding to insert before the stage (on the left). Default: None (implies use value from pad parameter). See examples in r_pad below.

• r_pad (str or None) –

  A padding to insert after the stage (on the right). Default: None (implies use value from pad parameter). Some examples:

  >>> log_stage("CMake.Configure", fill_char="-", l_pad="+ ", r_pad=" +")
  ----------------------------+ CMake.Configure +-----------------------------
  # Without specifying r_pad, pad is used (default: " ")
  >>> log_stage("CMake.Configure", fill_char="-", l_pad="+ ")
  -----------------------------+ CMake.Configure -----------------------------
  

• color (str or None) – The ANSI color code to use with colorize(). If no coloring is desired, call this function with color=None to disable.

• style (str) – The ANSI style specification to use with colorize(). If no coloring is desired, leave this parameter as is and specify color=None.

• width (int) –

  If specified, the terminal size will be ignored and a message formatted to this positive valued parameter will be used instead. If the value is less than the length of the stage message, this parameter is ignored.

  Note

  The specified width here does not necessarily equal the length of the string printed. The ANSI escape sequences added / trailing newline character will make the printed string longer than width, but the perceived width printed to the terminal will be correct.

  That is, if logging to a file, you may also desire to set color=None to remove the ANSI escape sequences / achieve the actual desired width.

• **kwargs – If provided, **kwargs is forwarded to the print(). E.g., to specify file=some_log_file_object or file=sys.stderr rather than printing to sys.stdout.

Tests

Tests for the ci_exec.colorize module.

tests.colorize.test_all_colors()

Validate Colors.all_colors returns all available colors.

tests.colorize.test_all_styles()

Validate Styles.all_styles returns all available styles.

tests.colorize.test_colorize(color, style)

Test colorize() colors as expected for each platform.

tests.colorize.test_dump_predefined_color_styles(capsys)

Validate dump_predefined_color_styles() dumps all.

tests.colorize.test_log_stage(capsys, stage, fill_char_, pad_, l_pad_, r_pad_, color_, style_, width_)

Test log_stage() prints the expected messages.

ci_exec.patch

Various utilities for patching files.

filter_file(path, pattern, repl[, count, …])	Filter the contents of a file.
unified_diff(from_path, to_path[, n, …])	Return the unified_diff between two files.

ci_exec.patch.filter_file(path, pattern, repl, count = 0, flags = 0, backup_extension = '.orig', line_based = False, demand_different = True, encoding = None) → pathlib.Path

Filter the contents of a file.

1. Backup path to {path} + {backup_extension}. Typically, this would mean copying e.g., file.txt to file.txt.orig.

2. Perform filtering using re.sub().

3. If demand_different=True (default), verify that replacements were actually made. If not, fail().

The only required arguments are path, pattern, and repl. If any errors occur, including invalid input, this function will fail().

Parameters

• path (pathlib.Path or str) – The file that needs to be filtered.

• pattern (str) – The pattern to replace. Pass-through parameter to re.sub().

• repl (Callable[[Match], str] or str) – The replacement to be made. Pass-through parameter to re.sub().

• count (int) – The number of replacements to make (default 0 means replace all). Pass-through parameter to re.sub().

• flags (int) – Any flags such as re.IGNORECASE or re.MULTILINE (default 0 means no special flags). Pass-through parameter to re.sub().

• backup_extension (str) – The name to tack onto the back of path to make a backup with. Must be a non-empty string. Default: ".orig".

• line_based (bool) – Whether or not replacements should be made on the entirety of the file, or on a per-line basis. Default: False, do re.sub() on the entire contents. Setting line_based=True can make for simpler or more restrictive regular expressions depending on the replacement needed.

• demand_different (bool) – Whether or not this function should fail() if no changes were actually made. Default: True, fail() if no filtering was performed.

• encoding (str or None) – The encoding to open files with. Default: None implies default. Pass-through parameter to open().

Returns

The path to the backup file that was created with the original contents.

Return type

pathlib.Path

ci_exec.patch.unified_diff(from_path, to_path, n = 3, lineterm = '\n', encoding = None, no_pygments = False) → str

Return the unified_diff between two files.

Any errors, such as not being able to read a file, will fail() the application abruptly.

Parameters

• from_path (pathlib.Path or str) – The file to diff from (the “original” file).

• to_path (pathlib.Path or str) – The file to diff to (the “changed” file).

• n (int) – Number of context lines. Default: 3. Pass-through parameter to difflib.unified_diff().

• lineterm (str) – Default: "\n". Pass-through parameter to difflib.unified_diff().

• encoding (str or None) – The encoding to open files with. Default: None implies default. Pass-through parameter to open().

• no_pygments (bool) –

  Whether or not an attempt to colorize the output using Pygments using the console formatter. If Pygments is not installed, no errors will ensue.

  Default: False, always try and make pretty output. Set to True if you need to enforce that the returned string does not have colors.

Returns

A string ready to be printed to the console.

Return type

str

Tests

Tests for the ci_exec.patch module.

tests.patch.test_filter_file(capsys)

Validate that filter_file() patches / errors as expected.

tests.patch.test_unified_diff(capsys)

Validate that unified_diff() diffs / errors as expected.

ci_exec.provider

Mechanisms to detect a given CI provider.

Provider

Check if code is executing on a continuous integration (CI) service.

ci_exec.provider.provider(func) → staticmethod

Mark a function as a CI provider.

Not intended for use outside of the Provider class.

Parameters

func – The function to decorate.

Returns

A static method that has an attribute register_provider=True.

Return type

staticmethod()

class ci_exec.provider.ProviderMeta

Metaclass for Provider.

This metaclass populates Provider._all_provider_functions by coordinating with the provider() decorator.

Not intended to be used as a metaclass for any other classes.

class ci_exec.provider.Provider

Check if code is executing on a continuous integration (CI) service.

Every now and then it is useful to know

1. If you are running on any CI service, or

2. If you are running on a specific CI service.

The static methods in this class provide a way of checking for pre-defined (by the CI service provider) environment variables:

from ci_exec import Provider, which

def build():
    # ... run cmake etc ...
    ninja = which("ninja")
    if Provider.is_travis():
        # Ninja uses too much memory during link phase.  See:
        # "My build script is killed without any error"
        # https://docs.travis-ci.com/user/common-build-problems/
        ninja("-j", "2", "install")
    else:
        ninja("install")

Available Providers:

is_ci()	Whether or not the code is executing on a CI service.
is_appveyor()	Whether or not the code is executing on AppVeyor.
is_azure_pipelines()	Whether or not the code is executing on Azure Pipelines.
is_circle_ci()	Whether or not the code is executing on CircleCI.
is_jenkins()	Whether or not the code is executing on Jenkins.
is_travis()	Whether or not the code is executing on Travis.

Adding a New Provider:

Pull requests are welcome. Alternatively, simply raise an issue with a link to the provider’s main homepage as well as a link to the documentation certifying the environment variables we can rely on.

1. Add a new is_{new_provider} method to this class, decorated with @provider. Keep these alphabetically sorted (except for is_ci, which should always be first).

2. Document any environment variable(s) involved in a table, including hyperlinks to the provider’s main homepage as well as documentation describing the environment variables in question.

3. Add to the _specific_providers list of environment variables in the tests/provider.py file (near provider_sum()).

4. Add a “pseudo-test” in tests/provider.py in the appropriate location.

_all_provider_functions

Not intended for external usage. The list of all known (implemented) CI provider functions in this class. For example, it will contain Provider.is_appveyor(), …, Provider.is_travis(), etc. This is a class attribute, the Provider class is not intended to be instantiated.

Type

list

static is_ci() → bool

Whether or not the code is executing on a CI service.

Environment variables considered:

Environment Variable	Environment Value
CI	true (case insensitive)
CONTINUOUS_INTEGRATION	true (case insensitive)

If neither of these are true, this function will query every provider directly. For example, it will end up checking if any([Provider.is_appveyor(), ..., Provider.is_travis(), ...]).

static is_appveyor() → bool

Whether or not the code is executing on AppVeyor.

Environment variables considered:

Environment Variable	Environment Value
APPVEYOR	true (case insensitive)

static is_azure_pipelines() → bool

Whether or not the code is executing on Azure Pipelines.

Environment variables considered:

Environment Variable	Environment Value
AZURE_HTTP_USER_AGENT	Existence checked, value ignored.
AGENT_NAME	Existence checked, value ignored.
BUILD_REASON	Existence checked, value ignored.

Note

All three must be set for this function to return True.

static is_circle_ci() → bool

Whether or not the code is executing on CircleCI.

Environment variables considered:

Environment Variable	Environment Value
CIRCLECI	true (case insensitive)

static is_jenkins() → bool

Whether or not the code is executing on Jenkins.

Environment variables considered:

Environment Variable	Environment Value
JENKINS_URL	Existence checked, value ignored.
BUILD_NUMBER	Existence checked, value ignored.

Note

Both must be set for this function to return True.

static is_travis() → bool

Whether or not the code is executing on Travis.

Environment variables considered:

Environment Variable	Environment Value
TRAVIS	true (case insensitive)

Tests

Tests for the ci_exec.provider module.

tests.provider.provider_sum()

Return number of Provider’s that return True.

tests.provider.test_provider_is_ci()

Validate Provider.is_ci() reports as expected.

tests.provider.test_provider_is_appveyor()

Validate Provider.is_appveyor() reports as expected.

tests.provider.test_provider_is_azure_pipelines()

Validate Provider.is_azure_pipelines() reports as expected.

tests.provider.test_provider_is_circle_ci()

Validate Provider.is_circle_ci() reports as expected.

tests.provider.test_provider_is_jenkins()

Validate Provider.is_jenkins() reports as expected.

tests.provider.test_provider_is_travis()

Validate Provider.is_travis() reports as expected.

ci_exec.utils

Assorted utility functions.

This module aims to house any utility functions that may facilitate easier consumption of the ci_exec package.

cd(dest, *[, create])	Context manager / decorator that can be used to change directories.
merge_kwargs(defaults, kwargs)	Merge defaults into kwargs and return kwargs.
set_env(**kwargs)	Context manager / decorator that can be used to set environment variables.
unset_env(*args)	Context manager / decorator that can be used to unset environment variables.

class ci_exec.utils.cd(dest, *, create = False)

Context manager / decorator that can be used to change directories.

This context manager will change directories to dest, and after its scope expires (outside of the with statement, or after the decorated function) it will change directories back to the original current working directory.

As a context manager:

from ci_exec import cd, which

if __name__ == "__main__":
    # Get the build tools setup.
    cmake = which("cmake")
    ninja = which("ninja")

    # Suppose current directory here is "/source"
    with cd("build", create=True):
        # Current directory is now "/source/build"
        cmake("..", "-G", "Ninja", "-DCMAKE_BUILD_TYPE=Release")
        ninja()

    # Any code out-dented (not under the `with`): current directory is "/source"

As a decorator:

from ci_exec import cd, which

@cd("build", create=True)
def build():
    # Inside the function: current directory is "/source/build"
    cmake = which("cmake")
    ninja = which("ninja")
    cmake("..", "-G", "Ninja", "-DCMAKE_BUILD_TYPE=Release")
    ninja()

if __name__ == "__main__":
    # Suppose current directory here is "/source"
    build()  # Function executes in "/source/build"
    # After the function current directory is "/source"

Parameters

• dest (pathlib.Path or str) – The destination to change directories to.

• create (bool) – Whether or not the dest is allowed to be created. Default: False, the dest must exist already (will fail() if it does not). If True, mkdir_p() will be called with dest.

ci_exec.utils.merge_kwargs(defaults, kwargs)

Merge defaults into kwargs and return kwargs.

Intended usage is for setting defaults to **kwargs when the caller did not provide a given argument, but making sure not to overwrite the caller’s explicit argument when specified.

For example:

>>> merge_kwargs({"a": 1, "b": 2}, {})
{'a': 1, 'b': 2}
>>> merge_kwargs({"a": 1, "b": 2}, {"a": 3})
{'a': 3, 'b': 2}

Entries in the defaults parameter only get included of not present in the kwargs argument. This is to facilitate something like this:

from ci_exec import merge_kwargs

# The function we want to customize the defaults for.
def func(alpha=1, beta=2):
    return alpha + beta

# Example: default to alpha=2, leave beta alone.
def custom(**kwargs):
    return func(**merge_kwargs({"alpha": 2}, kwargs))

# custom()                == 4
# custom(alpha=0)         == 2
# custom(beta=0)          == 2
# custom(alpha=0, beta=0) == 0

Parameters

• defaults (dict) – The dictionary of defaults to add to kwargs if not present.

• kwargs (dict) – The dictionary to merge defaults into.

Returns

The kwargs dictionary, possibly with values from defaults injected.

Return type

dict

class ci_exec.utils.set_env(**kwargs)

Context manager / decorator that can be used to set environment variables.

Usage example:

from ci_exec import set_env

@set_env(CC="clang", CXX="clang++")
def build_clang():
    # CC="clang" and CXX="clang++" inside function.

# ... or ...

with set_env(CC="clang", CXX="clang++"):
    # CC="clang" and CXX="clang++" in `with` block

Prior environment variable state will be recorded and later restored when a decorated function / with block’s scope ends.

1. An environment variable was already set. Its value is saved before overwriting, and then later restored:

   # Example: CC=gcc was already set.
   with set_env(CC="clang"):
       # Inside block: CC="clang"
   # Out-dented: CC=gcc again.
   

2. An environment variable was not already set. Its value is unset again:

   # Example: CC was _not_ set in environment.
   with set_env(CC="clang"):
       # Inside block: CC="clang"
   # Out-dented: CC _not_ set in environment.
   

Note

See note in unset_env for more information on removing environment variables.

Parameters

**kwargs – Keyword argument parameter pack. Keys are the environment variable to set, and values are the desired value of the environment variable. All keys and all values must be strings.

Raises

ValueError – If no arguments are provided (len(kwargs) == 0), or if any keys / values are not a str.

class ci_exec.utils.unset_env(*args)

Context manager / decorator that can be used to unset environment variables.

Usage example:

from ci_exec import unset_env

@unset_env("CC", "CXX")
def build():
    # Neither CC nor CXX are set in the environment during this function call.

# ... or ...

with unset_env("CC", "CXX"):
    # Neither CC nor CXX are set in the environment inside this block.

Prior environment variable state will be recorded and later restored when a decorated function / with block’s scope ends. So if an environment variable was already set, its value is saved before deletion, and then later restored:

# Example: CC=gcc was already set.
with unset_env("CC"):
    # Inside block: CC not set in environment.
# Out-dented: CC=gcc again.

Note

Removing the environment variable is done via del os.environ[env_var]. This may or may not affect child processes in the manner you expect, depending on whether your platform supports os.unsetenv(). See the end of the description of os.environ for more information.

Parameters

*args – Argument parameter pack. Each argument is an environment variable to unset. Each argument must be a string. If a specified argument is not currently set in the environment, it will effectively be skipped.

Raises

ValueError – If no arguments are provided (len(args) == 0), or if any arguments are not a str.

Tests

Tests for the ci_exec.utils module.

tests.utils.test_cd(capsys)

Validate cd behaves as expected.

tests.utils.test_merge_kwargs()

Validate merge_kwargs() merges as expected.

tests.utils.test_set_env()

Validate set_env sets environment variables.

tests.utils.test_unset_env()

Validate unset_env unsets environment variables.

Demos

More demos, particularly related to building C++, will be added when possible.

custom_log_stage

Simple demo for how to modify the ci_exec defaults to suit user preferences. Go to demo ↱

custom_log_stage

Simple demo for how to modify the ci_exec defaults to suit user preferences.

By default log_stage() will log in bold green, using "=" as a separator. This makes stages stick out / easy to spot during CI builds, but users may not prefer this style. Instead of manually calling with explicit arguments each time:

from ci_exec import Colors, Styles, log_stage
# ... some time later ...
log_stage("CMake.Configure", color=Colors.Cyan, style=Styles.Regular, fill_char="-")

it is preferable to just create your own wrapper function. If you want a "-" fill character in regular cyan each time, why force yourself to type that out every time? It is much cleaner to just define your own wrapper with your preferred defaults. The most simple wrapper you can create:

import ci_exec
from ci_exec import Colors, Styles

def log_stage(stage: str):
    ci_exec.log_stage(stage, color=Colors.Cyan, style=Styles.Regular, fill_char="-")

Chances are, this will satisfy 95% of use-cases. The code in this file demonstrates how you can enable keyword arguments on your wrapper function so that if you have a scenario where you want to override your custom log_stage function defaults for just one or two use cases you can.

You could of course just set ci_exec.log_stage.__kwdefaults__, but changing this can lead to some surprising behavior if you don’t know what you are doing. Additionally, other readers will have a harder time following what your build script is doing.

demos.custom_log_stage.log_stage(stage, **kwargs)

Sample wrapper #1: provide custom behavior of log_stage.

demos.custom_log_stage.log_sub_stage(sub_stage, **kwargs)

Sample wrapper #2: enable sub-stages to be printed (e.g., for big scripts).

demos.custom_log_stage.bold_green(msg)

Sample wrapper #3: return msg in bold green text.

demos.custom_log_stage.do_work(n, width = 80)

Ignore this function, pretend this is the real work you need to do.

demos.custom_log_stage.main()

Execute all build stages and log progress.

Demos Program Execution

Wrapper module for executing each respective location from the repository root.

Each individual demo is executable on its own. However, users may also run a demo from repository root doing python demos/ <demo_name>.

By default the programs are not run in “animated” mode. The --animated flag is what is used to record the videos hosted on each individual demo page, which utilizes clear, PAUSE and a delay in calls to type_message(). See mock_shell() for more information.

clear()	Clear the console screen.
pause([amount])	Pause by amount using time.sleep().
type_message(message, *, delay)	Flush message to sys.stdout, sleep by delay after character.
mock_shell(program, *, cwd, delay, animated)	Run a “shell” program from the specified working directory.
run_demo(program, cwd, animated)	Run the specified demo program.

demos.__main__.CI_EXEC_DEMOS_COVERAGE = False

Whether or not this is a coverage run of the demo files.

Warning

This should not be set unless invoking from tox. See notes in [testenv:docs] section of tox.ini at repository root.

demos.__main__.windows_cmd_builtin(builtin)

Return a function that runs the specified builtin.

Note

There is a reason this is not in the main library. To deal with shell builtins requires a significant amount of extra work for little to no benefit. The demos just need "cls" to clear and "type" to cat.

The return is a function that can support:

*args

Any command line arguments to provide to the builtin.

**kwargs

Any keyword arguments to provide to subprocess.run(). This function will add check=True and shell=True unless these keys are already explicitly specified.

Parameters

builtin (str) – Any of the CMD builtins, such as "type" or "cls". No checking is performed!

demos.__main__.clear()

Clear the console screen. Uses cls on Windows, and clear otherwise.

demos.__main__.pause(amount = 3.0)

Pause by amount using time.sleep().

This function exists so that it can be used for PAUSE statements in mock_shell().

Parameters

amount (float) – The amount of time to time.sleep() for. Default: 3.0 seconds.

demos.__main__.type_message(message, *, delay)

Flush message to sys.stdout, sleep by delay after character.

Parameters

• message (str) – What to type. A trailing newline "\n" will be written at the end.

• delay (float) – The positive amount to time.sleep() after each character in message is written. Suggested value for simulating typing to the console: 0.05. Use 0.0 to avoid delays.

demos.__main__.mock_shell(program, *, cwd, delay, animated)

Run a “shell” program from the specified working directory.

• Lines starting with # are “comment” lines, they will be printed to the screen using type_message().

• Lines starting with "$ " are a command to execute.

  • Commands executed will be printed to the screen using type_message().

There is also a notion of “animated” mode. When animated=True, the following special behavior is enabled:

• $ clear: calls clear(). In non-animated mode, clear is skipped.

• PAUSE or PAUSE=X.Y: call pause(), X.Y should be parseable as a float e.g., PAUSE=7.0 would pause for 7 seconds. In non-animated mode this is skipped.

• Calls to type_message() will have a delay=0.05. In non-animated mode the delay=0.0.

These scripts are not intended to be robust. Features are implemented as needed… this is not intended to be a real shell programming language! See demos/__main__.py for example program’s that can execute.

Parameters

• program (str) – The “shell” program to execute.

• cwd (str) – The directory to execute program from.

• delay (float) – Pass-through parameter for type_message().

• animated (bool) – Whether or not this is an “animated” shell, meaning commands such as clear or PAUSE should be executed.

demos.__main__.run_demo(program, cwd, animated)

Run the specified demo program.

When animated=True, clear() the screen and sleep for 2 seconds to allow recording to begin. The delay parameter passed-through to type_message() will be set to 0.05. In non-animated mode the screen will not be cleared, and the delay will be 0.0.

Parameters

• program (str) – The “shell” program to execute with mock_shell().

• cwd (str) – Pass-through parameter to mock_shell().

• animated (bool) – Pass-through parameter to mock_shell().

demos.__main__.main()

Create the argument parser and run the specified demo.

Important Usage Notes

Namespace Pollution

The ci_exec package namespace is polluted intentionally. Always import from the polluted namespace, never import from the module originally defining something:

from ci_exec import which       # Yes
from ci_exec.core import which  # NO!

In practice it shouldn’t matter that much, but

1. Any functions moved to different modules will not be considered ABI breaking changes. So if ci_exec.core.which moved to a different ci_exec.other_module, this would be done in a patch release (ci_exec uses semantic versioning).

2. Anything that is not exposed at the top-level is more than likely something that should not be used.

Beware of Commands Expecting Input

The Executable.__call__() method internally invokes subprocess.run(), which does some fancy manipulations of stdout, stderr, and stdin. In particular, the subprocess by default will allow communication on stdin for commands that are expecting input. In the context of CI, this is problematic, and users need to be aware that the subprocess will indeed pause and solicit user input. Consider a scenario where say filter_file() was used to make some significant changes on the CI provider. If you ran:

from ci_exec import which

git = which("git")
git("diff")

The build may fail from a timeout rather than just displaying the changes, because for a large enough change to report git will want to use your $PAGER to let you scroll through it. Most commands that allow input also provide a command-line flag to disable this behavior and in this scenario, it would be:

git("--no-pager", "diff")

In situations where you know that input is required, subprocess.run() enables this via the input parameter. Consider the following toy script that is expecting two pieces of user input at different stages:

name = input("What is your name? ")
print("Hi {name}, nice to meet you.".format(name=name))

age = input("How old are you? ")
print("Wow!  {age} is such a great age :)".format(age=age))

Then you could call it using ci_exec like this:

import sys
from ci_exec import Executable

if __name__ == "__main__":
    python = Executable(sys.executable)
    python("./multi_input.py", input=b"Bilbo\n111")

where

1. The input parameter needs a bytes object (the b prefix in b"...").

2. You use \n to send a newline. From the docs: For stdin, line ending characters '\n' in the input will be converted to the default line separator os.linesep. So "Bilbo\n" will end up being the answer to the first name = input(...), and "111" will be sent to the age = input(...).

Build Logs out of Order (Azure Pipelines)

If the results of call logging appear out of order, then your CI provider is affected by this. This is a known problem on Azure Pipelines, it seems unlikely to affect many other providers due to the nature of how the Azure Pipelines build logs are served. Example “script”:

cmake("..")
ninja()

Produces:

-- The C compiler identification is GNU 8.3.1
-- The CXX compiler identification is GNU 8.3.1
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
...
[12/74] Building C object ...joystick.c.o
$ /usr/bin/cmake ..
$ /usr/bin/ninja

The log is out of order, all cmake / ninja output appeared before the call logging ($ /usr/bin/cmake .. and $ /usr/bin/ninja). There are two possible solutions:

1. Invoke your build script using python -u: python -u ./.ci/build.py

2. Set the environment variable PYTHONUNBUFFERED=true.