loadconfig¶

Inline config¶

Lets start with the most simple and practical example:

>>> from loadconfig import Config
>>> Config('greeter: Hi there')
{greeter: Hi there}

>>> Config('{greeter: Hi there}')
{greeter: Hi there}

As we can see, loadconfig uses yaml strings as its way to input data and represent its state.

Our config object is in itself a yaml flavored OrderedDict:

>>> c = Config('''
...         greeter:
...           message: Hi
...           group:
...             - Jill
...             - Ted
...             - Nancy
...     ''')
>>> c
{greeter: {message: Hi, group: [Jill, Ted, Nancy]}}

Notice our greeter was defined with a message, and a group of people to greet, in that order. We see that both, the message and the group are kept in exactly that order in the config representation. It might sound the most natural thing in the world, but remember that ‘normal’ dictionaries do not keep key order. There are multiple practical examples where key order is crucial, and in fact loadconfig itself needs it for processing its clg special keyword.

Lets now see how our config prints:

>>> print(c)
greeter:
  message: Hi
  group:
    - Jill
    - Ted
    - Nancy

Not bad. The parsed yaml string and later rendered output generated exactly the same input. Let’s try now feeding back that pretty Config representation

>>> Config('{greeter: {message: Hi, group: [Jill, Ted, Nancy]}}')
{greeter: {message: Hi, group: [Jill, Ted, Nancy]}}

Not bad at all! Yaml allowed us to skip all those quotes in literal strings, making the code much more cleaner. Just for a second lets consider how we would write a similar expression in python:

>>> c = {'greeter':
...         {'message': 'Hi',
...          'group':   ['Jill', 'Ted', 'Nancy']}}

In more complex cases and especially when dealing with the shell, quotes are a real source of very subtle bugs. So we are gaining in readability and correctness.

Idempotence¶

To summarize, let’s highlight another desirable property of Config:

>>> c = Config('greeter: {message: Hi, group: [Jill, Ted, Nancy]}')
>>> c
{greeter: {message: Hi, group: [Jill, Ted, Nancy]}}
>>> c == Config(c)
True

In other words our config representation is idempotent. Very useful for having a common unique representation of data regardless of what was done to make it.

Access interface¶

Now, let’s check our config access interface:

>>> c['greeter']
{message: Hi, group: [Jill, Ted, Nancy]}

>>> c['greeter']['group']
['Jill', 'Ted', 'Nancy']

>>> c.greeter.group
['Jill', 'Ted', 'Nancy']

Right there, we avoided two pairs of quotes and two pairs of square brakets! We can even intermix the dictionary and the attribute access:

>>> c['greeter'].group
['Jill', 'Ted', 'Nancy']

Overriding keys¶

Overriding keys is a fundamental feature of loadconfig that gives the ability to quickly adapt configuration content to the program needs.

>>> c.update('place: Yosemite')
>>> c
{greeter: {message: Hi, group: [Jill, Ted, Nancy]}, place: Yosemite}

>>> c.greeter.group.append('Steve')
>>> c
{greeter: {message: Hi, group: [Jill, Ted, Nancy, Steve]}, place: Yosemite}

As with regular code, the latest key defined wins:

>>> conf = '''\
...     greeter: {message: Hi, group: [Jill, Ted, Nancy]}
...     greeter: {message: Hi, group: [Jill, Ted, Nancy, Steve]}'''
>>> Config(conf)
{greeter: {message: Hi, group: [Jill, Ted, Nancy, Steve]}}

But what about the DRY (Don’t Repeat Yourself) principle? And what about descriptive programming? Lets explore some of the best features like include and CLG (Command Line Generator) in the Intermediate tutorial

Expansion¶

Yaml config sources are meant to reduce redundancy whenever possible:

>>> from loadconfig import Config
>>> conf = '''\
...     name: &dancer
...       - Zeela
...       - Kim
...     team:
...       *dancer
...     '''
>>> Config(conf)
{name: [Zeela, Kim], team: [Zeela, Kim]}

To make the syntax more DRY and intuitive, loadconfig introduces an alternative form of expansion:

>>> conf = '''\
...     name: [Zeela, Kim]
...     team: $name
...     choreography: $team
...     '''
>>> Config(conf)
{name: [Zeela, Kim], team: [Zeela, Kim], choreography: [Zeela, Kim]}

loadconfig yaml goodies¶

>>> birds = '''\
...     hummingbird:
...       colors:
...         - iris
...         - teal
...         - coral
...     '''
>>> with open('birds.yml', 'w') as fh:
...     _ = fh.write(birds)
>>> conf = '!include birds.yml'
>>> Config(conf)
{hummingbird: {colors: [iris, teal, coral]}}

>>> conf = 'colors: !include birds.yml:hummingbird:colors'
>>> Config(conf)
{colors: [iris, teal, coral]}

>>> conf = '''\
...     _: !include birds.yml:&
...     colors: !expand hummingbird:colors
...     '''
>>> Config(conf)
{colors: [iris, teal, coral]}

>>> from os import environ
>>> environ['CITY'] = 'San Francisco'
>>> c = Config('!env city')
>>> c.city
'San Francisco'

>>> with open('libpath.cfg', 'w') as fh:
...     _ = fh.write('/usr/local/lib')
>>> Config('libpath: !read libpath.cfg')
{libpath: /usr/local/lib}

Introducing -E and -C cli switches¶

As with the inline config and include, we have the -E switch for extra yaml config and -C for yam config files. Let’s looks again at our beautiful hummingbird:

>>> birds = '''\
...     hummingbird:
...       colors:
...         - iris
...         - teal
...         - coral
...     '''
>>> extra_arg = '-E="{}"'.format(birds)
>>> Config(args=[extra_arg])
{hummingbird: {colors: [iris, teal, coral]}}

Similarly we can introduce the same data through a configuration file. In this case, we will reuse our birds.yml file with simply:

>>> Config(args=['-C="{}"'.format('birds.yml')])
{hummingbird: {colors: [iris, teal, coral]}}

These operations are in themselves pretty useful. They are even more revealing when considering them in the shell context. loadconfig is at its core a python library, so the issue is how do we bridge these two worlds. Shell environment variables, and some little magic from our loadconfig script would help. Let’s reintroduce loadconfig script call here:

$ BIRDS=$(cat << 'EOF'
>   hummingbird:
>     - iris
>     - teal
>     - coral
> EOF
> )
$ loadconfig -E="$BIRDS"
export HUMMINGBIRD="iris teal coral"

If our bird decided to take a nap in a file, it would be:

$ echo "$BIRDS" > birds.yml
$ loadconfig -C="birds.yml"
export HUMMINGBIRD="iris teal coral"

At this point, we can use both switches. loadconfig accepts them in sequence, updating and overriding older data with new values from the sequence:

$ BIRDS2="hummingbird: [ruby, myrtle]"
$ BIRDS3="swallow: [cyan, yellow]"
$ loadconfig -E="$BIRDS2" -C="birds.yml" -E="$BIRDS3"
export HUMMINGBIRD="iris teal coral"
export SWALLOW="cyan yellow"

$ loadconfig -E="$BIRDS3" -C="birds.yml" -E="$BIRDS2"
export SWALLOW="cyan yellow"
export HUMMINGBIRD="ruby myrtle"

CLI interface¶

One key feature of loadconfig is its CLG integration. CLG is a wonderful yaml based command line generator that wraps the standard argparse module. Loadconfig uses a special clg keyword to unleash its power.

$ CONF=$(cat << 'EOF'
>   clg:
>     description: Build a full system
>     args:
>       host:
>         help: Host to build
> EOF
> )
$ loadconfig -E="$CONF" --help
usage: loadconfig [-h] host

Build a full system

positional arguments:
  host        Host to build

optional arguments:
  -h, --help  show this help message and exit

$ loadconfig -E="$CONF" antares
export HOST="antares"

>>> from loadconfig import Config
>>> conf = '''
...     clg:
...         description: Build a full system
...         args:
...             host:
...                 help: Host to build
...     '''
>>> try:
...     c = Config(conf, args=['sysbuild', '--help'])
... except SystemExit as e:
...     pass
>>> print(e.code)
usage: sysbuild [-h] host

Build a full system

positional arguments:
  host        Host to build

optional arguments:
  -h, --help  show this help message and exit

>>> Config(conf, args=['', 'antares'])
{prog: '', host: antares}

$ CONF=$(cat << 'EOF'
>     clg:
>         prog: $prog
>         description: $prog $version is a documentation server.
>         epilog: |
>             Build sphinx docs, launch a browser for easy reading,
>             detect and render doc changes with inotify.
>         options:
>             version:
>                 short: v
>                 action: version
>                 version: $prog $version
>             debug:
>                 short: d
>                 action: store_true
>                 default: __SUPPRESS__
>                 help: show docker call
>         args:
>             sphinx_dir:
>                 nargs: '?'
>                 default: /data/rst
>                 help: |
>                     directory holding sphinx conf.py and doc sources
>                     (default: %(default)s)
> EOF
> )

$ loadconfig -E="$CONF" --help
usage: $prog [-h] [-v] [-d] [sphinx_dir]

$prog $version is a documentation server.

positional arguments:
  sphinx_dir     directory holding sphinx conf.py and doc sources
                 (default: /data/rst)

optional arguments:
  -h, --help     show this help message and exit
  -v, --version  show program's version number and exit
  -d, --debug    show docker call

Build sphinx docs, launch a browser for easy reading,
detect and render doc changes with inotify.

$ loadconfig -E="$CONF"
export SPHINX_DIR="/data/rst"

$ loadconfig -E="$CONF" -v
$prog $version

$ loadconfig -E="$CONF" -E="prog: sphinx, version: 0.1" -v
sphinx 0.1

$ loadconfig -E="$CONF" -d /data/salt/prog/loadconfig/docs
export SPHINX_DIR="/data/salt/prog/loadconfig/docs"
export DEBUG="True"

Multiple commands and execution¶

Once we start adding more capabilities to our program, we might find having multiple commands with their own arguments and options does make the interface cleaner. To keep it simple, let’s look at a small program that can install, run and uninstall itself:

$ ls -l netinstall.py
-rwxr-xr-x 1 admin admin 745 Jul  2 21:00 netapplet.py

$ cat netinstall.py

#!/usr/bin/env python

from loadconfig import Config
import sys

conf = """\
    clg:
        subparsers:
            run:
                help: 'run as:  $prog run'
            install:
                help: 'run as:  $prog install | sudo bash'
            uninstall:
                help: 'run as:  $prog uninstall | sudo bash'
    """

def install(c):
    print('cp {} /usr/bin'.format(c.prog))

def uninstall(c):
    print('rm -f /usr/bin/{}'.format(c.prog))

def run(c):
    print('Running {}'.format(c.prog))

def main(args):
    c = Config(conf, args=args)
    c.run()

if __name__ == '__main__':
    main(sys.argv)

A couple of runs will show:

# Our program is not installed in the system /usr/bin directory
netapplet.py
bash: netapplet.py: command not found


# Running netapplet.py from the current directory.
# ( ./ needs to be prepended to the command as . is not usually on $PATH )
$ ./netapplet.py
usage: netapplet.py [-h] {run,install,uninstall} ...


# Retrieving help
$ ./netapplet.py --help
usage: netapplet.py [-h] {run,install,uninstall} ...

positional arguments:
  {run,install,uninstall}
    run                 run as:  ./netapplet.py run
    install             run as:  ./netapplet.py install | sudo bash
    uninstall           run as:  ./netapplet.py uninstall | sudo bash

optional arguments:
  -h, --help            show this help message and exit


# Looking the output of install
./netapplet.py install
cp ./netapplet.py /usr/bin


# Sounds good. Lets install it according to the help.
$ ./netapplet.py install | sudo bash


# Checking our program is working on the system. Yeey!
$ netapplet.py run
Running /usr/bin/netapplet.py


# Time to uninstall it
$ ./netapplet.py uninstall | sudo bash

# And... it's gone
$ netapplet.py run
bash: /usr/bin/netapplet.py: No such file or directory

The program is clean, self-installable and self-documented.

The first line is a typical unix shebang to look for the system or virtualenv python shell. (Note: $VIRTUAL_ENV could had been used to optionally autoinstall within the virtualenv). It follows a few import lines and the conf global variable. The install, uninstall and run functions are simple print statements that will be leveraged by a sudo shell in our program, though these functions can be as complex as wanted, being part of other modules, etc. They receive the configuration as the first argument. The main function is called if our program is executed directly. This is a good programming and testing practice. Having main as a function with parameters allows to test it with handcrafted arguments. Lets now focus on the conf and the main functions.

conf is a clg key with subparsers. This is an argparse concept which basically means a subcommand. In our case we have three of them with their documentation. If you are wondering why the quotes in the help of the subparsers keys, this is to escape the usual yaml meaning of the colon (:) char on these help strings. As we are using a clg key, loadconfig assigns args[0] to the $prog attribute, decoupling the program name from sys.argv[0]. Next, it loads the configuration in the c variable, and finally c.run() executes the function invoked by the cli arguments. c.run() only makes sense if the config holds a clg key with subparsers.

This simple program succinctly highlights very common needs in a program lifecycle (configuration, interface, documentation, deployment) and it can easily be used as a base for more complex ones.

loadconfig as a generic program wrapper¶

In modern unix systems, the sheer amount of options in most of the cli tools is plain staggering. Often we have use cases that matter to us, but in order to use them effectively we find ourselves writing a wrapper around them. The real goal is to create a new ‘interface’ with sensible defaults. Now, instead of thinking in solving our original problem, we are thinking how to solve implementation details of an imperative language.(do we need to escape that space?, enclose that string in double quotes?)

Regardless of the language, our tool have a set of common interfaces: argument processing, documentation, configuration files, and variables. loadconfig is meant to unify the management of these interfaces with simple, descriptive yaml strings.

#!/usr/bin/env python
'''usage: loadconfig [-h] [-v] [-C CONF] [-E STR] [args [args ...]]

loadconfig 0.2.6 generates envvars from multiple sources.

positional arguments:
  args                  arguments for configuration

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -C CONF, --conf CONF  Configuration file in yaml format to load
  -E STR, --str STR     yaml config string "key: value, .."

Make a list of envvars from config file, yaml strings and cli args.
Keywords:
    check_config: python code for config key validation.
    clg: Specify command line interpretation.
As convention, keys are lowercase with underscore as space.
Full documentation:
    web:  https://loadconfig.readthedocs.org
    pdf:  https://readthedocs.org/projects/loadconfig/downloads
'''

from loadconfig import Config, __version__
import sys

conf = """\
    clg:
        description: $prog $version generates envvars from multiple sources.
        epilog: |
            Make a list of envvars from config file, yaml strings and cli args.
            Keywords:
                check_config: python code for config key validation.
                clg: Specify command line interpretation.
            As convention, keys are lowercase with underscore as space.
            Full documentation:
                web:  https://loadconfig.readthedocs.org
                pdf:  https://readthedocs.org/projects/loadconfig/downloads
        options:
            version:
                short: v
                action: version
                version: $prog $version
            conf:
                short: C
                default: __SUPPRESS__
                help: Configuration file in yaml format to load
            str:
                short: E
                default: __SUPPRESS__
                help: 'yaml config string "key: value, .."'
        args:
            args:
                nargs: '*'
                default: __SUPPRESS__
                help: arguments for configuration"""


def main(args):
    c = Config(conf, args, version=__version__)
    print(c.export())

if __name__ == '__main__':
    main(sys.argv)

Following loadconfig’s philosophy, its script implementation is in itself straightforward. All imperative programming aspects are kept to minimum. As we can see, all keywords and concepts of our conf python variable were already introduced in CLI interface. When using a clg key, loadconfig defines the $prog attribute using args[0]. This allows to decouple the program name from sys.argv[0]. Sometimes, it is nice to get the program version from an external source (eg: another module) and feed it into Config. The convenient Config version parameter is used in this case. At this point, we are familiar with the Config class. c.export is just a Config method that iterates over all keywords defined, making them uppercase, replacing space by underline and prepending the word export. Want to take a guess? We will see shortly why. Finally, all the actual commands are enclosed in the main function as good organizational practice and as it allows for easy testing.

Ok Daniel, all of this looks fine. Did I miss something?

This seamlessly simple script hides really well its true expressiveness power when combined with some shell scripting. Here we go.

Sphinx renderer¶

Lets assume we have an application that renders sphinx html documentation, detects changes in the documentation sources in real time and controls a browser. There is a wonderful project called docker that encapsulates incredibly well all the application pieces (libraries, fonts, programs) in a single unit called image and offers a neat cli to interact with the operating system. Now, there are lots of possibilities to precisely control how docker will communicate with the filesystem, with the video and audio subsystems, with the network ... Wait! we just want to run our application, remember? Sure! Docker makes the task trivially simple in just one line... one looong line:

docker run -d -u admin -v /data/rst:/data/sphinx -e DISPLAY=:0.0 \
-v /tmp/.X11-unix:/tmp/.X11-unix reg.csl/sphinx

The point is that although it is an incredible simple interface to interact with the operating system, typing those ‘lines’ are not exactly fun. loadconfig allow us to take back the command line interface, defining the defaults we want in configuration files or within a wrapper and to leave the command line for the variable arguments we care the most. In this case, most of the docker run command is setup. The only ‘interesting’ variable part, is the path of our sphinx source documents, in this case /data/rst.

#!/bin/bash

CONF=$(cat << 'EOF'
    version: 0.1
    desktop_args: -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix
    docker_args: -d -u admin -v $(realpath $sphinx_dir):/data/sphinx \
        $DESKTOP_ARGS reg.csl/sphinx
    clg:
        prog: $prog
        description: $prog $version is a documentation server.
        epilog: |
            Build sphinx docs, launch a browser for easy reading,
            detect and render doc changes with inotify.
        options:
            version:
                short: v
                action: version
                version: $prog $version
            debug:
                short: d
                action: store_true
                default: __SUPPRESS__
                help: show docker call
        args:
            sphinx_dir:
                nargs: '?'
                default: /data/rst
                help: |
                    directory holding sphinx conf.py and doc sources
                    (default: %(default)s)
    check_config: |
        import os, sys
        if not os.path.isfile('$sphinx_dir/conf.py'):
            sys.exit('Error: $sphinx_dir/conf.py not found.')
EOF
)

set -e
ENV=$(loadconfig -E="prog: $(basename $0)" -E="$CONF" "$@")
eval "$ENV"

[ $DEBUG ] && echo "docker run $DOCKER_ARGS"

cid=$(docker run $DOCKER_ARGS)
docker wait $cid
docker rm $cid >/dev/null

As this is a full application, there are plenty of details to see. Still, with a simple glance, we can see 2 distinctive sections. A config section with just one shell variable, CONF, and an executable section. Most of the ‘code’ happens on the CONF variable. The executable section is driven by loadconfig script, docker and the shell interpreter.

loadconfig CLG features¶

One key feature of loadconfig is its CLG integration. We saw in CLI interface some of its features. Here, we mention extra features that loadconfig adds to CLG

# Using the conf variable as in the original example
$ netapplet.py run
Running /usr/bin/netapplet.py

$ ./netapplet.py
usage: netapplet.py [-h] {run,install,uninstall} ...

conf = """\
    clg:
        default_cmd: run
        subparsers:
            run:
                help: 'run as:  $prog run'

$ ./netapplet.py
Running ./netapplet.py