Constructing runners¶

Mush works by assembling a number of callables into a Runner:

from mush import Runner

def func1():
    print('func1')

def func2():
    print('func2')

runner = Runner(func1, func2)

Once assembled, a runner can be called any number of times. Each time it is called, it will call each of its callables in turn:

>>> runner()
func1
func2

More callables can be added to a runner:

def func3():
    print('func3')

runner.add(func3)

If you want to add several callables in one go, you can use the runner’s extend() method:

def func4():
    print('func4')

def func5():
    print('func5')

runner.extend(func4, func5)

Now, when called, the runner will call all five functions:

>>> runner()
func1
func2
func3
func4
func5

Runners can also be added together to create a new runner:

runner1 = Runner(func1)
runner2 = Runner(func2)
runner3 = runner1 + runner2

This addition does not modify the existing runners, but does give the result you’d expect:

>>> runner1()
func1
>>> runner2()
func2
>>> runner3()
func1
func2

This can also be done by passing runners in when creating a new runner or calling the extend method on a runner, for example:

runner1 = Runner(func1)
runner2 = Runner(func2)
runner4_1 = Runner(runner1, runner2)
runner4_2 = Runner()
runner4_2.extend(runner1, runner2)

In both cases, the results are as you would expect:

>>> runner4_1()
func1
func2
>>> runner4_2()
func1
func2

Finally, runners can be cloned, providing a way to encapsulate commonly used base runners that can then be extended for each specific use case:

runner5 = runner3.clone()
runner5.add(func4)

The existing runner is not modified, while the new runner behaves as expected:

>>> runner3()
func1
func2
>>> runner5()
func1
func2
func4

Configuring Resources¶

Where Mush becomes useful is when the callables in a runner either produce or require objects of a certain type. Given the right configuration, Mush will wire these together enabling you to write easily testable and reusable callables that encapsulate specific pieces of functionality. This configuration is done either imperatively, declaratively or using a combination of the two styles as described in the sections below.

For the examples, we’ll assume we have three types of resources:

class Apple:
    def __str__(self):
        return 'an apple'
    __repr__ = __str__

class Orange:
    def __str__(self):
        return 'an orange'
    __repr__ = __str__

class Juice:
    def __str__(self):
        return 'a refreshing fruit beverage'
    __repr__ = __str__

def apple_tree():
     print('I made an apple')
     return Apple()

def magician(fruit):
     print('I turned {0} into an orange'.format(fruit))
     return Orange()

def juicer(fruit1, fruit2):
     print('I made juice out of {0} and {1}'.format(fruit1, fruit2))

runner = Runner()
runner.add(apple_tree)
runner.add(magician, Apple)
runner.add(juicer, fruit1=Apple, fruit2=Orange)

>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange

from mush import requires

def apple_tree():
    print('I made an apple')
    return Apple()

@requires(Apple)
def magician(fruit):
    print('I turned {0} into an orange'.format(fruit))
    return Orange()

@requires(fruit1=Apple, fruit2=Orange)
def juicer(fruit1, fruit2):
    print('I made juice out of {0} and {1}'.format(fruit1, fruit2))
    return Juice()

>>> runner = Runner(apple_tree, magician, juicer)
>>> runner()
I made an apple
I turned an apple into an orange
I made juice out of an apple and an orange

@requires(Juice)
def packager(juice):
    print('I put {0} in a bottle'.format(juice))

def orange_tree():
    print('I made an orange')
    return Orange()

trees = Runner(apple_tree, orange_tree)
runner = trees.clone()
runner.extend(juicer, packager)

>>> runner()
I made an apple
I made an orange
I made juice out of an apple and an orange
I put a refreshing fruit beverage in a bottle

runner = trees.clone()
runner.add(juicer, Orange, Apple)

>>> runner()
I made an apple
I made an orange
I made juice out of an orange and an apple

Resource usage periods¶

It can be important for a callable to have either first access or last access to a particular resource. For this reason, configuration can specify one of three periods during which a callable needs to be called with a particular resource; the default is normal and a decorator can be used to indicate either first or last. Within these periods, callables are called in the order they are added to the runner.

As an example, consider a ring and some things that can be done to it:

class Ring:
    def __str__(self):
        return 'a ring'

def forge():
    return Ring()

def polish(ring):
    print('polishing {0}'.format(ring))

def more_polish(ring):
    print('polishing {0} again'.format(ring))

def engrave(ring):
    print('engraving {0}'.format(ring))

def package(ring):
    print('packaging {0}'.format(ring))

These can now be added to a runner with configuration expressing the correct periods:

from mush import Runner, first, last

runner = Runner(forge)
runner.add(package, last(Ring))
runner.add(polish, first(Ring))
runner.add(more_polish, first(Ring))
runner.add(engrave, Ring)

Even though the callables were added out order, they will be executed correctly:

>>> runner()
polishing a ring
polishing a ring again
engraving a ring
packaging a ring

The configuration of periods works identically with both the imperative and declarative forms.

def func1():
    return Apple()

def func2(apple):
    print('func2 got {0}'.format(apple))
    return Orange()

def func3(apple):
    print('func3 got {0}'.format(apple))

def func4(orange):
    print('func4 processed {0}'.format(orange))

>>> runner = Runner(func1)
>>> runner.add(func2, Apple)
>>> runner.add(func3, Apple)
>>> runner.add(func4, Orange)
>>> runner()
func2 got an apple
func3 got an apple
func4 processed an orange

from mush import Runner, after

runner = Runner(func1)
runner.add(func2, Apple)
runner.add(func3, after(Orange), Apple)
runner.add(func4, Orange)

>>> runner()
func2 got an apple
func4 processed an orange
func3 got an apple

Special return types¶

There are certain types that can be returned from a callable that have special meaning. While these are provided in case of their specific need, you should think twice when you find yourself wanting to use them as it is often a sign that your code can be better structured differently.

For the examples below, we’ll use the fruit classes from above.

def orchard():
    return Apple(), Orange()

>>> runner = Runner(orchard, juicer)
>>> runner()
I made juice out of an apple and an orange

class Pear:
    def __str__(self):
        return 'a pear'

def desperation():
    print('oh well, a pear will have to do')
    return {Apple: Pear()}

>>> runner = Runner(orange_tree, desperation, juicer)
>>> runner()
I made an orange
oh well, a pear will have to do
I made juice out of a pear and an orange

from mush import returns

@returns(Apple)
def make_pears():
    print('I made a pear')
    return Pear()

>>> runner = Runner(orange_tree, make_pears, juicer)
>>> runner()
I made an orange
I made a pear
I made juice out of a pear and an orange

from mush import returns

def someone_elses_pears():
    print('I made a pear')
    return Pear()

>>> runner = Runner(orange_tree, juicer)
>>> runner.add_returning(someone_elses_pears, returns=Apple)
>>> runner()
I made an orange
I made a pear
I made juice out of a pear and an orange

from mush import nothing, marker

@returns(marker('SetupComplete'))
def setup():
    print('setting things up')

@requires(after(marker('SetupComplete')))
def body():
    print('doing stuff')

>>> runner = Runner(body, setup)
>>> runner()
setting things up
doing stuff

Using parts of a resource¶

When pieces of functionality use settings provided either by command line arguments or from configuration files, it’s often cleaner to structure that code to recieve the specific setting value rather than the setting’s container. It’s certainly easier to test. Mush can take care of the needed wiring when configured to do so using the attr and item helpers:

from mush import Runner, attr, item, requires

class Config(dict): pass

class Args(object):
    fruit = 'apple'
    tree = dict(fruit='pear')

def parse_args():
    return Args()

def read_config():
    return Config(fruit='orange')

@requires(attr(Args, 'fruit'),
          item(Config, 'fruit'),
          item(attr(Args, 'tree'), 'fruit'))
def pick(fruit1, fruit2, fruit3):
    print('I picked {0}, {1} and {2}'.format(fruit1, fruit2, fruit3))
    picked = []
    for fruit in fruit1, fruit2:
        if fruit=='apple':
            picked.append(Apple())
        elif fruit=='orange':
            picked.append(Orange())
        else:
            raise TypeError('You have made a poor fruit choice')
    return picked

While the pick() function remains usable and testable on its own:

>>> pick('apple', 'orange', 'pear')
I picked apple, orange and pear
[an apple, an orange]

It can also be added to a runner with the other necessary functions and Mush will do the hard work:

>>> runner = Runner(parse_args, read_config, pick, juicer)
>>> runner()
I picked apple, orange and pear
I made juice out of an apple and an orange

Context manager resources¶

A frequent requirement when writing scripts is to make sure that when unexpected things happen they are logged, transactions are aborted, and other necessary cleanup is done. Mush supports this pattern by allowing context managers to be added as callables:

from mush import Runner, requires

class Transactions(object):

    def __enter__(self):
        print('starting transaction')

    def __exit__(self, type, obj, tb):
        if type:
            print(obj)
            print('aborting transaction')
        else:
            print('committing transaction')
        return True

def a_func():
    print('doing my thing')

def good_func():
    print('I have done my thing')

def bad_func():
    raise Exception("I don't want to do my thing")

The context manager is wrapped around all callables that are called after it:

>>> runner = Runner(Transactions, a_func, good_func)
>>> runner()
starting transaction
doing my thing
I have done my thing
committing transaction

This gives it a chance to clear up when things go wrong:

>>> runner = Runner(Transactions, a_func, bad_func)
>>> runner()
starting transaction
doing my thing
I don't want to do my thing
aborting transaction

Debugging¶

Mush makes some heuristic decisions about the order in which to call objects added to a runner. If your expectations of the call order don’t match that used by Mush, it can be confusing to figure out where the difference comes from.

For this reason, when constructing a Runner you can pass an optional debug parameter which can be either a boolean True or a file-like object. If passed, debug information will be generated whenever an object is added to the runner.

If True, this information will be written to stderr. If a file-like object is passed instead, the information will be written to that object.

As an example, consider this code:

from mush import Runner, requires

class T1(object): pass
class T2(object): pass
class T3(object): pass

def makes_t1():
    return T1()

@requires(T1)
def makes_t2(obj):
    return T2()

@requires(T2)
def makes_t3(obj):
    return T3()

@requires(T3, T1)
def user(obj1, obj2):
    m.user(type(obj1), type(obj2))

The debug information that would be written looks like this:

>>> import sys
>>> runner = Runner(makes_t1, makes_t2, makes_t3, user, debug=sys.stdout)
Added <function makes_t1 ...> to 'normal' period for <... 'NoneType'> with Requirements()
Current call order:
For <... 'NoneType'>:
  normal: <function makes_t1 ...> requires Requirements()

Added <function makes_t2 ...> to 'normal' period for <class 'T1'> with Requirements(T1)
Current call order:
For <... 'NoneType'>:
  normal: <function makes_t1 ...> requires Requirements()
For <class 'T1'>:
  normal: <function makes_t2 ...> requires Requirements(T1)

Added <function makes_t3 ...> to 'normal' period for <class 'T2'> with Requirements(T2)
Current call order:
For <... 'NoneType'>:
  normal: <function makes_t1 ...> requires Requirements()
For <class 'T1'>:
  normal: <function makes_t2 ...> requires Requirements(T1)
For <class 'T2'>:
  normal: <function makes_t3 ...> requires Requirements(T2)

Added <function user ...> to 'normal' period for <class 'T3'> with Requirements(T3, T1)
Current call order:
For <... 'NoneType'>:
  normal: <function makes_t1 ...> requires Requirements()
For <class 'T1'>:
  normal: <function makes_t2 ...> requires Requirements(T1)
For <class 'T2'>:
  normal: <function makes_t3 ...> requires Requirements(T2)
For <class 'T3'>:
  normal: <function user ...> requires Requirements(T3, T1)

Encapsulating the re-usable parts of scripts¶

So, let’s start by looking at how these common sections of code can be extracted into re-usable functions that can be assembled by Mush into scripts:

from argparse import ArgumentParser, Namespace
from .configparser import RawConfigParser
from mush import Runner, requires, first, last, attr, item
import logging, os, sqlite3, sys

log = logging.getLogger()

@requires(ArgumentParser)
def base_options(parser):
    parser.add_argument('config', help='Path to .ini file')
    parser.add_argument('--quiet', action='store_true',
                        help='Log less to the console')
    parser.add_argument('--verbose', action='store_true',
                        help='Log more to the console')

@requires(last(ArgumentParser))
def parse_args(parser):
    return parser.parse_args()

class Config(dict): pass

@requires(first(Namespace))
def parse_config(args):
    config = RawConfigParser(dict_type=Config)
    config.read(args.config)
    return Config(config.items('main'))

def setup_logging(log_path, quiet=False, verbose=False):
    handler = logging.FileHandler(log_path)
    handler.setLevel(logging.DEBUG)
    log.addHandler(handler)
    if not quiet:
        handler = logging.StreamHandler(sys.stderr)
        handler.setLevel(logging.DEBUG if verbose else logging.INFO)
        log.addHandler(handler)

class DatabaseHandler:
    def __init__(self, db_path):
        self.conn = sqlite3.connect(db_path)
    def __enter__(self):
        return self
    def __exit__(self, type, obj, tb):
        if type:
            log.exception('Something went wrong')
            self.conn.rollback()

We start with a function that adds the options needed by all our scripts to an argparse parser. This uses the requires() decorator to tell Mush that it must be called with an ArgumentParser instance. See Configuring Resources for more details.

Next, we have a function that calls parse_args() on the parser and returns the resulting Namespace. It uses the last() decorator to indicate that it should be the last callable to use the ArgumentParser instance. See Resource usage periods for more details.

The parse_config() function uses the first() decorator to indicate that it needs first use of the command line arguments. It uses those to read the specified configuration and return the configuration as a dictionary. A sub-class of dict is used so that other callables can easily indicate that they require the configuration to be passed in.

Finally, there is a function that configures log handling and a context manager that provides a database connection and handles exceptions that occur by logging them and aborting the transaction. Context managers like this are handled by Mush in a specific way, see Context manager resources for more details.

Each of these components can be separately and thoroughly tested. The Mush decorations are inert to all but the Mush Runner, meaning that they can be used independently of Mush in whatever way is convenient. As an example, the following tests use a TempDirectory and a LogCapture to fully test the database-handling context manager:

    def setUp(self):
        self.dir = TempDirectory()
        self.db_path = self.dir.getpath('test.db')
        self.conn = sqlite3.connect(self.db_path)
        self.conn.execute('create table notes '
                          '(filename varchar, text varchar)')
        self.conn.commit()
        self.log = LogCapture()
        
    def tearDown(self):
        self.log.uninstall()
        self.dir.cleanup()
        
    def test_normal(self):
        with DatabaseHandler(self.db_path) as handler:
            handler.conn.execute('insert into notes values (?, ?)',
                                 ('test.txt', 'a note'))
            handler.conn.commit()
        # check the row was inserted and committed
        curs = self.conn.cursor()
        curs.execute('select * from notes')
        self.assertEqual(curs.fetchall(), [('test.txt', 'a note')])
        # check there was no logging
        self.log.check()

    def test_exception(self):
        with ShouldRaise(Exception('foo')):
            with DatabaseHandler(self.db_path) as handler:
                handler.conn.execute('insert into notes values (?, ?)',
                                     ('test.txt', 'a note'))
                raise Exception('foo')
        # check the row not inserted and the transaction was rolled back
        curs = handler.conn.cursor()
        curs.execute('select * from notes')
        self.assertEqual(curs.fetchall(), [])
        # check the error was logged
        self.log.check(('root', 'ERROR', 'Something went wrong'))
    

Writing the specific parts of your script¶

Now that all the re-usable parts of the script have been abstracted, writing a specific script becomes a case of just writing two functions:

def args(parser):
    parser.add_argument('path', help='Path to the file to process')

def do(conn, path):
    filename = os.path.basename(path)
    with open(path) as source:
        conn.execute('insert into notes values (?, ?)',
                     (filename, source.read()))
    conn.commit()
    log.info('Successfully added %r', filename)

As you can imagine, this much smaller set of code is simpler to test and easier to maintain.

Assembling the components into a script¶

So, we now have a library of re-usable components and the specific callables we require for the current script. All we need to do now is assemble these parts into the final script. The full details of this are covered in Constructing runners but two common patterns are covered below.

from mush import Runner, requires, first, last, attr, item

base_runner = Runner(ArgumentParser, base_options, parse_args, parse_config)
base_runner.add(setup_logging,
                log_path = item(first(Config), 'log'),
                quiet = attr(first(Namespace), 'quiet'),
                verbose = attr(first(Namespace), 'verbose'))

main = base_runner.clone()
main.add(args, ArgumentParser)
main.add(DatabaseHandler, item(Config, 'db'))
main.add(do,
         attr(DatabaseHandler, 'conn'),
         attr(Namespace, 'path'))

if __name__ == '__main__':
    main()

def options(parser):
    parser.add_argument('config', help='Path to .ini file')
    parser.add_argument('--quiet', action='store_true',
                        help='Log less to the console')
    parser.add_argument('--verbose', action='store_true',
                        help='Log more to the console')
    parser.add_argument('path', help='Path to the file to process')

def make_runner(do):
    runner = Runner(ArgumentParser)
    runner.add(options, ArgumentParser)
    runner.add(parse_args, last(ArgumentParser))
    runner.add(parse_config, first(Namespace))
    runner.add(setup_logging,
               log_path = item(first(Config), 'log'),
               quiet = attr(first(Namespace), 'quiet'),
               verbose = attr(first(Namespace), 'verbose'))
    runner.add(DatabaseHandler, item(Config, 'db'))
    runner.add(do,
               attr(DatabaseHandler, 'conn'),
               attr(Namespace, 'path'))
    return runner

main = make_runner(do)

Testing¶

The examples above have shown how using Mush can make it easier to have smaller components that are easier to re-use and test, however care should still be taken with testing. In particular, it’s a good idea to have some intergration tests that excercise the whole runner checking that it behaves as expected when all command line options are specified and when just the defaults are used.

When using the factory pattern, the factories themselves should be unit tested. It can also make tests easier to write by having a “testing runner” that sets up the required resources, such as database connections, while maybe doing some things differently such as not reading a configuration file from disk or using a LogCapture instead of file or stream log handlers.

Debugging¶

Mush does its best to call the things added to runners in the best order possible. If this doesn’t match your expectations, passing the debug flag when the Runner is instantiated can give you more insight into what’s going on. Please see Debugging for more information.

Setting up a virtualenv¶

The recommended way to set up a development environment is to turn your checkout into a virtualenv and then install the package in editable form as follows:

$ virtualenv .
$ bin/pip install -U -e .[test,build]

Running the tests¶

Once you have a buildout, the tests can be run as follows:

$ bin/nosetests

Building the documentation¶

The Sphinx documentation is built by doing the following from the directory containg setup.py:

$ cd docs
$ make html

Making a release¶

To make a release, just update the version in setup.py, update the change log, tag it and push to https://github.com/Simplistix/mush and Travis CI should take care of the rest.

Once Travis CI is done, make sure to go to https://readthedocs.org/projects/mush/versions/ and make sure the new release is marked as an Active Version.

1.3 (21 October 2015)¶

• Official support for Python 3.
• Drop official support for Windows, although things should still work.
• Move to Travis CI, Read The Docs and Coveralls for development.
• ‘How’ decorators like attr() and item() can now be nested as well as individually performing nested actions.
• Add returns() and add_returning() as new ways to override the type of a returned value.
• A better pattern for marker types.

1.2 (11 December 2013)¶

• Use nothing instead of None for marker return types, fixing a bug that occurred when a callable tried to type-map a result that was None.
• Add an after() type wrapper for callables that need to wait until after a resource is used but that can’t accept that resource as a parameter.

1.1 (27 November 2013)¶

• Allow runners to be instantiated using other runners.
• Allow Runner.extend() to be passed Runner instances.
• Allow requires() decorations to be stacked.
• Add a Runner.replace() method to aid with testing assembled runners.

1.0 (29 October 2013)¶

• Initial Release