xal¶

fs API¶

>>> from xal.fs.resource import FileSystem
>>> def demo_resource(session):
...     """Demonstrate use of ``FileSystem`` resource."""
...     fs = session.fs  # Just a shortcut.
...
...     # ``session.fs`` is a resource factory.
...     resource = fs('test-fs')
...     assert isinstance(resource, FileSystem)
...
...     # Just use path as strings.
...     assert str(resource) == 'test-fs'
...
...     # Paths remain relative until they are actually used. Absolute
...     # values are used as soon as filesystem operation is performed.
...     assert resource.is_relative()
...     created_resource = resource.mkdir()
...     assert created_resource.is_absolute()
...     assert resource.is_relative()  # Original value is preserved.
...
...     # Cleanup.
...     fs.rm(created_resource)

>>> def demo_cd(session):
...     """Demonstrate use of ``cd``."""
...     fs = session.fs  # Just a shortcut.
...
...     # Let's create a new directory.
...     path = fs.mkdir('test-fs')
...
...     # Change working directory.
...     initial_path = fs.cwd()
...     assert fs.cwd() != path
...     with fs.cd(path) as sub_path:
...         assert fs.cwd() == path == sub_path
...         assert sub_path.parent == initial_path
...     assert fs.cwd() == initial_path
...
...     # Also works without context manager.
...     # Notice that once used, paths are absolute.
...     rel_path = fs('test-fs')
...     assert rel_path.is_relative()
...     abs_path = rel_path.cd()
...     assert abs_path.is_absolute()
...
...     # Cleanup.
...     fs.cd(initial_path)
...     fs.rm(path)

>>> def demo_mkdir(session):
...     """Demonstrate use of ``mkdir``."""
...     fs = session.fs  # Just a shortcut.
...
...     # Let's create a new directory.
...     assert not fs.exists('test-fs')
...     path = fs.mkdir('test-fs')
...     assert fs.exists('test-fs')
...
...     # mkdir() returns absolute path, even if created from relative
...     # value.
...     assert str(path) != 'test-fs'
...     assert path.is_absolute()
...
...     # Cleanup.
...     fs.rm(path)

Providers¶

>>> from xal.session.local import LocalSession
>>> session = LocalSession()

>>> demo_resource(session)
>>> demo_cd(session)
>>> demo_mkdir(session)

>>> from xal.session.fabric import FabricSession
>>> session = FabricSession()
>>> session.client.connect('localhost')
True

>>> demo_resource(session)
>>> demo_cd(session)
>>> demo_mkdir(session)

Differences with subprocess¶

XAL’s sh interface is made to run sh commands in a session. Think of the commands always run with sh -c. Whereas Python’s subprocess sets shell=False by default.

This postulate influences design. XAL’s sh interface helps you create and run commands through sh: pipes, redirects...

“sh” provider in session’s registry¶

Let’s start with a local session:

>>> from xal.session.local import LocalSession
>>> session = LocalSession()

By default, in LocalSession, “sh” provider is xal.sh.local.LocalShProvider:

>>> session.registry.default('sh')  # Doctest: +ELLIPSIS
<xal.sh.local.LocalShProvider object at 0x...>

For convenience, we will set sh as a shortcut for session.sh:

>>> sh = session.sh

The ShCommand resource¶

The sh interface can be used as a factory to create xal.sh.resource.ShCommand resources:

>>> sh("echo -n 'Hello world!'")  # Doctest: +ELLIPSIS
<xal.sh.resource.ShCommand object at 0x...>

Command resources are not executed automatically once created. You have to run them explicitely. They are callables. When called, they return a xal.sh.resource.ShResult instance:

>>> command = sh("echo -n 'Hello world!'")
>>> result = command()
>>> result  # Doctest: +ELLIPSIS
<xal.sh.resource.ShResult object at 0x...>
>>> result.stdout
'Hello world!'
>>> result.return_code
0
>>> result.succeeded
True

Command constructor accepts strings or iterables:

>>> sh("echo -n 'Hello world!'")().stdout
'Hello world!'
>>> sh(["echo", "-n", "Hello world!"])().stdout
'Hello world!'

The sh interface has a run() shortcut that creates and runs Cmd instances:

>>> sh.run("echo -n 'Hello world!'").stdout
'Hello world!'
>>> sh.run(["echo", "-n", "Hello world!"]).stdout
'Hello world!'
>>> command = sh("echo -n 'Hello world!'")
>>> sh.run(command).stdout
'Hello world!'

You can create a resource for later use in one or several sessions:

>>> hello = sh("echo -n 'Hello world!'")
>>> from xal.session.local import LocalSession
>>> session = LocalSession()
>>> session.sh.run(hello).stdout
'Hello world!'

Pipes¶

You can create and handle pipes, they are commands too:

>>> echo = sh("echo -e 'hello\nworld'")
>>> grep = sh("grep 'world'")
>>> piped = echo.pipe(grep)
>>> piped().stdout
'world\n'
>>> piped = echo | grep
>>> piped().stdout
'world\n'
>>> sh.run(echo | grep).stdout
'world\n'

Contextual execution¶

In Python, there are several tools to perform actions on the local system. As an example, the subprocess module makes it possible to run arbitrary shell commands. So, let’s suppose you wrote a script that echoes “Hello world!” using the shell:

import subprocess
subprocess.call(['echo', u'Hello world!'])

Then what if you want to run commands on a remote machine? You can write a fabfile:

from fabric import api as fab_api

@fab_api.task
def hello():
    fab_api.run(['echo', 'Hello world!'])

Then what if you want to run the command as an admin? With Fabric:

from fabric import api as fab_api

@fab_api.task
def hello():
    fab_api.sudo(['echo', 'Hello world!'])

Then what if you want to run the command as a sudoer, on the local machine? Fabric doesn’t provide a wrapper for that:

from fabric import api as fab_api

@fab_api.task
def hello():
    fab_api.local(['sudo', 'echo', 'Hello world!'])

Then what if you want to run it on a Windows machine? You’ll have to adapt the code.

Then what if you migrate to another deployment tool, such as zc.buildout or Salt? You’ll have to change the code.

Even with Fabric, we had to write 3 distinct scripts to be able to face all situations. As developers, we’d like to write only one function, then pass it parameters:

• run on local machine or on remote client;
• run as current user, as admin/root, or maybe as another user.

That’s why xal were created: write portable high-level system scripts.

A framework¶

There are so many commands and so many systems... it would be impossible to support them all. And even if it was, it would be made of tons of code and dependencies. That’s the first reason why XAL is a framework:

• focus on the API;
• allow plugins;
• keep framework lightweight.

Fully configurable, no global states¶

XAL is not designed to use global states. XAL registry is entirely configurable: you can change providers’ mapping, configure providers or write custom ones.

However, for convenience, XAL offers some configuration helpers, so that common use cases are covered easily.

Interfaces¶

• spur [1] runs commands and manipulate files locally or over SSH using the same interface.

Deployment utilities¶

• Puppet
• Chef
• salt
• zc.buildout
• fabric
• fabtools
• collective.hostout

Subprocess and wrappers¶

Other¶

0.2 (unreleased)¶

• Nothing changed yet.

0.1 (2013-07-06)¶

• Feature #6 - Introduced proof of concept implementation of sh interface. Using a basic implementation for local session, based on subprocess.
• Feature #7 - Proof of concept implementation of dir interface, with a basic implementation for local session based on Python builtins (os, os.path, ...).
• Feature #8 - Introduced contextual execution architecture: session, registry, providers, resources, client.

Python for sysadmins¶

Python is great:

• shell, scripts, provisioners, frameworks...
• runs on almost any system

But...

It’s hard to write and share portable scripts¶

• environment vary: users, packages...
• provisioners are overkill: just want a shell or a simple script
• libraries are divided: fabric, buildout, salt...

Develop to XAL session¶

Write a script which takes a XAL session as argument:

def write_greetings(session):
    """Write 'Hello world!' in 'greetings.txt' file relative to user's home."""
    home = session.user.home
    file_path = session.file.join(home, 'greetings.txt')
    file_resource = session.file(file_path)
    if not file_resource.exists():
        file_resource.write('Hello world!')

Fabric¶

Use it in a fabfile:

from fabric.api import task
import write_greetings
import xal

@task
def hello_fabric():
    session = xal.fabric(sudoer=True)
    write_hello_world(session)

zc.buildout¶

In a buildout recipe:

import write_greetings
import xal

class HelloBuildout(object):
    def __init__(self, buildout, name, options):
        self.session = xal.buildout(buildout, name, options)

    def install(self):
        write_hello_world(self.session)

    def update(self)
        pass

Salt¶

As a salt module:

import write_greetings
import xal

def hello_salt():
    session = xal.salt(__salt__)
    write_greetings(session)

Shell¶

In an interactive shell:

import write_greetings
import xal
session = xal.local()
write_greeting(session)

Resources¶

XAL session is a proxy to resources:

• files, directories,
• users,
• processes,
• packages,
• your own customized resources...

Share and reuse scripts!¶

• Reduced cost of change
• Enhanced collaboration between projects related to deployment

=> What about scripts fabric, buildout, salt... can share on PyPI?

XAL is a proof of concept¶

• https://github.com/benoitbryon/xal
• feedback is welcome!
• are popular projects interested in?

Abstract¶

XAL is a proof of concept designed to write high-level scripts you can reuse within various tools such as Fabric, zc.buildout, Salt…

Python has strong features for sysadmin scripts: portability, a shell and great libraries. But, currently, you develop using tool’s specific implementation. As a consequence, users of each project develop different tools that do similar things.

XAL proposes a new approach for system-related scripts: develop to a session. Write scripts that get a session as argument. That session is an abstraction layer for system resources such as files, users, packages… It is just like an ORM abstracts the database implementation. This design makes it possible to share sysadmin scripts that you can use within all those great Python tools related to system and deployment. Reduced cost of change and improved collaboration!

This poster will present the concepts of XAL. This project is a proof-of-concept, so the author will be glad to get feedback, discuss the ideas and, if you are interested in, sprint on it!

xal¶

xal

execution abstraction layer for high-level system scripts

Develop to a session¶

def greetings(session):
    """Write 'Hello world!' in 'greetings.txt' file relative to user's home."""
    home = session.user.home
    file_path = session.file.join(home, u'greetings.txt')
    file_resource = session.file(file_path)
    with file_resource.open('w'):
        file_resource.write(u'Hello world!')

Run it anywhere¶

>>> from europython import greetings
>>> import xal
>>> session = xal.LocalSession()
>>> greetings(session)

from europython import greetings
import xal

def hello_fabric():
    session = xal.FabricSession(sudoer=True)
    greetings(session)

from europython import greetings
import xal

class HelloBuildout(object):
    def __init__(self, buildout, name, options):
        self.session = xal.BuildoutSession(buildout, name, options)

    def install(self):
        greetings(self.session)

from europython import greetings
import xal

def hello_salt():
    session = xal.SaltSession(__salt__)
    greetings(session)

Share libraries¶

xal enables wider cooperation in Python community

Improve your workflow¶

• Do not wait for full provisioning stack, enter your project ASAP
• First focus on what your script does
• Then configure execution environment
• Build the provisioning stack incrementally
• Change provisioner, keep deployment recipes

Exit the subprocess labyrinth¶

• subprocess
• async_subprocess
• chut
• clom
• Command
• commandwrapper
• cpopen
• desub
• EasyProcess
• envoy
• execute
• extcmd
• extproc
• gevent_subprocess
• iterpipes
• pbs
• pipeline
• pyutilib.subprocess
• sarge
• seminode.utils.command
• sh
• shellout
• spur
• subwrap
• ... and more...
• ... and maybe yours

Could xal help APIs converge?

Challenges¶

• xal needs strong APIs:

  • run commands (subprocess wrapper)
  • consistent set of resources (files, users...)

  => PEPs?

• Efficient and comprehensive session registry

• Preconfigured registries: fabric, salt, buildout, local session...

• Resolution of session’s dependencies

• Smart handling of NotImplementedError

• Unit tests with mocks, integration tests

xal is a proof of concept¶

https://xal.readthedocs.org

Credits, license¶

xal is released under BSD license © 2012-2013, Benoît Bryon