Table of Contents¶

Quickstart¶

container-transform is a small utility to transform various docker container formats to one another.

Currently, container-transform can parse and convert:

Kubernetes Pod specs
ECS task definitions
Docker-compose configuration files
Marathon Application Definitions or Groups of Applications
Chronos Task Definitions

and it can output to:

Systemd unit files

Any missing required parameters are printed to STDERR.

Example usage:

$ cat docker-compose.yml | container-transform  -v
{
    "family": "python-app",
    "volumes": [
        {
            "name": "host_logs",
            "host": {
                "sourcePath": "/var/log/myapp"
            }
        }
    ],
    "containerDefinitions": [
        {
            "memory": 1024,
            "image": "postgres:9.3",
            "name": "db",
            "essential": true
        },
        {
            "memory": 128,
            "image": "redis:latest",
            "name": "redis",
            "essential": true
        },
        {
            "name": "web",
            "memory": 64,
            "command": [
                "uwsgi",
                "--json",
                "uwsgi.json"
            ],
            "mountPoints": [
                {
                    "sourceVolume": "host_logs",
                    "containerPath": "/var/log/uwsgi/"
                }
            ],
            "environment": [
                {
                    "name": "AWS_ACCESS_KEY_ID",
                    "value": "AAAAAAAAAAAAAAAAAAAA"
                },
                {
                    "name": "AWS_SECRET_ACCESS_KEY",
                    "value": "1111111111111111111111111111111111111111"
                }
            ],
            "essential": true
        }
    ]
}
Container web is missing required parameter "image".

or:

$ container-transform  --input-type ecs --output-type compose task.json
db:
  image: postgres:9.3
  mem_limit: 1073741824b
redis:
  image: redis:latest
  mem_limit: 134217728b
web:
  command: uwsgi --json uwsgi.json
  environment:
    AWS_ACCESS_KEY_ID: AAAAAAAAAAAAAAAAAAAA
    AWS_SECRET_ACCESS_KEY: '1111111111111111111111111111111111111111'
  mem_limit: 67108864b

Container web is missing required parameter "image".

Installation¶

To install the latest release (Python 3 only), type:

pip install container-transform

To install the latest code directly from source, type:

pip install git+git://github.com/micahhausler/container-transform.git

Usage¶

$ container-transform -h
Usage: container-transform [OPTIONS] [INPUT_FILE]

  container-transform is a small utility to transform various docker
  container formats to one another.

  Default input type is compose, default output type is ECS

  Default is to read from STDIN if no INPUT_FILE is provided

  All options may be set by environment variables with the prefix "CT_"
  followed by the full argument name.

Options:
  -i, --input-type [ecs|compose|marathon|chronos|kubernetes]
  -o, --output-type [ecs|compose|systemd|marathon|chronos|kubernetes]
  -v, --verbose / --no-verbose    Expand/minify json output
  -q, --quiet                     Silence error messages
  --version                       Show the version and exit.
  -h, --help                      Show this message and exit.

Kubernetes Format¶

When consuming Kubernetes input, container-transform supports the following object types:

ReplicaSet
Deployment
DaemonSet
Pod
ReplicationController

and will only load the first of those objects in the file.

Kubernetes Pods & Kubernetes API Objects

Marathon Applications¶

When consuming Marathon input, container-transform supports:

A single Marathon application
Content from the Marathon Group API
A JSON array of Marathon application objects

When emitting Marathon output, container-transform will emit a list of applications if there is more than one. Otherwise, it will emit a single application.

Marathon Application Basics & Marathon API docs

Chronos Tasks¶

Chronos tasks are meant to be run as headless tasks and while most docker options may be passed as parameters, they may not work as intended on a Mesos cluster.

When consuming Chronos input, container-transform supports:

A single Chronos task
A JSON array of Chronos tasks

When emitting Chronos output, container-transform will emit a list of tasks if there is more than one. Otherwise, it will emit a single task.

Note

A JSON array of tasks is not valid for submitting to the Chronos API, but is meant to be a convenience so that the output can be manipulated with other tools.

Chronos API Documentation & Chronos Job Serializer source code

API Documentation¶

KubernetesTransformer¶

class container_transform.kubernetes.KubernetesTransformer(filename=None)¶

A transformer for Kubernetes Pods

TODO: look at http://kubernetes.io/docs/api-reference/v1/definitions/#_v1_pod

__init__(filename=None)¶

Parameters:	filename (str) – The file to be loaded

emit_containers(containers, verbose=True)¶

Emits the applications and sorts containers by name

Parameters:	containers (list of dict) – List of the container definitions verbose (bool) – Print out newlines and indented JSON
Returns:	The text output
Return type:	str

flatten_container(container)¶: Accepts a kubernetes container and pulls out the nested values into the top level

ingest_memory(memory)¶

Transform the memory into bytes

Parameters:	memory (memory string or integer) – Compose memory definition. (1g, 24k)
Returns:	The memory in bytes
Return type:	int

ingest_port_mappings(port_mappings)¶

Transform the port mappings to base schema mappings

Parameters:	port_mappings (list of dict) – The port mappings
Returns:	The base schema mappings
Return type:	list of dict

ingest_volumes_param(volumes)¶: This is for ingesting the “volumes” of a pod spec

ECSTransformer¶

class container_transform.ecs.ECSTransformer(filename=None)¶

A transformer for ECS Tasks

To use this class:

transformer = ECSTransformer('./task.json')
output = transformer.ingest_containers()
print(json.dumps(output, indent=4))

__init__(filename=None)¶

We override .__init__() on purpose, we need to get the volume data.

Parameters:	filename (str) – The file to be loaded

add_volume(volume)¶: Add a volume to self.volumes if it isn’t already present

emit_containers(containers, verbose=True)¶

Emits the task definition and sorts containers by name

Parameters:	containers (list of dict) – List of the container definitions verbose (bool) – Print out newlines and indented JSON
Returns:	The text output
Return type:	str

ingest_port_mappings(port_mappings)¶

Transform the ECS mappings to base schema mappings

Parameters:	port_mappings (list of dict) – The ECS port mappings
Returns:	The base schema mappings
Return type:	list of dict

ingest_volumes_param(volumes)¶: This is for ingesting the “volumes” of a task description

ComposeTransformer¶

class container_transform.compose.ComposeTransformer(filename=None)¶

A transformer for docker-compose v1 and v2

To use this class:

transformer = ComposeTransformer('./docker-compose.yml')
normalized_keys = transformer.ingest_containers()

__init__(filename=None)¶

We override .__init__() on purpose, we need to get the volume, version, network, and possibly other data.

Parameters:	filename (str) – The file to be loaded

emit_port_mappings(port_mappings)¶

Parameters:	port_mappings (list of dict) – the base schema port_mappings
Returns:
Return type:	list of str

ingest_containers(containers=None)¶: Transform the YAML into a dict with normalized keys

ingest_memory(memory)¶

Transform the memory into bytes

Parameters:	memory (memory string or integer) – Compose memory definition. (1g, 24k)
Returns:	The memory in bytes
Return type:	int

ingest_port_mappings(port_mappings)¶

Transform the docker-compose port mappings to base schema port_mappings

Parameters:	port_mappings (list) – The compose port mappings
Returns:	the base schema port_mappings
Return type:	list of dict

SystemdTransformer¶

class container_transform.systemd.SystemdTransformer¶

A transformer for docker-compose

To use this class:

transformer = SystemdTransformer()

__init__()¶: Initialize self. See help(type(self)) for accurate signature.

emit_port_mappings(port_mappings)¶

Parameters:	port_mappings (list of dict) – the base schema port_mappings
Returns:
Return type:	list of str

MarathonTransformer¶

class container_transform.marathon.MarathonTransformer(filename=None)¶

A transformer for Marathon Apps

When consuming Marathon input, the transformer supports:

A single Marathon application
Content from the Marathon Group API
A JSON array of Marathon application objects

When emitting Marathon output, the transformer will emit a list of applications if there is more than one. Otherwise, it will emit a single application.

To use this class:

transformer = MarathonTransformer('./app.json')
output = transformer.ingest_container()
print(json.dumps(output, indent=4))

__init__(filename=None)¶

Parameters:	filename (str) – The file to be loaded

emit_containers(containers, verbose=True)¶

Emits the applications and sorts containers by name

Parameters:	containers (list of dict) – List of the container definitions verbose (bool) – Print out newlines and indented JSON
Returns:	The text output
Return type:	str

flatten_container(container)¶: Accepts a marathon container and pulls out the nested values into the top level

ingest_port_mappings(port_mappings)¶

Transform the port mappings to base schema mappings

Parameters:	port_mappings (list of dict) – The port mappings
Returns:	The base schema mappings
Return type:	list of dict

ChronosTransformer¶

class container_transform.chronos.ChronosTransformer(filename=None)¶

A transformer for Chronos Jobs

When consuming Chronos input, the transformer supports:

When emitting Chronos output, the transformer will emit a list of applications if there is more than one. Otherwise, it will emit a single application.

To use this class:

transformer = ChronosTransformer('./task.json')
output = transformer.ingest_container()
print(json.dumps(output, indent=4))

__init__(filename=None)¶

Parameters:	filename (str) – The file to be loaded

emit_containers(containers, verbose=True)¶

Emits the applications and sorts containers by name

Parameters:	containers (list of dict) – List of the container definitions verbose (bool) – Print out newlines and indented JSON
Returns:	The text output
Return type:	str

flatten_container(container)¶: Accepts a chronos container and pulls out the nested values into the top level

ingest_port_mappings(port_mappings)¶

Transform the port mappings to base schema mappings

Parameters:	port_mappings (list of dict) – The port mappings
Returns:	The base schema mappings
Return type:	list of dict

BaseTransformer¶

class container_transform.transformer.BaseTransformer¶

The base class for Transformer classes to inherit from.

Basic usage should look like

transformer = MyTransformer('./my-file.txt')
normalized_keys = transformer.ingest_containers()

__init__()¶: Initialize self. See help(type(self)) for accurate signature.

ingest_containers(containers=None)¶

Ingest self.stream and return a list of un-converted container definitions dictionaries.

This is to normalize where all the container information is. For example, Compose v1 places the container name outside the rest of the container definition. We need to have a ‘name’ key in the container definition.

Return type:	list of dict

static validate(container)¶

Validate that the container has all essential parameters and add any if possible

Parameters:	container (dict) – The converted container
Returns:	The container with all valid parameters
Return type:	dict

Contributing¶

Contributions and issues are most welcome! All issues and pull requests are handled through github on the issues page. Also, please check for any existing issues before filing a new one. If you have a great idea but it involves big changes, please file a ticket before making a pull request! We want to make sure you don’t spend your time coding something that might not fit the scope of the project.

Running the tests¶

To get the source source code and run the unit tests, run:

git clone git://github.com/micahhausler/container-transform.git
cd container-transform
virtualenv env
. env/bin/activate
pip install -e .[all]
python setup.py nosetests

While 100% code coverage does not make a library bug-free, it significantly reduces the number of easily caught bugs! Please make sure coverage is at 100% before submitting a pull request!

Code Quality¶

For code quality, please run flake8:

$ pip install flake8
$ flake8 .

Code Styling¶

Please arrange imports with the following style

# Standard library imports
import os

# Third party package imports
from mock import patch

# Local package imports
from container_transform.version import __version__

Please follow Google’s python style guide wherever possible.

Building the docs¶

When in the project directory:

pip install -e .[all]
python setup.py build_sphinx
open docs/_build/html/index.html

Adding a new parameter¶

If there is a docker parameter that you’d like to add support for, here’s a quick overview of what is involved:

Make a new parameter in the ARG_MAP in the file container-transform/schema.py
Check what the parameter name is for each supported transformation type. There are links to the documentation for each type on the Usage page
Create an ingest_<param> and emit_<param> method on the BaseTransformer class
Add any data transformations by overriding the base methods that each format requires.
Add tests to cover any new logic. Don’t just use a client test to make coverage 100%

Adding a new Transformer¶

If you’d like to add a new format, please create an issue before making a pull request in order to discuss any major design decisions before putting in valuable time writing the actual code.

Below is a rough checklist of creating a new transformer type:

Create a file and class in the base container_transform module
Implement all abstract methods on the BaseTransformer class
Add the class to the TRANSFORMER_CLASSES in the converter.py file.
Add the type to the enums at the top of the schema.py file.
Add a key to each of the dictionaries in the ARG_MAP parameters
If a docker parameter is not supported in your transformer, still create a dictionary for it, but set the name to None
Create a test file in the tests module for your transformer. Try to get at least 90% coverage of your transformer before adding any tests to the client_tests.py module.
Add client tests just to make sure the command doesn’t blow up
Add documentation and API links on the Usage page.
Update the usage text output on the README.rst and the Usage page
Add the type to the format list on the Quickstart and README.rst

Possible Transformer implementations:

Release Checklist¶

Before a new release, please go through the following checklist:

Bump version in container_transform/version.py
Add a release note in docs/release_notes.rst
Git tag the version

Upload to pypi:

pip install -e .[packaging]
python setup.py sdist bdist_wheel upload

Increment the version to x.x-dev

Vulnerability Reporting¶

For any security issues, please do NOT file an issue or pull request on github! Please contact hausler.m@gmail.com with the GPG key provided on keybase.

Release Notes¶

v1.1.5¶

Added Kubernetes Support
Updatd support for docker-compose and ECS
Properly split lines for cmd and entrypoint in ECS
Added automated deployment to Pypi
Bumped docker python to 3.6

v1.1.4¶

Added Chronos task support
Fixed privileged for Marathon output
Added support for Mesos fetcher

v1.1.3¶

Switched docker image to python:3.5-alpine (604 MB smaller, 77 MB total)
Fixed case when compose specifies command as an array

v1.1.2¶

Fixed udp port handling
Added support for pid and env-file parameters
Added docs for adding parameters and creating new transformers

v1.1.1¶

Added environment variable support for command line options
Added short form for command line options
CPU is no longer required for ECS

v1.1.0¶

Added Marathon task support
Correctly handle compose version type - GH #46
Assume compose memory is in bytes - GH #41

v1.0.0¶

Fixed GH #35
Removed fig support
Added support for labels
Added support for docker-compose v2
Added support for log drivers - GH #33

v0.6.1¶

Fix after names in Systemd
Fixed invalid volume name for ECS volumes
Added support for exec form of command and entrypoint

v0.6.0¶

Added Systemd as an output type

v0.5.1¶

Fixed issue where ECS host port was accidentally assigned when unspecified

v0.5.0¶

Features¶

Added support for local volumes
Output full ECS task json, including volume info

Internal¶

converted static methods to class methods to keep track of volume information

v0.4¶

Added support for docker compose
docker-compose is now the default input type

v0.3¶

Added support for volumesFrom in ECS Task Definitions

v0.2¶

Redesign of transformer classes
Added ability to read in ECS tasks and write fig configuration

v0.1¶

This is the initial release of container-transform.
Includes a Fig to ECS transformer

Table of Contents¶

Quickstart¶

Installation¶

Usage¶

Kubernetes Format¶

ECS Format¶

Docker Compose Format¶

Systemd Service Units¶

Marathon Applications¶

Chronos Tasks¶

API Documentation¶

KubernetesTransformer¶

ECSTransformer¶

ComposeTransformer¶

SystemdTransformer¶

MarathonTransformer¶

ChronosTransformer¶

BaseTransformer¶

Contributing¶

Running the tests¶

Code Quality¶

Code Styling¶

Building the docs¶

Adding a new parameter¶

Adding a new Transformer¶

Release Checklist¶

Vulnerability Reporting¶

Release Notes¶

v1.1.5¶

v1.1.4¶

v1.1.3¶

v1.1.2¶

v1.1.1¶

v1.1.0¶

v1.0.0¶

v0.6.1¶

v0.6.0¶

v0.5.1¶

v0.5.0¶

Features¶

Internal¶

v0.4¶

v0.3¶

v0.2¶

v0.1¶