What is the Ubuntu CI Engine?¶

The Ubuntu CI Engine is the implementation of the CI Airline designed to perform continuous integration (CI) of source code and binary packages under a converged workflow. The project is being implemented in multiple phases.

Prerequisites¶

The Ubuntu CI Engine current only processes source packages as input. These source packages will be uploaded unmodified into a Launchpad PPA. As a result, the source packages used as input must be signed by a user with permission to upload to the PPAs used by the Ubuntu CI Engine. These teams will be managed through Launchpad team membership granted via a CI team member.

Launchpad Setup

Prior to creating a source package to upload and a ticket, a user will need setup a Launchpad account and OpenPGP key:

• Create an https://launchpad.net/ account.
• Create and upload an OpenPGP key to Launchpad. Instructions are available at https://help.launchpad.net/YourAccount/ImportingYourPGPKey.

Creating a Source Package

Once the account setup is complete, users can create source packages. A full tutorial on creating and modifying source packages are beyond the scope of this document. There are many Ubuntu development and packaging guides available including:

• http://packaging.ubuntu.com/html/
• https://wiki.ubuntu.com/UbuntuDevelopment

Setting up the system for ticket creation

The following steps are needed to be able to use the command line interface:

sudo add-apt-repository ppa:canonical-ci-engineering/ci-airline-phase-0
sudo apt-get update
sudo apt-get install uci-cli

Creating a Ticket¶

Ubuntu CI Engine requests, known as tickets, are created through a command line interface.

ubuntu-ci [-S|--url] create_ticket -t "Ticket name" -d "Ticket description" -b 123 -o user@example.com -s /full/path/to/_source.changes -s /full/path/to/_source.changes

This returns a ticket ID which can later be used to monitor and check for results. Once the ticket is created, the Ubuntu CI Engine does the rest. The ticket will be queued by the Ticket System and executed in FIFO (First In First Out) order. As the system is limited to processing one ticket at a time, it may take multiple hours for a single ticket to be completed.

usage: ubuntu-ci [-h] [-v {1,2,3}] [-u URL] [-S]
               {create_ticket,update_ticket,status,get_image} ...

optional arguments:
  -h, --help            show this help message and exit
  -v {1,2,3}, --verbosity {1,2,3}
                      Verbosity level; 1=errors only, 2=verbose output,
                      3=very verbose output
  -u URL, --url URL     Development ticket system url
  -S, --staging
  -t TITLE, --title TITLE
                        Ticket title
  -d DESCRIPTION, --description DESCRIPTION
                        Ticket description
  -b BUG, --bug BUG     Related bug number
  -o OWNER, --owner OWNER
                        Email address of the ticket owner
  -s SOURCES, --sources SOURCES
                        Path to source.changes files. Source package files
                        (e.g. source.dsc, source.orig.tar.gz, etc.) are
                        required to be in the same directory as their
                        respective source.changes.
  -w, --wait            Do not queue the ticket for processing.
actions:
  {create_ticket,status,get_image}
                      commands
  create_ticket       Create a new ticket
  update_ticket       Update an existing ticket.
  status              Get ticket status. Use no flags for all tickets
  get_image           Retrieve the image produced by a ticket.

Monitoring a Ticket¶

As a ticket progresses through the Ubuntu CI Engine, the status of the ticket is updated to reflect it’s current processing step. This status can be checked via the web interface.

Collecting Ticket Results¶

Once a ticket is completed, it’s status will be updated appropriately and all build and test results and logs will be available via the web interface.

Exploiting Test Results¶

The test runner produces results in the subunit v1 format.

The subunit stream can be downloaded from the web interface and processed locally in different ways.

Note that the subunit stream contains the dep-8 test results.

The subunit v1 format is text only and as such can be read but it’s not especially user-friendly:

test: dsc0t-build
successful: dsc0t-build [ multipart
Content-Type: text/plain;charset=utf8
stderr
0^M
Content-Type: text/plain;charset=utf8
stdout
12^M
build: OK
run: OK
0^M
]

subunit provide filters to convert a stream into more readable outputs.

$ subunit-1to2  <subunit.stream | subunit2pyunit

$ subunit-1to2  <fail.stream | subunit2pyunit
tests.test_pass
tests.test_pass ... ok
tests.test_fail
tests.test_fail ... FAIL
======================================================================
FAIL: tests.test_fail
tests.test_fail
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "tests.py", line 31, in test_fail
    self.assertTrue(False)
  File "/usr/lib/python2.7/unittest/case.py", line 424, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true


----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=1)

$ subunit-1to2  <subunit.stream | subunit2junitxml >results.xml

General principles¶

Ticket itself

Ticket collects metadata about what components and branches to treat as feature branches. It will contain a description of this feature and an ETA (from the registrant). Other informations like against which image number we want to have our test running (generated from ticket creation time) and a proposed list of integration tests will also be provided. Finally, we select which ones we want to run against (automated ticket will have this pre-selected with a default per component).

We want as well the user to know about the status of their current delivery. This one can be a direct MP or a first-class ticket entered into the system.

The ticket will include evolving metadata with time and reflect dynamically the states related to it.

• what is building right now as per the ticket definition, and the different build status in the isolated environment on all those components.
• driving the build ordering as well if we have dependencies (but not strict bump) to help upstream only taking care of ABI bump at delivery time.
• getting a way to do a non change rebuild upload for one or more components.
• attaching source package instead of branches for features involving more than just projects we handle in bzr branches.
• knowing where we stands on all those feature branches and sources compared to latest in distro (if the source package isn’t the latest version in the development version anymore)
• what MP are pending against this ticket, meaning against all attached feature branches.
• what delta do we have between the various feature branches in this ticket and their corresponding trunks.
• knowing if we are able to merge or not against those trunks. If we can merge at a T time (and tests are all passing), propose a way to merge trunks easily into those branches.
• after getting the progress on their build, attaching latest available specific image (3 images per ticket): feature branch + fixed image number, feature branch + latest available image, trunk + feature branch merged + latest available image.
• knowing what latest image number is available, be able to change with it if test on latest image passed.
• getting tests progress while they are run dynamically. Represents them clearly against those previous 3 image tests
• ensuring that involved parties like core-devs and design team are involved if the ticket needs their review. A packaging change will require core-devs to ack their change. The design team will be in the process if there is a design change involved. Those should work through credentials.
• CI general health and global warnings if needed
• status on the corresponding components health
• gives an easy way once all those criteria are met (third-party acking, everything built and all tests passing) to give a “go” to the engine to deliver those different trunks.
• show the progress on the merging back to trunk, building packages there, tests passing, migration in UNAPPROVED/NEW, -proposed, release pocket and close the ticket completely once the next image is kicked in. Demonstrate explicitly when something is blocking there the whole pipeline for other delivery.

Tickets interactions

We also want to be able to show where their ticket or MP is in the component queue, and what time they can expect (in average) before seeing it delivered.

Global view

Finally, in this global view, we want to show the health of all projects:

• seeing all components (projects/branches) that we have in the CI system with global/general metadata (what test environment is going to be used, what tests are associated with that components, number of tickets opened against them and so on)
• giving a view for the managers to see what ticket their team are working on, and what’s the progress on them as well as global status (build/tests/ability to merge to trunk).
• if a direct commit to trunk blocked the project and that’s the only way to fix it back (another direct commit to trunk), surface that. All other tickets being blocked by that state (as touching that same component) should reflect that info as well.
• when tickets are expected to be delivered (based on the ETA), so that we can identify hot spots (times where a lot of landing will happen simultaneously and will clash) and try to shuffle them around to not having them in one landing (eventually by a global override on all tickets)
• a single point to see across all projects where different teams need to assess/review before the delivery takes place (pending packaging changes triggering a core-dev review, design review needed)
• CI general health

Generating Client Keys¶

Test data is submitted to the data store via the RESTful api, and this API call is secured with oauth. Client access keys need to be generated for every external client that wants to be able to post data into the database. This is achieved by changing to the nf-stats-service directory and running:

$ python3 -m nfss keys-add

This is an interactive script that will ask for client details, and finally will write a python script that can be used by the external client to insert data into the data store.

A list of client keys can be generated in a similar fashion:

$ python3 -m nfss keys-list

A specific client id can be revoked by specifying it’s client access key like so:

$ python3 -m nfss keys-del cj2DriLAGxmxinDyJzvDVQVltRSLNI

(obviously the client key will change, this is just an example).

Cleaning the database¶

Part of the oauth authentication scheme involves storing nonce values in the database. In order to prevent this table from filling up, we install a daily cron job that cleans the database. This can be achieved manually by running:

$ python3 -m nfss database-clean

Although this should never need to be done manually, since the restish charm installs a cron daily job to run this command.

Defining your Graph¶

angular.module('NonFunctional.graphing')
.controllerProvider.register("defaultGraphData", ['$scope', '$rootScope',
    function($scope, $rootScope) {
        $scope.graphData = [];
        // Expose a couple of helper methods to the template.
        $scope.dateFormatter = dates.dateFormatter;
        $scope.dateParser = dates.dateParser;

        $rootScope.$on("newdata", function(event, data) {
            $scope.graphData = [{ values: data.data }];
        });
    }
]);

<div ng-controller="defaultGraphData">
  <ol>
    <li ng-repeat="item in graphData[0].values">
      <b>{{dateFormatter()(dateParser(item.date_entered))}}:</b>
      <br />
      {{item.data}}
    </li>
  </ol>
</div>

{
    "image_release": "utopic",
    "image_build_number": "1:20140428:20140411.3",
    "kernel": 4.42,
    "xorg": 4.77,
    "kernel_init": 0,
    "desktop": 15.15,
    "plumbing": 8.43,
    "image_arch": "mako",
    "image_md5": "n/a",
    "ran_at": "2014-04-28 14:18:17.76909-04",
    "image_variant": "touch",
    "machine_mac": "",
    "machine_name": "mako",
    "boot": 32.77
}

{
    "data": {
        "image_release": "utopic",
        "image_build_number": "1:20140428:20140411.3",
        "kernel": 4.42,
        "xorg": 4.77,
        "kernel_init": 0,
        "desktop": 15.15,
        "plumbing": 8.43,
        "image_arch": "mako",
        "image_md5": "n/a",
        "ran_at": "2014-04-28 14:18:17.76909-04",
        "image_variant": "touch",
        "machine_mac": "",
        "machine_name": "mako",
        "boot": 32.77
    },
    "date_entered": "2014-04-28 18:18:17.76909+00",
    "id": 21272
}

[
    {
        key: 'Kernel',
        values: [ { date_entered: "...", value: 1 }, { date_entered: "...", value: 2 }, ... ]
    },
    {
        key: 'Plumbing',
        values: [ { date_entered: "...", value: 1 }, { date_entered: "...", value: 2 }, ... ]
    },
    {
        key: 'XOrg',
        values: [ { date_entered: "...", value: 1 }, { date_entered: "...", value: 2 }, ... ]
    },
    {
        key: 'Desktop',
        values: [ { date_entered: "...", value: 1 }, { date_entered: "...", value: 2 }, ... ]
    }
]

// setup event handling any new data and refreshing the graph.
$rootScope.$on("newdata", function(event, data) {
    // Massage the raw data into something usable by the tables/charts
    // (declared in the html).
    massageGraphData(data);
    // Refresh the graph once the new data is ready.
    $scope.api.refresh();
});

// Helper function used in massaging the data.
function isolator(key) {
    return function(item) {
        return {
            // using the dates service here that provides date manipulation
            // functions.
            date_entered: dates.dateParser(item.date_entered),
            timespan: Math.max(item.data[key], 0) }
    }
}

$scope.massageGraphData = function(blob) {
    $scope.rawData = blob;
    $scope.graphData = [
    {
        key: 'Kernel',
        values: blob.data.map(isolator('kernel'))
    },
    {
        key: 'Plumbing',
        values: blob.data.map(isolator('plumbing'))
    },
    {
        key: 'XOrg',
        values: blob.data.map(isolator('xorg'))
    },
    {
        key: 'Desktop',
        values: blob.data.map(isolator('desktop'))
    }
    ];
}

<div ng-controller="bootspeedCtrl">
    <nvd3 options="options" data="graphData" api="api"></nvd3>
</div>

$scope.options = {
    chart: {
        type: "stackedAreaChart",
        height: 400,
        margin: {left:100, top:10, bottom:40, right:100},
        x: modifiers.accessor('date_entered'),
        y: modifiers.accessor('timespan'),
        useInteractiveGuideLine: true,
        xAxis: {
            tickFormat: dates.dateFormatter(),
            staggerLabels: true,
        },
        yAxis: {
            tickFormat: modifiers.numberFormatter(',.2f', 's'),
            tickPadding: 10,
        },
    },
};

<table>
    <tr ng-repeat="series in graphData | reverse">
        <td><b>{{series.key}}:</b></td>
        <td ng-repeat="item in series.values">{{numberFormatter(',.2f', 's')(item.timespan)}}</td>
    </tr>
    <tr>
        <td><b>Date Entered:</b></td>
        <td ng-repeat="item in graphData[0].values">{{dateFormatter(ISO_ISH)(item.date_entered)}}</td>
    </tr>
</table>

// You need to expose the helper methods to the scope (within the
// controller):
$scope.ISO_ISH = dates.ISO_ISH;
$scope.dateFormatter = dates.dateFormatter;
$scope.numberFormatter = modifiers.numberFormatter;

this.definitions = {
    // Existing default definition.
    'default': {
        'templateUrl': 'graphs/default.html',
        'deps': 'graphs/default.js'
    },
    // Defining your graph def here.
    'my_test_name': {
        'templateUrl': 'graphs/my-custom-test-display.html',
        'deps': 'graphs/default.js'  // <-- Note the use of the default controller here.
    },
};

Introduction¶

The whole workflow is composed of 3 layers:

• One layer is responsible for delivering trunk to distro and handling the transaction to it. The delivery can be one or multiple components in one shot.
• The two other layers are the 2 different workflow to deliver a work. Either a MP to a project (we handle direct commit to trunk as well that way) or a feature branch that matured for a while. In any case, this is created as a ticket entry for the first base layer.

Putting this in images:

Components:

We can achieve all those functionalities with mainly 7 components:

• Projects manager
• Landing manager - Ticket manager (they do share most of their code)
• Branch listener
• Branch/Source builder
• ppa assigner
• Image builder
• Test runner

Delivery system¶

Case 1: delivering a ticket to trunk, (no pre-build package), success

Note: in case of failure in any step, the Projects manager will comment on the MP and reject it.

If direct push to trunk -> only set an error on the component and don’t enable any other landing to it.

Note 2: while monitoring the ppa or tests, we still can get signal from Projects manager to Landing manager telling “ignore this arch” or “ignore this step”. This impacts the landing manager on its view and can unblock/bypass some steps

Case 2: delivering a ticket to trunk, merge fail

Case 3: delivering a ticket to trunk, build fail

Case 4: delivering a ticket to trunk, tests fail

Case 5: delivering a ticket to trunk, blocked in UNAPPROVED/NEW

Case 6: delivering a ticket to trunk, blocked in proposed

Note: the package is in the archive at this moment. There is no way to backout the change, so the merge needs to go in one way or another. As the component is blocked in proposed, there is no advantage of unblocking the queue as blocked in proposed means that further unrelated landings will still be blocked in proposed. It will continue blocking potentially other unrelated packages (if a transition happens). The only way then is to unblock the package.

variant a: fix in the same component itself

variant b: fix outside the CI system: in another component or by a direct upload to that component

Workflow I: direct MP/trivial commit to one component (low-cost ticket)¶

Reminder: this is the case of a quick bug fix/feature (< 2 commits). The main case is manual ticket landing.

Workflow I A: direct commit to trunk or MP, all infos set

Workflow I B: direct MP, not following project rules (no commit message/enough approver)

Workflow II: opening a ticket¶

Notes: ticket manager and landing manager shares most of their code. Only the order and some interface changes.

Workflow II A: feature branch/transitions/fix involving multiple components (opening a first class ticket)

Components versus number of instances:¶

Please note that all services are NOT running at the same time. This diagram is just to show what is spawn by what and what takes care of one or multiple components.

Case delivering to trunk:

• one ticket A with mir, libunity-mir, platform-api
• one ticket B with libunity-mir only
• one ticket C with mir
• one ticket D with unity8
• one ticket E with unity-scope-home

First landing: Ticket A, D and E:

Second landing: Ticket B (starts as soon as A is treated) and D (starts as soon as D is treated) “Treated” corresponds to “ticket removed from the queue in the different “Delivery system” cases.

Workflow I

Note that all those processes are always running (and they only files ticket)

Workflow II

Feature involving mir, libunity-mir, and platform-api

Setting up Juju LXC¶

You’ll need a .juju/environments.yaml file with a “local” entry like:

default: local
environments:
  local:
    type: local
    default-series: precise
    lxc-clone: true
    lxc-clone-aufs: true
    admin-secret: secret
    apt-http-proxy: http://10.0.3.1:8000

Install squid-deb-proxy. Then create /etc/squid-deb-proxy/mirror-dstdomain.acl.d/20-juju-local with the following contents:

ppa.launchpad.net
private-ppa.launchpad.net

Restart the squid-deb-proxy service. All apt downloads will now be cached by the proxy server running on your local machine.

Next, you will need swift credentials set up. This ensures the services still have access to object storage (Swift), virtual machine image storage (Glance), and virtual machine instantiation (Nova). An example .hpcloud-rc file should look like:

# FOR SWIFT
export JUJU_ENV=local
export OS_USERNAME="<your email address>"
export OS_TENANT_NAME="<your tenent name from horizon>"
export OS_PASSWORD=<something special>
export OS_AUTH_URL="https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0"
export OS_REGION_NAME=region-a.geo-1

# FOR GLANCE (same credentials as above by default)
export GLANCE_OS_USERNAME="$OS_USERNAME"
export GLANCE_OS_AUTH_URL="$OS_AUTH_URL"
export GLANCE_OS_REGION_NAME="$OS_REGION_NAME"
export GLANCE_OS_TENANT_NAME="$OS_TENANT_NAME"
export GLANCE_OS_PASSWORD="$OS_PASSWORD"

# FOR OAUTH TOKENS
export CI_LAUNCHPAD_PPA_OWNER=<lp login-id>
export CI_LAUNCHPAD_USER=<lp login-id>
export CI_OAUTH_CONSUMER_KEY="ci-airline"
export CI_OAUTH_TOKEN=<Please see Note below+++>
export CI_OAUTH_TOKEN_SECRET=<Please see Note below+++>

If your swift server only supports the 1.0 auth protocol (TempAuth does not support 2.0), you additionally need to set some $ST_* variables for python-swiftclient, and include the tenant/project name into the user name:

export OS_USERNAME="<project:username>"
export OS_TENANT_NAME="<project>"
export OS_PASSWORD="<password"
export OS_AUTH_URL="http://your.swift.server:8080/auth/v1.0"
export OS_REGION_NAME=

# env for python-swiftclient
export ST_AUTH=$OS_AUTH_URL
export ST_USER=$OS_USERNAME
export ST_KEY=$OS_PASSWORD

# GLANCE_*, CI_* as above

If you use the Python virtualenv produced by testing/venv.py, you must additionally set export EPHEMERAL_CLOUD_NET_ID=.. in that rc file, as neutron is broken in the venv, and thus it cannot determine the net ID by itself.

You’ll now have the settings in-place. In order to iterate rapidly, It would be advisable to ensure juju’s template image has all the dependencies pre-installed so you don’t wait for that every time you re-deploy. You can do that with the following simple manual hack:

juju bootstrap
juju deploy cs:ubuntu # Wait for juju status to show it deployed
juju destroy-environment --force -y local

# You'll now have an lxc container named juju-trusty-template.
# Modify it with these commands:
sudo lxc-start -d --name juju-trusty-template
sudo lxc-attach --name juju-trusty-template -- add-apt-repository -y ppa:canonical-ci-engineering/ci-airline-phase-0
sudo lxc-attach --name juju-trusty-template -- apt-get update
sudo lxc-attach --name juju-trusty-template -- apt-get install -y rabbitmq-server python-amqplib python-pip python-jinja2 mercurial git-core    subversion bzr gettext python-django-south python-lazr.enum python-tastypie python-amqplib python-swiftclient postgresql-9.3 postgresql-contrib-9.3 python-psutil dput python-dput lazr.enum python-tz python-gnupg qemu-utils python-glanceclient python-amqplib python-requests python-novaclient python-psycopg2 pwgen postgresql-client gunicorn python-support pgtune postgresql-9.3-debversion postgresql-plpython-9.3 python-dnspython python3-pyramid

sudo lxc-stop --name juju-trusty-template

Host Configuration¶

Some additional changes will be necessary on the LXC host system for the imagebuilder to work properly. With some of the changes we have planned, many of these should not be needed soon. Be aware that making these changes may have an effect on other LXC containers you run on your host system.

First, ensure that nbd is loaded on the host. Module loading will not work in LXC, but the module will be available under lxc if it is loaded on the host.

Add the following lines to /var/lib/lxc/juju-precise-template/config:

# Allow mounting filesystems under LXC
aa_profile = lxc-container-default-with-mounting
# Allow full access to the block device with major number 43, which
# should be nbd (see /proc/devices)
lxc.cgroup.devices.allow = b 43:* rwm

Modify the LXC default apparmor rules to allow bind mounting filesystems under LXC. In /etc/apparmor.d/lxc/lxc-default, add the following line before the “}”:

mount options=(rw, bind, ro),

Then run:

sudo /etc/init.d/apparmor reload

Working with the code¶

Code modifications can be done using the following iterations:

1) bzr branch lp:uci-engine
2) cd uci-engine
3) <do changes>
4) juju destroy-environment --force -y local; juju bootstrap
5) rm -rf tmp/
6) ./juju-deployer/deploy.py # Or './juju-deployer/deploy.py branch-source-builder' to only deploy Branch Source Builder service
7) <check the services>
8) Repeat from step 3-7 for iterative development

Upgrade¶

The development effort can be further sped up using the --upgrade option in deploy.py. The generic steps of upgrading are given in the Upgrade section. The following steps are specific to the local development:

1) bzr branch lp:uci-engine
2) cd uci-engine
3) <do changes>
4) juju destroy-environment --force -y local; juju bootstrap
5) rm -rf tmp/
6) ./juju-deployer/deploy.py --build-only --working-dir ./tmp
7) ./juju-deployer/deploy.py   # use ./juju-deployer/deploy.py branch-source-builder for deploying only bsb serivce.
8) <do changes in charms for the deployed services locally>
9) ./juju-deployer/deploy.py --upgrade all # this will upgrade the modified charms and config
10) Repeat the steps 8-9 for fixing and testing the charms that were already deployed

Examples¶

Assuming the initial setup of Using Juju LXC For Local Development or Setting Up a Cloud Deployment has been performed, the generic steps where upgrade becomes useful are:

1) bzr branch lp:uci-engine
2) juju bootstrap
3) cd uci-engine
4) ./juju-deployer/deploy.py
   or
4) ./juju-deployer/deploy.py test-runner  # for deploying only test-runner service
5) <do changes in charms and code for the deployed services>
6) ./juju-deployer/deploy.py --upgrade all

1-4) <same as --upgrade all example above>
5) <do changes to charms for the deployed services>
6) ./juju-deployer/deploy.py --upgrade charms

1-4) <same as --upgrade all example above>
5) <do changes to charm config and/or code for the deployed services>
6) ./juju-deployer/deploy.py --upgrade config

1-4) <same as --upgrade all example above>
5) <do changes to code for the deployed services>
6) ./juju-deployer/deploy.py --upgrade code

Upgrading adt-run for the test runner¶

The test runner depends on adt-run to run the tests.

When needed (new upstream version, bug fix) the CI gated branch needs to be updated.

Gating these updates guarantees that they can be tested before being deployed and that hot fixes can also be deployed without requiring upstream intervention (those fixes should be rare and upstreamed in any case).

Then, there is a recipe to build autopkgtest into the phase-0 PPA.

Trigerring that recipe will build autopkgtest for the series that are used in the CI engine.

The last step is to update the deployed test runner workers themselves:

juju run --service ci-airline-tr-rabbit-worker 'apt-get update; apt-get install autopkgtest'