Introduction¶

Create prototype to trigger N jobs for a range of revisions¶

This allows triggering multiple jobs across a range of revisions.

Here’s an example:

python scripts/trigger.py \
    --buildername "Ubuntu VM 12.04 fx-team opt test jittest-1" \
    --rev e16054134e12 --from-rev fb64168bf663 --times 2

This has been accomplished on the 0.2.1 release (13/02/2015).

Add pushlog support¶

This helps interacting with ranges of revisions.

This has been accomplished on the 0.2.1 release (13/02/2015).

Determine accurately the current state of jobs¶

We can determine any information on jobs run on the Buildbot CI by retrieving scheduling information through Self-Serve’s BuildAPI. The information retrieved can then be matched to the buildjson status dumps that are produced every minute (for the last 4 hours) and every 15 minutes (for the day’s worth of data).

This feature is completed. (25/02/2015).

Create prototype to find last good job and backfill up to it¶

Given a bad job, we can simply scan the previous revisions for the last known good job for it. Known that, we can trigger all jobs required to trigger the missing jobs.

The script will find the last good job or trigger up to a maximum of revisions. We can also indicate that we want multiple jobs instead of just one.

Here’s an example:

python scripts/trigger.py \
    --buildername "Ubuntu VM 12.04 fx-team opt test jittest-1" \
    --rev e16054134e12 --backfill --max-revisions 30 --times 2

This feature was completed on release 0.3.0.

Ability to navigate through merges into other repositories¶

We need the ability to navigate through pushes and be able to follow through from one repository to another when merge pushes are found.

This is important to find an intermittent failing job.

We have filed issue 127 to track this.

Determine if a test failed in a job¶

We currently can only tell if a job has failed, however, we don’t know which test(s) have failed.

If we want to find intermittent oranges we would need to determine if a job has failed because the test we care about has actually failed.

A possible way to determine this would be to grab the structured log uploaded through blobber.

NOTE: Reftests currently don’t generate structured logs. In such case we would have to fallback to parsing logs. We would have to request this to be implemented.

NOTE 2: In the future, we might be able to query this information through the big data project.

We have filed issue 128 to track this.

Determine frequency of test failure¶

We need to find a way to analyze test failure frequency. That way we can determine the right number of jobs of to retrigger a job in order to find an intermittent orange.

I’m not sure if this will really be needed. We will see.

Make handling Buildbot job information sustainable¶

We currently have information about Buildbot jobs by grabbing scheduling info and status info. This is something that users should never know about. We should only expose the bits of information which are relevant to the user and allow them to access the raw data only if wished for.

Adding a base class to abstract jobs and have one implementation for Buildbot will make the project ready for the representation of jobs in TaskCluster.

We have filed issue 21 to track this.

Parallelize the analysis of each revision¶

We currently iterate and analyze one revision at a time. In many places we can parallelize this process since they’re isolated processes.

This will speed up the execution time.

NOTES:

• We should only print to the log once a revision has been completely processed.

• We should not print the summary about a revision until all most recent revisions are processed first.

  • i.e. log them in descending order based on their push id.
  • This will ensure that inspecting the log will visually make sense

We have filed issue 129 to track this.

Test framework to test CI data sources¶

We need to have a way to prevent regressions on Mozilla CI tools. Adding coverage reports would help us fix this issue.

We also need a way to test the data sources structures for changes that could regress us (e.g. new builder naming for Buildbot). We might be able to simply mock it but we might need to set up the various data sources.

This is to be tackled in Q2/Q3 2015.

We have filed issue 130 to track this.

Provide data structure to generate up-to-date trychooser¶

Currently trychooser’s UI is always out-of-date and nothing intelligent can be done with it. With mozci we currently can generate most of the data necessary to create a dynamic UI.

To generate the current data structure you can run this::

python scripts/misc/write_tests_per_platform_graph.py

graphs.json will be generated. We have filed issue 69 to track this.

NOTE: This will be needed once someone picks up bug 983802.

Integrate backfilling feature into treeherder¶

This will be similar to the re-trigger button that is part of the treeherder UI. We select a job that is failing and request that we backfill. mozci will determine when was the last time there was a successful job and trigger all missing jobs up to the last known good job.

TreeHerder currently uses client-side triggering for the re-trigger button and it intends to move it to the server side (bug 1077053).

We have filed issue 109 to track this.

Pulse support¶

Pulse allows you to listen and consume about jobs changing status. This is very important for monitoring jobs going through various states.

We have filed issue 126 to track this.

Add ability to monitor jobs¶

We currently run a script and let it schedule everything that is needed. However, we assume an ideal case scenario: everything that we schedule gets run.

This is a very optimistic approach. We should allow the user to use a mode in which the script watches and notifies the user when our expectations are not met.

For instance:

• A build finishes, however, the tests that is expected to run gets coalesced
  • In this case we would be expecting a completed build job + a completed test job
  • We would need to schedule the test job and watch it
• A job fails, however, we assume it would succeed
  • We need to re-trigger it and watch it

We have filed issue 131 to track this.

Support TaskCluster¶

As we’re transitioning to TaskCluster we should add the support for it.

We are tracking this with the TaskCluster Support milestone.

Case scenario 1: Bisecting permanent issue¶

• We have a job failing
• There are jobs that were coalesced between the last good and the first bad job
• We need to backfill between good revision and the bad revision

This has been completed by the trigger.py with –backfill.

Case scenario 2: Bisecting intermittent issue¶

• We have an intermittent job
• We want to determine when it started happening
• It is not only a matter of coalescing but also a matter of frequency
• We want to give a range of changesets and bisect until spotting culprit

NOTE: We trigger more than one job compared to case scenario 1

The script trigger.py helps with triggering multiple times the same jobs. The script generate_cli.py helps with tracking filed intermittent oranges in bugzilla.

This use case requires implementing the following milestones:

• Add the ability to find intermittent oranges
• Allow monitoring jobs

Case scenario 3: Retrigger an intermittent job on a changeset until hit¶

https://bugzilla.mozilla.org/show_bug.cgi?id=844746

• This is more of an optimization.
• The intent is to hit the orange with extra debugging information.
• We’re not bisecting in here.
• We can trigger batches (e.g. 5 at a time)

Not in scope at the moment.

This could be done with a modification of trigger.py where we monitor the jobs running until one of them fails.

This use case requires implementing the following milestones:

• Allow monitoring jobs

and this issue:

• Add the ability to inspect the reason of a job failure

Case scenario 4: Bisecting Talos¶

• We have a performance regression
• We want to determine when it started showing up
• Given a revision that _failed_
• Re-trigger that revision N times and all revisions prior to it until the last data point + 1 more

We can do this manually by using –back-revisions::

python scripts/trigger.py \
    --buildername "Rev5 MacOSX Yosemite 10.10 fx-team talos dromaeojs" \
    --rev e16054134e12 --back-revisions 10

We currently don’t have the ability to determine the previous baseline and trigger everything up to the last known data point close to the baseline.

Case scenario 5: After uplift we need a new baseline for release branches¶

• We need several data points to establish a baseline
• After an uplift we need to generate a new baseline
• Once there is a baseline we can determine regression

This can already be accomplished like this::

python alltalos.py --repo-name try --times 2 --rev 921277f744d9

Case scenario 6: New test validation¶

• New test validation
• Re-triggering to determine indeterminacy
• Single revision
• All platforms running test

NOTE: Review this thread on how to determine on which job and which platforms do we run a specific test.

Not in scope at the moment.

Case scenario 7: Fill in a changeset¶

• We know that a changeset is missing jobs
• We want to add all missing jobs

This would need grabbing the list of builders for such repo, determine if there has been a successful job for each builder and trigger it if not. We might need to filter out some periodic builders (i.e. a repo can have pgo periodic jobs).

If we want to watch that revision until it is completely filled we will need to accomplish the following milestones:

• Allow monitoring jobs

Case scenario 8: Developer needs to add missing platforms/jobs for a Try push¶

• The developer pushes to try specifying only a subset of all jobs
• The developer realizes that it needs more jobs to run on that push
• The developer uses mozci to not have to push again to try with the right syntax

This has been filed as issue 109

Case scenario 9: We generate data to build a dynamic TryChooser UI¶

• TryChooser UI is always out of date
• mozci can generate the data we need to create an up-to-date TryChooser UI

See write_tests_per_platform_graph.py for an example on how to generate the data needed.

I asked mozci to trigger a test job for me, however, a build got triggered¶

Test jobs require a build to be tested, hence, needing to trigger a build for it.

In some cases you might see that a build exists for that push and still trigger a build. This is because we have checked for its uploaded files and have been expired. In this case we need to trigger the build job again.

How does mozci deal with running/pending jobs?¶

mozci expects that running/pending _build_ jobs will trigger the tests we want.

If we want more than one job and we observe one running/pending job, we will trigger as many jobs as needed to meet the request.

How does mozci deal with failed jobs?¶

mozci is not currently designed to trigger jobs until they succeed. The use must say what it wants to trigger and we will only do a one pass to trigger everything requested. This is purposeful for simplicity and to prevent triggering jobs endlessly.

In the near future, we will add a monitoring module which could be used to keep track of triggered jobs and determine actions upon completion.

Can we schedule a PGO job on any tree?¶

Yes and no. We can’t trigger anything that the Release Engineering’s Buildbot CI can trigger. If a tree does not have a builder that generates a PGO build then we can’t.

Notice that some trees can trigger both non-PGO jobs and PGO jobs, hence, that tree has two different buildernames (one including “pgo” in its name). Other trees can only trigger PGO jobs and might not include “pgo” in its name (think of mozilla-aurora).

What products do you support?¶

All Firefox desktop and Firefox for Android. For Firefox OS we have partial support since some of the jobs run on the TaskCluster CI (which is not yet supported).

Can I trigger a nightly build?¶

Absolutely! You can trigger any nightly build for any repository. Simply find the name of the job that represents it and trigger it.

NOTE: Make sure you have consent from sheriffs to do this and a good reason. Nightly builds are not to be triggered lightly.

If I ask for different test jobs on the same changesets will I get as many builds jobs?¶

There is a possible 30 seconds delay between making a request to buildapi and it appearing as “pending/running”. If you hit this issue please let us know and we can discuss it on how to better address it. There are various options we can consider.

Does this work with TaskCluster?¶

Not yet.

Can anybody use mozci?¶

As long as you have LDAP credentials you should be able to use it.

What systems does mozci rely on?¶

If you look at mozci.sources you will see all CI components we depend on. If the structure of any of these changes, we might need to adjust mozci for it.

What happens if a new platform or suites are added to the CI?¶

Nothing to worry about! mozci determines platforms dynamically rather than statically.

What use cases are you hoping to address?¶

Please refer to Use cases.

I see that you store my credentials in plain text on my machine¶

If you want a different approach please let us know.

Can I run mozci in my web service?¶

Yes! However, we will need to figure out how to provide credentials. More to come.

Is mozci limited by the try chooser syntax?¶

No. We hit an API that is not affected by the try parser. We can trigger anything that can be triggered without any limitations. You can add more jobs on a try push than indicated in the try syntax of that push.

Can you trigger jobs on pushes with DONTBUILD?¶

No, we can not. This is a bug in buildapi. The pushes doesn’t even exist for buildapi. You can notice this if you load self-serve/buildapi for a DONTBUILD push.

How do you deal with coalesced and not scheduled jobs?¶

Coalesced jobs are requests that have been fulfilled by more recent pushes. We coalesce jobs to be able to catch up under high load.

We sometimes not schedule jobs for various reasons including:

• The user has marked the job not to be built with DONTBUILD in the commit message
• The files changed on that push do not affect certain products/platforms
• We have determined that we don’t need to trigger that job on every push

self-serve/buildapi does not keep track of jobs that have been coalesced or not scheduled.

mozci determines how many jobs to trigger a job depending on how many successful, running jobs and potential jobs trigger by a build. Coalesced and not scheduled jobs are not considered.

What are the concerns of trigger a large number of jobs in a short period of time?¶

Self-serve/buildapi is known to be unresponsive if too much is demanded of it. The operations of treeherder will continue as usual since the Buildbot master query the buildbot databases directly rather than through self-serve/buildapi. Re-triggering of jobs would be temporarily unavailable until self-serve auto-recovers. At worse, nagios checks will be triggered and buildduty will have to investigate.

Treeherder could also be affected if buildapi/self-serve did not go down and actually managed to trigger a lot of jobs. It is known that treeherder gets into trouble if several thousands of jobs get triggered in a short period of time.

Proper usage of mozci should not cause any issues, however, intentional misuse could cause the issues mentioned above.

What performance constraints does mozci have?¶

We are currently mainly restrained by two factors: sequential approach to triggering and responsiveness of the data sources.

We currently go through each push in a sequential order. In order to speed this up we could parallelize the work done on each push.

The data sources we use can be slow at times depending on the load on them. If this becomes troublesome we should investigate how to optimize them.

How do you release software?¶

We use zest.releaser. You simply install it:

pip install zest.releaser

`fullrelease` is used to bump the versions, tag and upload the new package to pypi.

The releases are documented in here.

How do I generate the docs?¶

To generate the docs, follow these steps:

• Move inside docs/ directory

• Run:

  pip install -r requirements.txt
  make html
  

• To view the docs on a webserver http://127.0.0.1:8000 and auto-rebuild the documentation when any files are changed:

  make livehtml
  

How can I contribute?¶

If you would like to contribute to this project, feel free to pick up one of the issues or tasks in the Trello board (Tasks) or the issues page (Issues).

In order to contribute the code:

• Fork the project
• Create a new branch
• Fix the issue - add the feature
• Run tox successfully
• Commit your code
• Request a pull request