Apollo¶

Package manager commands¶

To install system pre-requisites, you can try the following commands

Setup a production server¶

To setup in a production environment, please see the setup guide. To setup (as opposed to a development server as above), you must properly configure a servlet container like Tomcat or Jetty with sufficient memory.

Adding data to Apollo¶

After we have a server setup, we will want to add a new organism to the panel. If you are a new user, you will want to setup this data with the jbrowse pre-processing scripts. You can see the data loading guide for more details, but essentially, you will want to load a reference genome and an annotations file at a minimum:

bin/prepare-refseqs.pl --fasta yourgenome.fasta --out /opt/apollo/data

bin/flatfile-to-json.pl --gff yourannotations.gff --type mRNA \
        --trackLabel AnnotationsGff --out /opt/apollo/data

Login to the web interface¶

After you access your application at http://localhost:8080/apollo/ then you will be prompted for login information

Figure 1. “Register First Admin User” screen allows you to create a new admin user.

Figure 2. Navigate to the “Organism tab” and select “Create new organism”. Then enter the new information for your organism. Importantly, the data directory refers to a directory that has been prepared with the JBrowse data loading scripts from the command line. See the data loading section for details.

Figure 3. Open up the new organism from the drop down tab on the annotator panel.

Database configuration¶

Apollo supports several database backends, and you can choose sample configurations from using H2, Postgres, or MySQL by default.

Each has a file called sample-h2-apollo-config.groovy or sample-postgres-apollo-config.groovy that is designed to be renamed to apollo-config.groovy before running apollo deploy. Additionally there is a sample-docker-apollo-config.groovy which allows control of the configuration via environment variables.

Furthermore, the apollo-config.groovy has different groovy environments for test, development, and production modes. The environment will be selected automatically selected depending on how it is run, e.g:

• apollo deploy or apollo release use the production environment (i.e. when you copy the war file to your production server apollo run-local or apollo debug use the development environment (i.e. when you are running it locally)
• apollo test uses the test environment (i.e. only when running unit tests)

Database schema¶

After you startup the application, the database schema (tables, etc.) is automatically setup. You don’t have to initialize any database schemas yourself.

Note on database settings¶

If you use the apollo run-local command, then the “development” section of the apollo-config.groovy is used (or an temporary in-memory H2 database is used if no apollo-config.groovy exists).

If you use the WAR file generated by the apollo deploy command on your own webserver, then the “production” section of the apollo-config.groovy is used.

Pre-requisites for Javascript minimization¶

In addition to the system pre-requisites, the javascript compilation will use nodejs, which can be installed from a package manager on many platforms. Recommended setup for different platforms:

sudo apt-get install nodejs
sudo yum install epel-release npm
brew install node

bin/cpanm -l extlib DateTime Text::Markdown

Performing the javascript minimization¶

To build a Apollo release with Javascript minimization, you can use the command

./apollo release

This will compile JBrowse and Apollo javascript code into minimized files so that the number of HTTP requests that the client needs to make are reduced.

In all other respects, apollo release is exactly the same as apollo deploy though.

Performing active development¶

To perform active development of the codebase, use

./apollo debug

This will launch a temporary instance of Apollo by running grails run-app and ant devmode at the same time, which means that any changes to the Java files will be picked up, allowing fast iteration.

If you modify the javascript files (i.e. the client directory), you can run scripts/copy_client.sh and these will be picked up on-the-fly too.

Default data adapter options¶

The options available for the data adapters are configured as follows

• type: GFF3 or FASTA
• output: can be file or text. file exports to a file and provides a UUID link for downloads, text just outputs to stream.
• format: can by gzip or plain. gzip offers gzip compression of the exports, which is the default.
• exportSequence: true or false, which is used to include FASTA sequence at the bottom of a GFF3 export

Apache Proxy¶

The most simple setup on apache is as follows.. Here is the most basic configuration for a reverse proxy:

ProxyPass  /apollo http://localhost:8080/apollo
ProxyPassReverse  /apollo http://localhost:8080/apollo

Note: that a reverse proxy does not use ProxyRequests On (which turns on forward proxying, which is dangerous)

Also note: This setup will use downgrade to use AJAX long-polling without the websocket proxy being configured.

To setup the proxy for websockets, you can use mod_proxy_wstunnel, first load the module

LoadModule proxy_wstunnel_module libexec/apache2/mod_proxy_wstunnel.so

Then add extra ProxyPass calls for the websocket “endpoint” called /apollo/stomp

ProxyPass /apollo/stomp  ws://localhost:8080/apollo/stomp
ProxyPassReverse /apollo/stomp ws://localhost:8080/apollo/stomp

ProxyPass  /testing http://localhost:8080
ProxyPassReverse  /testing http://localhost:8080
ProxyPassReverseCookiePath / /testing

Nginx Proxy (from version 1.4 on)¶

Your setup may vary, but setting the upgrade headers can be used for the websocket configuration http://nginx.org/en/docs/http/websocket.html

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }
    
    
    server {
        # Main
        listen   80; server_name  myserver;
        
        # http://nginx.org/en/docs/http/websocket.html
        location /ApolloSever {
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_pass      http://127.0.0.1:8080;
        }
    }

Upgrading existing JBrowse data stores¶

It is not necessary to change your existing JBrowse data directories to use Apollo 2.x, you can just point to existing data directories from your previous instances.

More information about JBrowse can also be found in their FAQ.

Adding custom CSS for track styling for JBrowse¶

There are a variety of different ways to include new CSS into the browser, but the easiest might be the following

Add the following statement to your trackList.json:

    "css" : "data/yourfile.css"

Then just place your CSS file in your organism’s data directory.

Adding custom CSS globally for the GWT app¶

If you want to style the GWT sidebar, generally the bootstrap theme is used but extra CSS is also included from web-app/annotator/theme.css which overrides the bootstrap theme

Adding / using proxies¶

If you are https, or choose to use separate services rather than the default provided, you can setup a pass-through proxy or modify a particular URL.

This service is only available to logged-in users.

The internal proxy URL is:

<apollo url>/proxy/request/<encoded_proxy_url>/

For example if your URL the URL we want to proxy:

http://golr.geneontology.org/solr/select

encoded:

http%3A%2F%2Fgolr.geneontology.org%2Fsolr%2Fselect

If you user is logged-in and you pass in:

http://localhost/apollo/proxy/request/http%3A%2F%2Fgolr.geneontology.org%2Fsolr%2Fselect?testkey=asdf&anotherkey=zxcv

This will get proxied to:

http://golr.geneontology.org/solr/select?testkey=asdf&anotherkey=zxcv

If you choose to use another proxy service, you can go to the “Proxy” page (as administrator). Internally used proxies are provided by default. The order the final URL is chosen in is ‘active’ and then ‘fallbackOrder’.

GFF3¶

You can use the tools/data/add_transcripts_from_gff3_to_annotations.pl script to bulk load GFF3 files with transcripts to the user annotation track. Let’s say we want to load our maker.gff transcripts.

    tools/data/add_transcripts_from_gff3_to_annotations.pl \
        -U localhost:8080/Apollo -u web_apollo_admin -p web_apollo_admin \
        -i scf1117875582023.gff -t mRNA -o "name of organism"

The default options should be able to handle most GFF3 files that contain genes, transcripts, and exons.

You can still use this script even if the GFF3 file that you are loading does not contain transcripts and exon types. Let’s say we want to load match and match_part features as transcripts and exons respectively. We’ll use the blastn.gff file as an example.

    tools/data/add_transcripts_from_gff3_to_annotations.pl \
       -U localhost:8080/Apollo -u web_apollo_admin -p web_apollo_admin \
       -i cf1117875582023.gff -t match -e match_part -o "name of organism"

You can view the add_transcripts_from_gff3_to_annotations.pl help (-h) option for all available options.

Note: Apollo makes a clear distinction between a transcript and an mRNA. Genes that have mRNA as its child feature are treated as protein coding annotations and Genes that have transcript as its child feature are treated as non-coding annotations, specifically a pseudogene.

If you would like to look at a compatible representative GFF3, export annotations from Apollo via GFF3 export.

Suggested Tomcat memory settings¶

export CATALINA_OPTS="-Xms512m -Xmx1g \
        -XX:+CMSClassUnloadingEnabled \
        -XX:+CMSPermGenSweepingEnabled \
        -XX:+UseConcMarkSweepGC \
        -XX:MaxPermSize=256m"

In cases where the assembled genome is highly fragmented, additional tuning of memory requirements and garbage collection will be necessary to maintain the system stable. Below is an example from a research group that maintains over 40 Apollo instances with assemblies that range from 1,000 to 150,000 scaffolds (reference sequences):

export CATALINA_OPTS="-Xmx12288m -Xms8192m \
        -XX:PermSize=256m \
        -XX:MaxPermSize=1024m \
        -XX:ReservedCodeCacheSize=64m \
        -XX:+UseG1GC \
        -XX:+CMSClassUnloadingEnabled \
        -Xloggc:$CATALINA_HOME/logs/gc.log \
        -XX:+PrintHeapAtGC \
        -XX:+PrintGCDetails \
        -XX:+PrintGCTimeStamps"

To change your settings, you can usually edit the setenv.sh script in $TOMCAT_BIN_DIR/setenv.sh where $TOMCAT_BIN_DIR is the directory where the Tomcat binaries reside. It is possible that this file doesn’t exist by default, but it will be picked up when Tomcat restarts. Make sure that tomcat can read the file.

In most cases, creating the setenv.sh should be sufficient but you may have to edit a catalina.sh or another file directly depending on your system and tomcat setup. For example, on Ubuntu, the file /etc/default/tomcat7 often contains these settings.

Confirm your settings¶

Your CATALINA_OPTS settings from setenv.sh can be confirmed with a tool like jvisualvm or via the command line with the ps tool. e.g. ps -ef | grep java should yield something like the following allowing you to confirm that your memory settings have been picked up.

root      9848     1  0 Oct22 ?        00:36:44 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/current/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms1g -Xmx2g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC -XX:MaxPermSize=512m -Dj

Re-install after changing settings¶

If you start seeing memory leaks (java.lang.OutOfMemoryError: Java heap space) after doing an update, you might try re-installing, as the live re-deploy itself can cause memory leaks or an inconsistent software state.

If you have named your web application named Apollo.war then you can remove all of these files from your webapps directory and re-deploy.

• Run apollo deploy (or apollo release for javascript-minimization)
• Undeploy any existing Apollo instances
• Stop tomcat
• Copy the war file to the webapps folder
• Start tomcat

JBrowse tools don’t show up in bin directory (or install at all) after install or typing install_jbrowse.sh¶

If the bin directory with JBrowse tools doesn’t show up after calling install_jbrowse.sh JBrowse is having trouble installing itself for a few possible reasons. If these do not work, please observe the JBrowse troubleshooting and JBrowse install pages, as well and the setup.log file created during the installation process.

Buildfile: build.xml

copy.apollo.plugin.webapp:

setup-jbrowse:

git.clone:
[exec] Result: 129

setup-jbrowse:

git.clone:
     [exec] Submodule 'src/FileSaver' (git://github.com/dkasenberg/FileSaver.js.git) registered for path 'src/FileSaver'
     [exec] Submodule 'src/dbind' (git://github.com/rbuels/dbind.git) registered for path 'src/dbind'
    . . . .
     [exec] Submodule 'src/xstyle' (git://github.com/kriszyp/xstyle.git) registered for path 'src/xstyle'
     [exec] Result: 1

e.g. “Can’t locate Hash/Merge.pm in @INC” or “Can’t locate JBlibs.pm in @INC”¶

If you are trying to run the jbrowse binaries but get these sorts of errors, try running install_jbrowse.sh which will initialize as many pre-requisites as possible including JBLibs and other JBrowse dependencies.

Rebuilding JBrowse¶

You can manually clear jbrowse files from web-app/jbrowse and re-run apollo deploy to rebuild JBrowse.

RequestError: Unable to load ... Apollo2/jbrowse/data/trackList.json status: 500¶

Apollo2 does fairly strict JSON validation so make sure your trackList.json file is valid JSON

If you still get this error after validating please forward the issue to our github issue tracker.

Disabling the cache:¶

    grails.cache.config = {
        cache {
            enabled = false
            name 'globalcache'
        }
    }

This can also be done by removing the plugin. In grails-app/conf/BuildConfig remove / comment out the line and re-building:

 compile ':cache-ehcache:1.0.5'

Disallow writing overflow to disk¶

Can be used for small instances

    grails.cache.config = {
        // avoid ehcache naming conflict to run multiple WA instances
        provider {
            name "ehcache-apollo-"+(new Date().format("yyyyMMddHHmmss"))
        }
        cache {
            enabled = true
            name 'globalcache'
            eternal false
            overflowToDisk false   // THIS IS THE IMPORTANT LINE
            maxElementsInMemory 100000
        }
    }

Specify the overflow directory¶

Best for high load servers, which will need the cache. Make sure your tomcat / web-server user can write to that directory:

    // copy from Config.groovy except where noted
    grails.cache.config = {
    ... 
        cache {
        ...  
            maxElementsOnDisk 10000000
            // this is the important part, below!
            diskStore{
                path '/opt/apollo/cache-directory'
            }
        }
        ...
    }

Information on the grails ehcache plugin (see “Overriding values”) and ehcache itself.

Migrate Annotations¶

We provide a [migration script](https://github.com/gmod/apollo/tree/master/docs/web_services/examples/groovy/ migrate_annotations1to2.groovy) that connects to a single Web Apollo 1 instance and populates the annotations for an organism for a set of sequences / (confusingly called tracks as well). It would be best to develop your script on a development instance of Apollo2 for restricted sequences.

To get the scripts working properly, you’ll need to provide the list of sequences (or tracks) to migrate for each organism. You can get the list of tracks by either using the database (select * from tracks ;) or looking in the Web Apollo annotations directory

ls -1 /opt/apollo/annotations/ | grep Annotations | grep -v history | paste -s -d"," -

Migrate Users¶

You have to add users de novo using something like the add_users.groovy script. In this case you create a csv file with the email, name, password, and role (‘user’ or ‘admin’). This is passed into the add_users.groovy script and users are added.

From Web Apollo 1, you should be able to pull user names out of the database select * from users ;, but there is not much overlap between users in Web Apollo1.x and Apollo2.x.

If you have only a few users, however, just adding them manually on the users will likely be easier.

Add Organisms¶

If possible adding organisms on the organisms tab is the easiest option if you only have a handful of organisms.

The [add_organism.groovy script](https://github.com/gmod/apollo/tree/master/docs/web_services/examples/groovy/ add_organism.groovy) can help automate this process if you have a large number of migrations to handle.

Principle 1: Work from a personal fork¶

• Prior to adopting the workflow, a developer will perform a one-time setup to create a personal Fork of apollo and will subsequently perform their development and testing on a task-specific branch within their forked repo. This forked repo will be associated with that developer’s GitHub account, and is distinct from the shared repo managed by GMOD.

Principle 2: Commit to personal branches of that fork¶

• Changes will never be committed directly to the master branch on the shared repo. Rather, they will be composed as branches within the developer’s forked repo, where the developer can iterate and refine their code prior to submitting it for review.

Principle 3: Propose changes via pull request of personal branches¶

• Each set of changes will be developed as a task-specific branch in the developer’s forked repo, and then create a pull request will be created to develop and propose changes to the shared repo. This mechanism provides a way for developers to discuss, revise and ultimately merge changes from the forked repo into the shared Apollo repo.

Principle 4: Delete or ignore stale branches, but don’t recycle merged ones¶

• Once a pull request has been merged, the task-specific branch is no longer needed and may be deleted or ignored. It is bad practice to reuse an existing branch once it has been merged. Instead, a subsequent branch and pull-request cycle should begin when a developer switches to a different coding task.
• You may create a pull request in order to get feedback, but if you wish to continue working on the branch, so state with “DO NOT MERGE YET”.

Step 1 - Backup your existing repo (optional)¶

The Apollo team has recently adopted the workflow described in this document. Many developers will have an existing clone of the shared repo that they have been using for development. This cloned local directory must be moved aside so that a proper clone of the forked repo can be used instead.

If you do not have an existing local copy of the shared repo, then skip to Step 2 below.

Step 2 - Fork apollo via the Web¶

The easiest way to fork the apollo repository is via the GitHub web interface:

• Ensure you are logged into GitHub as your GitHub user.
• Navigate to the apollo shared repo at https://github.com/GMOD/apollo.
• Notice the ‘Fork’ button in the upper right corner. It has a number to the right of the button.
• Click the Fork button. The resulting behavior will depend upon whether your GitHub user is a member of a GitHub organization. If not a member of an organization, then the fork operation will be performed and the forked repo will be created in the user’s account.
• If your user is a member of an organization (e.g., GMOD or acme-incorporated), then GitHub will present a dialog for the user to choose where to place the forked repo. The user should click on the icon corresponding to their username.
• If you accidentally click the number, you will be on the Network Graphs page and should go back.

Step 3 - Clone the Fork Locally¶

At this point, you will have a fork of the shared repo (e.g., apollo) stored within GitHub, but it is not yet available on your local development machine. This is done as follows:

# Assumes that directory ~/MI/ will contain your Apollo repos.
# Assumes that your username is MarieCurie.
# Adapt these instructions to suit your environment
> cd ~/MI
> git clone git@github.com:MarieCurie/apollo.git
> cd apollo

Notice that we are using the SSH transport to clone this repo, rather than the HTTPS transport. The telltale indicator of this is the git@github.com:MarieCurie... rather than the alternative https://github.com/MarieCurie....

Note: If you encounter difficulties with the above git clone, you may need to associate your local public SSH key with your GitHub account. See Which remote URL should I use? for information.

Step 4 - Configure the local forked repo¶

The git clone above copied the forked repo locally, and configured the symbolic name ‘origin’ to point back to the remote GitHub fork. We will need to create an additional remote name to point back to the shared version of the repo (the one that we forked in Step 2). The following should work:

# Assumes that you are already in the local apollo directory
> git remote add upstream https://github.com/GMOD/apollo.git

Verify that remotes are configured correctly by using the command git remote -v. The output should resemble:

upstream    https://github.com/GMOD/apollo.git (fetch)
upstream    https://github.com/GMOD/apollo.git (push)
origin  git@github.com:MarieCurie/apollo.git (fetch)
origin  git@github.com:MarieCurie/apollo.git (push)

Step 5 - Configure .bashrc to show current branch (optional)¶

One of the important things when using Git is to know what branch your working directory is tracking. This can be easily done with the git status command, but checking your branch periodically can get tedious. It is easy to configure your bash environment so that your current git branch is always displayed in your bash prompt.

If you want to try this out, add the following to your ~/.bashrc file:

function parse_git_branch()
{
  git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ \1/'
}
LIGHT_GRAYBG="\[\033[0;47m\]"
LIGHT_PURPLE="\[\033[0;35m\]"
NO_COLOR="\[\033[0m\]"
export PS1="$LIGHT_PURPLE\w$LIGHT_GRAYBG\$(parse_git_branch)$NO_COLOR \$ "

You will need to open up a new Terminal window (or re-login to your existing terminal) to see the effect of the above .bashrc changes.

If you cd to a git working directory, the branch will be displayed in the prompt. For example:

~ $
~ $ # This isn't a git directory, so no branch is shown
~ $
~ $ cd /tmp
/tmp $
/tmp $ # This isn't a git directory, so no branch is shown
/tmp $
/tmp $ cd ~/MI/apollo/
~/MI/apollo fix-feedback-button $
~/MI/apollo fix-feedback-button $ # The current branch is shown
~/MI/apollo fix-feedback-button $
~/MI/apollo fix-feedback-button $ git status
On branch fix-feedback-button
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
    ... remaining output of git status elided ...

Refresh and clean up local environment¶

Git will not automatically sync your Forked repo with the original shared repo, and will not automatically update your local copy of the Forked repo. These tasks are part of the developer’s normal cycle, and should be the first thing done prior to beginning a new development effort and creating a new branch. In addition, this

> git fetch upstream        # Updates the local copy of shared repo BUT does not affect the working directory, it simply makes the upstream code available locally for subsequent Git operations. See step 2.

> git checkout master
> git reset --hard upstream/master

Create a new branch¶

Once you have updated the local copy of the master branch of your forked repo, you can create a named branch from this copy and begin to work on your code and pull-request. This is done with:

> git checkout -b fix-feedback-button   # This is an example name

This will create a local branch called ‘fix-feedback-button’ and will configure your working directory to track that branch instead of ‘master’.

You may now freely make modifications and improvements and these changes will be accumulated into the new branch when you commit.

If you followed the instructions in Step 5 - Configure .bashrc to show current branch (optional), your shell prompt should look something like this:

~/MI/apollo fix-feedback-button $

Changes, Commits and Pushes¶

Once you are in your working directory on a named branch, you make changes as normal. When you make a commit, you will be committing to the named branch by default, and not to master.

You may wish to periodically git push your code to GitHub. Note the use of an explicit branch name that matches the branch you are on (this may not be necessary; a git expert may know better):

> git push origin fix-feedback-button   # This is an example name

Note that we are pushing to ‘origin’, which is our forked repo. We are definitely NOT pushing to the shared ‘upstream’ remote, for which we may not have permission to push.

Reconcile branch with upstream changes¶

If you have followed the instructions above at Refresh and clean up local environment, then your working directory and task-specific branch will be based on a starting point from the latest-and-greatest version of the shared repo’s master branch. Depending upon how long it takes you to develop your changes, and upon how much other developer activity there is, it is possible that changes to the upstream master will conflict with changes in your branch.

So it is a good practice to periodically pull down these upstream changes and reconcile your task branch with the upstream master branch. At the least, this should be performed prior to submitting a PR.

> git fetch upstream

> # Make that your changes are committed to your branch
> # before doing any rebase operations
> git status
    # ... Review the git status output to ensure your changes are committed
    # ... Also a good chance to double-check that you are on your
    # ... task branch and not accidentally on master
> git rebase upstream/master

Submitting a PR (pull request)¶

Once you have developed code and are confident it is ready for review and final integration into the upstream version, you will want to do a final git push origin ... (see Changes, Commits and Pushes above). Then you will use the GitHub website to perform the operation of creating a Pull Request based upon the newly pushed branch.

See submitting a pull request.

Reviewing a pull request¶

The set of open PRs for the apollo can be viewed by first visiting the shared apollo GitHub page at https://github.com/GMOD/apollo.

Click on the ‘Pull Requests’ link on the right-side of the page:

Note that the Pull Request you created from your forked repo shows up in the shared repo’s Pull Request list. One way to avoid confusion is to think of the shared repo’s PR list as a queue of changes to be applied, pending their review and approval.

Respond to TravisCI tests¶

The GitHub Pull Request mechanism is designed to allow review and refinement of code prior to its final merge to the shared repo. After creating your Pull Request, the TravisCI tests for apollo will be executed automatically, ensuring that the code that ‘worked fine’ on your development machine also works in the production-like environment provided by TravisCI. The current status of the tests can be found near the bottom of the individual PR page, to the right of the Merge Request symbol:

TBD - Something should be written about developers running tests PRIOR to TravisCI and the the PR. This may already be in the README.html, but should be cited.

Respond to peer review¶

The GitHub Pull Request mechanism is designed to allow review and refinement of code prior to its final merge to the shared repo. After creating your Pull Request, the TravisCI tests for apollo will be executed automatically, ensuring that the code that ‘worked fine’ on your development machine also works in the production-like environment provided by TravisCI. The current status of the tests can be found

Repushing to a PR branch¶

It’s likely that after created a Pull Request, you will receive useful peer review or your TravisCI tests will have failed. In either case, you will make the required changes on your development machine, retest your changes, and you can then push your new changes back to your task branch and the PR will be automatically updated. This allows a PR to evolve in response to feedback from peers. Once everyone is satisfied, the PR may be merged. (see below).

Merge a pull request¶

One of the goals behind the workflow described here is to enable a large group of developers to meaningfully contribute to the Apollo codebase. The Pull Request mechanism encourages review and refinement of the proposed code changes. As a matter of informal policy, Apollo expects that a PR will not be merged by its author and that a PR will not be merged without at least one reviewer approving it (via a comment such as +1 in the PR’s Comment section).

Celebrate and get back to work¶

You have successfully gotten your code improvements into the shared repository. Congratulations! The branch you created for this PR is no longer useful, and may be deleted from your forked repo or may be kept. But in no case should the branch be further developed or reused once it has been successfully merge. Subsequent development should be on a new branch. Prepare for your next work by returning to Refresh and clean up local environment.

Main components¶

The main components of the Apollo 2.x application (the four most important are 1 through 4):

1. The domain classes; these are the main objects
2. Controllers, which route those domains and provide URL routes; provides rest services
3. Views: annotator and index and the only ones that matter for Apollo
4. Services: very important because all of the controllers should typically have routes, then particular business logic should go into the service.
5. Configuration files: The grails-app/conf folder contains central conf files, but the apollo-config.groovy file in your root directory can override these central configs (i.e. it is not necessary to edit DataSource.groovy)
6. Grails-app/assets: all your javascript live here. efficient way to deliver this stuff
7. Resources: web-app directory: css, images, and the jbrowse directory + WA plugin are initialized here.
8. Client directory: The WA plugin is copied or compiled along with jbrowse to the web-app directory

Feature class¶

All features inherit an ontologyId and specify a cvTerm, although CvTerms are being phased out.

Subclasses of “Feature” will specify the ontologyId, but “Feature” itself is too generic, for example, so it does not have an ontologyId.

Sequence class¶

Sequences are the method for WA to grabs sequences used to have a cache built-in mechanism doesn’t want to have that anymore to avoid running into memory problems.

Feature locations¶

Features such as genes all have a feature location belongs to a particular sequence. If you have a feature with subclasses, it can exist within many locations, and each location belongs to its own sequence.

Feature relationship¶

Feature relationships can define parent/child relationships as well as SO terms i.e. SO “part_of” relationships

Feature enums¶

The FeatureString enum: allows for mapping names for concepts, and it is useful to use these enums without worrying about string mappings inside the application.

Websockets and listeners¶

All clients subscribe to AnnotationNotifications for new transcripts and events.

If an add_transcript operation occurs, this is broadcasted via the websocket. The server side broadcasts this event, and then it does a JSON roundtrip to render the results and sends the return object that belongs to an AnnotationEvent.

Procedure transcript is created –> goes to the server –> adds a transcript locally –> announces it to everyone.

We used to use long polling request model for “push notifications” but now we use Spring with the SockJS, which uses websockets but it can fall back to long-polling.

There is another component of the broadcasting called brokerMessagingTemplate is the converter to broadcast the event

Controllers¶

Grails controllers are a fairly easy concept for “routing” URLs and info to methods in the code.

Services¶

Grails services are classes that perform business logic. (In IntelliJ, these are indicated by green buttons on the definitions to show that these are Injected Spring Bean classes)

The word @Transactional means that every operation that is not private is handled via a transaction. In the old model there were a lot of files that were recreated each time, even though they did the same. Now we define a class and can use it again and again. And there can be transactions within transaction. I could also call other services within services.

addTranscript generateTranscript

The different services do exactly what their name implies. It may not always be clear in what particular service each class should be in, but it can be changed later. It is easy also to make changes to the names as well.

Database configuration¶

The “root” database configuration is specified by grails-app/conf/DataSource.groovy but it is generally over-ridden by the user’s apollo-config.groovy

It is recommended that the user takes sample-postgres-apollo-config.groovy or sample-mysql-apollo-config.groovy and copies it to apollo-config.groovy for their application.

The default database driver is the h2 database, which is an “embedded” database that doesn’t require installing postgres or mysql, but it is not generally seen as performant as postgres or mysql though.

Note: there are three environments that can be setup: a development environment, a test environment, and a production environment, and these are basically assigned automatically depending on how you deploy the app.

• Development environment - “apollo run-local” or “apollo debug”
• Test environment - “apollo test”
• Production environment - “apollo deploy” or “apollo release”

Note: If there are no users and no annotations, a bootstrap procedure can also automatically create some annotations and users to start up the app so there is something in there to begin with.

UrlMapping configuration:¶

The UrlMappings are stored in grails-app/conf/UrlMappings.groovy

The UrlMappings sets up a mapping from routes to controllers

Standard and customized mappings go in here. The way we route jbrowse to organism data directories is also controlled here. The organismJBrowseDirectory is set for a particular session, per user. If none specified, it brings up a default one.

Build configuration¶

The build configuration is stored in grails-app/conf/BuildConfig.groovy

If there are libraries that are missing are are to be added, you can add them here.

Additionally, the build system uses the “apollo” script and the “build.xml” to control the compilation and resource steps.

Central config¶

The central configuration is stored in grails-app/conf/Config.groovy

The central Grails config contains logging, app config, and also can reference external configs. The external config can override settings without even touching the application code using this method

In our application, we use the apollo-config.groovy then everything in there supersedes this file.

The log4j area can enable logging levels. You can turn on the “debug grails.app” to output all the webapollo debug info, or also set the “grails.debug” environment variable for java too.

There is also some Apollo configuration here, and it is mostly covered by the configuration section.

User interface definitions¶

A Bootstrap/GWT interface handles the tabs on the right for the new UI. The annotator object is at the root of everything.

Example definition: MainPanel.ui.xml

Unit tests¶

Unit tests and some basic javascript tests are running on Travis-CI (see .travis.yml for example script).

You can also run “apollo test” to run the tests locally. It will use the “test” database configuration automatically.

Also see the testing notes for more details.

get_gff3.groovy¶

Example:

get_gff3.groovy -organism Amel_4.5 -username admin@webapollo.com \
    -password admin_password -url http://localhost:8080/apollo > my output.gff3

This command can accept an -output argument to output to file, or the stdout can be redirected.

The -username and -password can be specified via the command line or if omitted, the user will be prompted.

get_fasta.groovy¶

Example:

get_fasta.groovy -organism Amel_4.5 -username admin@webapollo.com \                                                      
    -password admin_password -seqtype cds/cdna/peptide -url http://localhost:8080/apollo > output.fa

This command can accept an -output argument to output to file, or the stdout can be redirected.

The -username and -password can be specified via the command line (similar to get_gff3.groovy) or if omitted, the user will be prompted.

add_users.groovy¶

Example:

add_users.groovy -username admin@webapollo.com -password admin_password \
    -newuser newuser@test.com -newpassword newuserpass \
    -destinationurl http://localhost:8080/apollo

The -username and -password refer to the admin user, and they can also be specified via stdin instead of the command line if they are omitted.

A list of users specified in a csv file can also be used as input.

add_organism.groovy¶

Example:

add_organism.groovy -name yeast -url http://localhost:8080/apollo/ \
    -directory /opt/apollo/yeast -username admin@webapollo.com -password admin_password

The -directory refers to the jbrowse data directory containing the output from prepare-refseqs.pl, flatfile-to-json.pl, etc. The -blatdb is optional, -genus, and -species are optional.

The -username and -password refer to the admin user, and they can also be specified via stdin instead of the command line if they are omitted.

delete_annotations_from_organism.groovy¶

Example:

docs/web_services/examples/groovy/delete_annotations_from_organism.groovy  -destinationurl http://localhost:8080/apollo\
     -organismname honeybee2

This script will delete any annotations associated with a given organism.

Errors If an error has occurred, a proper HTTP error code (most likely 400 or 500) and an error message. is¶

returned, in JSON format:

{ "error": "error message" }

Additional Notes¶

If you are sending password you care about over the wire (even if not using web services) it is highly recommended that you use https (which adds encryption ssl) instead of http.

Examples are provided in the docs/web_services/examples/groovy/ directory for using SSL and optionally ignoring certificates.

Example¶

curl -b cookies.txt -c cookies.txt -e "http://localhost:8080" \
    -H "Content-Type:application/json" \
    -d "{'username': 'demo', 'password': 'demo'}" \
    "http://localhost:8080/apollo/Login?operation=login"

Login expects two parameters: username and password, and optionally rememberMe for a persistent cookie.

A successful login returns a empty JSON object

Cookies¶

The Apollo Login creates a JSESSIONID cookie and rememberMe cookie (if applicable) and these can be used in downstream API requests (for example, by setting -b cookies.txt in curl will preserve the cookie in the request).

You can also pass username/password to individual API requests and these will authenticate each individual request.

Representing features in JSON¶

Most requests and responses will contain an array of feature JSON objects named features. The feature object is based on the Chado feature, featureloc, cv, and cvterm tables.

{
    "residues": "$residues",
    "type": {
        "cv": {
            "name": "$cv_name"
        },
        "name": "$cv_term"
    },
    "location": {
        "fmax": $rightmost_intrabase_coordinate_of_feature,
        "fmin": $leftmost_intrabase_coordinate_of_feature,
        "strand": $strand
    },
    "uniquename": "$feature_unique_name"
    "children": [$array_of_child_features]
    "properties": [$array_of_properties]
}

where:

• residues - A sequence of alphabetic characters representing biological residues (nucleic acids, amino acids) [string]
• type.cv.name - The name of the ontology [string] type.name - The name for the cvterm [string]
• location.fmax - The rightmost/maximal intrabase boundary in the linear range [integer]
• location.fmin - The leftmost/minimal intrabase boundary in the linear range [integer]
• strand - The orientation/directionality of the location. Should be 0, -1 or +1 [integer]
• uniquename - The unique name for a feature [string]
• children - Array of child feature objects [array]
• properties - Array of properties (including frameshifts for transcripts) [array]

Note that different operations will require different fields to be set (which will be elaborated upon in each operation section).