Crops in silico¶
Overview¶
Crops in silico is an integrative and multi-scale modeling platform designed to combine modeling efforts toward the generation of virtual crops.
The Crops in silico platform consists of:
- Model integration framework (
cis_interface) - Model composer and execution framework
- Model catalog
Model integration framework: cis_interface¶
The cis_interface model integration framework provides support for combining
scientific models written in different programming languages. To combine two
models, modelers add simple communications interfaces to the model code and
provide simple declarative specification files that identfy the models that
should be run and the inputs and outputs those models expect.
For more information, see the documentation
Model composer and execution framework¶
The Model Composer and execution framework is available to the Cis community at https://cropsinsilico.ndslabs.org.
The Model Composer is a tool for composing systems of models built using the
cis_interface framework. For documentation on using the Model Composer UI, please see the
User Guide. For documentation on developing the Model Composer UI, please see the
Developer Guide
Model catalog¶
The model catalog is a GitHub repository containing community contributed model definitions and predefined systems of models as graphs. Models in this repository are available to all users of the platform. Models are contributed through GitHub issues and pull requests.
Getting started¶
The Crops in silico model composer is based on the popular JupyterLab
web-based interactive analysis and development environment. JupyterLab
provides a complete development environment for the creation and execution
of models using the cis_interface integration framework. The model
composer is a JupyterLab plugin that supports a visual-programming approach
to model composition.
Logging in¶
To access the web service, go to https://cropsinsilico.ndslabs.org. The CiS model composer uses GitHub for authentication. You will be prompted to sign-in via GitHub and to authorize CiS to access information about you. username.
Click on the “Sign in with GitHub” button:
Login
Enter your GitHub credentials or create an account, if necessary:
Github login
If prompted, authorize “Crops in silico” to access your profile information:
Authorize CiS
Starting JupyterLab¶
Once logged in, you will be able to start your JupyterLab instance. Select “Start My Server”:
Start server
Launching the Model Composer¶
Once started, you will see the JupyterLab launcher. This JupyterLab environment
is a complete development environment for use with the cis_interface
framework. You can create notebooks, execute commands from the terminal, or
launch the model composer. Select the “Model Composer” icon:
JupyterLab launcher
Loading an Existing Model Graph¶
The model composer allows to to create and use models from the model library to
compose execution graphs. Select the “Load” button to load the “GrCM” model:
and
Model composer
Executing the Graph¶
Select “Execute” to run the model. Model output will be displayed on screen and
written to an model output directory for further exploration:
Model logs
Viewing Execution Output¶
The model output directory contains the graph, model source, inputs and outputs used during execution:
Model output
Using the Model Library¶
The model library lists all official models approved by the community as well as any private models you have developed and added to the system. You can use the library to compose new graphs using this user interface:
Model libraryl
Next step¶
For more detailed usage information, please see the User Guide.
User Guide¶
Welcome to the User’s Guide for the Crops in silico Model Composer UI. The purpose of this document is to describe the usage of the User Interface. If any of the steps described in this document are unclear or confusing, please direct your questions to Crops in silico Support
To assist the Support Team in their investigation, please be sure to include the browser type/version and operating system that you were using when the troublesome behavior was encountered.
Table of Contents¶
Orientation¶
Orientaton
The Model Composer UI consists of several smaller components:
- Navbar
- Canvas (aka “The Graph”)
The Navbar runs along the top of the UI and contains the Crops in silico brand name and logo. On the left side, you will find a link to submit a new model. On the right side, you will also find the Log In button and a Help dropdown which contains links to this user guide, the developer’s guide, and the documentation for the cisrun CLI tool.
The Canvas is the grid with the black background in the center of the screen. This displays the current state of the graph that the user is working on.
In front of the Canvas, there are also a few helpful floating windows:
- Model Library (aka “Palette”)
- Button Bar
- Context Menu + Edit Sidebar
The Model Library starts collapsed, and can be found on the top-left of the canvas. This consists of a simple table listing of the existing models that the system knows about. Here we can also add InPorts and OutPorts to our graph. Simply click the Add (+) button and the new node will appear in the Canvas. If the node appears off-screen, you can easily find it by pressing the F key to focus on the entire graph. For models that you have created, here you can also find the option to delete them (see Creating a New Model).
Expanded Model Palette
The Button Bar runs along the top-right of the canvas. It contains actions such as Save, Load, Clear, and Generate Manifest. Save will store the state of your current graph to the database, while Load will import a given state from the database. Clear will remove all nodes and edges from the current canvas, but does not affect graphs that have been stored in the database. Generate Manifest will convert the current graph on the canvas to the format that is required by the cisrun CLI.
The Context Menu appears when an entity in the canvas is right-clicked. It appears as a round menu that centers on where the mouse click occurred. Edit and Delete are offered the only actions offered, and only when right-clicking on Edges or Nodes (including InPorts and OutPorts).
If Edit is chosen, the Edit Sidebar will be displayed on the right edge of the canvas. The sidebar allows you to edit the metadata of the entity you have selected. This sidebar will also appear when adding an InPort or OutPort to specify the source/destination of the data.
Canvas Controls¶
The Canvas accepts a few mouse/keyboard inputs:
F: Auto-focus the graph so that you can see all elementsLeft-click: Select a port on a nodeLeft-click (hold): Drag a node around the canvasRight-click: Display the context-menu for the clicked graph element
Loading an Example Graph¶
The Button Bar at the top-right should offer a Load button. Clicking on this button will expand a dropdown list of all saved graphs and examples currently accessible. Choosing one of these examples will load its contents into your Canvas, allowing you to visualize and edit the graph before generating a manifest for use with cisrun.
Adding a Node¶
The Model Library on the left side offers an “Add” button beside each model. Click this button to add a new node to the canvas representing the model you’ve chosen. Once added, you can left-click this node and hold to drag it around the canvas. On the new node, you should see grey dots on the left/right sides - these are the inputs (left side) and outputs (right side) that this model accepts.
For more details, see the documentation for Model file input/output
About the Bounding Box¶
While dragging, you may notice a lighter gray circle around the dragged node. This is the “bounding box” that indicates valid areas where your node can be dragged - as your node moves the bounding box moves with it! This behavior is meant to prevent you from dragging a node so fast that the mouse cursor exits the node before the “drag” event is picked up.
Adding an Edge¶
To create an edge, simply click on an output on a model - this should display a floating edge that ends at your cursor. Then, select an input on another model. You can also choose an input first and connect that to an output.
NOTE: Inputs can only be connected to outputs and vice versa. An input cannot be connected to another input. An output cannot be connected with another output.
For more details, see the documentation for Model-to-model communication
Adding an InPort / OutPort¶
While each node has its own set of inputs and outputs, the entire graph likely will need InPorts/OutPorts of its own. The Model Palette at the left of the view offers buttons to add these InPorts and OutPorts.
Logging In¶
For users who simply wish to build new graphs from our existing sets of models, we do not require them to create an account or log into the system. Anyone who wishes to Create a New Model or Save a Composed Graph to the database, however, will need to sign up for an account. This is simply to track which users created which models, to avoid showing unvetted or defunct options to all users.
To begin the login process, click Log In at the top-right in the Navbar:
OAuth Start
You will then be prompted for your GitHub account credentials. If you do not already have a GitHub account, you can sign up for one for free on GitHub.com
OAuth Authentication
After signing in, you will be asked to Authorize the Crops in silico Platform to access your GitHub account information. We only examine your user profile information and use it to create al inked account in our system.
OAuth Authorization
Click “Authorize” to be redirected the Crops in silico application. You are now logged in as your GitHub user, and should see a couple of new buttons have appeared in the User Interface:
- Save Graph
- Submit a New Model
OAuth Authorized
Saving a Composed Graph¶
After logging in, the Button Bar will include a Save button. Clicking this button will offer you a prompt to name this graph. If a valid name is entered, the graph will be saved to the database.
Loading a Previously-Saved Graph¶
After a graph has been saved, it should appear in the Load dropdown on the Button Bar. Choosing a previously-saved graph will load its contents into the Canvas.
NOTE: In order to facilitate composing different models and examples, Load does not Clear your existing graph state.
Deleting a Previously-Saved Graph¶
For graphs that you have saved, you should see a Delete button beside them. Simply click this button to remove the saved graph from the database.
Creating a New Model¶
Do you have a new model that you would like to contribute? After logging in, the Navbar offers a link that will allow you to submit your own custom model metadata. Simply Log In at the top-right, then click Submit a New Model at the top-left. A pop-up should appear allowing you to enter all necessary metadata fields of you model. Once created, your model appears in your personal catalog for testing and debugging.
Deleting a Model¶
If you have created a model that you would like to remove, you should see a Delete button listed on the Model Palette once when it is expanded. Click the search icon to expand the Model Palette. You should see a red Delete button beside any unofficial specs that you have created.
NOTE: Once a model or graph has been officially submitted, it cannot be deleted
Official Submission¶
When you are satisfied with the working state of your model or graph, you can submit it to our official catalog as an issue or pull request to the cis-specs repository. Here it will go through a peer review process where it will be testing and vetted. If it passes the review, it will be accepted into our official catalog, where all users will be able to consume and use the new model.
We are working to automate this submission process further, and we thank you for your patience while we determine the best course of action.
Developer’s Guide¶
The purpose of this document is to familiarize you with the patterns used to develop the Crops in silico framework.
Running the Crops in silico Platform¶
No matter how you choose to run the platform, you will need a few things:
- MongoDB: An open-source NoSQL database
- Girder: An open-source data management
- The
cis-uiweb application
Running Under Kubernetes (Production-Ready)¶
The platform has been most thoroughly-tested while running under Kubernetes.
To get started with a single Kubernetes master node, you can follow the instructions here: https://github.com/nds-org/kubeadm-bootstrap#setting-up-a-cluster
You can continue testing with a single node, or add additional workers by following these instructions: https://github.com/nds-org/kubeadm-bootstrap#setting-up-a-worker-node
You should not need more than one Kubernetes node to run the CiS Platform.
Once you have Kubernetes up and running, you can use the templates in cis-startup to quickly get up and running:
git clone https://github.com/cropsinsilico/cis-startup && cd cis-startup
NOTE: You may need to edit the hostnames the Ingress resources located in platform/ to match your desired hostname.
If everything came up as it should, navigating to http://desired.hostname.com:80 should bring you to the CiS Model Composer UI.
Running Under Docker (Development Only)¶
If Kubernetes feels like too much overhead, running under Docker is simple enough.
The following command will start up MongoDB:
docker run -it --name=mongodb -v $(pwd):/data/db -p 27017:27017 mongo:3.3
Then run a Girder container:
docker run -it --name=girder --link mongodb:mongodb -p 8080:8080 cropsinsilico/girder:stable --host 0.0.0.0 --database mongodb://mongodb:27017/girder
Finally, run our NGINX container with the cis-ui source:
docker run -it --name=cis-ui --link girder:girder -p 80:80 cropsinsilico/cis-ui:stable
If everything came up as it should, navigating to http://localhost:80 should bring you to the CiS Model Composer UI.
Running Without Containers (Not recommended)¶
It is highly recommended that you run each of these services as a separate container, but installing them on the host should work as well.
A few steps are necessary to get the platform up and running on a new host:
- Install MongoDB
- Install Girder and configure it to talk to MongoDB
- Install NodeJS and use it to run
cis-ui, which will talk to Girder
Such steps are highly manual and will likely vary from version to version of the softwares mentioned above.
For this reason, we do not provide such instructions and instead suggest running everything in a container, as it greatly simplifies the complexity of running, configuring, and connecting these applications.
Girder Configuration¶
Coming Soon!
Adjusting the Server Root¶
Coming Soon!
Enabling OAuth¶
Coming Soon!
Enabling cis-girder-plugin¶
Coming Soon!
Crops in silico Development Environment¶
The following instructions should help you get a development environment up and running for modifying the Crops in silico Model Composer UI
Prerequisites¶
- Git
- Docker or NodeJS
Clone this repository:
git clone https://github.com/cropsinsilico/cis-ui && cd cis-ui
Run¶
Start a development webserver on port 8080:
docker run -it -p 8080:80 --name=cis-ui -v $(pwd):/usr/nginx/share/html cis/ui
NOTE: -v src:dest tells Docker to map src from your host to dest within the container
Without Docker¶
If you don’t have Docker, then you will need to install NodeJS and run the following:
npm start
Develop¶
Start a Cloud9 IDE on port 8081:
docker run -it -p 8081:80 --name=cloud9-cis -v $(pwd):/workspace -w /workspace ndslabs/cloud9-nodejs
Regenerate API client¶
If you modify the swagger spec, you will need to regenerate the API client.
A grunt task has been provided to do this for you:
npm install -g grunt
npm install
grunt swagger
Ensure that the client and server always have matching API specs.
Rebuilding Jekyll Locally¶
Execute the following command to run a jekyll development server:
docker rm -f cis-jekyll
docker run -itd \
--name=cis-jekyll \
--label=jekyll \
--volume=$(pwd):/srv/jekyll \
-p 127.0.0.1:4000:4000 \
-p 127.0.0.1:35729:35729 \
jekyll/jekyll \
jekyll server --watch --livereload
You can then view the logs of your running container by executing:
docker logs -f cis-jekyll
Administrator’s Guide¶
Jetstream¶
The platform is currently hosted on XSEDE Jetstream.
Kubernetes¶
The Cis platform is deployed via the Kubernetes container orchestration system. Kubernetes is deployed on Jetstream/ OpenStack using either kubeadm-bootstrap or kubeadm-terraform.
Configuration options include:
- NFS dynamic volume provisioner
- Cert manager for TLS certificate management
- NGINX ingress controller
- Flannel network provider
JupyterHub¶
The primary entrypoint for the model composer and execution framework is a customized JupyterHub instance. Configuration options include:
- Use of JupyterLab as default environment
- Github authentication
- Pre-populated model source code via
gitpuller - MATLAB mounted via
hostPath
JupyterLab¶
JupyterHub and the model execution framework share a common JupyterLab image defined in cropsinsilico/jupyterlab.
This image includes:
- Model composer extension
MATLAB support¶
MATLAB support is enabled by installing MATLAB on the host and mounting the installation directory into the JupyterLab and job execution environments.
Dockerhub¶
All images are hosted on Dockerhub under the cropsinsilico organization. Most images should have autobuild configured.
Github¶
Source code is managed on Github under the cropsinsilico organization.
Sphinx¶
git clone https://github.com/cis-ui/ -b gh-pages cis-ui-ghpages
git clone https://github.com/cis-ui/
cd cis-ui/docs
make html
cp -a _build/html/ ../../cis-ui-ghpages/
VM Setup¶
To setup a new Jetstream project, see the [Jetstream documentation] (https://iujetstream.atlassian.net/wiki/spaces/JWT/pages/44826638/Setup+for+Horizon+API+User+Instances). You’ll need to create a network, subnet, router, and security groups for HTTP/S and SSL.
Jetstream has tons of images intended for use with Atmosphere. It’s generally
easiest to upload or glance in your own images for use in the system.
Download Ubuntu 16.04
cloud
and either upload or use the openstack client:
openstack image create --disk-format qcow2 --container-format bare \
--file xenial-server-cloudimg-amd64-disk1.img "Ubuntu 16.04 LTS"
Launch a new instance following the Horizon documentation above or via the
openstack client.
Once the instance is up, install nfs-common:
sudo apt-get install nfs-common
Install Kubernetes via kubeadm-bootstrap:
git clone https://github.com/data-8/kubeadm-bootstrap
cd kubeadm-bootstrap
sudo ./install-kubeadm.bash
sudo -E ./init-master.bash
Install cert-manager (See also https://opensource.ncsa.illinois.edu/confluence/display/~lambert8/Kubernetes+Cert-Manager):
mkdir cert-manager
vi cert-manager-values.yaml
sudo helm repo update
sudo helm install --debug --name cert-manager -f cert-manager-values.yaml stable/cert-manager
kubectl create -f letsencrypt-staging.yaml
Upgrade Helm (required by JupyterHub 0.7):
curl https://storage.googleapis.com/kubernetes-helm/helm-v2.10.0-linux-amd64.tar.gz | tar xvz
sudo mv linux-amd64/helm /usr/local/bin
helm init --upgrade
Install JupyterHub:
sudo helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
sudo helm repo update
cd /home/ubuntu/cis-startup/jupyterhub
helm upgrade --install hub jupyterhub/jupyterhub \
--namespace hub \
--version 0.7.0 \
--values config.yaml
To upgrade an existing chart:
helm upgrade hub jupyterhub/jupyterhub \
--version=0.7.0 \
--values config.yaml
To install MATLAB, download R2018a ISO disk images from UIUC webstore and transfer to host
Create input.txt:
fileInstallationKey=<your key>
licensePath=/home/ubuntu/matlab-cis/license.dat
agreeToLicense=yes
mode=silent
Install. When prompted to enter disk 2, unmount first iso and mount second:
sudo su -
mkdir /mnt/matlab
mount -t iso9660 -o loop /home/ubuntu/matlab/R2018a_glnxa64_dvd1.iso /mnt/matlab
/mnt/matlab/install -inputFile /home/ubuntu/matlab-cis/input.txt
umount /mnt/matlab
mount -t iso9660 -o loop /home/ubuntu/matlab/R2018a_glnxa64_dvd2.iso /mnt/matlab
umount /mnt/matlab
MATLAB is now installed in /usr/local/MATLAB/R2018a
To test via standard Jupyter environment:
docker run -v /usr/local/MATLAB:/usr/local/MATLAB -it jupyter/scipy-notebook:2bfbb7d17524 bash
export PATH=/usr/local/MATLAB/R2018a/bin/:$PATH
export LM_LICENSE_FILE=<license server>
matlab -nodisplay -nosplash -nodesktop -nojvm
Optionally, test cis_interface:
# Install python engine
cd /usr/local/MATLAB/R2018a/extern/engines/python
python setup.py build -b /tmp install
# Install cis_interface
pip install cis_interface
#Run example
git clone https://github.com/cropsinsilico/cis_interface
cd cis_interface/cis_interface/examples/hello/
cisrun hello_matlab.yml
To install CiS, clone cis-startup, cis-ui, cis-girder-plugin. Modify the specs as needed.
cd cis-startup/platform
kubectl apply -f rbac/
kubectl apply -f girder.dev.yaml -f girder.staging.yaml -f girder.prod.yaml
cd pvcs/nfs
kubectl create -f deployment.yaml -f rbac.yaml -f storageclass.yaml
Register the admin user, enable and configure Oauth, Jobs and CiS plugins. Restart Girder.