C3S Magic¶
The Metrics and Access to Global Indices for Climate Projections (MAGIC) portal provides a quick and easy assessment of climate projections using well-established performance metrics. It allows the user to configure these criteria, apply them on selected CMIP5 model output and visualize results in the browser or download them onto your local machine.
The MAGIC portal provides both metrics on model fidelity, targeted to climate scientists and meteorologists, as well as tailored products for different economic sectors, including insurances, agriculture, water management and sustainable energy.
Technically, the MAGIC portal consists of a number of services, which together form the portal site. Most of these services can also function standalone, many are based on existing community software.
This documentation serves as an installation guide, overview of all services, and user guide.
Portal Installation¶
This portal can be deployed with Docker. A docker compose file is present to start and orchestrate all the services needed. An .env file is used for settings, see the example file included.
Obtaining Docker and Docker Compose¶
The MAGIC portal is normally deployed using Docker. See The Docker Documentation for more information on how to install and run docker.
In addition to docker, we also make use of Docker Compose.
This Guide assumes a working installation of docker and docker-compose, and knowledge of this software.
Obtaining the portal code¶
The MAGIC portal code is available on GitHub. The main portal repo uses submodules as references to underlying services.
$ git clone https://github.com/c3s-magic/c3s-magic-portal.git
$ cd c3s-magic-portal
$ git submodule init
$ git submodule update
Obtaining the required data¶
The dataset needed can be downloaded on zenodo: Zenodo Dataset.
Choosing a WPS location¶
Choose either the build-in wps (and supply the needed CP4CDS CMIP and OBS data), or point to the CP4CDS WPS (prefered). The location of the WPS used can be set in the .env file (see below)
Creating a docker-compose .env file¶
All settings are done via a docker environment file. The env file comes with an example (env.example). Copy this file .env and fill in the required fields. The software is currently setup for CEDA account usage only.
#This is the docker environment file.
#It should be copied to .env and filled out.
# CEDA OAUTH secret and client id
CEDA_OAUTHCLIENTSECRET=<Ask matt.pryor@stfc.ac.uk >
CEDA_OAUTHCLIENTID=<Ask matt.pryor@stfc.ac.uk >
#Email used on obtaining letsencrypt certificate
SSL_ADMIN_EMAIL=admin@mydomain.org
#Hostname used to run the portal. An invalid name will
#cause an invalid certificate but otherwise still works
EXTERNAL_HOSTNAME=host.domain.eu
#Location of the static datasets hosted by the portal
#Should contain a single folder "c3smagic" containing all data files
#Can be obtained from zenodo: https://doi.org/10.5281/zenodo.3367497
#Run the scan_adaguc-server_datasets.sh script if updated
STATIC_DATA_LOCATION=/some/data/location
#location of the data used for ESMValtool in the WPS (if build-in)
ESMVALTOOL_CMIP5=/data/cmip5
ESMVALTOOL_OBS=/data/obs
#URLs for the WPS services. By default run inside the portal
#machine, but externally deployed WPS services can also be used.
COPERNICUS_WPS_URL=http://wps:5000/wps?
CLIMEXP_WPS_URL=http://climexp_numerical_wps:5000/wps?"
Deploying using docker-compose¶
Once the settings file is correctly filled in, the entire portal can be deployed with docker-compose like normal.
$ docker-compose build --pull
$ docker-compose up
$ docker-compose down
Updating the ADAGUC Datasets¶
If the datasets served by ADAGUC are updated, update the database of these in the following manner:
$ docker exec c3s-magic-portal_backend_1 /adaguc/adaguc-server-updatedatasets.sh
User Guide¶
The Metrics and Access to Global Indices for Climate Projections (MAGIC) portal provides a quick and easy assessment of climate projections using well-established performance metrics. It allows the user to configure these criteria, apply them on selected CMIP5 model output and visualize results in the browser or download them onto your local machine.
Whom is this portal for?¶
This portal provides both metrics on model fidelity, targeted to climate scientists and meteorologists, as well as tailored products for different economic sectors, including insurances, agriculture, water management and sustainable energy.
Usage¶
- To see pre-calculated quality indices of global circulation models, go to the Metrics & Diagnostics section and navigate to your metric of choice.
- For cross-domain derived diagnostics, go to the Tailored Products page and select a diagnostic.
- To re-evaluate a metric with customized parameters on a CMIP5 dataset, sign in with your BADC/CEDA OpenID and navigate to the metric of choice. Then click the Calculate metric button to enter the metric configuration page. Choose the configuration parameters and select the input dataset of the assessment tool, and the resulting processing job will be scheduled onto the JASMIN cluster and listed in your job list. Your job list and basket can be found on the [Calculate](https://portal.c3s-magic.eu/#/calculate/) page.
- For a simple exploration and visualization or download of CMIP-compliant GCM output, go to Explore Data and start browsing.
About¶
The MAGIC portal provides the functions as requested in the C3S tender “Global Climate Projections: Data Access, product generation and impact of front-line developments” (GCP).
Metrics¶
The Magic portal contains metrics, all calculated using ESMValTool recipes.
The Earth System Model Evaluation Tool (ESMValTool) is a community-development that aims at improving diagnosing and understanding of the causes and effects of model biases and inter-model spread. The ESMValTool is open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the Coupled Model Intercomparison Project (CMIP) ensemble. This will facilitate and improve ESM evaluation beyond the state-of-the-art and aims at supporting the activities within CMIP and at individual modelling centers. We envisage running the ESMValTool routinely on the CMIP model output utilizing observations available through the Earth System Grid Federation (ESGF) in standard formats (obs4MIPs) or made available at ESGF nodes.
The goal is to develop a benchmarking and evaluation tool that produces well-established analyses as soon as model output from CMIP simulations becomes available, e.g., at one of the central repositories of the ESGF. This is realized through standard recipes that reproduce a certain set of diagnostics and performance metrics that have demonstrated its importance in benchmarking Earth System Models (ESMs) in a paper or assessment report, such as Chapter 9 of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) (Flato et al., 2013). The expectation is that in this way a routine and systematic evaluation of model results can be made more efficient, thereby enabling scientists to focus on developing more innovative methods of analysis rather than constantly having to “reinvent the wheel”.
In parallel to standardization of model output, the ESGF also hosts observations for Model Intercomparison Projects (obs4MIPs) and reanalyses data (ana4MIPs). obs4MIPs provides open access data sets of satellite data that are comparable in terms of variables, temporal and spatial frequency, and periods to CMIP model output (Taylor et al., 2012). The ESMValTool utilizes these observations and reanalyses from ana4MIPs plus additionally available observations in order to evaluate the models performance. In many diagnostics and metrics, more than one observational data set or meteorological reanalysis is used to assess uncertainties in observations.
—ESMValTool Documentation https://esmvaltool.readthedocs.io
For information on specific recipes used in this portal see the ESMValTool page: https://esmvaltool.readthedocs.io/en/latest/recipes/index.html
System Overview¶
The MAGIC Web Processing Service (WPS) provides the scientific functionality of MAGIC. Building on s2dverificatoin, climate explorer, iris, ESMValTool, pyWPS, and other libraries and tools, the WPS takes ESGF CMIP5 data and processes these into the graphs and data shown on this portal.
This is an overview of the infrastructure in which the WPS is running. It consists of the portal system based on Climate4Impact and ADAGUC Services. These run on cloud infrastructure outside of ESGS, and is managed by the MAGIC team.
The actual processing of ESGF data will be done on infrastructure provided by the CP4CD project.