Introduction

Hummingbird (the bird)

Hummingbirds are among the smallest of birds. They hover in mid-air at rapid wing flapping rates, typically around 50 times per second, but possibly as high as 200 times per second, allowing them also to fly at speeds exceeding 15 m/s (54 km/h; 34 mph), backwards or upside down. [..]. (Wikipedia).

Hummingbird provides a Web Processing Service (WPS) using PyWPS for the climate science community. It has WPS processes for common tools used in climate science like CDO and cfchecker.

Hummingbird is part of the Birdhouse project.

Contents:

Installation

Check out code from the Hummingbird github repo and start the installation:

$ git clone https://github.com/bird-house/hummingbird.git
$ cd hummingbird
$ make clean install

For other install options run make help and read the documention of the Makefile.

After successful installation you need to start the services. Hummingbird is using Anaconda Python distribution system. All installed files (config etc …) are below the Anaconda root folder which is by default in your home directory ~/anaconda. Now, start the services:

$ make start    # starts supervisor services
$ make status   # shows supervisor status

The depolyed WPS service is by default available on http://localhost:8092/wps?service=WPS&version=1.0.0&request=GetCapabilities.

Check the log files for errors:

$ tail -f  ~/birdhouse/var/log/pywps/hummingbird.log
$ tail -f  ~/birdhouse/var/log/supervisor/hummingbird.log

Using docker-compose

Start hummingbird with docker-compose (docker-compose version > 1.7):

$ docker-compose up

By default the WPS is available on port 8080: http://localhost:8080/wps?service=WPS&version=1.0.0&request=GetCapabilities.

You can change the ports and hostname with environment variables:

$ HOSTNAME=hummingbird HTTP_PORT=8092 SUPERVISOR_PORT=48092 docker-compose up

Now the WPS is available on port 8092: http://hummingbird:8092/wps?service=WPS&version=1.0.0&request=GetCapabilities.

Configuration

If you want to run on a different hostname or port then change the default values in custom.cfg:

$ cd hummingbird
$ vim custom.cfg
$ cat custom.cfg
[settings]
hostname = localhost
http-port = 8092

After any change to your custom.cfg you need to run make update again and restart the supervisor service:

$ make update    # or install
$ make restart

Running unit tests

Run quick tests:

$ make test

Run all tests (slow, online):

$ make testall

Check pep8:

$ make pep8

WPS Processes

We describe here the WPS processes available in Hummingbird.

WPS Capabilities

Using the default Hummingbird installation the GetCapabilities request is as follows:

http://localhost:8092/wps?service=WPS&version=1.0.0&request=GetCapabilities

The XML response of the WPS service is the following document:

<?xml version="1.0" encoding="utf-8"?>
<wps:Capabilities service="WPS" version="1.0.0" xml:lang="en-CA" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsGetCapabilities_response.xsd" updateSequence="1">
	<ows:ServiceIdentification>
		<ows:Title>Hummingbird</ows:Title>
		<ows:Abstract>WPS processes for general tools used in the climate science community like cdo</ows:Abstract>
		<ows:Keywords>
			<ows:Keyword>WPS</ows:Keyword>
			<ows:Keyword>PyWPS</ows:Keyword>
		</ows:Keywords>
		<ows:ServiceType>WPS</ows:ServiceType>
		<ows:ServiceTypeVersion>1.0.0</ows:ServiceTypeVersion>
		<ows:Fees>None</ows:Fees>
		<ows:AccessConstraints>None</ows:AccessConstraints>
	</ows:ServiceIdentification>
	<ows:ServiceProvider>
		<ows:ProviderName></ows:ProviderName>
		<ows:ServiceContact>
			<ows:ContactInfo>
				<ows:HoursOfService>0:00-24:00</ows:HoursOfService>
				<ows:ContactInstructions>None</ows:ContactInstructions>
			</ows:ContactInfo>
		</ows:ServiceContact>
	</ows:ServiceProvider>
	<ows:OperationsMetadata>
		<ows:Operation name="GetCapabilities">
			<ows:DCP>
				<ows:HTTP>
					<ows:Get xlink:href="http://localhost:8092/wps?"/>
					<ows:Post xlink:href="http://localhost:8092/wps"/>
				</ows:HTTP>
			</ows:DCP>
		</ows:Operation>
		<ows:Operation name="DescribeProcess">
			<ows:DCP>
				<ows:HTTP>
					<ows:Get xlink:href="http://localhost:8092/wps?"/>
					<ows:Post xlink:href="http://localhost:8092/wps"/>
				</ows:HTTP>
			</ows:DCP>
		</ows:Operation>
		<ows:Operation name="Execute">
			<ows:DCP>
				<ows:HTTP>
					<ows:Get xlink:href="http://localhost:8092/wps?"/>
					<ows:Post xlink:href="http://localhost:8092/wps"/>
				</ows:HTTP>
			</ows:DCP>
		</ows:Operation>
	</ows:OperationsMetadata>
	<wps:ProcessOfferings>
		<wps:Process wps:processVersion="0.1">
			<ows:Identifier>cdo_sinfo</ows:Identifier>
			<ows:Title>CDO sinfo</ows:Title>
			<ows:Abstract>Apply CDO sinfo on NetCDF File.</ows:Abstract>
                        <ows:Metadata xlink:title="CDO" xlink:href="https://code.zmaw.de/projects/cdo" />
		</wps:Process>
		<wps:Process wps:processVersion="0.1">
			<ows:Identifier>cdo_operation</ows:Identifier>
			<ows:Title>CDO Operation</ows:Title>
			<ows:Abstract>Apply CDO Operation like monmax on NetCDF File.</ows:Abstract>
                        <ows:Metadata xlink:title="CDO" xlink:href="https://code.zmaw.de/projects/cdo" />
		</wps:Process>
		<wps:Process wps:processVersion="0.1">
			<ows:Identifier>cfchecker</ows:Identifier>
			<ows:Title>CF Checker</ows:Title>
			<ows:Abstract>The cfchecker checks NetCDF files for compliance to the CF standard.</ows:Abstract>
		</wps:Process>
	</wps:ProcessOfferings>
	<wps:Languages>
		<wps:Default>
			<ows:Language>en-CA</ows:Language>
		</wps:Default>
		<wps:Supported>
			<ows:Language>en-CA</ows:Language>
		</wps:Supported>
	</wps:Languages>
	<wps:WSDL xlink:href="http://localhost:8092/wps?WSDL"/>
</wps:Capabilities>

SpotChecker

Spot Checker checks a single dataset (NetCDF or OpenDAP) against a variety of compliance standards. Available compliance standards are the Climate and Forecast conventions (CF) and project specific rules for CMIP6 and CORDEX.

CFChecker

The cfchecker checks NetCDF files for compliance to the Climate Forcast Conventions (CF) standard.

The process expects one or more NetCDF files which should be checked and an optional parameter for the CF version.

WPS process description

Using the default Hummingbird installation the DescribeProcess request is as follows:

http://localhost:8092/wps?service=WPS&version=1.0.0&request=DescribeProcess&identifier=cfchecker

The XML response of the WPS service is the following document:

<?xml version="1.0" encoding="utf-8"?>
<wps:ProcessDescriptions xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsDescribeProcess_response.xsd" service="WPS" version="1.0.0" xml:lang="en-CA">
    <ProcessDescription wps:processVersion="0.1" storeSupported="true" statusSupported="true">
        <ows:Identifier>cfchecker</ows:Identifier>
        <ows:Title>CF Checker</ows:Title>
        <ows:Abstract>The cfchecker checks NetCDF files for compliance to the CF standard.</ows:Abstract>
        <DataInputs>
            <Input minOccurs="1" maxOccurs="1000">
                <ows:Identifier>resource</ows:Identifier>
                <ows:Title>NetCDF File</ows:Title>
                <ows:Abstract>NetCDF File</ows:Abstract>
                <ComplexData>
                    <Default>
                        <Format>
                            <MimeType>application/x-netcdf</MimeType>
                        </Format>
                    </Default>
                    <Supported>
                        <Format>
                            <MimeType>application/x-netcdf</MimeType>
                        </Format>
                    </Supported>
                </ComplexData>
            </Input>
            <Input minOccurs="0" maxOccurs="1">
                <ows:Identifier>cf_version</ows:Identifier>
                <ows:Title>CF version</ows:Title>
                <ows:Abstract>CF version to check against, use auto to auto-detect the file version.</ows:Abstract>
                <LiteralData>
                    <ows:DataType ows:reference="http://www.w3.org/TR/xmlschema-2/#string">string</ows:DataType>
                    <ows:AllowedValues>
                            <ows:Value>auto</ows:Value>
                            <ows:Value>1.6</ows:Value>
                            <ows:Value>1.5</ows:Value>
                            <ows:Value>1.4</ows:Value>
                            <ows:Value>1.3</ows:Value>
                            <ows:Value>1.2</ows:Value>
                            <ows:Value>1.1</ows:Value>
                    </ows:AllowedValues>
                    <DefaultValue>auto</DefaultValue>
                </LiteralData>
            </Input>
        </DataInputs>
        <ProcessOutputs>
            <Output>
                <ows:Identifier>output</ows:Identifier>
                <ows:Title>CF Checker Report</ows:Title>
                <ComplexOutput>
                    <Default>
                        <Format>
                            <MimeType>text/plain</MimeType>
                        </Format>
                    </Default>
                    <Supported>
                        <Format>
                            <MimeType>text/plain</MimeType>
                        </Format>
                    </Supported>
                </ComplexOutput>
            </Output>
        </ProcessOutputs>
    </ProcessDescription>
</wps:ProcessDescriptions>

The WPS Parameters are:

resource

Is the input parameter to provide one or more URLs (http://, file://) to NetCDF files. It is a WPS ComplexData type with MIME-type application/x-netcdf.

cf_version

Is an optional input parameter to provide the CF version to check against. It is a WPS LiteralData type with a set of allowed values (1.1, 1.2, …, auto).

output

Is the output parameter to provide the report of the CF check as text document. It is a WPS ComplexData type with MIME-type text/plain.

WPS process execution

An example execution of the cfchecker process with public available data:

http://localhost:8092/wps?service=WPS&version=1.0.0&request=Execute&identifier=cfchecker&DataInputs=resource=http://www.esrl.noaa.gov/psd/thredds/fileServer/Datasets/ncep/vwnd.sfc.2015.nc&RawDataOutput=output

The process is called with key/value parameters, synchronously and with direct output (RawDataOutput).

The resulting text document of the cfchecker report looks like the following:

CDO

to be continued …

Example: Using Docker

If you just want to try the Hummingbird Web Processing Service you can also use the Docker image:

$ docker run -i -d -p 9001:9001 -p 8000:8000 -p 8080:8080 birdhouse/hummingbird

Open your browser and enter the url of the supervisor service:

http://localhost:9001/

Run a GetCapabilites WPS request:

http://localhost:8080/wps?service=WPS&version=1.0.0&request=getcapabilities

Run DescribeProcess WPS request for CFChecker:

http://localhost:8080/wps?service=WPS&version=1.0.0&request=describeprocess&identifier=cfchecker

Execute CFChecker process with public available data:

http://localhost:8080/wps?service=WPS&version=1.0.0&request=Execute&identifier=cfchecker&DataInputs=dataset=http://www.esrl.noaa.gov/psd/thredds/fileServer/Datasets/ncep/vwnd.sfc.2015.nc&RawDataOutput=output

Install Birdy WPS command line tool from Anaconda (Anaconda needs to be installed and in your PATH):

$ conda install -c birdhouse birdhouse-birdy

Use Birdy to access Hummingbird WPS service:

$ export WPS_SERVICE=http://localhost:8080/wps
$ birdy -h
$ birdy cfchecker -h
$ birdy cfchecker --dataset http://www.esrl.noaa.gov/psd/thredds/fileServer/Datasets/ncep/vwnd.sfc.2015.nc

Sphinx AutoAPI Index

This page is the top-level of your generated API documentation. Below is a list of all items that are documented here.

exceptions

Module Contents

class ProcessError

processing

Module Contents

processing.get_cdo()

processing.ncdump(dataset)

Returns the metadata of the dataset

Code taken from https://github.com/ioos/compliance-checker-web

processing.cmor_tables_path()

processing.cmor_tables()

processing.cmor_dump_output(dataset, status, output, output_filename)

processing.cmor_checker(dataset, table="CMIP6_CV", variable=None, output_filename=None)

processing.hdh_cf_check(filename, version="auto")

processing.hdh_qa_checker(filename, project, qa_home=None)

utils

Module Contents

utils.make_dirs(path)

utils.fix_filename(filename)

utils.output_filename(filename, addition=None, extension=None)

build an appropriate output filename based on filename, addition and extension.

patch

Module Contents

patch.patch_compliance_checker()

Patch compliance_checker for ESGF OpenDAP with .dodsrc.

__init__

Package Contents

__init__.application(environ, start_response)

config

Module Contents

config.cache_path()

processes

Submodules

processes.wps_cdo_bbox

Processes with cdo commands

Module Contents

class processes.wps_cdo_bbox.CDOBBox

This process calls cdo sellonlatbox on netcdf file

_handler(request, response)

processes.wps_cdo_copy

Processes with cdo commands

Module Contents

class processes.wps_cdo_copy.CDOCopy

This process calls cdo with the copy operation on datasets (netcdf, opendap)

_handler(request, response)

processes.wps_cdo_ensembles

Processes with cdo ensemble opertions

Module Contents

class processes.wps_cdo_ensembles.CDOEnsembles

_handler(request, response)

processes.wps_cdo_indices

Processes with cdo commands

Module Contents

class processes.wps_cdo_indices.CDOClimateIndices

This process calls cdo to calculate climate indices written to a netcdf file

_handler(request, response)

processes.wps_cdo_inter_pywps4

Processes with cdo commands

Module Contents

processes.wps_cdo_inter_pywps4.cdo_wrap(tmargs)

class processes.wps_cdo_inter_pywps4.CDOinter_MPI

_handler(request, response)

processes.wps_cdo_op

Processes with cdo commands

Module Contents

class processes.wps_cdo_op.CDOOperation

This process calls cdo with operation on netcdf file

_handler(request, response)

processes.wps_cdo_sinfo

Processes with cdo commands

Module Contents

class processes.wps_cdo_sinfo.CDOInfo

This process calls cdo sinfo on netcdf file

_handler(request, response)

processes.wps_cfchecker

Module Contents

processes.wps_cfchecker.cf_check(nc_file, version)

class processes.wps_cfchecker.CFChecker

_handler(request, response)

processes.wps_cmor_checker

Module Contents

class processes.wps_cmor_checker.CMORChecker

_handler(request, response)

processes.wps_compliance_checker

Module Contents

class processes.wps_compliance_checker.CChecker

_handler(request, response)

processes.wps_hdh_cfchecker

Module Contents

class processes.wps_hdh_cfchecker.HDHCFChecker

_handler(request, response)

processes.wps_hdh_qachecker

Module Contents

class processes.wps_hdh_qachecker.QualityChecker

_handler(request, response)

processes.wps_ncdump

Module Contents

class processes.wps_ncdump.NCDump

_handler(request, response)

processes.wps_ncplot

Processes for plotting netcdf files with matplotlib/basemap

Module Contents

class processes.wps_ncplot.SimplePlot

Plots a simple 2D map of netcdf file

execute()

processes.wps_spotchecker

Module Contents

class processes.wps_spotchecker.SpotChecker

_handler(request, response)

tests

Submodules

tests.common

Module Contents

class tests.common.WpsTestClient

get(*args, **kwargs)

tests.common.client_for(service)

tests.test_utils

Module Contents

tests.test_utils.test_fix_filename()

tests.test_utils.test_output_filename()

tests.test_wps_caps

Module Contents

tests.test_wps_caps.test_wps_caps()

tests.test_wps_cdo_ensembles

Module Contents

tests.test_wps_cdo_ensembles.test_wps_ensembles()

tests.test_wps_cdo_op

Module Contents

tests.test_wps_cdo_op.test_wps_cdo_operation()

tests.test_wps_cdo_sinfo

Module Contents

tests.test_wps_cdo_sinfo.test_wps_cdo_sinfo()

tests.test_wps_cfchecker

Module Contents

tests.test_wps_cfchecker.test_wps_cfchecker()

tests.test_wps_compliance_checker

Module Contents

tests.test_wps_compliance_checker.test_wps_cchecker()

tests.test_wps_hdh

Module Contents

tests.test_wps_hdh.test_wps_qa_cfchecker()

tests.test_wps_ncdump

Module Contents

tests.test_wps_ncdump.test_wps_ncdump()

Package Contents

class tests.WpsTestCase

Base TestCase class, sets up a wps

setUpClass()

Indices and tables

• Index
• Module Index
• Search Page