Templates, grants and pre-fill¶

A core focus of the EnMaSSe system has been to reduce the amount of information that needs to be entered and pre fill as much as possible.

To achieve this goal a number of features have been implemented:

• Project templates allow administrators to provide projects that have any/all fields already filled, while this is a great feature it has the limitation that most projects will be dissimilar.
• Method templates allow administrators to provide method+dataset combinations for known data collection methods. This will be great for sensors that are commonly used and it is anticipated that this will both save a lot of time and simplify the use of sensors.
• Standardised fields allow administrators to provide common data configurations that the end user can select rather than recreating the way the same data is stored all the time.
• Project creation wizard allows selection of the project template as well as collecting the primary contacts and (optionally) the associated grant, data is retrieved and used to pre fill fields such as the brief description and project dates.
• Record citation are automatically generated from the entered information (administrators have the option to provide a custom citation).
• Duplicate Project allows any user to create a new project which is a copy of the currently viewed project. This is great for similar projects and is more flexible than templates that administrators first need to set up.

How metadata records are generated¶

When an EnMaSSe project is submitted and approved a metadata record is created for the project itself as well as each individual dataset, each dataset metadata record starts as a copy of the project record then the:

• Title is set to <project title> at <location name>(<lat, long, height>) collected by <method name>.
• Location is set to the dataset location (each dataset requires a valid location).
• Each dataset is associated with a method, the description entered for the method is added as a note description.
• Publish date is set as specified on datasets page.
• Citation is generated when sent to ReDBox.

If a dataset is required as an intermediary processing step and it doesn’t make sense to create a metadata record, generation for the dataset can be disabled by unchecking the publish metadata record box.

All generated metadata Records link back to EnMaSSe for previewing of the ingested data, for project records this will redirect to the project pages (contextual sidebar options will allow browsing of associated data) and dataset records will redirect to the relevant manage dataset page.

Limitations¶

There are a couple of metadata record creation limitations with the current system.

Records can’t be edited after creation, this is by design as published records should be persistent. One use case in support of this is where your data is cited in a publication, if the metadata record was then changed it could invalidate their work.

Record generation isn’t configurable, this was out of scope - if specific minor changes are required the administrators may be able to update the record generation.

Data Sources¶

Data sources are ways of getting data into the system and how that data should be processed to fit the data configuration.

EnMaSSe has been designed to be as flexible as possible by providing generic ways of ingesting data and allowing end users to provide custom python scripts to process the data,

Choose the data source that is easiest for you to use:

• Web form/manual data sources simply allow for manual data entry (web form) without any streamed data ingestion.
• Pull from external file system data sources ingest data from a folder on a web accissible server, this is a very generic and is anticipated to be the most commonly used data source.
• SOS data sources (Sensor Observation Service) provide data ingestion from an external SOS server, all data for every sensor is ingested and it is up to the processing script to retrieve and store the relevant data.
• Push to this website allows 3rd parties to develop software tools that integrate directly with the EnMaSSe system, allowing them to provide the data for ingestion however they like.
• Output from other datasets allows for chained processing and storage. Such as the data for many datasets is ingested as a single large file in a ‘parent’ dataset and each ‘child’ dataset then ingests and further processes the data individually.

Data sources aren’t conceptually too difficult they just need to get the data from somewhere and process it to fit the provided data configuration.

The complicated part is that a custom Python script is needed to do the processing (Requires a developer/programmer), so most users will need to enter a description of their processing needs for the administrators to help with.

Data Configuration¶

Data configuration is about storing your data as efficiently as possible and identifying which fields potential users will want to search on.

The data configuration also sets up a manual data entry form where the custom fields and standardised fields are the fields on that form.

Efficiency & Searching¶

Think of each data ingestion as being a single file (eg. basic text file that you open in notepad), the custom fields and standardised fields you set up in data configuration are bits of data that you want read from the file and remembered so that they are easy to search.

Basically the less indexes used, the lower the processing and storage overheads, but the whole point is to make the data as reusable as possible so provide indexes on fields that are likely to be searched.

Storage and data configuration has been implemented this way to make EnMaSSe as efficient and scalable as possible, this is necessary as research data grows quite large (eg. our first import was over 8 million data points).

For example, when deciding which fields should be searchable - potential users would almost always search for the location and time (which is why they are compulsory) but they would probably also search on relevant data such as the temperature, it is unlikely that they would search on quality assurance or other minor/associated data though.

Process to work out how to store your data¶

While modelling data we want to store it in a way that makes sense to the researcher, so that it makes sense to the researcher when it comes time to reuse the data.

It is good practice to store the original data as well as the processed data or results, this is both incase the processing had errors and the original data is needed to recover as well as to allow other researchers to re-process the original data in the way they need.

The first step is to think about what data is being collected, how the data is originally stored and what needs to be searchable.

Now identify the different methods that are being used to collect the data, this may include:

• Different ways of collecting data, such as manual observation forms or sensors.
• Different methodology being used for the same data collection method, such as temperature sensors placed under trees vs temperature sensors placed in the sun for measuring the differences.

Each method just identified is a data collection method in the EnMaSSe system, now we need to work out the data configuration for each method.

Now we need to break the data your methods collect up to fit the EnMaSSe data configurations:

1. In most cases the raw data should be stored as a file of some kind so add a custom field of type file (it is good practice to permanently store the raw data for future needs). One possible exception is when the web form/manual data source is selected.
2. Identify what in your data needs to be searchable. This will typically be the final, processed result and is generally a common measurement (eg. temperature, weight, humidity) rather than associated information such as quality assurance.
3. Where available add fields that need to be searchable as standardised data fields section, if there is no applicable standardised field add them as custom fields.

After following this process you should now have your data logically modelled in a fine-grained manner that promotes efficiency of storage, efficiency of searching and the flexibility for researchers to reprocess and/or reuse your data in ways you don’t even anticipate.

Limitations¶

There are some limitations with the current EnMaSSe implementation:

• Data is stored as flat files, so indexing (data configuration) needs to be done right from the start - it is possible to reprocess and re-index data, but this shouldn’t be the norm.
• It is likely that users will require new standardised fields and template which requires constant administrator support.
• Standardised fields can only be added once, so if the user has two of the same type of data they will need to add the second as a custom field. We have since rethought this and it would make more sense for standard measurements such as temperature to be added as custom field types and standardised fields be used as a template like system for data configurations.

1. Project Creation¶

The first step is to create a new project (click the New Project item in the main menu), this consists of a creation wizard that pre-fills fields based on the selected project template and the associated research grant as well as collecting the primary contacts.

Project templates allow for pre-filling of any/all fields, and provides the maximum time-savings when there are projects that are similar - equivalent functionality can be achieved using duplicate project in the sidebar.

Figure 3: New Project page (Templates are hidden by default)

2. General Details¶

After project creation the general details page is displayed and collects metadata including the title, associated grant and information about all associated people, groups and organisations.

If a research grant was provided in the project creation step:

• Project title is pre-filled with the grant title as a starting point
• Any additional people associated with the research grant are added to the people section.

Figure 4: General details page

3. Descriptions¶

The descriptions page provides plenty of space to enter the brief and full descriptions of the project as well as optional notes.

Detailed methods used within the project should not be entered in this section.

Figure 5: Descriptions page

4. Information¶

Collects the bulk of metadata (information about the collected research data) for the ReDBox record such as keywords, research codes, dates, location and other related information.

If a research grant was selected the date from and date to fields will be prefilled (when available).

Figure: Information page

5. Methods¶

The methods page sets up ways of collecting data (data sources), what the data is or its type (data configuration) as well as collecting the methods name (used to generate record titles of associated datasets) and description of the detailed methodology (added as a note description to records).

Adding methods uses a simple wizard that allows selection of a method template. Method templates pre-fill any/all data in methods and their associated datasets.

The type of data being collected allows configuration of what data is collected and how that data is indexed:

• Most methods will store raw data as a file and index specific information so it is searchable.
• Standardised fields are provided for common data types (eg. temperature, humidity, etc).
• Using the standardised fields will make the indexed data searchable globally within the data storage.
• Data configuration allows full configuration of the data types as well as how to display the fields in a web form.

Selection of the data source specifies how data will be ingested but configuration of the data source is done in the datasets step.

Figure 6: Methods page

6. Datasets¶

Each dataset represents an individual collection of data with an associated metadata record (metadata record generation can be disabled).

Adding datasets uses a simple wizard where the data collection method is selected as shown in figure 7 below.

The dataset page collects the following data:

• Whether to create a metadata record and when the record should be published.
• Location of the data, the location may be a set location or an offset from a location where that is more relevent. For example it is more relevent that the sensor shown is 1m from the base of the artificial tree.
• Configuration of the data source.

Each data source is configured differently but will usually require the data location, when to sample and how to process the found data.

Figure X: Datasets page.

7. Submit¶

Submit provides full project validation and an overview of the generated records and data ingesters. The project has four states:

• Open - The initial state when a project is created, the creator and administrators have read/write access. The creator can also share permissions with other users.

• Submitted - When the project is submitted by the creator it is ready to be reviewed by the administrators and either approved or reopened. A project can only be submitted when there are no validation errors. In the submitted state creators have read access and administrators have read/write access.

• Approved - When an administrator approves the project:

  • Metadata records are exported to ReDBox.
  • Data ingesters are configured and started.
  • The project can no longer be modified, the creator and administrators only have read access.

• Disabled - This state represents the end of the project, when an administrator disables an approved project it disables all ingesters (no more data will be ingested).

The generated record for each dataset can be viewed, edited or reset. Viewing a dataset record is exactly the same as general details, descriptions and information all on a single form.

Figure 7: Submit page.

Open¶

The open state is for initial setup of the project including all configurations ready to create metadata records and setup data ingestion.

Once the project has been fully configured the user should go to the submit page to and click the submit button to indicate that the project is ready for administrator approval.

Submitted¶

The submitted state is for administrators to check the project configurations are correct and the project is ready to be approved for metadata record generation and data ingestion set up.

Once the administrator has checked that the configurations are correct they should press the approve button on the submit page. Pressing the approve button starts the metadata record generation and export as well as setting up and starting the data ingesters.

If the administrator requires the creator to update the configurations they should press the reopen button on the submit page.

Active¶

An active project has had all metadata records generated and exported and the data ingesters are currently running.

Once a project has come to the end of its life either the project creator, administrator or user with adequate permission shared should press the disable button on the submit page. Pressing the disable button will deactivate data ingestion, all data and metadata records will be maintained.

Disabled¶

The disabled state is where the project is finished but all data and metadata records are persisted.

If the project needs to be re-activated (there is more data to ingest) the re-enable button on the submit page should be pressed by the creator, administrator or other user with adequate shared permissions.

If this is a project that was set up wrong to begin with and has no valid data, the super administrator can delete it. Once the project is deleted it cannot be recovered, this functionality is only intended for administration and cleanup purposes and would never be used in an ideal world.

Interface/Application¶

Configuration of the provisioning interface is provided through the production.ini file located in the base provisioning interface directory (<install location>/src/jcu.dc24.provisioning/production.ini).

Application specific configuration values (Compulsory items are bold):

Other settings can be changed as per their documentation:

• General pyramid (the framework) configurations: http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/project.html#myproject-ini
• SQLAlchemy (database): http://docs.sqlalchemy.org/en/rel_0_8/core/engines.html
• pyramid_beaker add-on settings: http://docs.pylonsproject.org/projects/pyramid_beaker/en/latest/
• Logging configuration: http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/logging.html

Web Server & Shibboleth¶

Shibboleth is a federated authentication service that sits in front of the application and injects user authentication headers.

Shibboleth requires a web server that supports CGI, JCU eResearch has chosen the NGINX webserver using Fast CGI to support Shibboleth.

The application login address is located at /login/shibboleth, shibboleth should be configured to handle authentication at this address and redirect back to the application when complete.

The EnMaSSe provisioning interface is based on Pyramid using a UWSGi container for production deployment.

SSH Access¶

A new user should be created on the ReDBox server with a public key and write access to the enmasse-alerts folder (see alerts configuration). The actual steps to do this are operating system dependent but for a linux machine it would be something like:

1. useradd <username>
2. passwd <username>
3. Copy/paste your public key into ~/.ssh/authorized_keys
4. cd <enmasse-alerts location>
5. sudo chown <username>:<username> ./

Once the new user has been created and has write permission the provisioning interface configuration file needs to be updated with the connection details:

1. Open <install location>/src/jcu.dc24.provisioning/production.ini.
2. Edit redbox.ssh_host to the server IP address or domain name (eg. example.com.au).
3. Set redbox.ssh_port as configured on the server (default is 22).
4. Set redbox.rsa_private_key to the fully qualified location of the private key that corresponds to the public key added above.
5. Set redbox.ssh_username as the user created above.
6. Set redbox.ssh_password as the SSH private key password (or leave blank).
7. Set redbox.ssh_harvest_dir to the location of the enmasse-alerts folder (eg. /opt/deployment/redbox/home/harvest/enmasse-alerts)
8. Update redbox.identifier_pattern to an appropriate identifier prefix for your organisation (eg. JCU uses jcu.edu.au/collection/enmasse)

Alerts Configuration¶

There are 2 steps to configuring the EnMaSSe alerts harvester:

1. Copy the enmasse-alerts folder from the root folder of the github repository and add it to <redbox installation dir>/home/harvest/
2. Update the <redbox installation dir>/home/system-config.json file as illustrated below:

{
        ...
        "houseKeeping": {
        "config": {
                "quartzConfig": "${fascinator.home}/quartz.properties",
            "desktop": true,
            "frequency": "3600",
            "jobs": [
                    ...
                    {
                        "name": "newalerts-enmasse",
                        "type": "external",
                                "url": "http://localhost:${jetty.port}/redbox/default/hkjobs/newalerts.script",
                                        "timing": "0 0 1 * * ?"
                        }
                ]
                }
    ...
    "new-alerts": {
        "alertSet": [
                {
                "name": "EnMaSSe Provisioning",
                "path": "${fascinator.home}/harvest/enmasse-alerts",
                "harvestConfig": "${fascinator.home}/harvest/enmasse-alerts/config/enmasse-dataset.json",
                "handlers": {"xml": "XMLAlertHandler"},
                "baseline": {
                        "workflow_source": "EMAS Alert"
                        "viewId": "default",
                        "packageType": "dataset",
                        "redbox:formVersion": "1.5.2",
                        "redbox:newForm": "true",
                        "redbox:submissionProcess.redbox:submitted": "true",
                        "xmlns:dc": "http://dublincore.org/documents/2008/01/14/dcmi-terms/",
                        "xmlns:foaf": "http://xmlns.com/foaf/spec/",
                        "xmlns:anzsrc": "http://purl.org/anzsrc/",
                        "dc:type.rdf:PlainLiteral": "dataset",
                        "dc:type.skos:prefLabel": "Dataset",
                        "dc:language.dc:identifier": "http://id.loc.gov/vocabulary/iso639-2/eng",
                        "dc:language.skos:prefLabel": "English"
                },
                "timestampFields": ["redbox:submissionProcess.dc:date"],
                "XMLAlertHandlerParams": {
                        "configMap": {
                                "xml": {"xmlMap": "${fascinator.home}/harvest/enmasse-alerts/config/enmasseXmlMap.json"}
                        }
                        }
                        }
                ],
        "baseline": {
                "viewId": "default",
                "packageType": "dataset",
                "redbox:formVersion": "1.5.2",
                "redbox:newForm": "true",
                "redbox:submissionProcess.redbox:submitted": "true"
        }
    ...
}

Templates¶

Everything on the provisioning interface is displayed through templates in the <installation directory/src/jcu.dc24.provisioning/jcudc24provisioning/templates>:

• custom_widgets are templates developed for specific situations or pages and aren’t related to the frameworks used (deform).
• widgets are default framework (deform) templates, most have been modified in some way (mostly for placeholder text, help text and descriptions).
• templates in the base directory are individual pages or part of the framework for rendering pages.

custom_widgets and widgets control how form elements are displayed more-so than what is displayed such as a text input is displayed using the help, description and value as provided by the model (eg. text isn’t hard-coded).

Templates in the base directory are mostly hard-coded such that you can directly edit the text seen on the website.

Models (Database & Form)¶

Models are broken into 2 basic classifications:

1. Project models are held in src/jcu.dc24.provisioning/jcudc24provisioning/models/project.py and describe how all forms are displayed as well as all project related database tables.
2. src/jcu.dc24.provisioning/jcudc24provisioning/models/website.py contains general website forms and database tables (eg. authentication models).

For the most part you should be looking at:

• ca_help/help sets the text or HTML code under the ? symbol for that form element.
• ca_description/description set the text/HTML code that is displayed permanently below form element titles.
• ca_title/title sets the displayed title.
• ca_placeholder/placeholder sets the greyed out text that disappears when clicked.
• ca_force_required makes a form element required.
• If you see ca_child_... it means that this element is a sequence (can add elements by clicking the add button) and the added value will be used on the item added rather than the sequence itself.
• If you see ca_group_... it is grouping this and following elements together (for display purposes) and the set value will be used on the grouping rather than the element itself.

Specifics of all models and the underlying frameworks is relatively complicated, if you would like more information please refer to the developer guides.

Note: All changes to models won’t take effect until the application is restarted.

Views (Display & Logic)¶

Views are also split between project pages and general website pages:

• src/jcu.dc24.provisioning/jcudc24provisioning/views/view.py contains general website pages such as the dashboard, login and help.
• src/jcu.dc24.provisioning/jcudc24provisioning/views/workflows.py contains project specific views such as creating new projects, configuring the project and managing project data.

Page help is set by the page_help value passed into self._create_response(page_help=<help>), this value is usually set in a page_help variable at the top of the view.

Menu text for project pages is found at the top of the workflows.py page, contextual options are set in WORKFLOW_ACTIONS and project configuration menus are set in WORKFLOW_STEPS:

• title sets the menu text itself.
• page_title sets the page heading as well as the browser tab text.
• tooltip sets the pop-up text when you hover the mouse over the menu.

For more information please refer to the developer guide.

Note: All changes to views won’t take effect until the application is restarted.

Stylesheet¶

The stylesheet is located at:

src/jcu.dc24.provisioning/jcudc24provisioning/static/template.css.

Stylesheets are used in almost all websites and there are many CSS guides available on the web for reference.

A quick hint is to right click on the element you want to change and click inspect element or similar (supported in most modern browsers), look for a CSS button and you will see what/how the website elements look is being set.

Project Templates¶

At the time of writing there is no administration interface for project templates and the database will need to be edited directly.

To make a project into a template a new entry needs to be added to the project_template table linked to the relevant project.

Project templates should have an appropriate name and description and they can be grouped by using the same category.

All fields in the associated project will be pre-filled on any project that uses this template.

Method Templates¶

At the time of writing there is no administration interface for method templates and the database will need to be edited directly.

To make a method into a template a new entry needs to be added to the method_template table linked to the relevant method.

Method templates should have an appropriate name and description and they can be grouped by using the same category.

Method templates may also be linked to a dataset.

All fields in the associated methods and datasets will be pre-filled on any method that uses this template.

Standard Data Configuration Fields¶

At the time of writing there is no administration interface for method templates and the database will need to be edited directly.

To make a data configuration into a standardised field its method_schema row needs to be updated with a template_schema value of 1.

Standardised fields are hierarchical such that each data configuration may have many parents each of which add their associated method_schema_field’s and parent method_schema’s.

This means that similar data configurations can be reused for similar data streams and their data would be cross searchable. Or common data configurations could be standardised so that no custom configuration is needed at all for compatible methods.

Adding Data Source’s¶

The most useful and generic data source currently implemented is the PullDataSource, so throughout this explanation we will use it as the example.

Custom Import Scripts¶

See Ingester Post Processing Scripts in the Ingester Platform developers guide.

Adding New Custom Field Types for Data Configuration¶

1. Add the new field type value and name to models/project.py->field_types
2. Update controllers/method_schema_scripts.py->get_schema_fields() to create the correct:
   • field type
   • python type
   • widget
3. Update controllers/ingesterapi_wrapper.py->process_schema() so that the new field type is mapped to the ingester platform correctly.

Note: Please refer to the ingester platform documentation and source code for adding new data types to the ingester platform.