Introduction to CREDO
CREDO is a Python toolkit for system testing,
benchmarking and analysis of Modelling tools based on the
StGermain framework,
such as Underworld.
It is distributed with the stgUnderworld application bundle, but in future
it’s planned that it will also be possible to be obtained and installed
independently.
Core design goals
CREDO’s core design goals are to:
- Support the effective development, inquiry and maintenance of a suite
of benchmarks that test the scientific features, numerical accuracy, and
computational performance of StGermain-based codes.
- Significantly enhance the ease and effectiveness of performing scientific
analysis of the results of StGermain modelling applications.
Both these tasks are possible using a combination of command lines and custom
3rd party tools, both proprietary and open source - but the goal of CREDO is to
provide an integrated system that makes doing both of them to a high level
much easier and clearer - and also in a readily repeatable manner so that
the status of a code as it evolves can be readily checked (e.g. as part of
a continuous integration system).
To meet these goals, we chose the Python scripting and programming language,
explained more in the Language choice - why Python? section below.
How CREDO fits into the workflow of using the Underworld modelling code
CREDO still runs StGermain codes such as Underworld “under the hood”,
but it’s role is to:
- Construct XML data files based on the analysis/testing the user asks for
in a CREDO script;
- Launch the necessary StGermain jobs required to perform the analysis;
- ..and finally provide access to the results created by the model at a
high-level, facilitate post-processing of these results, and perform any
post-run analysis required by tests/benchmarks.
This is summarised in the diagram below.
As such, it currently explicity doesn’t provide run-time access to the
StGermain objects (written in a custom framework implemented in the C language).
Rather, it works in detail with the StGermain XML format for defining models,
and the defined StGermain data output formats.
CREDO’s Benchmarking Goals / Motivation
There has long been an interest in a more systematic way of testing both
the software performance, and scientific/numerical reliability of the
StgUnderworld code – for example the issue was explored in a
paper presented at the APAC05 conference [FarringtonEtAl2005].
The benefits of such a system are:
- A clear and up-to-date record of the performance and reliability of the
code to present to scientists interested in using it or comparing it
to similar codes. The chosen benchmarks would be of a “scope” of much
more interest to modellers and scientists than the detailed unit
and integration tests. This ties in to the concept of ‘reproducible research’
([FomelHennenfent2007]).
- A definitive record to examine the changing performance of the code over
time, which has always been a big concern among the existing research
community using the code;
- As a corollary to the above, the system would give the development
team a clear and timely warning if a design change accidentally
negatively affected performance or accuracy – which has been a
problem in the past.
- The ability to use the science benchmark suite and framework
to quickly evaluate the performance of Stg-Underworld on new
hardware systems, to assist purchasing decisions and new collaborators.
- The goal would be to automate as much as possible the regular
collection of these benchmarks (initally proposed as weekly),
assessment of them, and publishing of them on-line in a succinct
and meaningful form (IE with key metrics emphasised, and linked
to clear definitions of the benchmark problem).
Scientific analysis of computational codes - core capabilities
To achieve the above goals, the features below are either provided by CREDO, or
under active development:
- Ability to quickly extract and compare observables that are produced during
runs of the code (such as VRMS). This includes testing that an
observable is within a given range at a given time.
- Ability to extract information about the 2D or 3D mesh, field
or swarm data produced by a StGermain application.
- For testing purposes, this includes functions like checking that a field
produced by the model run compares as expected with a reference field,
loaded in from a separate data file.
- Ability to record, and compare performance metadata about the code, such as
time that simulations ran for, memory usage, etc.
Examples of doing this sort of analysis and testing is provided in the
Examples of how to use CREDO section.
Language choice - why Python?
The CREDO code is written in the dynamic scripting and programming language
Python. Python was chosen as the implementation language because of:
- It’s ability to run in either interactive or scripted mode:- and thus
facilitate either scripted, repeatable workflows, or interactive exploration;
- The fact that it’s clear, concise syntax and high level of abstraction is
recommended for human developer productivity - while computationally
intensive tasks can be performed in compiled languages and libraries (such as
Underworld itself).
- The fact that it’s a highly portable language between operating systems and
architectures.
- The increasingly stable, feature-rich and wide-ranging set of open source
packages for mathematical and scientific analysis in Python, such as
SciPy,
NumPy,
Matplotlib,
SAGE,
Paraview,
and MayaVI.
The fact that CREDO is written in Python doesn’t prevent you from using a
favourite tool or language for your final analysis work - in this case, CREDO is
being written to allow you to extract the needed observables from a set of model
results in common formats such as CSV or XML files.
Footnotes
[FarringtonEtAl2005] | Farrington, R, Moresi, L, Quenette, S, Turnbull, R, &
Sunter, P, 2005, ‘Geodynamic benchmarking tests in HPC’, Presented at the
2005 APAC Conference, Gold Coast, Australia. |