Fusion Data Platform

Documentation Status

Fusion Data Platform (FDP) is a data framework in Python for magnetic fusion experiments. FDP streamlines data discovery, management, analysis methods, and visualization.

Contents

Fusion Data Platform

Documentation Status

Fusion Data Platform (FDP) is a data framework in Python for magnetic fusion experiments. FDP streamlines data discovery, management, analysis methods, and visualization.

Description and features

  • An extensible software layer that unites data access, management, analysis, and visualization
  • A descriptive data object that users can query to find data and analysis methods
  • Data access tasks (servers, trees, nodes, queries) are handled behind the scenes
  • A collaborative development platform for data analysis tools
  • Built with popular, open-source packages like Numpy and Matplotlib

Example usage

import fdp
nstxu = fdp.nstxu()
nstxu.s204551.logbook()
nstxu.s204551.mpts.te.plot()
nstxu.s204551.equilibria.efit02.kappa.plot()

nstxu is a data object that abstracts the NSTX-U device with easy access to shots, diagnostics, signals, and data methods. The typical heirarchy is:

<machine>.<shot>.<diagnostic container>.[<possible sub-containers>].<signal>.<method>

Users can discover data containers like mpts, data signals like te, and data methods like plot() with Python’s tab-complete functionality.

Contributing

To contribute to the FDP project, see CONTRIBUTING.rst in the top-level directory or Contributing in the docs.

Lead developers

  • David R. Smith, U. Wisconsin-Madison
  • Kevin Tritz, The Johns Hopkins U.
  • Howard Yuh, Nova Photonics

User Guide

Getting started

This guide is for people who want to use FDP on the PPPL Linux cluster. If you wish to contribute to the FDP project as a developer, see the developer guide.

To use FDP on the PPPL Linux cluster, load the module nstx/fdp (you may need to unload other nstx modules):

[sunfire06:~] % module load nstx/fdp

[sunfire06:~] % module list
Currently Loaded Modulefiles:
1) torque/2.5.2      5) idl/8.2           9) java/v1.6
2) moab/5.4.0        6) nstx/treedefs    10) nstx/mdsplus5
3) ppplcluster/1.1   7) nstx/epics       11) nstx/fdp
4) freetds/0.91      8) nstx/idldirs

Verify that python points to /p/fdp/anaconda/bin/python:

[sunfire06:~] % which python
/p/fdp/anaconda/bin/python

If python does not point to /p/fdp/anaconda/bin/python, then PATH contains to a different python distribution. In this case, you need to modify PATH so /p/fdp/anaconda/bin is the first python distribution in PATH.

Finally, you can launch python and import the FDP package:

[sunfire06:~] % python
Python 2.7.10 |Anaconda 2.3.0 (64-bit)| (default, Sep 15 2015, 14:50:01)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import fdp
>>>

See Usage Examples to learn about the capabilities of FDP.

Usage examples

First, import the FDP module:

>>> import fdp

Initiate a machine instance

Define a NSTX machine instance:

>>> nstxu = fdp.nstxu

Shots are added as referenced. For instance, without previous reference to 139980, you can enter:

>>> nstxu.s139980.chers.plot()

Add shots to the NSTX instance:

>>> nstxu.addshot(140000)

or a shotlist:

>>> nstxu.addshot([141400, 141401, 141402])

or by XP:

>>> nstxu.addshot(xp=1048)

or by date (string or int YYYYMMDD):

>>> nstxu.addshot(date=20100817)

List shots presently loaded:

>>> dir(nstxu)

or:

>>> nstxu.listshot()

Get a custom shotlist:

>>> my_shotlist = nstxu.get_shotlist(xp=1032)  # returns numpy.ndarray

Developer Guide

Getting started

This guide is for developers who want to contribute to the FDP project, and this guide describes the development workflow on the PPPL Linux cluster. If you simply want to use FDP on the PPPL Linux cluster, see the user guide.

The FDP code repository is hosted on GitHub: https://github.com/Fusion-Data-Platform/fdp

To participate in the FDP project as a developer, you must create a GitHub account. The FDP project uses GitHub and Git for collaborative development and version control.

Configure Git

On the PPPL Linux cluster, load the module git/1.8.0.2 (on Red Hat 6 systems, use git/2.4.2):

[sunfire08:~] % module avail git
--------------------- /usr/pppl/Modules/modulefiles -------------------
git/1.7.4.1(default)     git/1.8.0.2      git/2.4.2

[sunfire08:~] % module load git/1.8.0.2

[sunfire08:~] % module list
Currently Loaded Modulefiles:
1) torque/2.5.2      3) ppplcluster/1.1
2) moab/5.4.0        4) git/1.8.0.2

You may want to add the module load command to your shell start-up files: ~/.cshrc for csh/tcsh or ~/.bash_profile for bash.

Next, you must configure Git with your name and email (the same email associated with your GitHub account):

[sunfire08:~] % git config --global user.name "John Doe"
[sunfire08:~] % git config --global user.email "JohnDoe@email.com"

Also, you may want to set a default editor (e.g. vi, emacs, nedit) for Git comments:

[sunfire08:~] % git config --global core.editor nedit

You can inspect your Git configuration in the file ~/.gitconfig. For more information about Git configuration, see https://help.github.com/articles/set-up-git/ or https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup

Clone the FDP repository

Git clones repositories into a new directory in your current directory. In the right column of the FDP repo page (https://github.com/Fusion-Data-Framework/fdp), you can find the HTTPS URL (https://github.com/Fusion-Data-Framework/fdp.git) to clone FDP to your local directory

[sunfire08:~] % ls -d fdp
ls: fdp: No such file or directory

[sunfire08:~] % git clone https://github.com/Fusion-Data-Framework/fdp.git
Cloning into 'fdp'...
remote: Counting objects: 619, done.
remote: Total 619 (delta 0), reused 0 (delta 0), pack-reused 619
Receiving objects: 100% (619/619), 783.01 KiB, done.
Resolving deltas: 100% (279/279), done.

[sunfire08:~] % ls -d fdp
fdp/

Cloning via SSH is also feasible: https://help.github.com/articles/set-up-git/#next-steps-authenticating-with-github-from-git

Finally, add your new FDP directory to the PYTHONPATH environment variable:

[sunfire08:~] % setenv PYTHONPATH ${HOME}/fdp:$PYTHONPATH

[sunfire08:~] % echo $PYTHONPATH
/u/drsmith/fdp:<other directories>

You may want to add this action to your shell start-up files, as described above. In bash, use the export command to set PYTHONPATH.

Git workflow for FDP development

(1) Create a development branch (here, we call it devbranch) and checkout the new branch:

[sunfire08:~] % cd fdp

[sunfire08:~/fdp] % git branch
* master

[sunfire08:~/fdp] % git branch devbranch

[sunfire08:~/fdp] % git branch
devbranch
* master

[sunfire08:~/fdp] % git checkout devbranch
Switched to branch 'devbranch'

[sunfire08:~/fdp] % git branch
* devbranch
master

Devbranch initializes as a copy of master. git branch lists branches in your local repository, and the asterisk denotes the active branch. You can switch between local branches with git checkout <LocalBranchName>.

(2) Push devbranch to the remote FDP repository at GitHub (you may need to enter your GitHub username and password):

[sunfire08:~/fdp] % git push origin devbranch
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/Fusion-Data-Framework/fdp.git
 * [new branch]      devbranch -> devbranch

devbranch is now listed in the FDP repository at GitHub. origin is the alias for the remote GitHub repository. You can view your remote repositories and aliases with git remote -v.

(3) Proceed with FDP development within devbranch: commit changes, add/delete files, and push updates to GitHub.

As you complete small tasks, you should commit changes to your local repository with git commit -a -m '<mymessage>'. Also, each commit requires a short message describing the changes:

[sunfire02:~/fdp] % git commit -a -m 'added dictionary rows in logbook.py'
[devbranch bb6c58a] added dictionary rows in logbook.py
1 file changed, 16 insertions(+), 21 deletions(-)

If you do not specify a commit message with -m option, then Git will open your default editor and ask for a commit message (see Configure Git above). The -a option commits all file changes throughout the branch index, not simply your current directory. The branch index is the list of files Git tracks in the branch. git commit -a tracks changes to files in the branch index, so you must add new files to the index and remove deleted files from the index. You can view the branch index with git ls-files, and you can add new files to the index and remove deleted files from the index with git add -A:

[sunfire02:~/fdp] % touch temp.py

[sunfire02:~/fdp] % ls temp.py
temp.py

[sunfire02:~/fdp] % git ls-files temp.py

[sunfire02:~/fdp] % git add -A

[sunfire02:~/fdp] % git ls-files temp.py
temp.py

Note that temp.py appeared in the index only after the command git add -A. Similarly, deleted files stay in the index until the git add -A is given.

When you complete a large task, you should “push” changes to the devbranch on GitHub with git push:

[sunfire05:~/fdp] % git push origin devbranch
Counting objects: 10, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.30 KiB, done.
Total 6 (delta 3), reused 0 (delta 0)
To https://github.com/Fusion-Data-Framework/fdp.git
    129c5d9..a166825 devbranch -> devbranch

Again, “origin” signifies the branches on the remote GitHub repo.

(4) While you are working locally in devbranch, others may be modifying master at GitHub. When you are ready to merge devbranch into master, you should first merge the latest version of master from GitHub into your local devbranch. To retrieve the latest version of master from GitHub, use git fetch:

[sunfire05:~/fdp] % git fetch origin master
From https://github.com/Fusion-Data-Framework/fdp
* branch            master     -> FETCH_HEAD

Next, verify that you are in devbranch and merge origin/master into devbranch:

[sunfire08:~/fdp] % git branch
* devbranch
master

[sunfire05:~/fdp] % git merge origin/master

Next, push your local devbranch to devbranch on GitHub:

[sunfire05:~/fdp] % git push origin devbranch

Finally, on the GitHub website, in the devbranch area, submit a pull request to pull devbranch into master.

Package reference

Subpackage fdp.classes

Subpackage fdp.methods

Class Fdp

Class Machine

Class Logbook

Class Shot

Class Container

Class Signal

Class Node

Module fdp.classes.globals

Module fdp.classes.factory

How to contribute

Code

Thank you for your interest in the FDP project. To contribute to the FDP code base, fork the repo at Github and submit pull requests with your contributions. Read more:

Style

Try to follow the PEP8 style guide. To scan for PEP8 conformance, run make lint in the top-level directory.

Makefile utilities

The top-level Makefile contains several utilities for generating docs, code style/quality reviews, and versioning:

$ make help
help                 show this help message
docs                 build HTML and PDF documents
docs-html            build HTML documents
docs-pdf             build PDF documents
test                 run pytest in current Python environment
coverage             check test coverage and show report in terminal
coverage-html        check test coverage and show report in browser
lint                 run flake8 for code quality review
autopep              run autopep8 to fix minor pep8 violations
authors              update AUTHORS.txt
bump-major           update AUTHORS and CHANGELOG; bump major version and tag
bump-minor           update AUTHORS and CHANGELOG; bump minor version and tag
bump-patch           update AUTHORS and CHANGELOG; bump patch version and tag
clean                remove all build, docs, and Python artifacts
clean-docs           remove docs/build
clean-pyc            remove Python file artifacts

Authors

Lead developers:
    David R. Smith
    Kevin Tritz
    Howard Yuh

Commits from authors:
   289  David R. Smith
    37  ktritz
    22  dmkriete
     2  hyyuh
     1  Howard Yuh

Commits from obsolete FDF repository:
   261  David R. Smith
    41  Kevin Tritz
    17  ktritz
    10  Howard Yuh
     7  John Schmitt
     4  hyyuh
     2  jcschmitt

Change Log

Release v0.2.3 – 2017-04-20

00e03db  bumpversion, AUTHORS, and Makefile
28119ae  Bump version: 0.2.1 → 0.2.2
64ee3f5  updated CHANGELOG.rst and AUTHORS.txt
c93f1c4  bumpversion and Makefile

Release v0.2.2 – 2017-04-20

c93f1c4  bumpversion and Makefile

Release v0.2.1 – 2017-04-20

4745b51  CHANGELOG generation in Makefile
f5a0585  changed CHANGELOG to .rst
40cdcfa  implemented doc hosting at readthedocs.io
c2de060  Merge pull request #55 from drsmith48/drs-dev

Release v0.2.0 – 2017-04-19

4abf46b tests
6564d4c changed 'fdp_globals' to 'globals' throughout repo
dea305b changed fdp_globals import
ea12a72 tests
54cb3a7 Merge branch 'drs-dev' of github.com:drsmith48/fdp into drs-dev
74c271f tests
ff08bd2 tests
63ce140 tests
af97cda testing for logbook.py and machine.py
8fbaf30 pytest setup
534f00a created new test directory and configured for pytest; run 'pytest' or 'make test'
3f59f74 minor
be1449c minor
a8e38e3 fixed axis removal from multi-dim signals
9400ca8 Merge branch 'master' into drs-dev
8e2d471 Merge pull request #54 from drsmith48/plotpsirz
33c025e debugging eq.plotrz.plot()
7adfae2 bfield() method for equilibria
ddd16a5 minor fixes
ea8852b changed 'zero level' indices to floats
09f779a changed iteritems to iter()
7d32fd6 docs
bab6bbc docs
fa1a1fb docs
17e9578 removed pushes from bumps in Makefile
a06ac9b run 'authors' recipe prior to 'bump-*' recipes in Makefile

Release v0.1.4 – 2017-04-07

a2842c8 added automatic CHANGELOG.txt updates with 'bump-*' recipes in Makefile

License

Copyright (c) 2015-2017  David R. Smith, Kevin Tritz, Howard Yuh

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.