AT-TPC DAQ Help¶

GET software components¶

There are two programs, in particular, that need to be running for each CoBo. They are:

getEccSoapServer

This program controls the CoBo. It sends the configuration to the CoBo and tells it when to start and stop acquisition.

dataRouter

This program records the data.

The web interface controls the system by acting as a client for the getEccSoapServer. It does not communicate with the dataRouter directly.

CoBo state machine¶

The ECC server controls the CoBo using the model of a state machine. This means that that CoBo can be in one of several well-defined states, and to change from one state to another, it will undergo a well-defined transition. The state machine for the CoBo looks like this:

The ellipses represent the different states that the system may be in, the arrows along the right side show the forward state transitions, and the arrows along the left show the reverse state transitions.

Config files¶

Each forward transition requires a particular part of the configuration file. This is why we have three config files for each setup (or, alternatively, two true files and one symbolic link). The expected files are named:

• describe-[name].xcfg for the describe step
• prepare-[name].xcfg for the prepare step
• configure-[name].xcfg for the configure step

These names will be shown stripped of their prefix and suffix in the DAQ interface. For example, a file called describe-cobo0.xcfg will be shown as simply cobo0.

Web-based GUI¶

The interface to the system is a web application written in Python 3 using the Django web framework for the back-end with Bootstrap providing the front-end. The structure of the code is described briefly in Developer documentation and in comments directly in the code itself.

As Django apps can be a bit tricky to serve, the app has been structured to run inside Docker containers. The Dockerized version of the app can be built using the Docker compose utility like this:

docker-compose build

Requirements¶

git clone https://github.com/attpc/attpc-daq.git

Creating the environment file¶

There are a few environment variables that need to be set to system-dependent values inside the Docker container. Several of these variables provide encryption keys or passwords, so this environment file is not in the Git repository (and it should never be committed to the repository!).

Create a file in the root of the repository with the following values. Remove the comment strings (starting with #) before saving it.

DAQ_IS_PRODUCTION=True         # Tells the system to use the production settings, rather than debug.
POSTGRES_USER=[something]      # A user name for the PostgreSQL database. Set it to something reasonable.
POSTGRES_PASSWORD=[something]  # A secure, random password that you will not likely need to remember.
POSTGRES_DB=attpcdaq           # The name of the database for PostgreSQL
DAQ_SECRET_KEY=[something]     # A secure, *STRONG* random string for Django's cryptography tools.

The user name for PostgreSQL is not important. Just set it to something reasonable. The remaining two things to be filled in are the PostgreSQL database password and the Django secret key. Set these both to long, random strings of characters since you will not need to remember them.

One way to generate these random strings is the following Python script:

from __future__ import print_function
import random

chars = 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)'
sr = random.SystemRandom()
key = ''.join(sr.choice(chars) for i in range(50))
print(key)

Or, if you prefer a one-liner:

python -c "import random; chars = 'abcdefghijklmnopqrstuvwxyz0123456789%^&*(-_=+)'; sr = random.SystemRandom(); print(''.join(sr.choice(chars) for i in range(50)))"

Building the containers¶

Once you’ve installed Docker and docker-compose, open a terminal in the root of the repository. This is the directory with the docker-compose.yml file. The Docker images can then be built with the command:

docker-compose build

This will create a set of Docker images and install all of the software’s dependencies inside them. This will require an internet connection.

Starting the app¶

Start all of the containers and the virtual network connecting them by running:

docker-compose up

This will instantiate the containers and start them, and then it will start printing the standard output from the containers. Keep this terminal window running to see the output as the program runs. If you want to quit the program later, press Control-c in this terminal.

The first time you run the code, it will need to do some housekeeping to get set up. This may take a minute or so. When the output printed to the terminal slows down or stops, continue with the next steps.

First-run setup¶

When the code is freshly installed, the database that backs the web app will be empty. We need to create a user in the web app so that we can log in and set up an experiment. To do this, open a new terminal and run this command:

docker exec -it attpcdaq_web_1 python manage.py createsuperuser

If we break this command down into parts, it opens a TTY inside the container running the Django app (docker exec -it attpcdaq_web_1) and runs the Django manage.py script to create a superuser account (python manage.py createsuperuser). It will prompt you for a username and password, which you should choose and remember for later.

Once you’ve made a superuser account, open a browser to http://localhost:8080/admin to access the Django admin interface. Log in with the username and password you just set up. This will put you on the Admin page.

This page allows you to access the internals of the DAQ web interface and directly change the contents of its database. For now, click on “Experiments” under the “DAQ” header and then click the “Add Experiment” button on the next page.

Click the green plus to add a new regular user account.

Also enter a name for the experiment. Data will be written into a directory with this name at the end of each run. Spaces are ok in this name. Finally, click “Save” to create the experiment.

Once you’ve finished this, click “Log Out” in the upper right to log out of the admin interface.

Starting the remote processes¶

Under macOS, the remote GET processes are managed by launchd, the operating system’s main management process. It will automatically re-launch the processes if they fail, and it will coordinate logging of the processes’ standard outputs to a log file.

The behavior of launchd with respect to the GET software components is controlled by a Launch Agent plist file. Example plist files are included in the Git repository, but here is an annotated example for the ECC server:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <!-- A label to identify the program -->
    <key>Label</key>
    <string>attpc.getEccSoapServer</string>

    <!-- Any necessary environment variables. This might include settings for paths
         needed for libraries installed using MacPorts, for example.-->
    <key>EnvironmentVariables</key>
    <dict>
        <key>DYLD_FALLBACK_LIBRARY_PATH</key>
        <string>/opt/local/lib</string>
    </dict>

    <!-- The commands needed to start the program. Each element of the command is given in
         a separate <string> tag. The first element should be the full path to the program,
         and the remaining elements give the command line arguments. -->
    <key>ProgramArguments</key>
    <array>
        <string>/path/to/getEccSoapServer</string>
        <string>--config-repo-url</string>
        <string>/path/to/configs/directory</string>
    </array>

    <!-- The working directory for the program. This is important for the dataRouter as it's
         where that program will write the data. -->
    <key>WorkingDirectory</key>
    <string>/path/to/working/directory</string>

    <!-- Where to write the standard out and standard error files. These may be the same file.
         It is probably best to put the logs in ~/Library/Logs since that will allow you to
         view them with the Console application. -->
    <key>StandardOutPath</key>
    <string>/Users/USER/Library/Logs/getEccSoapServer.log</string>

    <key>StandardErrorPath</key>
    <string>/Users/USER/Library/Logs/getEccSoapServer.log</string>

    <!-- Keep the program running at all times, even if there are no incoming connections. -->
    <key>KeepAlive</key>
    <true/>
</dict>
</plist>

A similar file should be created for the data router with the appropriate arguments.

Once plist files have been created, they are conventionally placed in ~/Library/LaunchAgents, and they should be launched on startup if they are in that directory. To launch the programs manually, use launchctl:

launchctl load ~/Library/LaunchAgents/attpc.getEccSoapServer.plist

Manually stopping the programs is very similar. Just replace load with unload in the above command.

This can also be automated for all of the remote computers using, e.g. Apple Remote Desktop.

Easy setup¶

On the status page, click the “Easy setup” link in the left-hand menu column. This will take you to a form that you can fill out to automatically set up the system with some default values. Fill in values for the following fields:

Number of CoBos

How many CoBos are you using? A data router will be created for each one.

Use one ECC server for all sources?

If this is checked, the system will create one ECC server and link all CoBos to it. If this is unchecked, a separate ECC server object will be created for each CoBo.

IP address of first CoBo ECC server

If we’re using one global ECC server, it will have this IP address. If each CoBo has its own ECC server, the first ECC server will get this IP address, and subsequent servers will get this address plus an offset in the last segment. For example, if this address is set to 192.168.1.10, CoBo 0’s ECC server will be at 192.168.1.10, CoBo 1’s ECC server will be at 192.168.1.11, CoBo 2’s ECC server will be at 192.168.1.12, etc.

IP address of first CoBo data router

The address assigned to the data router of the first CoBo. Subsequent CoBos have an address that is incremented by an offset like above.

Is there a MuTAnT?

If so, the system will create a data router object for it.

IP address of the MuTAnT ECC server

If there is only one global ECC server, the MuTAnT will also be connected to that server, so this field will have no effect. Otherwise, the MuTAnT’s ECC server will be found at this address.

IP address of MuTAnT data router

The address where we should look for the MuTAnT data router.

Once you click “Submit”, the system will create all of the necessary objects for this setup.

Manual configuration¶

If you need to tweak the results of the easy setup page, or if you need something more sophisticated than what it provides, you can always set things up manually. Under “Setup” in the left-hand navigation menu, there are links for setting up ECC servers, data routers, and data sources. Each of these leads to a table of the instances of that object that are currently set up. You can add a new instance using the “Add” button in the table header, and instances can be edited or removed using the buttons in each row.

To manually set up the system, you should first create your ECC servers and data routers. Then, create data source objects to link the two. For more information about the model used to describe the system, see Modeling the system in code.

Default run information¶

The default set of information will be recorded for every run, regardless of configuration. This set of fields includes the following:

• A sequential run number
• A run class identifying the type of the run. Options include “Testing”, “Production”, “Beam”, “Pulser”, and “Junk.”
• A title or label for the run
• The date and time when the run started and ended
• The name of the config file(s) used for this run

Adding additional fields¶

In addition to these defaults, any number of custom fields can be added. These fields, known in the DAQ software as observables can be used to record detector parameters like voltages and pressures. These should be set up at the beginning of an experiment, but they can also be added later.

To set up observables, click “Observables” under “Setup” in the left-hand menu. This will bring you to a list of the observables that are currently set up in the system. Add a new one by clicking the “Add” button in the top right corner of the “Observables” panel.

An observable has four properties that you can set:

Name

The name of the measurement. Choose something descriptive, but don’t include units. They will be added later.

Value type

What type of data is this? Options include integer, floating point, and string values.

Units

The units this will be recorded in. This is just for display, and no unit conversions will be done by the software.

Comment

This optional comment will be shown next to the field on the run data sheet for this observable. This could be used to make a brief note of how to take a particular measurement, for example.

Fill these fields in and click “Submit” to add a new observable.

Web GUI status page¶

After logging into the system at http://localhost:8080 or whatever address the system is available at, you will arrive at the main status page:

This page shows an overview of what’s currently happening in the system. It is divided into a set of panels:

Run Information

This panel has details about the current current run, like how long it has been going and what run number is currently being recorded.

ECC Server Status

This panel lists the status of each ECC server the system knows about. The “State” indicator shows what state machine state the ECC server is in (i.e. “Idle”, “Ready”, “Running”, etc.). The “Selected Config” column lists the name of the config file set that will be used to configure the devices. he “Controls” column contains a set of buttons for changing the state of an individual ECC server. These button should only be used for troubleshooting purposes. Finally, clicking the icon in the “Logs” column will display the last few lines of the log file for that ECC server.

Data Router Status

This panel shows the state of all of the data routers the system knows about. The “Online” column shows if the data router process is running, and the “Clean” column shows if the data router’s staging directory contains unsorted files. Both of these should display green checkmarks if the system is ready to take data.

Log Entries

This panel will show the latest error messages from the web interface. This does not include error messages that may be produced by the GET software. You can click on an individual error to get more information and possibly a traceback. Finally, clicking “Clear” will discard all error messages.

Controls

This set of large buttons configures the entire system at once. This is what you should use to control the system. The reset button will step the system back one state. For example, if the system is in the “Ready” state, pressing Reset will step it back to “Prepared”.

Selecting a configuration¶

Once all necessary processes are up and running, the ECC Server Status panel should display a status of “Idle” for each ECC server and the Data Router Status panel should show green check marks next to each data router. At this point, you should select a config file for each ECC server.

Config files can be selected by clicking the pencil icon next to the current config name in the Selected Config column of the ECC Server Status panel.

This will bring up a page with a drop-down menu listing the configurations available for that ECC server. The list of available configurations contains all possible permutations of the set of describe-*.xcfg, prepare-*.xcfg, and configure-*.xcfg files known to the ECC server. Each configuration is identified by a name composed of the names of the three *.xcfg files that go into it, formatted as [describe-name]/[prepare-name]/[configure-name]. For example, if you want to configure a data source using the files describe-cobo0.xcfg, prepare-experiment.xcfg, and configure-experiment.xcfg, then you should choose the configuration called cobo0/experiment/experiment. See Config files for more information about these files and their naming convention.

Preparing to take runs¶

After selecting a configuration, the CoBos and MuTAnT must be configured to prepare them to take data. This can be done using the first three buttons on the main Controls panel.

Begin by clicking the “Describe all” button. The system will then send a message to the ECC servers telling them to execute the “Describe” transition on the CoBos. The status label for each ECC server should then disappear and be replaced by a spinning cursor. Once the transition is finished, each ECC server should list a status of “Described”, and the overall system status in the top-right corner should also be shown as “Described.”

The next two steps are nearly identical. Click the “Prepare all” button, and wait until the status on each ECC server is shown as “Prepared.” Finally, click “Configure all,” and wait for a status of “Ready.” At this point, the system is ready to take data.

Starting a run¶

Runs are controlled using the “Start all” and “Stop all” buttons in the main Controls panel.

Once you click “Start all,” the CoBos will begin recording data and the Run Information panel should update to reflect the new run.

Recording run metadata¶

Once a new run has been started, metadata about the run can be entered by clicking on either the “Update values” or “Same as previous” button on the Run Information panel. Both of these will bring up a form where you can enter information about the current run. The only difference between the two is that the “Same as previous” button will pre-fill some fields with their values from the previous run. This is useful for values that don’t change often.

Fill in any values on this page that were not filled automatically, and then click “Submit” to save them. You can get back to the status page by clicking “Status” in the left-hand menu.

Stopping a run¶

When it is time to stop a run, click the “Stop all” button. This will tell the CoBos to stop recording data, and it will also tell the system to connect to each computer where the data router is running and rearrange the data files into a directory for the just-completed run. Watch the “Clean” column in the Data Router Status panel to see when this process has finished.

Resetting the system¶

When an experiment is complete, or when you want to re-configure the CoBos, the system should be reset to the “Idle” state. This can be done using the “Reset all” button in the main Controls panel. One click of this button will step each ECC server back by one state in the state machine (see CoBo state machine).