BioXTAS RAW¶

What is BioXTAS RAW?¶

BioXTAS RAW is a GUI based, free, open-source Python program for reduction and analysis of small-angle X-ray solution scattering (SAXS) data. The software is designed for biological SAXS data. It is available on windows, macOS (and OS X), and linux. It provides an alternative to closed source programs such as Primus and Scatter for primary data analysis. Because it can calibrate, mask, and integrate images it also provides an alternative to synchrotron beamline pipelines that scientists can install on their own computers and use both at home and at the beamline.

Find out how to get it!

Features¶

Analysis of radius of gyration (R_g) and I(0) via Guinier fit.
Analysis of molecular weight via I(0) comparison to standards, absolute calibration, correlation volume (V_c) and corrected Porod volume (V_p) methods.
Calculation of inverse Fourier transforms (IFTs) via GNOM and a Bayesian indirect Fourier transform (BIFT).
Calculation of envelopes (dummy atom models) using DAMMIF, DAMMIN, DAMAVER, and DAMCLUST.
Calculation of electron density using the DENSS algorithm
Easy processing of in-line chromatography coupled SAXS data, including size-exclusions coupled SAXS (SEC-SAXS) data.
Deconvolution of SEC-SAXS data using singular value decomposition (SVD) and evolving factor analysis (EFA).
Standard data operations such as averaging, subtraction, merging, and rebinning.
Creation and plotting of 1D scattering profiles from 2D detector images, including Pilatus, CBF, Eiger, and more than 20 other types of images.

History and Usage¶

RAW was first developed in 2008 by Soren Skou as part of the biological x-ray total analysis system (BioXTAS) project. Since then it has been extensively developed, with recent work being done by Jesse Hopkins.

RAW is actively used as the primary analysis software at the MacCHESS G1 BioSAXS Beamline. It is also used at various other beamlines, including:

BioCAT (18-ID) beamline at the APS
BL19U2 (SSRF)
I911-4 (aka I911-SAXS) beamline at MAX-LAB

SAXSLAB distributes RAW with some of its homesources, and RAW is used at various other homesources around the world.

Do you use RAW? Let us know!

Installation¶

RAW can be downloaded from sourceforge. There are prebuilt installers for windows (.msi) and mac (.dmg). Installation on Linux requires building from source.

RAW is also available through SBGrid.

Some features in RAW require a separate installation of the ATSAS package.

Some features in RAW require a separate installation of the EMAN2 package.

Detailed installation instructions¶

RAW Install Guide for Microsoft Windows¶

There are two ways to install RAW on Windows. The easiest, and recommended, way is to use a prebuilt installer (.msi file). If you want to install a version for which there is no prebuilt installer, you will need to install RAW from the source code.

Here you can find information on:

Using a prebuilt installer¶

The recommend way to install RAW on Windows is using a prebuilt installer. To install from a prebuilt installer, simply download the RAW-x.y.z-win32.msi (where x.y.z is the version number) file from sourceforge ( http://sourceforge.net/projects/bioxtasraw), and double click it to run the installer.

Important Notes:

Because the RAW team is an ‘unknown’ developer, you will probably see some security warnings when you install RAW. When you do, just give the installer and program permission to run on your computer.
RAW is used by (relatively) few people, which means many virus scanners have not seen the RAW software before. Occasionally virus scanners will mark a file (typically RAW.exe) as a threat (it will usually be in the ‘general malware’ category). If this happens, please do the following:
- Upload the file to https://www.virustotal.com/ and see if any other virus scanners identify it as a virus (it is always possible someone hijacked the installer somehow!).
- If most or all of the virus scanners on virustotal.com clear the file, make an exception for it in your virus scanner.
- Contact the RAW developers, so we can report the false identification to the virus scanner company and get the file whitelisted in future definitions files.

General instructions for installing from source (advanced users)¶

Install the Microsoft Visual C++ compiler for Python 2.7
Install Python 2.7 (if it isn’t already installed) and add it to your system path.
Install a C compiler (such as the gcc c++ compiler) and add it to your system path.
Install the following Python packages (version indicated if less than most recent):
- numpy
- matplotlib
- scipy
- pillow
- wxpython < 4.0
- fabio
- lxml
- h5py
- hdf5plugin
- pyFAI
- weave
Download the RAW source file (RAW-x.y.z-Source where x.y.z is the version number) from sourceforge ( http://sourceforge.net/projects/bioxtasraw)
Extract RAW to a directory of your choice and run RAW.py using python.
- Note: the first time you run RAW.py it may need to be run from the command line in order to successfully compile various extensions. It may take some time to compile the extensions, be patient.
- After you run RAW for the first time, you can run it by double clicking the RAW.py file, assuming .py files are associated with your python executable.
Enjoy!
- If you have problems, please consult the detailed installation guides and the solutions to common problems below. If that doesn’t help, please contact the developers.

Windows 7, 8.1, and 10 install from source instructions¶

Download and install the Microsoft Visual C++ Compiler for Python 2.7.
- Download from https://www.microsoft.com/en-us/download/details.aspx?id=44266
- Run the installer with the default options.
Download and install the Microsoft Visual C++ 2008 SP1 Redistributable Package (x86).
- Download from https://www.microsoft.com/en-us/download/details.aspx?id=5582
- Run the installer with the default options.
Install 32 bit python 2.7 from python.org. Make sure pip is installed, and add python to the windows path.
- Download the 32 bit python 2.7 installer for windows from https://www.python.org/downloads/windows/
  - Select the latest release (2.7.14 as of 3/16/18).
  - Download the Windows x86 MSI Installer (NOT the x86-64 MSI Installer).
- Run the installer
- Accept the default installation location
- At the customization option, make sure that pip is installed and that you add python.exe to Path.
Download and install the MinGW compiler
- Download mingw-get-setup.exe from https://sourceforge.net/projects/mingw/
  - The download button should default to the correct file for windows.
- Run the installer.
- Install to the default location
- Once the installer is finished, click continue
- You will see this screen:
- Click the boxes for mingw32-base and mingw32-gcc-g++ and select mark for installation.
- Once selected, they should look like this:
- Click on the Installation menu and click Apply Changes.
- In the windows that pops up, click Apply.
- Close the MinGW installer.
Add mingw and mingw\bin to the start of the global path variable. Use part a for Windows 7, part b for Windows 8.1, and part c for Windows 10.
- Windows 7:
  1. Click on the Start Menu and click on Computer.
  2. In the window that opens, click on System properties
  3. In the window that opens, click on Advanced system settings
  4. In the window that opens, click on Environment Variables…
  5. In the window that opens, In the System variables area, select the Path variable and click Edit…
  6. In the window that opens, add the following to the start of the path: C:\MinGW;C:\MinGW\bin;
  7. Click Ok several times to save and exit out of the settings.
  8. Note: You can modify the User variable path instead of the System variable path if you want to only modify the setup for your user account, rather than the entire system. The same steps apply, but if you do not already have a use path variable, you will have to create one using the New button.
- Windows 8.1:
  1. Open a file browser window and click on This PC. At the top, click on System properties.
  2. Continue with steps 3 and onward in the Windows 7 section above.
- Windows 10:
  1. Click on the windows/start menu and click File Explorer.
  2. In the file explorer window, click on This PC and then Computer
  3. Then click on System properties in the new menu
  4. In the window that opens, click on Advanced system settings
  5. In the window that opens, click on Environment Variables…
  6. In the window that opens, In the System variables area, select the Path variable and click Edit…
  7. In the window that opens, use the New button to add these items to the path:
    - C:\MinGW
    - C:\MinGW\bin;
  8. Use the Move Up button to move the items to the start of the path
  9. Click Ok several times to save and exit out of the settings.
  10. Note: You can modify the User variable path instead of the System variable path if you want to only modify the setup for your user account, rather than the entire system. The same steps apply, but if you do not already have a user path variable, you will have to create one using the New button.
Restart your computer.
Downgrade the gcc compilers.
- Open a command prompt
  - Windows 7: Click on the start menu, search for cmd, then run the cmd program.
  - Windows 8: Click on the windows tile and search for cmd, then run the Command Prompt program.
  - Windows 10: Click on the windows/start menu, select All Files, select Windows System, and click on Command Prompt.
- Type mingw-get upgrade "gcc<6" and hit enter.
- Type mingw-get upgrade "g++<6" and hit enter.
Upgrade pip, setuptools, and wheel.
- Open a command prompt window as in the previous step (if necessary).
- Type python -m pip install --upgrade pip
- Hit enter
- Once that finishes, type pip install --upgrade setuptools wheel
- Hit enter
Download and install numpy+mkl, scipy, and wxpython prebuilt versions.
- Go to http://www.lfd.uci.edu/~gohlke/pythonlibs/ and download:
  - numpy-x.y.z+mkl-cp27-cp27m-win32.whl (where x.y.z is the version number).
  - scipy-x.y.z-cp27-cp27m-win32.whl (where x.y.z is the version number).
  - wxPython_common‑3.0.2.0‑py2‑none‑any.whl
  - wxPython‑3.0.2.0‑cp27‑none‑win32.whl
- Note: It is important to download the cp27 win32 version, otherwise installation will fail!
- Open a command prompt window as in the previous step (if necessary).
- Navigate to the folder where you downloaded these files.
  - The easiest way to do this is to type cd followed by a space, then drag the folder from the Explorer window into the command prompt window. You should see the path to the folder appear in the command prompt. Hit enter after that.
  - Alternatively, in Explorer right click on the folder in the title bar and select Copy address. Then right click on the command prompt and select Paste.
- Type pip install numpy-x.y.z+mkl-cp27-cp27m-win32.whl, replacing the x.y.z with the appropriate version number. Hit enter.
- Type pip install scipy-x.y.z-cp27-cp27m-win32.whl, replacing the x.y.z with the appropriate version number. Hit enter.
- Type pip install wxPython_common‑3.0.2.0‑py2‑none‑any.whl and hit enter.
- Type pip install wxPython‑3.0.2.0‑cp27‑none‑win32.whl and hit enter.

# Install matplotlib, pillow, lxml, h5py, fabio, weave, hdf5plugin, and pyfai using pip.

Type pip install matplotlib pillow lxml h5py fabio weave hdf5plugin pyfai and hit enter.

Download RAW from sourceforge ( http://sourceforge.net/projects/bioxtasraw)
- Go to the Files tab on the linked website and download the RAW-x.y.z-Source.zip file, where x.y.z is the version number (for example, 1.0.0).
Expand the downloaded zip file into the downloads folder
- Right click on the download and select Extract All
- Accept the default location for files to be extracted.
Check that the files are located at the top of the directory, as in the following image:
- Windows 7: Browse to Computer -> Local Disk -> raw
- Windows 8 and 10: Browse to This PC -> Local Disk -> raw
- You may have only a single subfolders, named something like raw or src.
- If so, browse down levels until you find the directory containing files that look like the top image on this page, and copy everything in that folder to the top level raw directory.
Run RAW.py from the command line
- Open a command prompt as in Step 6 of these instructions.
- Type cd C:\raw
- Hit enter
- Type python RAW.py
- Hit enter
- When you start RAW for the first time, it compiles various extensions, this may take a while. Please be patient.
Enjoy!
- After running RAW for the first time, you can start it without using the command line simply by double clicking on RAW.py in the C:raw folder.
- If you want to create a desktop shortcut, right click on RAW.py and select Send To ‣ Desktop.
- If you have trouble with the installation, please see the solutions to common problems section below.

Common problems/troubleshooting¶

Prebuilt installer:

Because the RAW team is an ‘unknown’ developer, you will probably see some security warnings when you install RAW. When you do, just give the installer and program permission to run on your computer.
RAW is used by (relatively) few people, which means many antivirus programs have not seen the RAW software before. Occasionally virus scanners will mark a file (typically RAW.exe) as a threat (it will usually be in the ‘general malware’ category). If this happens, please do the following:
- Upload the file to https://www.virustotal.com/ and see if any other antivirus programs identify it as a problem (it is always possible someone hijacked the installer somehow!).
- If most or all of the antivirus programs on virustotal.com clear the file, make an exception for it in your virus scanner.
- Contact the RAW developers, so we can report the false identification to the virus scanner company and get the file whitelisted in future definitions files.

From source:

The compiler can fail if there are any spaces in the directory paths. Make sure raw, the compiler (MinGW), and python are all installed in directory paths without spaces in the names.
The compiler can fail if it tries to compile the modules when some of them are already compiled. If the compilation is failing, try deleting all .pyd files in the raw directory.
The compiler can fail if you try to compile when you’re not using the command line. This most commonly happens if someone tries to run RAW.py for the first time by double clicking on it, rather than using the python RAW.py command in the command prompt window.
If the extensions won’t compile properly (you’ll get a popup message when you start RAW warning you of this), you can try copying the precompiled extensions (.pyd files) from the appropriate WinLib folder into the main raw folder.
If you are updating your RAW installation, you should completely delete the old RAW source files, and then replace them with the new ones.
You may have trouble with various pieces of the installation if your path variable isn’t set right. The windows PATH variable cannot have spaces. That is, your path should look like: item1;item2;item3 not: item1; item2; item3. For Windows 10, where you enter separate entries in your path variable (which Windows automatically concatenates), make sure that you don’t have leading or training spaces in any of the items.
On some systems, we’ve found it necessary to install the packages from pip in multiple steps. If a pip install fails, trying running it on each package separately. For example, if pip install matplotlib pillow fabio fails, try running:
- pip install matplotlib
- pip install pillow
- pip install fabio
In recent numpy builds (I think it’s numpy, might be scipy) I’ve found that the compiler needs msvcr90d.dll This is the debug dll, which means someone built it wrong, that is, in a debug environment. The only solution to this is to use the numpy and scipy packages from https://www.lfd.uci.edu/~gohlke/pythonlibs/ instead of the pip archive.
The latest version of the gcc compiler that seems to work is 5.3. The 6.3 compiler from MinGW fails. Downgrade with the following commands from a command prompt: mingw-get upgrade "gcc<6" and mingw-get upgrade "g++<6".

RAW Install Guide for macOS and OS X¶

There are two ways to install RAW on Mac. The easiest, and recommended, way is to use a prebuilt app package (.dmg file). If you want to install a version for which there is no prebuilt installer, you will need to install RAW from the source code.

Here you can find information on:

Using a prebuilt app package¶

The recommended way to install RAW on Mac is using a prebuilt app package. To install from a prebuilt app package simply download the RAW-x.y.z-mac.dmg (where x.y.z is the version number) file from sourceforge ( http://sourceforge.net/projects/bioxtasraw), double click it to open the dmg, and drag the RAW.app file to your Applications folder (or wherever you want to install RAW).

Important Notes:

Because the RAW team is an unidentified developer, you may get a warning message the first time you run the program. If that happens, right click on RAW and select Open from the right click menu, and then click the Open button in the window that appears.
- Note: This requires administrator permissions.

General instructions for installing from source (advanced users)¶

Install a standalone version of python 2.7 (recommended, not required).
Install a gcc c++ compiler (for example the Xcode command line tools) and add it to your system path.
Install the following python packages (most recent version of each recommended):
- numpy
- scipy
- matplotlib
- pillow
- wxpython < 4.0
- fabio
- lxml
- h5py
- pyFAI
- hdf5plugin
- weave < 0.16
Download the latest RAW sourcecode from sourceforge ( http://sourceforge.net/projects/bioxtasraw)
Extract RAW to a directory of your choice and run RAW.py using python.
- Note: the first time you run RAW.py it may need to be run from the command line in order to successfully compile various extensions. It may take some time to compile the extensions, be patient.
Enjoy!
- If you have problems, please consult the detailed installation guide and the solutions to common problems below. If that doesn’t help, please contact the developers.

OS X and macOS install from source instructions¶

Install Enthought Canopy python distribution
- Download the free Canopy installer from: https://store.enthought.com/downloads/#default
  - We recommend the 64-bit version, unless you know you need the 32-bit version.
- Open the downloaded disk image, and drag the Canopy application into the Applications folder
- Start the Canopy application (for example, double click on it in the Applications folder).
  - Canopy may not open, depending on user security settings. You have two options if it doesn’t open:
    - Open canopy by right clicking on it in the applications folder and selecting Open.
    - Change security settings by going to System Settings -> Security and Privacy, and selecting Allow apps downloaded from ‘Anywhere’.
  - Accept the default environment install location
- The first time you run Canopy, it will ask you if you want to make it the default environment. Select Yes and Start using Canopy
- Note: In recent installation attempts I haven’t seen this dialog. In that case you have to add the following to your .bash_profile: source /Users/<username>/Library/Enthought/Canopy/edm/envs/User/bin/activate
- More detailed install instructions are available here: http://docs.enthought.com/canopy/quick-start/install_macos.html
Install xcode command lines tools
- Opening a Terminal window by starting the Terminal app.
  - Using the Launcher, it is located in the Other section.
  - Browsing the Applications folder in Finder, Terminal is in the Utilities folder.
- In the terminal window type xcode-select --install
- Hit enter
- In the popup window, select install (you don’t need the whole xcode installation)
- Close the terminal window.
Install the fabio, pyFAI, hdf5plugin, and weave python packages using pip
- Open a terminal window as in step 2.
- Type pip install --upgrade pip
- Hit enter
- Once that installs, type pip install fabio pyFAI hdf5plugin weave
- Hit enter
- Once the installation finishes, close the terminal window.

Download RAW from sourceforge
- http://sourceforge.net/projects/bioxtasraw
- Navigate to the Files tab and download the latest source code, RAW-x.y.z-Source.zip. Or download the latest development version from the git by navigating to the Code tab.

Expand the downloaded zip file in the Downloads folder by double clicking on it.
- This step may not be necessary, some browsers may automatically expand zip files.
Check if the top level directory contains files that look like the following image.
- If you see only a single folder, navigate down through the folders until you find a folder with all of the files in it, as in the above image.
  - An example of this is shown below, where the expanded files has an src directory in it, which contains all of the downloaded files.
Move these files to Applications folder
- Move the folder that contains all of the RAW files to the Applications folder. In the above image, this would be the src folder.
- Rename the folder that you just moved to raw.
Run RAW from the terminal.
- Open a terminal window as in step 2.
- Type cd /Applications/raw
- Hit enter
- Type python RAW.py
- Hit enter
- When you start RAW for the first time, it compiles various extensions. This may take some time. Please be patient.
Enjoy!
- In the future, you can start RAW as in the previous step.
- If you want, see the section on making a clickable shortcut for RAW
- If RAW doesn’t work, check out the solutions to common problems

Setting up a RAW shortcut

The easiest way to set up a RAW shortcut is to expand the RunRAW.zip file in the RAW MacLib folder, and move the RunRAW app to wherever you want to have it. Start raw by double clicking on the RunRAW app.

Note: since this is an application from an unknown developer (the RAW team!), you may have to give it permission to run. You can do this by right clicking on it and selecting Open or by changing your security settings by going to System Settings -> Security and Privacy, and selecting Allow apps downloaded from ‘Anywhere’.

If the shortcut doesn’t work, and/or you would rather not change your security settings, you can follow these instructions to make your own shortcut:

Expand the RunRAW.zip file as above.
Open the Automator app
- Automator is located in Applications folder.
Select Open an Existing Document
Open the RunRAW app (located in Applications/raw/MacLib)
You should see a window like this:
- Note: If you didn’t install RAW in the recommended location, you will need to change the line do script “python /Applications/raw/RAW.py” to have the appropriate path.
Select File ‣ Duplicate from the menu.
In the duplicate window (should be named RunRAW copy or Untitled), select File ‣ Save.
Save the file as RunRAW (.app, if you are showing extensions) wherever you want to have the shortcut.
- Note: you can name it whatever you want. RunRAW is just the default name.
Now you can launch raw just by double clicking the RunRAW icon.
- If it doesn’t work, make sure that you saved the duplicate, rather than the original!

Common problems/troubleshooting¶

Installing the prebuilt app package:

Because the RAW team is an unidentified developer, you may get a warning message the first time you run the program. If that happens, right click on RAW and select Open from the right click menu, and then click the Open button in the window that appears.
- This requires administrator privileges
If the above doesn’t work, you can run the RAW app from the command line. Navigate to RAW.app/Contents/MacOS and run the RAW unix executable file (./RAW) in that directory.

Installing from source:

The compiler can fail if there are any spaces in the directory paths. Make sure that the RAW.py file is installed in a directory path without any spaces.
If the extensions won’t compile properly (you’ll get a popup message when you start RAW warning you of this), try copying the appropriate precompiled extensions (.so files) from the MacLib folder into the main raw folder.
The shortcut can fail if you didn’t install raw in the recommended location. If that’s the case, go through the process of creating a new shortcut, and make sure you change the line in the script mentioned in that section.
Because of some weirdness with the anaconda/miniconda framework pythonw, it can’t run EMAN2. Every other variant I’ve tried (enthought and homebrew) have been fine.

RAW Install Guide for Linux¶

Here you can find information on:

General instructions for installing from source (advanced users)¶

Install python 2.7 (if it isn’t already installed).
Install the python development tools.
Install the gcc c++ compiler.
Install the following python packages (version indicated if less than most recent):
- numpy
- scipy
- matplotlib
- pillow
- wxpython < 4.0
- fabio
- h5py
- lxml
- pyFAI
- hdf5plugin
- weave
Download RAW from sourceforge ( http://sourceforge.net/projects/bioxtasraw)
Extract RAW to a directory of your choice, and run RAW.py using python.
- Note: the first time you run RAW.py it may need to be run from the command line in order to successfully compile various extensions. It may take some time to compile the extensions, be patient.
Enjoy!
- If you have problems, please consult the detailed installation guides below and check out the solutions to common problems.
- If you want, see the section on making a desktop shortcut for RAW.

Ubuntu/Linux Mint/Debian install from source instructions¶

Tested on Ubuntu 16.04, Linux Mint 18.3, and Debian 9.4.

First install the g++ compiler:

Open a terminal (right click on the desktop and select New Terminal or Open in terminal) and run the following commands (hit enter/return to execute each command):
- su (if necessary)
- sudo apt-get update
- sudo apt-get install g++

Then install python and RAW:

Open a new terminal window (in many distros you can right click on the desktop and select New Terminal or Open in terminal).
Download Miniconda Python (Python 2.7) distribution and install.
- http://conda.pydata.org/miniconda.html
- Make sure you chose the python 2.7 installer.
- Save to the downloads folder
- Open a terminal and run the following commands:
  - cd ~/Downloads
  - bash ./Miniconda2-version.sh (where -version which will depend on the version you download)
- Accept the default installation location.
- At the end, say “yes” to have the conda python install put in your system path.
Install python packages. Open a new terminal window and run the following commands.
- conda update conda setuptools wheel pip
- conda install numpy scipy matplotlib pillow h5py lxml numexpr
- conda install 'wxpython<4'
- pip install fabio pyFAI hdf5plugin weave
Download RAW from sourceforge
- http://sourceforge.net/projects/bioxtasraw/
- The download button on the main page should default to the right download for your OS.
Expand the RAW download to your location of choice.
- We suggest ~/raw
- Make sure there are no spaces in the file path (you can check by navigating to the raw directory in a terminal window and using pwd).
- In the terminal or in the graphical file manager, confirm that the file named RAW.py is in your raw directory. If it isn’t, it’s likely that when you expanded the RAW download, you ended up with unnecessary layers of directories. Find the directory with RAW.py in it, and make that the top level folder.
In a terminal, change directory into the top level RAW folder
- If you used the suggested path of ~/raw type: cd ~/raw
Run RAW
- python RAW,py
- The first time RAW runs, it may take a little while to load, as it has to compile various extensions.
RAW is now installed. Enjoy!
- If you want, see the section on making a desktop shortcut for RAW.
- If RAW doesn’t work, check out the solutions to common problems.

OpenSUSE install from source instructions¶

Tested on OpenSUSE Leap 42.3.

First install the g++ compiler:

Open a terminal (Application Menu ‣ System ‣ Konsole) and run the following commands (hit enter/return to execute each command):
- su (if necessary)
- sudo zypper update
- sudo zypper install gcc-c++

Then install python and RAW:

Open a new terminal window (in many distros you can right click on the desktop and select New Terminal or Open in terminal).
Download Miniconda Python (Python 2.7) distribution and install.
- http://conda.pydata.org/miniconda.html
- Make sure you chose the python 2.7 installer.
- Save to the downloads folder
- Open a terminal and run the following commands:
  - cd ~/Downloads
  - bash ./Miniconda2-version.sh (where -version which will depend on the version you download)
- Accept the default installation location.
- At the end, say “yes” to have the conda python install put in your system path.
Install python packages. Open a new terminal window and run the following commands.
- conda update conda setuptools wheel pip
- conda install numpy scipy matplotlib pillow h5py lxml numexpr
- conda install 'wxpython<4'
- pip install fabio pyFAI hdf5plugin weave
Download RAW from sourceforge
- http://sourceforge.net/projects/bioxtasraw/
- The download button on the main page should default to the right download for your OS.
Expand the RAW download to your location of choice.
- We suggest ~/raw
- Make sure there are no spaces in the file path (you can check by navigating to the raw directory in a terminal window and using pwd).
- In the terminal or in the graphical file manager, confirm that the file named RAW.py is in your raw directory. If it isn’t, it’s likely that when you expanded the RAW download, you ended up with unnecessary layers of directories. Find the directory with RAW.py in it, and make that the top level folder.
In a terminal, change directory into the top level RAW folder
- If you used the suggested path of ~/raw type: cd ~/raw
Run RAW
- python RAW,py
- The first time RAW runs, it may take a little while to load, as it has to compile various extensions.
RAW is now installed. Enjoy!
- If you want, see the section on making a desktop shortcut for RAW.
- If RAW doesn’t work, check out the solutions to common problems.

Scientific Linux/Red Hat/CentOS install from source instructions¶

Tested on SL 6.8 and SL 7.4.

First install the g++ compiler:

Open a terminal (right click on the desktop and select Open Terminal) and run the following commands:
- su (if necessary)
- sudo yum update
- sudo yum install gcc-c++

Then install python and RAW:

Open a new terminal window (in many distros you can right click on the desktop and select New Terminal or Open in terminal).
Download Miniconda Python (Python 2.7) distribution and install.
- http://conda.pydata.org/miniconda.html
- Make sure you chose the python 2.7 installer.
- Save to the downloads folder
- Open a terminal and run the following commands:
  - cd ~/Downloads
  - bash ./Miniconda2-version.sh (where -version which will depend on the version you download)
- Accept the default installation location.
- At the end, say “yes” to have the conda python install put in your system path.
Install python packages. Open a new terminal window and run the following commands.
- conda update conda setuptools wheel pip
- conda install numpy scipy matplotlib pillow h5py lxml numexpr
- conda install 'wxpython<4'
- pip install fabio pyFAI hdf5plugin weave
Download RAW from sourceforge
- http://sourceforge.net/projects/bioxtasraw/
- The download button on the main page should default to the right download for your OS.
Expand the RAW download to your location of choice.
- We suggest ~/raw
- Make sure there are no spaces in the file path (you can check by navigating to the raw directory in a terminal window and using pwd).
- In the terminal or in the graphical file manager, confirm that the file named RAW.py is in your raw directory. If it isn’t, it’s likely that when you expanded the RAW download, you ended up with unnecessary layers of directories. Find the directory with RAW.py in it, and make that the top level folder.
In a terminal, change directory into the top level RAW folder
- If you used the suggested path of ~/raw type: cd ~/raw
Run RAW
- python RAW,py
- The first time RAW runs, it may take a little while to load, as it has to compile various extensions.
RAW is now installed. Enjoy!
- If you want, see the section on making a desktop shortcut for RAW.
- If RAW doesn’t work, check out the solutions to common problems.

Instructions for setting up a RAW desktop shortcut¶

All files referred to are initially located in the RAW LinuxLib folder.

Add the start_raw file to your path:
- sudo cp ~/raw/LinuxLib/start_raw /usr/local/bin
Make the start_raw file executable:
- sudo chmod +x /usr/local/bin/start_raw
Copy the RAW.desktop file to the desktop:
- cp ~/raw/LinuxLib/RAW.desktop ~/Desktop/
Right click on the RAW file on the desktop, and select Properties
Click on the Permissions tab, and make sure Allow executing file as program is checked.
Note: depending your distribution/shell, you have to edit the start_raw file to use a different shell. By default it uses bash.

Common problems/troubleshooting¶

On Scientific Linux 6, and thus probably on Red Hat 6 (untested), RAW completely fails to work with wxpython 3.0 and certain python distributions (namely the Enthought python).
- If RAW completely doesn’t start, check and make sure you have wxpython 2.8 installed. This requires that you have matplotlib<=1.4.
- In general, RAW should work with wxpython 3.0.
Sometimes, compilers can have trouble if there are spaces in the filepath. Try installing RAW so that there are no spaces in the file path (navigate the folder in the terminal, type pwd and see what the result is).
The Enthought Canopy python package DOESN’T WORK on Ubuntu or Linux Mint with wxpython.
If you have installed a standalone python distribution (such as Enthought Canopy or miniconda/anaconda), it is possible that it isn’t set to default, so when you run python RAW.py, you are using the wrong python.
- You can verify which python you are using the command which python in the terminal.
- You can set the correct python to default by modifying your appropriate profile file (such as the .bash_profile), or setting the $PATH environmental variable.
- You can also specify the full path to the version of python you want to use in the command, such as ~/miniconda2/bin/python
In some cases it is necessary to run RAW as an administrator, in order to compile (we’ve observed this on Scientific Linux 6). If RAW runs but doesn’t compile, and you’re sure you’ve got the gcc c++ compiler installed, try running it using sudo.
- Warning: the python used for sudo may not be the python for the regular user (particularly if you su and then sudo).
Note that when you change environmental variables in one terminal window, you need to restart other windows for this to take effect. If you aren’t using the right python (or compiler, etc), trying closing all of your terminal windows and opening a new one.

Getting Help¶

General Questions/Comments¶

For questions on how to use the software, first check out the documentation on this site.

If that doesn’t answer your question, or for other comments, please contact the RAW mailing list. If that doesn’t work for you, you can contact Soren and Jesse directly: soren (dot) skou (at) saxslab (dot) com and hopkins (dot) jesse (at) gmail (dot) com.

Reporting A Bug¶

We appreciate any and all bug reports, it helps us improve the software! If you have a bug to report, the best way is to contact the RAW mailing list. If that doesn’t work for you, you can contact Soren and Jesse directly: soren (dot) skou (at) saxslab (dot) com and hopkins (dot) jesse (at) gmail (dot) com.

If you have a bug report, please include: operating system, RAW version number, installation type (package or from source), what you were doing when the bug occured, the appropriate error message(s) and sample data so we can recreate the bug. That will enable us to find and solve the problem as quickly as possible.

Tutorial¶

Introduction¶

Overview¶

This tutorial covers SAXS data processing with RAW. You will learn how to:

Process images into scattering profiles
Average, subtract and save scattering profiles
Find R_g and I(0) by Guinier analysis
Find molecular weight by four different methods
Do Kratky analysis and normalized Kratky analysis
Test the similarity of scattering profiles
Load and process SEC-SAXS data
Carry out singular value decomposition (SVD) and evolving factor analysis (EFA) to evaluate and analyze SEC-SAXS data
Merge SAXS/WAXS data from two detectors
Carry out Pair-distance distribution analysis (BIFT and GNOM)
Evaluate ambiguity of 3D shape reconstructions (AMBIMETER)
3D reconstructions with bead models (DAMMIF/N and DAMAVER)
3D reconstructions with electron density (DENSS)
Calibrate RAW for integrating images
Mask images for integration
Set up normalization and save processing settings
Set absolute scaling in RAW using water and glassy carbon
Set a molecular weight standard in RAW

Section 1 covers basic processing with RAW, and Section 2 covers advanced processing with RAW. Section 3 covers how to set up RAW for integrating images for those who do not already have a configuration file.

Requirements¶

BioXTAS RAW >= v1.4.0 (most recent is best).
- Install instructions

Tutorial data.
- Available from: https://sourceforge.net/projects/bioxtasraw/files/?source=navbar

ATSAS programs, >= v2.8.0 (for Section 2 of the tutorial only).
- Download and installation instructions are available from: http://www.embl-hamburg.de/biosaxs/download.html
- Requires a free registration for academic users. Industrial users must pay to use.

EMAN2, >=2.2 (for Section 2 of the tutorial only) * Download and installation instructions are available from: http://blake.bcm.edu/emanwiki/EMAN2/Install
- Free for all, optional registration for user tracking purposes.

Other useful materials¶

An overview and tutorial of RAW produced by Jesse Hopkins for SBGrid, which can be viewed here: https://youtu.be/XGnJDs3N2MI
There are RAW tutorial videos produced by Richard Gillilan, which can be viewed here: http://bit.ly/bioxtast. Data for these tutorial videos is available here: http://bit.ly/bioxtasd.
On data collection, molecular weight estimation, and data analysis: Skou, Gillilan, and Ando (2014) Nature Protocols, 9,1727–1739.
On publication guidelines: Jacques, Guss, Svergun, & Trewhella (2012). Acta Cryst D, 68(Pt 6), 620–626.
Uniqueness of ab initio shape determination in small-angle scattering. V. V. Volkov and D. I. Svergun. Journal of Applied Crystallography (2003) 36, 860-864.
Small Angle X-ray Scattering as a Complementary Tool for High-Throughput Structural Studies. Thomas Grant et al. Biopolymers (2011) 95, 517-530.
Ambiguity assessment of small-angle scattering curves from monodisperse systems. M. V. Petoukhov and D. I. Svergun. Acta Crystallographica D (2015) 71, 1051-1058.
ATSAS resources:
- Manuals: http://www.embl-hamburg.de/biosaxs/manuals/
- User forum: http://www.saxier.org/forum/
Electron density (DENSS) resources available at DENSS.org
- Particularly useful is the section on visualizing the results and aligning with known structures.

Notes¶

If you are only interested in using RAW to process data, and are not interested in how to set up RAW to calibrate your data, you do not need to look at Section 3.

RAW depends on user feedback to get better. If you have questions, find bugs, or think a part of this tutorial is unclear, please let the developers know. The best way to do this is via the RAW google group: http://bit.ly/rawhelp

You can find additional developer contact information on the RAW website: https://sourceforge.net/projects/bioxtasraw/

Basic processing¶

This section will guide you through basic processing of files in RAW. It includes: integrating images into scattering profiles, averaging and subtracting scattering profiles, doing Guinier fits, MW analysis, processing SEC-SAXS data, processing data from 2 detectors (SAXS and WAXS data), and a few other things. It refers to the RAW tutorial data.

Select a section below to view the tutorial, or use the next and back buttons at the bottom of the page to navigate through it in order.

Loading configuration files and images, creating subtracted scattering profiles, saving profiles¶

Open RAW. The install instructions contain information on installing and running RAW.
In the files tab, click on the folder button and navigate to the Tutorial_Data/standards_data folder. Click the open button to show that folder in the RAW file browser.
At the bottom of the File tab in the Control Panel, use the dropdown menu to set the file type filter to “CFG files (*.cfg)”.
Double click on the SAXS.cfg file to load the SAXS configuration. This loads the beamline configuration into the program.
- Note: Any time you are going to process images, you need to load the appropriate configuration!
Change the file type filter to “TIFF files (*.tiff)”.
At the CHESS G1 station, typically ~10 images are collected from a given sample. To load in the 10 images for the glucose isomerase (GI) sample, start by selecting the files GI2_A9_19_001_xxxx.tiff, where xxxx will range from 0000 to 0009. These files are measured scattering from 0.47 mg/ml GI.
- Tip: you can hold down the ctrl key (apple key on macs) while clicking to select multiple files individually. You can also click on a file, and then shift click on another file to select those files and everything between them.
- Warning: Don’t load the files with PIL3 in their name. Those are the wide-angle scattering (WAXS) data, which we will process separately later.
Click the plot button to integrate all of the images and plot the integrated scattering profiles on the Main Plot.
- Note: Typically, once the images are integrated we work only with the scattering profiles. However, it is useful to keep the images around in case you want to reprocess the data.
Plot the GIbuf2 scattering profiles from the images. These are measured scattering from the matching buffer, without any protein, for the GI sample.
Click on the Manipulation tab. This is where you can see what scattering profiles are loaded into RAW, and manipulate/analyze them.
- Checkpoint: If you’ve successfully loaded the images given, you should see twenty scattering profiles in the manipulation list, with names like GI2_A9_19_001_0000.tiff or GIbuf2_A9_18_001_0000.tif.
Click on a filename to select the scattering profile. The background should turn gray, indicating it is selected.
Select all of the GI scattering profiles
- Tip: Again, the ctrl(/apple) key or the shift key can be used to select multiple scattering profiles.
- Warning: Select only the GI profiles, not the GI buffer profiles.
Use the average button to average all of the scattering profiles collected into a single curve.
- Checkpoint: The averaged scattering profile should appear at the bottom of the manipulation list. You may have to scroll down to see it. The filename will be in green, and will start with A_, indicating it is an average scattering profile.
Average all of the GI buffer scattering profiles.
In order to see the averaged scattering profiles, you will need to hide the individual profiles from the plot. Clicking on the check box to the left of the filename will show/hide a scattering profile. When the box is checked, the profile is shown on the plot, when it is unchecked, the profile is hidden. Hide all of the profiles except the two averaged curves.
- Tip: The eye and eye with the x through it at the top of the manipulation panel can be used to show/hide sets of loaded profiles at once. If no profiles are selected, these buttons show/hide all loaded profiles. If some profiles are selected, these buttons show/hide just the selected profiles. Try selecting all but the averaged files and using the show/hide all buttons.
Next you need to subtract the buffer scattering profile from the measured protein scattering (which is really the scattering of the protein plus the scattering of the buffer). Star the averaged buffer file, and select the averaged protein file, then click the subtract button.
- Checkpoint: The subtracted scattering profile should be shown in the lower plot. A new manipulation item should be shown in the manipulation menu, with the name in red and a S_ prefix indicating it is a subtracted file.
You don’t need the individual image scattering profiles any more. Select all of those (but not your averaged or subtracted profiles!) and click remove.
- Note: This only removes the scattering profiles from RAW. The images on your hard drive are unaffected.
Load in the lys2 and lysbuf2 files, average, and subtract to create a subtracted lysozyme scattering profile. The concentration of this sample was 4.27 mg/ml. Remove all of the profiles that are not averaged or subtracted profiles.
- Tip: In order to tell which curve is which in a plot, click on the target icon in the manipulation list. This should bold that curve in the plot. Click the target icon again to return the curve to normal.
We’re done with the averaged profiles. Select all of the averaged profiles and click the “Save” button to save them in the standards_data folder. Note that in the filename in the manipulation list, the * at the front goes away. This indicates there are no unsaved changes to those scattering profiles. You can now remove them.
- Note: This saves them with a .dat extension. This is the standard format for SAXS scattering profiles, and is also human readable.
Right click on the subtracted plot, move the cursor over ‘Axes’ and select the Log-Log option. Well-behaved globular proteins will intersect the intensity axis roughly perpendicularly.
- Note: It is best practice to display SAXS data, particularly in publications, on either a semi-log (Log-Lin, default option in RAW) or double-log plot (depending on the features of interest).

Guinier analysis¶

Recall Guinier’s approximation at low-q: $I(q)\approx I(0) \exp(-R_g^2 q^2 /3)$.

R_g and I(0) can be determined by performing a linear fit in the Guinier plot (a plot of $\ln(I)$ vs. $q^2$). The fitting region should normally have $q_{max}R_g<1.3$ for globular proteins. This fitting region is called the “Guinier region.”

In RAW, right click (ctrl click on macs without a right mouse button) on the subtracted GI scattering profile in the Manipulation list and select “Guinier fit”. In the plots on the right, the top plot shows you the Guinier plot and the fit, while the bottom plot shows you the residual of the fit.
- Note: RAW automatically tries to find the best Guinier region for you when the Guinier window is opened for the first time.
- Note: The R_g value is in Angstroms, while the two $qR_g$ boxes give, left to right, $q_{min}R_g$ and $q_{max}R_g$ respectively.
In the “Control” panel, you’ll see that n_min is now 6. This means RAW has cut off the first six points of the scattering profile in the fit. Use the arrow buttons next to the n_min box to adjust that to zero. Check whether the R_g changes.
In the “Parameters” panel, note that $q_{max}R_g$ is only ~1.26. Recall that for globular proteins like GI, it is typical to have $q_{max}R_g$ ~1.3. Adjust n_max until that is the case, watching what happens to the R_g and the residual.
- Question: The literature radius of gyration for GI is 32.7 Å. How does yours compare?
RAW also provides an estimate of the uncertainty in both the R_g and I0 values for the Guinier fit, shown in the Uncertainty section.
- Note: This is the largest of the uncertainties from the fit (standard deviation of fit values calculated from the covariance matrix), and either the standard deviation of R_g and I(0) across all acceptable intervals found by the autorg function or an estimated uncertainty in R_g and I(0) based on variation of the selected interval start and end points.
Click the “OK” button to keep the results.
- Checkpoint: If you now select the GI scattering profile, in the information panel at the top you should see the R_g and I(0) that you just found.
- Note: Clicking the “Cancel button will discard the results.
Repeat the Guinier analysis for lysozyme.
- Try: Increase q_min and/or decrease q_max to verify that the R_g does not change significantly in the Guinier region.
- Tip: If you hover your mouse cursor over the info icon (just left of the target icon) for a given scattering profile it should show you the R_g and I(0) of your Guinier analysis.

Molecular weight analysis¶

RAW provides four forms of molecular weight analysis:

Referencing I(0) to that of a known standard
From the volume of correlation using the method of Rambo and Tainer
From the adjusted Porod volume using the method of Fisher et al.
From the value of I(0) on an absolute scale.

In RAW, right click on the subtracted GI scattering profile in the manipulation panel and select “Molecular weight.” At the top of the panel is a description of the methods used, and the results of your Guinier fit. All four methods require a good Guinier fit, so you can use that button to redo the fit if necessary. In the lower part of the panel, the results of the four estimates for MW are shown.
- Note: Neither the I(0) Ref. MW panel nor the Abs. MW panel should be reporting a MW.
In either concentration box, enter the sample concentration of 0.47 mg/ml. Notice that you now get results from all four methods of MW calculation.
- Question: The expected MW value for GI is 172 kDa. How do your results compare?
Click on the “Show Details” button for the Vc MW panel. You should see a graph, which shows the integrated area of $qI(q)$ vs. q. For this method to be accurate, this value needs to converge at high q.
Click the “OK” button to save your analysis.
- Note: The “Cancel” button discards the analysis.
Repeat the MW analysis for the lysozyme sample, which had a concentration of 4.27 mg/ml. The expected MW of lysozyme is 14.3 kDa.
- Question: Does the Vc method work for the lysozyme data?

Saving analysis information¶

Save your subtracted scattering profiles in the standards_data folder.
Select both subtracted profiles, right click on one of them, and select ‘Save all analysis info.’ Give it an appropriate name and save it in the standards_data folder.
- Note: This saves a .csv file with all of the analysis information for the selected scattering profiles.
- Try: Open the .csv file in Microsoft Excel or Libre/Open Office Calc. You should see all of the analysis that you just did.
Remove the subtracted scattering profiles from RAW by selecting both of them and clicking the “Remove” button.
Load the saved subtracted scattering profiles back into RAW. Note that if you select one in the Manipulation list, the information panel in the upper left corner of RAW populates with analysis information. The analysis information is saved with the scattering profile, so if you forget to save it in a .csv, you can load in the profiles later and do it then.
- Note: To get new files to show up in the file tab, you may have to click the refresh button. Also, make sure to that your file type filter is either All files or DAT files.
- Try: Open the saved subtracted scattering profile S_A_GI2_A9_19_001_0000.dat in a text editor such as Notepad (windows) or TextEdit (mac). You should see all of the data in three columns, followed by header information. If you scroll down far enough, the header information contains all of the analysis information, as well as the files that were averaged and subtracted to make the scattering profile.

Kratky analysis¶

A Kratky plot is a plot of $q^2I(q)$ vs. q. Kratky plots can qualitatively assess the flexibility and/or degree of unfolding in samples. Unfolded (highly flexible) proteins should have a plateau in the Kratky plot at high q, while compact, globular proteins will have a bell-shaped (Gaussian) peak. A partially unfolded (flexible) protein may have a combination of the bell-shape and plateau, or a plateau that slowly decays to zero.

Normalized Kratky plots are plots of $q^2I(q)/I(0)$ vs. q. This normalizes scattering profiles by mass and concentration. Dimensionless Kratky plots are presented as either $(qR_g)^2I(q)$ vs. $qR_g$or $(q^2V_c)I(q)$ vs. $q(V_c)^{1/2}$. These dimensionless plots can provide semi-quantitative analysis of flexibility and disorder. More information about can be found here and references therein: http://www.bioisis.net/tutorial/21.

Put the top plot on Kratky axes.
- Tip: Right click on the plot to change the plot type.
Show only the top plot by clicking on the 1 in the plot control bar below the plots.
Both GI and lysozyme show the classic bell shape, indicating they are completely folded.
- Warning: Bad buffer subtraction can also result in a Kratky plot that appears to show some degree of flexibility. Excellent buffer subtraction is required for accurately analysis with this technique.
Load the two scattering profiles in the Tutorial_Data/flexibility_data folder.
- Note: The unfolded.dat file is the scattering profile of an unfolded lysine riboswitch. The partially_folded.dat file is same lysine riboswitch, but in the biologically functional configuration. The data were downloaded from the BIOISIS database, and has the BIOISIS ids of 2LYSRR and 3LYSRR.
SAXS data can be presented on an arbitrary scale, which is why these two profiles have intensity that is much larger than the lysozyme and GI data (which is on an absolute scale). Use the triangle button for each item in the manipulation menu to show more options. Hide one of the newly loaded data sets, and adjust the scale factor on the other until you can comfortably see it and your lysozyme and GI data. Repeat the scale adjustment for the other data set.
- Tip: The up and down arrows will only adjust the last digit of the scale factor.
Kratky analysis can also be done on normalized or dimensionless data. RAW supports normalization by I(0), and non-dimensionalization by R_g and Vc (the volume of correlation).
Select all four loaded scattering profiles, right click, and select the Normalized Kratky Plot option.
Normalized Kratky plots require Guinier analysis to be done. If one or more profiles are missing this information, RAW will show the following window. You can either cancel, and do the fits manually, or you can proceed with RAW’s automatic determination.
Click the Proceed using AutoRg button to proceed to the Normalized Kratky Plot window using RAW’s automatic fitting for R_g.
By default, the plot is the Dimensionless R_g plot. Use the dropdown “Plot” menu at the top to select the Normalized (by I(0) and Dimensionless Vc plots.
Return to the Dimensionless R_g plot. Use the check boxes to hide the partially_folded and unfolded data sets on the plot. Note that both the lysozyme and GI data look very similar on this plot, showing they have similar shapes and (lack of) flexibility.
- Tip: You can click on the colored line in the Color column to change the color of an item on the plot.
Right click on the plot and select “Export Data As CSV” to save the dimensionless data for further processing or plotting with another program.
Click the Close button to close the Normalized Kratky Plot window.

Similarity Testing¶

RAW has the ability to test scattering profiles for statistical similarity. Currently, only one test is available: the Correlation Map test. This can be done manually, and is also done automatically when scattering profiles are averaged. This can be useful when you’re dealing with data that may show changes in scattering from radiation damage or other possible sources.

Clear any data loaded into RAW. Load all of the profiles in the Tutorial_Data/damage_data folder into the main plot. Show only the top plot.
- Tip: In the Files tab, click the “Clear All” button.
Put the plot on a log-log scale. You should see that the profiles are different at low q.
- Note: These data are showing what radiation damage looks like in a data set. They are consecutive profiles from the same sample, and as total exposure of the sample increase (frame number increases), the sample damages. In this case, the damage is manifesting as aggregation, which shows up as an uptick in the profiles at low q.
Select all of the profiles and average them. You will get a warning message informing you that not all the files are statistically the same.
- Note: This is only as good as the statistical test being used, and the cutoff threshold selected. In the advanced options panel you can select the test, whether or not it is corrected for multiple testing, and the threshold used.
Click the “Average Only Similar Files” button.
- Note: This averages only those profiles found to the same as the first file, for the given statistical test.
Select all of the profiles except the new averaged one, and right click and select “Similarity Test”.
The similarity testing window (above) shows the results of the pairwise tests done using the CorMap method. Expand the window and the Filename columns to allow you to see the full filenames along with the probabilities.
Using the menu at the top, turn off multiple testing correction. Change the highlight less than value to 0.15, and highlight those pairs.
Without multiple testing correction, and using a less stringent threshold for similarity, we see that more profiles are selected here (profiles 6-10) than were excluded from the average using the automatic test. Because we know radiation damage increases with dose, it is reasonable to suspect that we should discard profiles 6-10, not just 8-10 as in the automated version.
Save the similarity test data as a .csv by clicking the “Save” button.
Close the similarity testing window by clicking the “Done” button.
Average profiles 1-5.
Hide all of the profiles except the two averaged profiles on the plot.
- Question: Is there a difference between the two? What about if you do a Guinier fit?
- Note: In this case, the differences are subtle, a ~1-2% increase in R_g. So the automated determination did a reasonable job. However, it is generally good to double check your set of profiles both visually and using the Similarity Test panel when the automated test warns you of outlier profiles.

Basic SEC-SAXS processing¶

In a typical SEC-SAXS run, images are continuously collected while the eluate (outflow) of a size exclusion column flows through the SAXS sample cell. As proteins scatter more strongly than buffer, a plot of total scattered intensity vs. time, the so-called SAXS chromatograph, will show a set of peaks similar to what is seen by UV absorption measurement of the SEC system. RAW includes the capability to do routine processing of SEC-SAXS data. This includes creating the SAXS chromatograph from the data, plotting R_g, MW, and I(0) across the peaks, and extracting specific frames for further analysis.

Note: In RAW, this is called Series analysis, as the same tools can be used for other sequentially sampled data sets.

Clear any data loaded into RAW. Click on the Series tab in the control panel. Click the “Select file in series” button. Navigate to the Tutorial_Data/sec_data/sec_sample_1 folder and select any of the .dat files in the folder.
- Tip: In the Files tab, click the “Clear All” button.
- Troubleshooting: If you get an error message, it means you don’t have a configuration file loaded. Load the SAXS.cfg file referenced earlier.
The SEC run will automatically load. RAW should automatically show you the Series plot panel. If not, click on the Series tab in the plot panel.
- Try: Each point on this curve is the integrated intensity of a scattering profile. You can figure out which one by right clicking on the filename in the Series list and selecting ‘Show data’. This will show you the frame number and integrated intensity displayed on the plot, and the filename corresponding to the displayed frame number.
Drag the plot so that you can clearly the see the first frame. You’ll notice it has a significantly lower intensity than the rest of the frames. This happens occasionally at the MacCHESS G1 beamline (where the data was taken). It can make it harder to tell what the data is doing.
- Tip: Select the crossed arrows in the plot control bar, and then click and drag on the plot to move the curve around on the screen.
Go to the files tab and navigate to the sec_sample_1 data directory. Click on the second data file, profile_001_0001.dat. Scroll down to the bottom of the file list, and shift click on the last file, profile_001_0964.dat. This should highlight all of the files in between, as well as the two you clicked on.
Click on the “Plot Series” button. You will see the same curve plotted as before, but without the very first scattering profile. Remove the other loaded data set. Now you should have a curve where the baseline is very close to the bottom of the plot.
In some cases it is more useful to look at the mean intensity or the intensity at a specific q value than the total intensity. Right click on the plot and select mean intensity for the left axis y data. Then try the intensity at q=0.02.
- Note: You need to have the drag button in the plot control bar unselected to get a right click menu.
Return to plotting the integrated intensity. Zoom in near the base of the peak. Notice that there are two smaller peaks on the left, likely corresponding to higher order oligomers that we don’t have the signal to properly resolve. Also notice that the baseline after the peak is not the same as the baseline before the peak. This can happen for several reasons, such as damaged protein sticking to the sample cell windows.
- Tip: Click on the magnifying glass at the bottom of the plot, then click and drag on the plot to select a region to zoom in on.
Zoom back out to the full plot.
- Tip: Click the Home (house) button at the bottom of the plot.
In order to determine if we really have a single species across the peak, we will calculate the R_g and MW as a function of frame number. In the “Calculate/Plot Structural Parameters” section, enter a “Buffer Range” of 400 to 500 and a “Window Size” of 5. Star the SEC curve of interest and click the “Set/Update Parameters” button. This may take a while to calculate.
- Note: All of the files in the given buffer range will be averaged and used as a buffer. A sliding average window is then moved across the SEC curve, of size specified by the Window Size parameter. So for a window of size five, the profiles corresponding to frames 0-4, 1-5, 2-6, etc will be averaged. From each of these averaged set of curves, the average buffer will be subtracted, and RAW will attempt to calculate the R_g, MW, and I(0). These values are then plotted as a function of frame number.
- Note: If you had RNA instead of protein, you would use the Mol. Type menu to select that option. This affects the calculation of the molecular weight.
- Warning: It is important that the buffer range actually be buffer! In this case, we made sure to not include the small peaks before the main peak.
Once the calculation is finished, you should see a set of markers, matching the color of the original curve. These points are plotted on the right Y axis. Click on the green line next to the star in the Series control panel. In the line properties control panel this brings up, change the Calc Marker color to something different. Add a line to the Calc Markers by selecting line style ‘-’ (solid), and adjust the line color to your liking.
- Tip: You can do the same thing to adjust the colors of the scattering profiles in the Manipulation and IFT control tabs.
Zoom in on the peak region. You’ll notice a region of roughly constant R_g across the peak. To either side there are regions with higher or lower R_g values. These variations, particularly on the right side, are from scattering profiles near the edge of the peak with lower concentrations of sample, leading to more noise in determining the R_g values.
- Note: There may also be some effects from the small peaks on the leading (left) side of the peak, and from the baseline mismatch between left and right sides of the peak.
You can move your mouse cursor across the R_g values on the plot, and the frame number and intensity and R_g at your cursor position are read out in the bar at the bottom of the RAW window. Use this to determine the range of frames over which the R_g is roughly constant.
- Note: For an automated determination of R_g, particularly with only 5 frames averaged together, a change of several percent is likely insignificant.
Zoom back out to the full plot. Right click on the plot and select molecular weight as the right axis Y data. Again zoom in on the peak region and find the set of frames for which the MW is roughly constant.
- Try: Vary the window size and/or the buffer range and see how that affects the constant R_g and MW regions.
Enter the buffer range, 400 to 500, in the “Select Data Frames” boxes of the “Data to main plot” section, and then click the “Average to Main Plot” button.
Enter the range over which you found the R_g and MW to be constant (should be ~700-715) in the “Select Data Frames” section and click the “Average to Main Plot” button.
Click on the Main Plot tab and the Manipulation tab. You should see two scattering profiles, one is the average of the buffer and one is the average across the peak. Carry out buffer subtraction and then do a Guinier and MW analysis.
- Note: The I(0) reference and absolute calibration will not be accurate for SEC-SAXS data, as the concentration is not accurately known.
- Question: How does the R_g and MW you get from the averaged curve compare to what RAW found automatically for the peak?
- Tip: Make sure your plot axes are Log-Lin or Log-Log. Make sure that both plots are shown by clicking the 1/2 button at the bottom of the plot window.
Generate a new average buffer from the frames on the right side of the peak, 850-950. Generate a new subtracted curve and repeat the R_g and MW analysis.
- Question: Which curve looks best?
Try taking a few small sections of the peak, 5-10 frames wide. Use one on the left side of the peak, one at the top, and one on the right side (e.g. 685-690, 700-705, 725-730). Generate subtracted curves from the first buffer (frames 400-500). Carry out the R_g and MW analysis.
- Question: Are there any differences in these curves?
- Try: Apply a scale factor to these new subtracted curves. Can you scale them onto each other?
- Note: It is useful to analyze several regions on the peaks of the SEC-SAXS curve in this way to verify that they are the same. You could have species that failed to separate out completely. This kind of analysis will give you confidence in your final result.
Load the Bovine Serum Albumin (BSA) SEC-SAXS data contained in the sec_sample_2 data folder. Hide the first SEC-SAXS chromatograph.
Select a good buffer region, and calculate the R_g and MW across the peak for the BSA.
- Warning: Don’t forget to star the curve you want to set/update parameters for!
- Tip: If you hover your mouse cursor over the info icon, you will see the buffer range and window size used to calculate the parameters.
- Question: Is the BSA peak one species?
Find the useful region of the peak (constant R_g/MW), and send the buffer and sample data to the main plot. Carry out the standard R_g and MW analysis on the subtracted scattering profile. For BSA, we expect R_g ~28 Å and MW ~66 kDa.
- Try: As with the previous sample, take a few smaller regions along the peak and see if the subtracted scattering profile varies.
In the Series control tab, right click on the name of BSA curve in the list. Select export data and save it in an appropriate location. This will save a CSV file with the frame number, integrated intensity, radius of gyration, molecular weight, filename for each frame number, and a few other items. This allows you to plot that data for publications, align it with the UV trace, or whatever else you want to do with it.
- Try: Open the .csv file you just saved in Excel or Libre/Open Office Calc.
Select both items in the Series control panel list, and save them in the sec_data folder. This saves the Series plot data in a form that can be quickly loaded by RAW.
- Try: Clear the Series data and then open one of your saved files from the Files tab using either the “Plot” or “Plot Series” button.

WAXS processing and merging¶

Several SAXS beamlines use two (or more) detectors to collect different q regions. The MacCHESS G1 beamline uses dual Pilatus detectors to measure SAXS and WAXS from q ~0.008 – 0.75 Å^-1. The SAXS detector has q ~< 0.25 Å^-1 and the wide-angle scattering (WAXS) data has q >~ 0.25 Å^-1. All of the data that you have been working with so far has been SAXS data. Some experiments can make use of the WAXS data. In this part of the tutorial you will learn the basics of processing it.

Clear any data in RAW.
Navigate to the standards_data and load the WAXS.cfg file.
Plot the lysbuff2 and lys2 PIL3 files. These are the images from the WAXS detector. Average these files and create a subtracted WAXS scattering profile.
Load the saved subtracted SAXS scattering profile for the lysozyme standards data.
- Note: You should have saved it in the standards_data folder, and it is likely named S_A_lys2_A9_17_001_0000.dat.
Move the SAXS scattering profile you just loaded to the bottom plot by right clicking on it in the Manipulation list and selecting “Move to bottom plot.”
The WAXS data is not on the same scale as the SAXS data. For this data, the known scale factor to apply is 0.000014 to the WAXS data.
- Note: The scale factor can be calculated as the ratio of solid angles subtended by the pixels on the SAXS and WAXS detectors, plus any scale factor for absolute calibration and normalization included for one curve but not the other.
Star the WAXS data. Right click on the SAXS data and select merge. This will create a new merged scattering profile. The new file will have the prefix M_ to indicate it is a merged file.
- Tip: If you can’t see it, that’s probably because it appeared on the upper plot, and is hidden by the very large intensities of the averaged WAXS files. Either try hiding those, or move the Merged curve to the lower plot.

A few additional tricks¶

Here are some additional tricks that may make your life easier while using RAW:

If you click on a scattering profile in the main plot, the corresponding manipulation list item will be highlighted.
You can save the workspace by going to File->Save Workspace. This will save all of the scattering profiles, IFT curves, and Series curves. These will all be loaded again when you load the workspace.
- Note: This does not save the settings!
If you go to Options -> Advanced Options -> Molecular weight, you can change the default type of molecule used in the MW estimation from the volume of correlation. This affects the default option selected in the MW window.
If you have the crossed arrows selected in the plot control bar to drag a plot, right clicking and dragging allows you to zoom a plot.
You can turn error bars on and off for scattering profiles using the error bar button in the plot control (to the right of the save button).
You can rename a curve by right clicking on the appropriate entry in the list and choosing rename.
You can view the history of a scattering profile by right clicking on it and selecting Show History. For a curve that has been processed from an image, this will show you processing parameters such as normalization and any corrections applied to the scattering intensity. For a curve that is processed (such as an averaged of subtracted curve) it will show you the steps used to make that curve. For example, for an averaged curve, it will show you all of the files that were averaged.

Advanced processing¶

This section will guide you through advanded processing of files in RAW. It includes: pair-distance distribution analysis using GNOM and BIFT methods, ambiguity assessment for shape reconstructions, 3D reconstructions with bead models and electron density, and data deconvolution using singular value decomposition (SVD) and evolving factor analysis (EFA). It refers to the RAW tutorial data.

Select a section below to view the tutorial, or use the next and back buttons at the bottom of the page to navigate through it in order.

Pair-distance distribution analysis – GNOM in RAW¶

The first step in most advanced data processing is to calculate the P(r) function, the inverse Fourier transform of I(q). This cannot be calculated directly from the scattering profile, so indirect Fourier transform methods are typically used. The most common such method is implemented in the GNOM program from the ATSAS package. We will use RAW to run GNOM. Note that you need ATSAS installed to do this part of the tutorial.

Open RAW. The install instructions contain information on installing and running RAW.
Open the lysozyme.dat file in the Tutorial_Data/atsas_data folder.
Right click on the lysozyme profile in the Manipulation list and select “IFT (GNOM)”.
- Note: RAW will automatically try to find an appropriate maximum dimension (D_max) by running the DATGNOM program from the ATSAS software package.
- Troubleshooting: If you do not have the GNOM option in the right click menu, RAW does not know where your ATSAS programs are installed. If you installed the ATSAS programs after starting RAW, restart RAW and it will attempt to automatically find them. If that has failed, go to the Options->Advanced Options menu and choose the ATSAS settings (“ATSAS”). Uncheck the option to “Automatically find the ATSAS bin location”, and specify the location manually.
The GNOM panel has plots on the right. These show the P(r) function (top panel), the data (bottom panel, blue points) and the fit line (bottom panel, red line).
- Note: The fit line is the Fourier transform of the P(r) function, and is also called the regularized intensity.
On the left of the GNOM panel are the controls and the resulting parameters. You can alter the data range used and the D_max value.
- Tip: The Guinier and P(r) R_g and I(0) values should agree well. The total estimate varies from 0 to 1, with 1 being ideal. GNOM also provides an estimate of the quality of the solution. You want it to be at least a “REASONABLE” solution.
Vary the D_max value up and down in the range of 30-50 in steps of 1. Observe what happens to the P(r) and the quality of the solution.
- Note: D_max is in units of Å.
- Tip: Recall that we want the following qualities in a P(R) function:
  1. No negative values.
  2. A gentle approach to D_max (not suddenly forced down).
  3. Minimal oscillation.
Return the D_max value to that found by DATGNOM by clicking the “DATGNOM” button. D_max should be 40. By default, GNOM forces the P(r) function to zero at D_max. For a high quality data set and a good choice of D_max, P(r) should go to zero naturally. Change the “Force to 0 Dmax” option to “N”.
- Try: Vary D_max with this option turned off.
Change the advanced parameters so that the P(r) function is again being forced to zero at D_max.
Set the D_max back to 40, and click OK. This saves the results into the RAW IFT panel.
Click on the IFT Control and Plot tabs. This will display the GNOM output you just generated. Save the lysozyme.out item in the atsas_data folder.
- Note: This saved file is all of the GNOM output, in the GNOM format. It can be used as input for any program that needs a GNOM .out file.

Pair-distance distribution analysis – BIFT in RAW¶

RAW also has a built in method for determining the P(r) function using a Bayesian IFT method. This has the advantage of only have one possible solution. More information on this method can be found in the RAW paper and manual and references therein.

Right click on the lysozyme profile in the Manipulation list you loaded in Part 1 and select “BIFT”.
The BIFT panel has plots on the right. These show the P(r) function (top panel), the data (bottom panel, blue points) and the fit line (bottom panel, red line).
On the left of the BIFT panel are the controls and the resulting parameters. Note that in this case you do not control the D_max value, the BIFT method finds that for you automatically.
Click OK to exit the BIFT window. This saves the results into the RAW IFT panel.
Click on the IFT Control and Plot tabs. This will display the BIFT output you just generated. Save the lysozyme.ift item in the standards_data folder.

Note: As of now, BIFT output from RAW is not compatible with DAMMIF or other ATSAS programs. However, it is compatible with electron density determination via DENSS.

Assessing ambiguity of 3D shape information - AMBIMETER in RAW¶

It is impossible to determine a provably unique three-dimensional shape from a scattering profile. This makes it important to determine what degree of ambiguity might be expected in our reconstructions. The program AMBIMETER from the ATSAS package does this by comparing the measured scattering profile to a library of scattering profiles from relatively simple shapes. The more possible shapes that could have generated the scattering profile, the greater ambiguity there will be in the reconstruction. We will use RAW to run AMBIMETER. Note that you need ATSAS installed to do this part of the tutorial.

Clear all of the data in RAW. Load the lysozyme.out file that you saved in the atsas_data folder in a previous part of the tutorial.
- Note: If you haven’t done the previous part of the tutorial, or forgot to save the results, you can find the lysozyme.out file in the atsas_data/lysozyme_complete folder.
Right click on the lysozyme.out item in the IFT list. Select the “Run AMBIMETER” option.
The new window will show the results of AMBIMETER. It includes the number of shape categories that are compatible with the scattering profile, the ambiguity score (a-score) (log base 10 of the number of shape categories), and the AMBIMETER interpretation of whether or not you can obtain a unique 3D reconstruction.
- According to the original paper, “an a-score below 1.5 practically guarantees a unique ab initio shape determination, whereas when the a-score is in the range 1.5–2.5 care should be taken, perhaps involving cluster analysis, and for a-scores exceeding 2.5 unambiguous reconstruction without restrictions (for example, on symmetry and/or anisometry) is highly unlikely.”
- Note: AMBIMETER can also save the compatible shapes (either all or just the best fit). You can do that by selecting the output shapes to save, giving it a save directory, and clicking run. We won’t be using those shapes in this tutorial.
Click “OK” to exit the AMBIMETER window.

3D reconstruction with bead models – DAMMIF/N and DAMAVER in RAW¶

Shape reconstruction in SAXS is typically done using bead models (also called dummy atom models, or DAMs). The most common program used to generate these shapes is DAMMIF (and, to a lesser degree, DAMMIN) from the ATSAS package. We will use RAW to run DAMMIF/N. Because the shape reconstruction is not unique, a number of distinct reconstructions are generated, and then a consensus shape is made from the average of these reconstructions. The program DAMAVER from the ATSAS package is the most commonly used program for building consensus shapes. Note that you need ATSAS installed to do this part of the tutorial.

Clear all of the data in RAW. Load the lysozyme.out file that you saved in the atsas_data folder in a previous part of the tutorial.
- Note: If you haven’t done the previous part of the tutorial, or forgot to save the results, you can find the lysozyme.out file in the atsas_data/lysozyme_complete folder.
Right click on the lysozyme.out item in the IFT list. Select the “Bead Model (DAMMIF/N)” option.
Running DAMMIF generates a lot of files. Click the “Select/Change Directory” button, make a new folder in the atsas_data directory called lysozyme_dammif and select that folder.
Change the number of reconstructions to 5.
- Note: It is generally recommended that you do at least 10 reconstructions. However, for the purposes of this tutorial, 5 are enough.
- Note: For final reconstructions for a paper, DAMMIF should be run in Slow mode. For this tutorial, or for obtaining an initial quick look at results, Fast mode is fine.
Uncheck the “Refine average with dammin” checkbox.
- Note: For final reconstructions for a paper, DAMMIN refinement should be done. However, it is quite slow, so for the purposes of this tutorial we won’t do it.
Click the “Start” button.
- Note: The status panel will show you the overall status of the reconstructions. You can look at the detailed status of each run by clicking the appropriate tab in the log panel.
Note that by default the envelopes are aligned and averaged using DAMAVER, and then the aligned and averaged profile is refined using DAMMIN.
- Some settings are accessible in the panel, and all settings can be changed in the advanced settings panel.
Wait for all of the DAMMIF runs and DAMAVER to finish. Depending on the speed of your computer this could take a bit.
Once the reconstructions are finished, the window should automatically switch to the results tab. If it doesn’t, click on the results tab.
The results panel summarizes the results of the reconstruction run. At the top of the panel there is the ambimeter evaluation of how ambiguous the reconstructions might be (see previous tutorial section). If DAMAVER was run, there are results from the normalized spatial discrepancy (NSD), showing the mean and standard deviation of the NSD, as well as how many of the reconstructions were included in the average. If DAMAVER was run on 3 or more reconstructions, and ATSAS >=2.8.0 is installed, there will be the output of SASRES which provides information on the resolution of the reconstruction. If DAMCLUST was run (not shown) there will be information on the clustering. Finally, there will be information on each individual model, including the model chi squared, R_g, D_max, excluded volume, molecular weight estimated from the excluded volume, and, if appropriate, mean NSD of the model.
- Tip: Any models are rejected from the average by DAMAVER will be shown in red in the models list.
Click the “Save Results Summary” button to save the results summary as a .csv file.
Click on the Viewer tab to open the model viewer
- Note: The model viewer is intended for a fast first look at the results. It is not currently up to the standards of a program like pyMOL.
Click and drag the model to spin it.
- Note: For lysozyme, it should look more or less like a flattened sphere.
Right click and drag the model to zoom in and out.
Use the “Model to display” menu in the Viewer Controls box to change which reconstruction is displayed.
Click the “Close” button when you are finished looking at the results and reconstructions.
The results from individual DAMMIF runs are saved in the selected output folder with the name <prefix>_xx, where xx is the run number: 01, 02, etc. For this tutorial, that would be lysozyme_01, lysozyme_02, and so on. The different files produced are described in the DAMMIF manual.
- Note: Generally, the file of interest is the -1.pdb file, in this case lysozyme_01-1.pdb, lysozyme_02-1.pdb, etc.
If averaging was done with DAMAVER, the results are saved in the selected output folder with the given prefix, in this case lysozyme. The output files generated are described in the DAMAVER manual.
- Note: Generally, the files of interest are the generated pdbs: <prefix>_damaver.pdb and <prefix>_damfilt.pdb. For this tutorial, those would be lysozyme_damaver.pdb and lysozyme_damfilt.pdb.
If clustering was done with DAMCLUST, the results are saved in the selected output folder with the given prefix (for this tutorial, lysozyme). The files generated are described in the DAMCLUST manual.
If refinement was done with DAMMIN, the results are saved in the selected otuput folder as refine_<prefix>, e.g. for this tutorial refine_lysozyme. The files generated are described in the DAMMIN manual.
- Note: Generally, the file of interest is the -1.pdb file, in this case refine_lysozyme-1.pdb.

3D reconstruction with electron density – DENSS and EMAN2 in RAW¶

A new, exciting method for doing 3D shape reconstructions in SAXS yields actual electron density, rather than bead models. There are many potential advantages to this, but one significant one is easy handling of systems like RNA-Protein complexes or membrane proteins surrounded by lipids or detergents, which have more than one electron density. Bead models typically only have two (molecule and solvent) or three bead densities, and so typically fail to reconstruct these complex objects. DENSS has been fully implemented in RAW and will be used to recontruct these electron densities. In order to align and average these densities, EMAN2 will be used. Note that you need EMAN2 installed to do this part of the tutorial.

IMPORTANT NOTE: Unfortunately, the EMAN2 package doesn’t work properly on Windows. Windows users can still use RAW to generate electron densities, but they cannot create a consensus average or filter out enantiomers. Mac OS (or OS X) and Linux users can do everything described in this tutorial.

Clear all of the data in RAW. Load the lysozyme.out file that you saved in the atsas_data folder in a previous part of the tutorial.
- Note: If you haven’t done the previous part of the tutorial, or forgot to save the results, you can find the lysozyme.out file in the atsas_data/lysozyme_complete folder.
Right click on the lysozyme.out item in the IFT list. Select the “Electron Density (DENSS)” option.
Running DAMMIF generates a lot of files. Click the “Select/Change Directory” button, make a new folder in the atsas_data directory called lysozyme_denss and select that folder.
Change the number of reconstructions to 5 and the mode to Fast.
- Note: It is generally recommended that you do at least 20 reconstructions. However, for the purposes of this tutorial, 5 are enough.
- Note: For final reconstructions for a paper, DENSS should be run in Slow mode. For this tutorial, or for obtaining an initial quick look at results, Fast mode is fine.
Uncheck the “Filter enantiomers (EMAN2)” checkbox.
- Note: For final reconstructions for a paper, you should filter enantiomers. However, it is quite slow, so for the purposes of this tutorial we won’t do it.
- Note: SAXS data can’t determine between enantiomers, so all this does is ensure that the reconstructions and averaging are done on the most similar enantiomers. There is not guarantee that this matches the enantiomer in solution.
Click the “Start” button.
- Note: The status panel will show you the overall status of the reconstructions. You can look at the detailed status of each run by clicking the appropriate tab in the log panel.
Note that by default the envelopes are aligned and averaged using EMAN2.
- Some settings are accessible in the panel, and all settings can be changed in the advanced settings panel.
Wait for all of the DENSS runs and EMAN2 averaging to finish. Depending on the speed of your computer this could take a bit.
Once the reconstructions are finished, the window should automatically switch to the results tab. If it doesn’t, click on the results tab.
The results panel summarizes the results of the reconstruction runs. At the top of the panel there is the ambimeter evaluation of how ambiguous the reconstructions might be (see previous tutorial section). If EMAN2 averaging was run there is an estimate of the reconstruction resolution based on the Fourier shell correlation.
For each individual model there are plots of: the original data and the model data (scattering from density); the residual between the original data and the model data; and chi squared, R_g and support volume vs. refinement step.
- Verify that the residual between the actual data and the model data is small.
- Check that the chi squared, R_g, and support volume have all plateaued (converged) by the final steps.
If the plots were averaged, the average tab will display the Fourier shell correlation vs. resolution.
- Note: The reconstruction resolution is taken as the resolution in angstrom where the correlation first crosses 0.5.
Click the “Save Results Summary” button to save the results summary as a .csv file and save the summary plots as a multi-page pdf file.
Click the “Close” button when you are finished looking at the results and reconstructions.
The results from the individual DENSS runs are saved in the selected output folder as <prefix>_xx.mrc where xx corresponds to the run number: 01, 02, etc. For this tutorial that would be lysozyme_01.mrc, lysozyme_02.mrc, etc.
If averaging was done, final average density is saved in the selected output folder as <prefix>_aver.mrc. For this turotial, that would be lysozyme_aver.mrc
- Note: .mrc files can be opened in Chimera and pyMOL. For tips about how to visualize the density and align it with known structures see the appropriate sections here: http://www.tdgrant.com/denss/tips/. When looking at this page, please note that RAW does the conversion from hdf5 to mrc for you, so there’s no need to do this, i.e. you can skip the first paragraph of the evaluating the results section.

Advanced SEC-SAXS processing – Singular value decomposition (SVD) and evolving factor analysis (EFA)¶

Sometimes SEC fails to fully separate out different species, and you end up with overlapping peaks in your SEC-SAXS curve. It is possible to apply more advanced mathematical techniques to determine if there are multiple species of macromolecule in a SEC-SAXS peak, and to attempt to extract out scattering profiles for each component in an overlapping peak. Singular value decomposition (SVD) can be used to help determine how many distinct scatterers are in a SEC-SAXS peak. Evolving factor analysis (EFA) is an extension of SVD that can extract individual components from overlapping SEC-SAXS peaks.

Clear all of the data in RAW. Load the phehc_sec.sec file in the sec_data folder.
- Note: The data were provided by the Ando group at Princeton University and is some of the data used in the paper: Domain Movements upon Activation of Phenylalanine Hydroxylase Characterized by Crystallography and Chromatography-Coupled Small-Angle X-ray Scattering. Steve P. Meisburger, Alexander B. Taylor, Crystal A. Khan, Shengnan Zhang, Paul F. Fitzpatrick, and Nozomi Ando. Journal of the American Chemical Society 2016 138 (20), 6506-6516. DOI: 10.1021/jacs.6b01563
Right click on the phehc_sec.sec item in the SEC list. Select the “SVD” option.
The SVD window will be displayed. On the left are controls, on the right are plots of the value of the singular values and the first autocorrelation of the left and right singular vectors.
- Note: Large singular values indicate significant components. What matters is the relative magnitude, that is, whether the value is large relative to the mostly flat/unchanging value of high index singular values.
- Note: A large autocorrelation indicates that the singular vector is varying smoothly, while a low autocorrelation indicates the vector is very noisy. Vectors corresponding to significant components will tend to have autocorrelations near 1 (roughly, >0.6-0.7) and vectors corresponding to insignificant components will tend to have autocorrelations near 0.
Adjust the starting frame number to 100, the ending frame number to near 300, and switch to using Subtracted data.
- Note: The blue points are in the plot on the left are the region being used for SVD, while the red points shows the rest of the SEC-SAXS curve.
We have now isolated the peak. Looking at the top plot, we see there are two singular values significantly above the baseline level, and from the autocorrelation we see two values with both left and right singular vectors autocorrelations near 1. This indicates that there are two scattering components in the peak, even though there are no obvious shoulders in the region we selected
- Try: Adjust the starting and ending values and seeing how that changes the SVD results. Is there a region of the peak you can isolate that has just one significant component?
- Note: Normally, changing between Unsubtracted and Subtracted SEC-SAXS profiles should remove one significant singular value component, corresponding to the buffer scattering. In this data, you will see almost no difference, as the profiles used to produce the SEC-SAXS curve were already background subtracted.
- Note: You can save the SVD plots by clicking the Save button, as with the plots in the main RAW window. You can save the SVD results, either just the plotted values or all of the values, using the two Save buttons in the SVD panel.
Close the SVD window by clicking the OK button.
We will now use EFA to attempt to extract out the two scattering components in the main peak in this data. Right click on the phehc_sec.sec item in the SEC list. Select the “EFA” option.
For successful EFA, you want to use Subtracted data, and you typically want to have a long buffer region before and after the sample. For this data set, using the entire frame range (from 0 to 385) is appropriate. With other data sets, you may need to change the frame range to, for example, remove other, well separated, peaks from the analysis.
RAW attempt to automatically determine how many significant singular values (SVs) there are in the selected range. At the bottom of the control panel, you should see that RAW thinks there are three significant SVs in our data. For this data set, that is accurate.
- Note: You should convince yourself of this by looking at the SVD results in the plots on this page, using the same approach as in Steps 3-5 above.
- Note: There is a hint of a fourth component, likely related to imperfect background subtraction (possibly indicating a little capillary fouling). You can rerun this exercise using four components and see if that changes the results.
Click the “Next” button in the lower right-hand corner of the window to advance to the second stage of the EFA analysis.
- Note: It may take some time to compute the necessary values for this next step, so be patient.
This step shows you the “Forward EFA” and “Backward EFA” plots. These plots represent the value of the singular values as a function of frame.
- Note: There is one more singular value displayed on each plot than available in the controls. This is so that in the following Steps you can determine where each component deviates from the baseline.
In the User Input panel, tweak the “Forward” value start frames so that the frame number, as indicated by the open circle on the plot, aligns with where the singular value first starts to increase quickly. This should be around 148, 165, and 324.
- Note: For the Forward EFA plot, SVD is run on just the first two frames, then the first three, and so on, until all frames in the range are included. As more frames are added, the singular values change, as shown on the plot. When a singular values starts increasingly sharply, it indicates that there is a new scattering component in the scattering profile measured at that point. So, for the first ~150 frames, there are no new scattering components (i.e. just buffer scattering). At frame ~151, we see the first singular value (the singular value with index 0, l abeled SV 0 on the plot) start to strongly increase, showing that we have gained a scattering component. We see SV 1 start to increase at ~167, indicating another scattering component starting to be present in the scattering profile.
In the User Input panel, tweak the “Backward” value start frames so that the frame number, as indicated by the open circle on the plot, aligns with where the singular value first starts to increase quickly, reading the plot left to right (i.e. where it drops back to near the baseline). This should be around 380, 324, and 190.
- Note: For the Backward EFA plot, SVD is run on just the last two frames, then the last three, and so on, until all frames in the range are included. As more frames are added, the singular values change, as shown on the plot. When a singular values starts increasingly sharply (as seen from right to left), it indicates that there is a new scattering component in the scattering profile measured at that point.
- Note: The algorithm for determining the start and end points is not particularly advanced. For some datasets you may need to do significantly more adjustment of these values
Click the “Next” button in the bottom right corner to move to the last stage of the EFA analysis.
This window shows controls on the left and results on the right. In the controls area, at the top is a plot showing the SEC-SAXS curve, along with the ranges occupied by each scattering component, as determined from the input on the Forward and Backward EFA curves in stage 2 of the analysis. The colors of the ranges correspond to the colors labeled in the Scattering Profiles plot on the top right and the Concentration plot in the lower right. This panel takes the SVD vectors and rotates them back into scattering vectors corresponding to real components.
- Note: This rotation is not guaranteed to be successful, or to give you valid scattering vectors. Any data obtained via this method should be supported in other ways, either using other methods of deconvolving the peak, other biophysical or biochemical data, or both!
Fine tune the ranges using the controls in the “Component Range Controls” box. Adjust the start of Range 2 down until it overlaps with Range 1.
- Question: What is the effect on the chi-squared plot?
Adjust the starts and ends of Range 0 and the start of Range 1 by a few points until the spikes in the chi-squared plot go away. After these adjustments, Range 0 should be about 147 to 197, Range 1 from 161 to 324, and Range 2 from 323 to 380.
To see these changes on the Forward and Backward EFA plots, click the “Back” button at the bottom right of the page. Verify that all of your start and end values are close to where the components become significant, as discussed in Steps 12 and 13.
Click the “Next” button to return to the final stage of the EFA analysis.
In the Controls box, you can set the method, the number of iterations, and the convergence threshold. As you can see in the Status window, the rotation was successful for this data. If it was not, you could try changing methods or adjusting the number of iterations or threshold.
Examine the chi-squared plot. It should be uniformly close to 1 for good EFA. For this data, it is.
Examine the concentration plot. You’ll see three peaks, corresponding to the concentrations for the three components. In the Range Controls, uncheck the Range 0 C>=0 box. That removes the constraint that the concentration must be positive. If this results in a significant change in the peak, your EFA analysis is likely poor, and you should not trust your results.
- Note: The height of the concentration peaks is arbitrary, all peaks are normalized to have an area of 1.
Uncheck all of the C>=0 controls.
- Question: Do you observe any significant changes in the scattering profiles, chi-squared, or concentration when you do this? How about if you uncheck one and leave the others checked?
Recheck all of the C>=0 controls. You have now verified, as much as you can, that the EFA analysis is giving you reasonable results.
- Reminder: Here are the verification steps we have carried out, and you should carry out every time you do EFA:
  1. Confirm that your selected ranges correspond to the start points of the Forward and Backward EFA values (Steps 12-13).
  2. Confirm that your chi-squared plot is close to 1, without any major spikes (Step 21).
  3. Confirm that your concentrations are not significantly altered by constraining the concentration to be positive (Steps 22-23).
Click the “Save EFA Data (not profiles)” to save the EFA data, including the SVD, the Forward and Backward EFA data, the chi-squared, and the concentration, along with information about the selected ranges and the rotation method used.
Click the “Done” button to send the scattering profiles to the Main Plot.
In the main RAW window, go to the Manipulation control tab and the Main plot. If it is not already, put the Main plot on a semi-Log or Log-Log scale.
The three scattering profiles from EFA are in the manipulation list. The labels _0, _1, and _2 correspond to the 0, 1, and 2 components/ranges.
- Note: Regardless of whether you use subtracted or unsubtracted data, these scattering profiles will be buffer subtracted, as the buffer represents a scattering component itself, and so (in theory) even if it is present will be separated out by successful EFA.

Creating a configuration file¶

This section will guide you through creating a configuration file for RAW that allows you to integrate 2D images into 1D scattering profiles. It refers to the RAW tutorial data.

Select a section below to view the tutorial, or use the next and back buttons at the bottom of the page to navigate through it in order.

Centering and calibration – Automated method¶

The first step is to set the beam center, x-ray wavelength, and sample to detector distance. Before this can be done, you have to set the image and file header type in the Options window. The best way to find the beam center and sample to detector distance is using the automated method in RAW.

Open RAW. The install instructions contain information on installing and running RAW.
Open the Options window by going to the Options menu (top of the RAW window or in the system bar, depending on your OS) and selecting “Advanced Options”
In the options window, select the Image/Header Format section on the left.
In the area on the right, set the Image format dropdown menu to “Pilatus” and the Header file format to “G1, CHESS”.
Click the OK button to close the window and save the changes to the settings.
In the files tab, click on the folder button and navigate to the Tutorial_Data/calibration_data folder.
Select the AgBeh_A1_43_001_0000.tiff file by clicking on it once and click the show image button at the bottom of the screen.
In the Image Plot Panel that is now showing, click on the Image Display Settings button (looks like slider bars) at the bottom of the screen.
In the window that appears, set the scale to logarithmic and and click “OK”.
Open the Centering/Calibration panel by going to the Tools menu and selecting “Centering/Calibration”.
In the Centering/Calibration panel set the wavelength to 1.2461 Å. Set the detector pixel size to 172.0 μm.
- Note: The x-ray energy/wavelength and detector pixel size are previously known values, and are not found in RAW.
- Tip: You can set the value in two ways. Either using the up/down arrows next to the box (spin controls) or directly typing the value into the box.
The goal of centering and calibration is to find a beam center position and sample to detector distance that causes the displayed Silver-Behenate ring pattern to match up with the rings on the image.
- Note: The beamstop is the dark blue patch extending out from the center of the left edge of the detector.
Checkpoint: You should currently have a screen that looks like the one below.
In the “Manual Centering/Calibration Adjustments” panel, make sure the correct standard is selected, in this case, AgBh (silver behenate).
In the Automatic Centering panel, make sure the correct detector is selected, in this case “pilatus100k”.
- Note: If you cannot find your detector in that list, select “detector”, and make sure that you have entered the correct detector pixel size in the manual centering panel.
Click the “Start” button in the “Automatic Centering/Calibration” panel.
Make sure the “Ring #” is set to 0. Click on a point with strong intensity in the Silver Behenate ring nearest the beamstop (left most ring, in this case, near x = 200).
- Note: For some experimental setups, one or more of the largest d-spacing rings may not be visible on the detector. In this case, you need to figure out what the first visible ring on the detector is, and set the ring number to that. So, if the third ring was the first one on the detector, the Ring # would be set to 2 (the ring number is zero index, so 0 corresponds to the first ring, 1 to the second ring, and so on).
- Try: If you set the Manual settings to approximately right, hover the mouse above the image and use the scroll wheel to zoom out. This will let you see all of the centering rings, and figure out which ring is the first one visible on the image. Once you’re done, hit the Home button to return to the zoom of the entire image.
The peak intensity points in that ring will be automatically found, and labeled with yellow-green dots.
- Note: If it didn’t find very many points, try clicking again on another part of the ring, and it will add more points to your selection.
- Note: If you have the same ring separated by a gap (due to detector module gaps, beamstop shadow, or geometry, click on the separated parts of the rings to add points from all sections. The autofind algorithm will only find peaks in contiguous regions.
- Tip: Due to the color map selected, the points may be hard to see. Try changing to the heat map to see the selected points, like in the image below.
Change the “Ring #” to 1.
Click on a peak intensity point of the second visible ring.
The peak intensity points in that ring will be automatically found, and labeled with blue dots.
Click the “Done” button in the “Automatic Centering/Calibration” panel and beam position, sample to detector distance, and detector pixel size will all be automatically filled in.
- Note: If the automatic centering fails, carry out Steps 5-7 of Part 2. Giving the system starting points that are approximately in the right place can help it refine to the precise location.
In the Image Display Settings, set the color scale back to Linear, and the Upper limit to 9000. You should now be able to easily see the centering rings and beam center on the image.
Click the OK button in the Centering/Calibration panel to save your settings and exit the panel.

Centering and calibration – Manual method¶

If the automated method fails, you can also carry out centering and calibration

If you haven’t already, carry out Steps 1-9 of Part 1, so that the AgBeh_A1_43_001_0000.tiff file is shown in the Image Plot Panel with an upper limit of 90000 set for the image.
In the Image Display settings (see Part 1, Step 8), set the upper limit to 9000 and click “OK” to close the settings window.
Open the Centering/Calibration panel by going to the Tools menu and selecting “Centering/Calibration”.
In the Centering/Calibration panel set the wavelength to 1.2461 Å. Set the detector pixel size to 172.0 μm.
- Note: The x-ray energy/wavelength is a previously known value, and is not found in RAW.
We know that the beam is probably near the center of the beamstop. Click on the crosshairs button and click in the center of the beamstop.
- Note: The beamstop is the dark blue patch extending out from the center of the left edge of the detector.
Checkpoint: Once you have done that, three rings should be displayed on the image, and the current beam center is shown as a red dot on the image. This is shown below.
Set the Sample-Detector Distance to 1500 mm.
- Note: The rough sample to detector distance is known from direct measurement of the experimental setup. Calibration can be done without this knowledge, but will take longer.
Using the zoom tool at the bottom of the Plot window, zoom in on the first ring shown on the detector.
Using the red arrow buttons, move your beam center until the dashed red line (first silver behenate ring) matches with the hottest (most red) pixels of the first ring shown on the detector.
Use the home button to zoom out to the whole extent of the detector image.
Open the image display settings and set the upper limit to 2000. Zoom in on the second silver behenate ring on the image (near x=380). Use the spin controls (up/down arrows) on the sample detector distance to adjust the distance until the red dashed ring matches perfectly with the hottest pixels on the detector image.
Zoom back out to the full extent of the image. Set the image display upper limit back to 9000. Zoom in on the first ring, and adjust the beam center position until that ring perfectly matches the hottest pixels on the detector.
- Tip: If you had the ring properly centered in y the first time, you should only need to adjust the x position (left/right).
Iterate steps 10-13 until both rings are well aligned. You should find an X center of ~4, a Y center of ~ 92.5, and a sample-detector distance of ~1517 mm.
- Tip: You can adjust the step size for moving the beam center using the “Steps” menu. You can either pick a value from the dropdown menu, or enter your own value in the box.
- Tip: The spin controls adjust the last digit of the value in the box. So if you want to adjust the sample-detector distance more finely than 1 mm, set the distance to (for example) 1500.0 and then the spin controls will change the distance in steps of 0.1 mm.
Compare the manually found center and sample detector distance with those you found automatically in Part 1 (if you did that). If you did a careful job in this part, they should compare well.

Masking¶

This section teaches you how to mask out unwanted portions of your image, such as the beamstop and bad detector pixels. It assumes you have just done Part 1 or 2. If not, open RAW as in Step 1 and set your data folder as in Step 6 of Part 1.

In the Files tab, select the MT2_48_001_0000.tiff file and click the show image button.
Set the image upper limit to 50.
Open the masking panel by clicking “Masking” in the Tools menu.
Zoom in around the beamstop.
- Note: The beamstop is the blue rectangular area on the left edge of the detector
Select the Pan tool and left click and drag the image to the right until you can see a blank (white) canvas to the left of the beamstop.
Click on the rectangle mask drawing button and click on a point in the white part of the canvas just above the edge of the beamstop.
Move the cursor until the red outline of the rectangle covers the beamstop from top to bottom, and out to the right edge of the middle of the beamstop. Click again to make the rectangle mask.
- Note: There will still be a bit of the beamstop at the bottom right edge that is not covered by this rectangle.
- Tip: If you mess up drawing the mask, click on the masked region (shaded red) and click your backspace/delete key to remove it.
This beamstop is quite square, so a simple square mask works. A circle is also easy to draw:
- Click on the Circle mask drawing button.
- Click at the center
- Move the mouse out to make it the size you want, and click again to finish the circle.
If you need to draw another (non-square or circle) shape, you would do the following:
- Click on the Polygon mask drawing button. Left click to place the first vertex.
- Continue left clicking to place more vertices to draw the desired shape.
- Right click to connect the last point you put down to the first point, and finish drawing the polygon.
Zoom back out to the full extent of the image.
Set the image upper limit to 0, and the lower limit to -2 (should be the default lower limit).
Look for any pixels that are blue, these are bad pixels. You should find one at (189, 80)
- Note: On this detector, bad pixels have values of -2.
- Tip: Finding a single pixel, even when you are looking for a blue on red, can be quite tricky. You might try a couple of ways to look for it. First, you can zoom in on the image, and use the Pan tool to drag the image around and look for bad pixels. Second, you can try changing the colormap in the Image Display Settings to Gray. Then the bad pixel will look like a black pixel on a white background, which might be easier to see.
Zoom in on the bad pixel and apply a rectangular mask over the pixel.
- Note: Due to how the image displays, you may see a bit of blue edge beyond the pixel mask. Don’t worry, as the pixel is actually masked. If you want to be sure, you can mask a few pixels around it as well.
In the masking panel, make sure that “Beamstop mask” is selected in the Mask Creation dropdown menu. Click the set button to set the mask you just made as the beamstop mask.
Click the OK button to exit the masking panel.

Setting normalization and other options¶

This section teaches you how to set up normalization by a beamstop counter, and other options. It assumes you have completed Parts 1 (or 2) and 3.

Open the Options window by selecting “Advanced Options” in the Options menu.
In the window that shows up select the Image/Header Format section on the left. In the area on the right click the Load Image button.
In the window that pops up, select the AgBeh_A1_43_001_0000.tiff file. Click the Open button.
- Note: You can select any image of the appropriate type, not just the behenate.
In the Image/Header Format window you should now see header values loaded into the list. Click the Apply button at the bottom of the screen.
Click on the Normalization section in the options list on the left.
In the fields at the bottom of the Normalization panel, make sure “/” is selected in the left dropdown menu, and enter I3/200000 in the large field.
- Note: It is typical in SAXS to normalize by the transmitted intensity. At the CHESS G1 beamline, the beamstop counter is name I3, which is why we are using that name in the normalization expression.
Click the Calc button to evaluate the expression for the counter values loaded in the Image/Header Format tab. You should get a value of 0.02404.
Click the Add button to add the expression to the normalization list.
Make sure the “Enable Normalization” checkbox at the top of the page is checked.
Click OK to exit the options window.
In the file list, select the AgBeh_A1_43_001_0000.tiff file and click the Plot button. You will see a curve get plotted in the top panel of the Main Plot.
Click on the manipulation tab. You will see a data item loaded in the manipulation list.
Adjust the start point for q Min to remove the points with zero value at the start of the curve (these are q points entirely in the mask). Set q Min so that the first point is the peak of the curve on the main plot. This should be around point 13 (depending on your mask).
Open the Options window as in Step 1.
Click on the Calibration section in the options list on the left. Set “Start plots at q-point number” to the number you just found in Step 13.
- Note: This makes it so that every curve loaded from now on will by default not display the first n points, which are covered by the beamstop.
Click the OK button to exit the options window and save your changes.
You have configured everything necessary, and are now ready to save your settings. Go to the File menu and select “Save Settings”.
Save the settings as SAXS.cfg.
These settings can now be used to process images, and can be reloaded when you open RAW by selecting “Load Settings” from the File menu.

Setting absolute scale with water¶

This section teaches you how to set up absolute scale using water as a reference. It assumes you have completed Parts 1 (or 2), 3 and 4. Note that you can use water or glassy carbon (Part 6) for absolute scale calibration in RAW.

Using the settings from the previous parts of the tutorial, plot all of the MT2_48_001_000x.tiff files, where x is 0-9, on the main plot.
- Tip: Section 1 Part 1 of this tutorial document teaches you how to do this.
Average the MT2 files you just loaded. Save the average in the calibration_data folder.
Repeat steps 1 and 2, plotting, averaging and saving, for the water2_49_001_000x.tiff files.
Open the Options window by selecting “Advanced Options” in the Options menu.
Click on the Absolute Scale section in the options list on the left.
Click on the Empty cell “Set” button and select the A_MT2_48_001_0000.dat file.
Click on the Water sample “Set” button and select the A_water2_49_001_0000.dat file.
Set the Water temperature to 4 C.
Click the Calculate button to calculate the Absolute Scaling Constant. You should get a value near 0.00077.
- Tip: You can also use images to set the absolute scale. This may give worse results, as the signal to noise of the averaged file should be better than for a single image.
- Note: It is important that you not change your normalization settings once you have set the absolute scaling constant. If you do, you will have to recalculate the absolute scaling constant. Also, make sure absolute scale is turned off before you calculate the scale constant, otherwise you will get a bad scaling constant (see the manual for details).
Check the “Normalize processed data to absolute scale” checkbox. Click “OK” to exit the advanced options window and save the changes.
Save the settings for later use.

Setting absolute scale with glassy carbon¶

This section teaches you how to set up absolute scale using glassy carbon (NIST SRM 3600) as a reference. It assumes you have completed Parts 1 (or 2), 3 and 4. Note that you can use water (Part 5) or glassy carbon for absolute scale calibration in RAW.

There are two ways to use glassy carbon as a standard in RAW. One way follows the NIST protocol, and will deliver the most accurate results. However, this method depends on all measurements having reliable flux measurements upstream and downstream of the sample. It also requires accurate measurements of the background of the glassy carbon measurement and the sample measurements. The second way is more similar to that used by water, in that it essentially ignores the background (assumes it to be small). This approach only requires regular normalization and a single measurement of the background for the glassy carbon sample.

The simple approach, “ignoring” background:

Load/use the settings from part 4 (without absolute scale set from water, part 5).
Plot all of the glassy_carbon_41_001_000x.tiff files, where x is 0-9, on the main plot.
- Tip: Section 1 Part 1 of this tutorial document teaches you how to do this.
Average the glassy_carbon files you just loaded. Save the average in the calibration_data folder.
Open the Options window by selecting “Advanced Options” in the Options menu.
Click on the Absolute Scale section in the options list on the left.
Click on the Glassy carbon “Set” button and select the A_glassy_carbon_41_001_0000.dat file.
Set the Sample thickness to 1.5 mm.
Click “Calculate” button. You should get something near 0.0014.
- Note: It is important that you not change your normalization settings once you have set the absolute scaling constant. If you do, you will have to recalculate the absolute scaling constant. Also, make sure absolute scale is turned off before you calculate the scale constant, otherwise you will get a bad scaling constant (see the manual for details).
Check the “Normalize processed data to absolute scale using glassy carbon” checkbox.
Click “OK” to exit the advanced options panel, saving the changes.
Save the settings for future use.

The NIST approach:

Important note: All of the normalization (including flux, transmission, etc) happens through the absolute scale panel. You shouldn’t have anything set in the Normalization panel (unless you are doing something like subtracting off a constant pedestal from the image).

Load/use the settings from part 4 (without absolute scale set from water, part 5).
Open the Options window by selecting “Advanced Options” in the Options menu.
Click on the Normalization section in the options list on the left.
Remove any/all items in the Normalization List by highlighting them in the list and clicking the “Delete” button.
Turn off any absolute scaling already in place.
Click on the Calibration section in the options list on the left.
Change the “Start plots at q-point number” to 0.
Click “OK” to exit the advanced options window and save the changes.
Plot the glassy_carbon_41_001_0000.tiff file.
- Tip: Section 1 Part 1 of this tutorial document teaches you how to do this.
Save the glassy_carbon profile in the calibration_data folder.
Plot, average, and save the vac_37_001_000x.tiff and ** MT2_48_001_000x.tiff ** files, where x is 0-9.
- Tip: Because you aren’t normalizing by beam intensity, these averages may have profiles that are not similar (see Section 1 Part 6). In that case, average just the similar profiles.
Open the Options window and select the Absolute Scale section.
Uncheck the Ignore background checkbox.
Click the Glassy carbon “Set” button and select the glassy_carbon_41_001_0000.dat file.
Click the Glassy carbon background “Set” button and select the A_vac_37_001_0000.dat file.
Click the Sample background “Set” button and select the A_MT2_48_001_0000.tiff file.
Set the Sample thickness to 1.5 mm.
Set the Upstream counter to I1.
Set the Downstream counter to I3.
Click the “Calculate” button. You should get an absolute scaling constant near 198.
- Note: This approach will only work if the .dat files you select for the glassy carbon, glassy carbon background, and sample background contain the upstream and downstream counter values. This happens automatically with RAW. Otherwise, you should use images, which will have more noise, but should allow RAW to find all of the appropriate counter values.
- Note: It is important that you not change your normalization settings once you have set the absolute scaling constant. If you do, you will have to recalculate the absolute scaling constant. Also, make sure absolute scale is turned off before you calculate the scale constant, otherwise you will get a bad scaling constant (see the manual for details).
Check the “Normalize processed data to absolute scale using glassy carbon” checkbox.
Click on the Calibration section in the options list on the left.
Change the “Start plots at q-point number” to 13.
Click “OK” to exit the advanced options panel, saving the changes.
Save the settings for future use.

Comparison note:

We find that for the example data given here, the two methods of glassy carbon calibration agree within ~1.5%. The best approach depends on how strong your background scattering is relative to the rest of the scattering in the system.

Setting a molecular weight standard¶

One method for determining molecular weight from a scattering profile is comparison to a known scattering profile with known molecular weight. This part will teach you how to set that known standard in RAW.

Load/use the settings from Parts 4, 5, or 6.
Plot all of the lysbuf2_52_001_000x.tiff files, where x is 0-9, on the main plot.
- Tip: Section 1 Part 1 of this tutorial document teaches you how to do this.
Average the lysbuf2 files you just loaded. Save the average in the calibration_data folder.
Repeat steps 2-3 for the lys2_52_001_000x.tiff files.
Subtract the averaged buffer profile (lysbuf2) from the averaged sample profile (lys2).
- Tip: Section 1 Part 1 of this tutorial document teaches you how to do this.
Select the subtracted profile by clicking on it. In the information panel, set the concentration in the Conc box to 4.14 (this is concentration in mg/ml).
Perform a Guinier fit on the subtracted profile.
- Tip: Section 1 Part 2 of this tutorial document teaches you how to do this.
Right click on the subtracted profile and select the “Use as MW Standard” option.
Enter the molecular weight of the standard in kDa in the box that appears. For this lysozyme sample, the molecular weight is 14.3 kDa.
Click “OK” to save the molecular weight standard.
Save the settings for future use.

Videos¶

We have made various demo and instructional videos for RAW. These are linked below.

RAW demo for SBGrid¶

RAW tutorial videos from MacCHESS¶

For these videos, tutorial data is available from http://bit.ly/bioxtasd

Loading and plotting profiles¶

Collecting and processing data¶

Creating a config file¶

Combining SAXS and WAXS data¶

Radiation damage¶

Aggregation¶

Concentration Effects¶

Cite¶

RAW¶

If you use RAW in your research, please cite the newest RAW paper:

BioXTAS RAW: improvements to a free open-source program for small-angle X-ray scattering data reduction and analysis. J. B. Hopkins, R. E. Gillilan, and S. Skou. Journal of Applied Crystallography (2017). 50, 1545-1553.

DOI: 10.1107/S1600576717011438

You can also cite the previous RAW paper if you like:

BioXTAS RAW, a software program for high-throughput automated small-angle X-ray scattering data reduction and preliminary analysis. S. S. Nielsen, K. Noergaard Toft, D. Snakenborg, M. G. Jeppesen, J. K. Jacobsen, B. Vestergaard, J. P. Kutteraand L. Arleth. Journal of Applied Crystallography (2009). 42, 959-964.

DOI: 10.1107/S0021889809023863

ATSAS¶

If you use RAW to control any of the ATSAS programs ( DAMMIF, DAMMIN, DAMAVER, DAMCLUST, AMBIMETER, SASRES), in addition to the RAW paper, please cite the appropriate paper given on their documentation pages.

Evolving Factor Analysis (EFA)¶

If you used the EFA function in RAW to deconvolve overlapping chromatography data, in addition to the RAW paper please cite the following paper:

Domain Movements upon Activation of Phenylalanine Hydroxylase Characterized by Crystallography and Chromatography-Coupled Small-Angle X-ray Scattering. S. P. Meisburger, A. B. Taylor, C. A. Khan, S. Zhang, P. F. Fitzpatrick, N. Ando. J Am Chem Soc. 2016 May 25;138(20):6506-16.

DOI: 10.1021/jacs.6b01563

Electron Density (DENSS)¶

DENSS¶

If you used the electron density (DENSS) function in RAW to calculate electron density, in addition to the RAW paper please cite the following paper:

Ab initio electron density determination directly from solution scattering data. T. D. Grant. Nature Methods (2018) volume 15, pages 191–193.

DOI: 10.1038/nmeth.4581

EMAN2¶

If in addition to using DENSS to create electron density, you used EMAN2 to do enantiomer selection and/or create an average electron density, please cite both the primary EMAN2 paper and the single particle tomography paper.

Molecular weight¶

Volume of Correlation¶

If you used the volume of correlation method to determine molecular weight in RAW, in addition to the RAW paper please cite the following paper:

Accurate assessment of mass, models and resolution by small-angle scattering. Rambo, R.P. & Tainer, J.A. (2013). Nature. 496, 477-481

DOI: 10.1038/nature12070

Corrected Porod Volume¶

If you used the corrected Porod volume method to determine molecular weight in RAW, in addition to the RAW paper please cite the following paper:

Determination of the molecular weight of proteins in solution from a single small-angle X-ray scattering measurement on a relative scale. H. Fischer, M. de Oliveira Neto, H. B. Napolitano, I. Polikarpov and A. F. Craievich. Journal of Applied Crystallography (2010). 43, 101-109.

DOI: 10.1107/S0021889809043076

Manual¶

Introduction to RAW and this documentation¶

RAW¶

BioXTAS RAW is a program for analysis of Small-Angle X-ray Scattering (SAXS) data. The software enables: creation of 1D scattering profiles from 2D detector images, standard data operations such as averaging and subtraction, analysis of radius of gyration (Rg) and molecular weight, and advanced processing using GNOM, DAMMIF, and AMBIMETER (requires ATSAS installation). It also allows easy processing of inline SEC-SAXS data.

RAW is written in python (mostly) and C++ (a few small bits for speed). It is open source and free for anyone to use. If you do use RAW, we ask that you cite the following paper if you publish or present your results:

BioXTAS RAW, a software program for high-throughput automated small-angle X-ray scattering data reduction and preliminary analysis, J. Appl. Cryst. (2009). 42, 959-964.

Some of the features of RAW include:

Calibrate, mask, radially integrate, and normalize 2D images to make 1D scattering profiles
Average, subtract, merge, rebin, and interpolate scattering profiles
Easily process in-line SEC-SAXS data
Calculate radius of gyration (Rg) and I(0) via Guinier fit
Calculate the molecular weight via I(0) comparison to standards, absolute calibration, correlation volume, and corrected Porod volume
Run GNOM, DAMMIF, and AMBIMETER (requires ATSAS installation)
Run singular value decomposition (SVD) and evolving factor analysis (EFA) on datasets
Calculate P(r) functions using a Bayesian indirect Fourier transform (BIFT)
Can read the following image formats:
- Pilatus Tiff
- CBF
- SAXSLab300
- ADSC Quantum
- Bruker
- Gatan Digital Micrograph
- EDNA-XML
- Eiger
- ESRF EDF
- FReLoN
- Nonius KappaCCD
- Fit2D spreadsheet
- FLICAM
- General Electric
- Hamamatsu CCD
- HDF5
- ILL SANS D11
- MarCCD 165
- Mar345
- Medoptics
- MPA (multiwire)
- Numpy 2D Array
- Oxford Diffraction
- Pixi
- Portable aNy Map
- Rigaku SAXS format
- 16 bit TIF
- 32 bit TIF

RAW is currently being developed by Jesse Hopkins and Soren Nielsen.

More information on RAW is available from the RAW website:

https://sourceforge.net/projects/bioxtasraw/

RAW needs your help! If you find bugs in the program, or errors in the documentation or tutorial, please let us know. Your input is the way we get better!

This documentation¶

The purpose of this documentation is to clearly and completely document the functions of RAW. This document is not intended to act as a tutorial to RAW, for a tutorial please refer to the tutorial document and videos available from the RAW website. Instead, the goal is to document what features are available in each section of RAW, how to use them, and what they do.

This document is laid out in chapters discussing each general area of RAW (for example, the Files tab in the Control Panel is a single chapter). When appropriate, a chapter will cover two related areas, such as the Manipulation tab of the Control Panel and the Main Plot Panel. Some areas will be physical panels or windows of RAW, as the previous examples, while some will be features, such as the Online Mode chapter. The exceptions to this are the second chapter, which covers how to install RAW, and the third chapter, which provides a brief introduction to the different windows of RAW and how they relate.

Typically, the algorithms used will not be fully documented here, as they are available in the source code or appropriate citations (and are subject to change as we improve and expand RAW).

Installing RAW¶

This chapter provides an overview of how to install RAW on Windows, Linux and Macintosh. Detailed installation guides for each system are available separately. It covers:

Licensing
System requirements
Installing from source code.
Installing from pre-compiled binaries.
Obtaining the newest version of RAW

Licensing¶

RAW is open source software, which means the source code is freely available and can be modified/extended as the experienced users sees fit. RAW is available under the GPL V3 license which means that if a modified version of RAW, or any software that uses any of the code available in RAW, is made available to users, then the full source code must also be made available. For a full description of the GPL licensing terms, see the file gpl-3.0.txt included with the RAW source code, or visit http://www.gnu.org/licenses/gpl.html

System requirements¶

The faster the machine the better the experience.

Minimum Requirements

1.6 GHz 32 bit (x86) Computer

1 GB of memory

1024x768 resolution display

Windows 7 or newer, Mac OSX 10.9 or newer, Linux

Note: The newest version of RAW may run on Windows XP or Vista and OS X 10.4+, but it has not been tested by the developers on these older systems.

Optimal Requirements

2.5+ GHz 32/64 bit (x86) Dual/Quad Core Computer

2+ GB of memory

1280x1024+ resolution display

Windows 7 or newer, Mac OSX 10.9 or newer, Linux

Installing RAW from pre-compiled binaries¶

A prebuilt installer is available for RAW for Windows and Mac. We recommend that most users install the prebuilt versions. The windows installer has been tested on Windows 7, 8.1, and 10. It may also work for older versions. The Mac installer has been tested on macOS/OS X 10.9 - 10.12. It may also work for older versions.

Instructions for prebuilt installers are available here:

Windows
Mac

Installing RAW from source code¶

Installing RAW from source code takes more work but has the advantage that the newest version can always be downloaded and quickly installed once the necessary tools have been installed. Installing a pre-compiled version of RAW is easier, but compiled versions are not updated as often as the source code due to the time consuming process of making a compiled executable. Compiled versions may also only be available for certain platforms.

Instructions for install RAW from source are available for

NOTE: As of writing, the RAW is only compatible with Python version 2.7 and not the newer 3.x versions.

Obtaining the newest version of RAW¶

The newest version of RAW can always be found on Sourceforge. The released source code and compiled binaries can be downloaded at:

https://sourceforge.net/projects/bioxtasraw/files/

To obtain the very latest unreleased version (that might or might not be stable) go to: https://sourceforge.net/p/bioxtasraw/code/HEAD/tree/trunk/src/

And click “Download Snapshot” near the top of the screen. Unpack the content into a folder and run RAW.py as usual.

NOTE: When installing new source code on top of old it is important to delete the old source code, including the compiled C libraries with the extensions .so, .pyc, and .pyd, otherwise RAW could be loading old code and become unstable.

Integrating ATSAS with RAW¶

RAW allows you to do analysis with some of the programs from the ATSAS package directly from RAW. Currently, you can use GNOM, DAMMIF, and AMBIMETER in RAW. This requires a separate ATSAS installation, as the RAW developers are not allowed to distribute the ATSAS package with RAW.

Installing the ATSAS package¶

The ATSAS package is available from EMBL, and can be downloaded here:

http://www.embl-hamburg.de/biosaxs/download.html

Installation instructions are available here:

http://www.embl-hamburg.de/biosaxs/manuals/install.html

We recommend installing the packages in the default installation location.

To use all of the programs through RAW, you need ATSAS version 2.7.1 or greater. GNOM and DAMMIF may work for earlier versions of the ATSAS package, but the RAW developers have not tested this.

Locating the ATSAS package for RAW¶

RAW will attempt to automatically locate the ATSAS package when you start up RAW (and when you load a configuration file). It may fail to do this, in which case you will need to set the location of the ATSAS programs manually. To do this:

Open the “ATSAS” section of the Options window.
Uncheck the “Automatically find the ATSAS bin location”
Either by typing the path or using the “Select Directory” button, select the “bin” folder inside the main ATSAS folder. This folder should have a dammif executable inside of it.

Running without compiled extensions¶

RAW compiles certain extensions that are written in C++ in order to maximize the speed of the program. These extensions are involved in the following tasks: Making polygon masks, integrating images into scattering profiles, and carrying out the BIFT analysis. All of these extensions are also available in native python code, but run much more slowly. If RAW is unable to compile these extensions, a warning message will display when the program is started.

While RAW is able to run without the extensions compiled, it will significantly impact performance of the listed tasks. We recommend troubleshooting the RAW installation, or reinstalling RAW to get these to compile. The RAW installation guides contain detailed install instructions and some solutions to common problems with the installation. Please refer to those for more details.

Overview of RAW¶

The main screen of RAW shows three distinct panels: The Information Panel (top left), the Control Panel (bottom left), and the Plot Panel (right). Additional windows that can be shown are the Options window, and various Analysis windows. In addition, there is a menu bar at the top of RAW (either in the RAW window or in the system menu bar, depending on your operating system).

Any of the three panels docked in the main RAW window can be moved relative to the other panels. To do this, click on the title bar of the panel, and drag the panel to the desired location (you should see a blue rectangle indicating where the panel will be put). Additionally, panels can be undocked from the main window by clicking on the pin icon on the right side of the title bar of the panel. Undocked panels can be re-docked with the main window using the same method as rearranging the panels, drag the title bar until you see a blue rectangle appear indicating where the panel will go.

The portion of the total area available for the left (information and control) and right (plot) sides of RAW is controlled by the separator bar. You can click and drag on this bar to change this ratio.

The main RAW window can be resized by clicking and dragging on a corner or edge.

The Control panel¶

The control panel is where you manipulate files and items loaded into RAW. It has four tabs: Files, Manipulation, IFT, and SEC. The Files tab is for viewing files on the system disk, and loading the files into RAW. The Manipulation tab is for working with individual scattering profiles, the IFT tab is for working with inverse Fourier transforms, and the SEC tab is for working with SEC-SAXS data. You change tabs just by clicking on them. The order of the tabs in the control panel can be rearranged by clicking and dragging the tabs.

The Plot panel¶

The plot panel contains four tabs: the Main Plot, IFT Plot, Image plot, and SEC plot tabs. The Main Plot tab is for viewing individual scattering profiles. The IFT Plot tab is for viewing inverse Fourier transforms, the Image plot tab is for viewing detector images, and the SEC plot tab is for viewing SEC-SAXS data. The order of the tabs can be changed by clicking and dragging.

Multiple different plot tabs can be displayed at the same time. Click and drag the tab to any edge of the Plot Panel (up, down, left, or right) and you should see a blue box appear indicating where the tab will go. Additional splits after the first can be accomplished by dragging other tabs, and tabs can be moved between different tab bars (in this case, you should see the bar light up blue, rather than a full blue box for the panel). The amount of screen occupied by the different plots can be set by clicking and dragging the separator bars. To recombine, drag all of the plot tabs into a single bar.

Navigation/control bar

10000000000000210000002267AFAB5BBBEB8688_png

Home. Zoom the to fit all the displayed data.

Move back and fourth in the zoom history.

1000000000000023000000229179FE499099E336_png

Pan/Zoom. Click and drag the left mouse button to pan and the right mouse button to zoom.

1000000000000023000000229BAB0E7642366178_png

10000000000000230000002174F1BA52DF6B9C53_png

Area Zoom. Click and drag to zoom to the selected area.

100000000000001F0000001F9C07D19476FF080E_png

White-space settings. Ability to adjust the white-space around the plots.

100000000000001C0000001F02670C0CCE6242ED_png

Image save. Ability to save the displayed plot(s) in formats such as .png, .svg, .eps, .pdf and more.

Cursor readout

Every plot has a cursor readout. If the mouse is hovering over a position on the plot, the coordinates (x-axis and y-axis values) will read out in the bottom bar of RAW. For the image plot, the intensity at those coordinates is also displayed. For the SEC plot, the value of the secondary y-axis is also displayed.

Plot settings

For each of the Main Plot, IFT Plot, and SEC plot, one of the options in the right-click menu is “Plot Options”. Selecting this will open a window that allows you to set:

Plot title, x-axis label, and y-axis label text, font size, and bold/italicization
Legend title, font size, and bold/italication
Whether the legend is visible (default: no)
Legend font size, transparency, border, and shadow
Turn auto limits on/off for the Axes and manually set the axes limits if desired
Turn off the left/right/top/bottom plot borders
Turn on a line at y = 0 (the Zero Line)
Adjust the axes tick label font sizes

The axis labels and title will accept some LaTeX commands if they are placed between $ symbols, for example: $\int I(q)$ will display as $\int I(q)$. The bold and italic options will not change LaTeX text.

The settings will be changed for the plot/subplot that was clicked on.

The Information panel¶

The information panel displays information about a selected data item in the Manipulation panel. It will show, if available, the Rg, I(0), and MW. It provides a place to enter the sample concentration, description/notes about the sample, and quickly look at the different header values of the data item.

If the current Control panel is changed from the Manipulation panel to the IFT or SEC panel, the Information panel is cleared. If the panel is changed back to the Manipulation panel, the data in the Information panel are restored.

The Options window¶

The options panel is opened by clicking on the Options->Advanced Options menu item. This opens a separate window where many of the RAW settings can be viewed/changed. The panel has two pieces. On the left is the set/category of options, and on the right the actual options panel. For example, if you click on the General Settings section in the left panel, the general settings will be displayed on the right.

Note that in the options tree on the left, the triangles can be used to expand/collapse more options for many of the categories.

Once you have changed the settings, you simply need to click “OK” and they will be saved in memory. In some cases you may want to set the options without exciting the panel. To do this, click “Apply”. To exit without saving any changes click “Cancel”.

Analysis windows¶

Various analysis windows can be opened in RAW. These will be discussed in detail later.

Menus¶

The top menu bar of RAW contains the File, Options, View, Tools, and Help menu. The View and Tools menu are simply another way to access options found elsewhere, while the File, Options, and Help menu have items that cannot otherwise be accessed. These will be discussed later.

RAW Settings and the Options window¶

This section covers the RAW settings, in particular focusing on the Options window and what all of the settings there do. It also covers how to save and load settings in RAW.

Saving and loading settings¶

To save the current RAW settings to disk (a .cfg file):

Go to the “File” menu bar and select “Save Settings”
Select a filename and a location for the configuration (config) file and click “Save”

To load RAW settings from disk (a .cfg file):

Go to the “File” menu bar and select “Load Settings”
Select the appropriate file and click “Open”

Note: Saving a configuration file saves all of the settings in RAW that are set in the Options window. It also saves any masks that have been created.

Changing settings¶

Settings in RAW are generally changed in the Options window. Below we describe how to open the options window, how to change all of the settings, and what the settings do.

Opening the Options window¶

To open the options window:

Go to the “Options” menu
Select “Advanced Options”

There are two parts to the Options window. There is an options tree on the left that determines which options panel is displayed on the right. The part on the right displays the options associated with the selected option in the option tree.

Closing the Options window¶

To save the changes made to the settings in the Options window, close the window by clicking the “OK” button. To exit without saving any of the changes in the Options window, use the “Cancel” button or the system close button (an “x” in the upper left or right corner of the panel).

General settings panel¶

The general settings are all check boxes and can be set by checking and unchecking the boxes. They control the following items:

Hide controls on manipulation items for new plots

If this option is selected, new Manipulation data items start out collapsed, if not selected they start out expanded. Defaults to off.

Write header on top of dat files

The .dat and .ift file “header” information can be saved at the top or bottom of the file. If this is selected, it is written at the top, if not it is written at the bottom. Defaults to off.

Use header for mask creation (SAXSLAB instruments)

If this option is set, if a SAXSLab instrument is being used, the image header will be used to make the beamstop mask. Defaults to off.

Detector is rotated 90 degrees (SAXSLAB instruments)

If this option is set, it indicates to RAW that the SAXSLAB detector is rotated 90 degrees. This affects mask creation from the image header. Defaults to off.

Start online mode on startup

If this option is selected and an Online mode startup directory has been picked, then when RAW starts, it will automatically turn on online mode, with the selected directory as the target online directory.

2D Reduction¶

In the 2D reduction panel, there are reduction options.

Correct of the change in the solid angle of the pixels

If this option is selected, the scattering intensity is corrected for the change in solid angle of pixels as you move along the detector. This is implemented as the standard $\cos^3 (2\theta)$ where $2\theta$ is the scattering angle.

Image/Header Format panel¶

This panel in the Options window allows you to set the image format, the header file format (if any), load the image header into RAW to set up normalizations, and set up bindings to use information in the image header or header file as calibration and reduction parameters.

Image format¶

The image format can be set using the drop down menu. A full description of supported image types is available in the file types section.

Header file format¶

The header file is a file that accompanies the detector image on some beamlines. This file often contains additional information such as, diode values, ion chamber readings, exposure time and date. Currently RAW supports the following header files: I711, MAXLab; I911-4, MAXLab; F2, CHESS; G1, CHESS; G1 WAXS, CHESS; G1 Eiger, CHESS; BL19U2, SSRF; and BioCAT, APS.

Note: If you wish to have a new header file format added to RAW, please contact the developers.

Loading/Viewing header information¶

If you wish to view header information from either the header file or the image header, click the “Load Image” button and select the image of interest.

If you wish to use the header information to normalize the image, load the image using the “Load Image” button and then click the “Apply” button at the bottom of the screen. This will save the counter values in such a way that RAW can set up the normalization appropriately.

Using image header information for calibration and reduction (turning on and setting bindings)¶

RAW has the ability to use header information for calibration and reduction settings. The method for doing this is to set a “binding” between the counter value and the calibration value. The calibration values that can be obtained from the image header or header file are: Beam X Center, Beam Y Center, Detector Pixel Size, Sample Detector Distance, and Wavelength.

To create a binding:

Check the “Use image-header/header file for calibration and reduction parameters” box.
Load the image and header file values into the list as described above.
In the list of the image header and header file names and values, click on the name of the header parameter that you want to use as one of the calibration values. This will fill in the Name and Value in the appropriate fields in the lower left hand portion of the panel.
Using the “Binding” menu, select what calibration parameter should use this header value. In the binding column of the header list, you will see this calibration parameter displayed.

Note: These values overwrite the same values set elsewhere in the settings. So if you bind the Beam X Center to use a value from the header, no matter what you set it to in the Calibration panel of the Options window it will use the value from the header.

Note 2: Make sure that your header file values match the expected units for the calibration parameter. The beam center values should be in pixels on the detector, the detector pixel size should be in microns, the sample detector distance in mm, and the wavelength in angstroms.

Adding a modifier to a binding¶

Once a binding is set, it is possible to add a modifier to the binding, which affects the value obtained from the header. This might be used in a case where the header value is not in the appropriate units.

To set a modifier:

In the list of the image header and header file names and values, click on the name of the bound header parameter that you want to set a modifier for.
In the Modifier field at the bottom of the panel, type in a mathematical expression. This expression may contain any of the header values (including but not limited to the header value selected for the binding). It may contain “+” “-“ “*” and “/” for addition, subtraction, multiplication, and division. The following strings are restricted, and apply specific mathematical functions: acos, asin, atan, atan2, ceil, cos, cosh, degrees, exp, fabs, floor, fmod, frexp, hypot, ldexp, log, log10, modf, pow, radians, sin, sinh, sqrt, tan, tanh, all of which correspond to the functions of the same name in the python math library ( https://docs.python.org/2/library/math.html#module-math ).
Click the “Add” button. You should get a popup window that evaluates the expression for the current loaded header values. Once you close that window, the modifier should be listed in the Modifer column of the header list.

Changing or removing a modifier to a binding¶

To change a modifier to a binding, do the steps to add a modifier, above. When you click on the header item in step 1, the modifier will be shown in the Modifier field at the bottom of the panel, and you can make changes as appropriate in step 2.

To remove a modifier to a binding:

In the list of the image header and header file names and values, click on the name of the bound header parameter that you want to remove a modifier to.
Click the “Remove” button (next to the Modifier field at the bottom of the panel).

Removing bindings¶

To remove a single binding:

In the list of the image header and header file names and values, click on the name of the bound header parameter that you want to remove a binding to.
In the “Binding” menu at the bottom of the panel, select “No binding”.

To remove all bindings, click the “Clear Bindings” button.

Disabling bindings for calibration and reduction¶

To disable the use of bindings for calibration and reduction, either remove all bindings or uncheck the “Use image-header/header file for calibration and reduction parameters” checkbox.

Clear All¶

The “Clear All” button clears all bindings, and removes the current loaded header/header file values from the panel.

Calibration panel¶

The calibration panel allows you to set the beam center, binning size, number of points skipped at the start and end of a scattering profile, the sample to detector distance, wavelength, detector pixel size, and whether or not the Q range is calibrated.

Setting calibration parameters¶

The calibration paramters are: Beam center (x and y), sample-detector distance, wavelength, and detector pixel size. These can all be set by entering a value in the appropriate field on this panel or using the spin controls. However, it is more natural to set these values from the Calibration/Centering panel.

Note: Changing the settings in the calibration/centering panel will change the values in this panel, and vice versa.

Note 2: All of these calibration values are overridden by the bindings described above, if a binding for the particular calibration parameter is set.

Start and end points¶

The “Start plots at q point number” value sets the first q point shown on the plot when a scattering profile is integrated. It is zero indexed (first point is zero). So if it is set to 5, the plot will start with the 6th q point in the q-vector. This is typically used to get rid of the beamstop shadow from the integrated profiles.

The “Skip n points at the end of the curve” value sets the last point shown on the plot when a scattering profile is integrated. If it is set to zero, all points are shown. So if it is set to 5, the last point shown will be the 5th to last point in the q-vector. This is typically used to remove end points if something, for example the downstream flight tube window, is shadowing a high q region of the detector.

Binning¶

The default binning for integrated scattering profiles can be set using the “Binning Size” option. It accepts integer values. A binning size of one corresponds to q bins that are one pixel wide. A binning size of 2 corresponds to q bins that are 2 pixels wide, and so on.

Note: The q size of a bin of a given pixel size will depend on the calibration parameters.

Calibrating the q-range¶

If you do not wish to calibrate the q-range of integrated scattering profiles, uncheck the “Calibrate Q-range” box. The scattering profile will then be displayed as intensity vs. bin number. This option is checked by default.

Normalization panel¶

The normalization panel allows you to normalize integrated scattering profiles by some value. Typically a counter value is used that is proportional to the beam intensity transmitted through the sample (such as a beamstop counter from an active beamstop).

Enabling and disabling normalization¶

To enable normalization for integrated scattering profiles, check the “Enable Normalization” checkbox (checked by default). To disable, uncheck the “Enable Normalization” checkbox.

Setting up normalization operations¶

To add a new operation to the normalization procedure:

Select the operator to be used (/, *, -, + corresponding to division, multiplication, subtraction, and addition respectively)
Enter the desired expression in the expression box.
Click the “Calc” button to view the result of the entered expression.
Click the “Add” button.

Note: This expression may contain any of the header values (including but not limited to the header value selected for the binding). It may contain “+” “-“ “*” and “/” for addition, subtraction, multiplication, and division. The following strings are restricted, and apply specific mathematical functions: acos, asin, atan, atan2, ceil, cos, cosh, degrees, exp, fabs, floor, fmod, frexp, hypot, ldexp, log, log10, modf, pow, radians, sin, sinh, sqrt, tan, tanh, all of which correspond to the functions of the same name in the python math library ( https://docs.python.org/2/library/math.html#module-math ).

Reordering and removing normalization operations¶

The order in which the operations are carried out can be changed by selecting the operation in the normalization list and using the Move Up and Move Down buttons. Operations can be removed by selecting the operation in the list and clicking the Delete button.

Normalizing by a header value¶

It is often desired to normalize the data by exposure time or incoming / transmitted beam intensity, and/or remove offsets on the detector.

To do so:

Load a header file into RAW as described. Be sure to hit the “Apply” button after loading!
Return to the Normalization panel.
Add a normalization value as in steps, in the expression box enter the name of the header value you wish to normalize by along with any other mathematical operations.

Normalizing by a region of interest (ROI)¶

RAW has the ability to normalize by a region of interest on the image. Every pixel in the region of interest is summed, and that can be used to normalize in the same way as a header value.

To normalize by an ROI:

Set an ROI mask.
Add an operation to the normalization list, but use “roi_counter” (without quotes) as the header value. For example, to divide by the roi value, select the “/” operator and enter roi_counter in the expression box, then add that to the list.

Absolute scale panel¶

RAW is able to scale loaded image data to absolute scale using water as a standard. Water has a known, temperature dependent absolute scale value at the forward scattering I(0). Water has a relatively flat scattering profile, which makes it possible to estimate the forward scattering, I(0), from an average of the intensity. To obtain the pure water signal, the water sample obtained in a sample cell must have the empty cell subtracted from it.

To set up Absolute scale:

Click the “Set” button for the empty cell. Select either an image or text (such as .dat) file of the empty cell scattering.
Click the “Set” button for the water sample. Select either an image or text (such as .dat) file of the water scattering.
Select the water temperature in degrees centigrade.
Click the “Calculate” button. An absolute scaling constant should appear.
Enable absolute scale normalization by checking “Normalize processed data to absolute scale” check box.

The algorithm uses the middle third part of the water scattering curve to estimate I(0) by the average intensity.

Note: The selected files must have been normalized in exactly the same way as the rest of the data that is to go on absolute scale. If loading an image, that means not changing the normalization parameters after calculating the absolute scale. If normalization parameters are changed the absolute scale constant will have to be re-calculated. It is particularly important that the images or profiles used to calculate absolute scale not have been saved with absolute scale already on (for example, from a previous calibration).

Turning off absolute scale¶

To turn off absolute scale, uncheck the “Normalize processed data to absolute scale” checkbox in the Absolute scale panel.

Flatfield correction panel¶

If a flatfield file is available, RAW can do a flatfield correction of the data. To do so, click the “Set” button, and select the flatfield image. Then check the “Enable flatfield correction” box.

When RAW applies a flatfield correction, it divides every image it processes by the flatfield image, on a per-pixel basis. The assumption is that every pixel in the flatfield image should have gotten the same intensity, so any variation comes from variation in the detector.

Turning off flatfield correction¶

To turn off flatfield correction, uncheck the “Enable flatfield correction” checkbox in the Flatfield correction panel.

Molecular weight panel¶

The molecular weight panel of the Options windows allows control of the parameters used to calculate molecular weight in the molecular weight window and the SEC calculated parameters. All four methods are described in more detail elsewhere.

Molecular Weight Estimation Using a Standard

This subpanel corresponds to parameters for the MW estimation by comparison to a known standard. While all of the parameters of the standard can be set/changed in this box, the standard MW (in kDa), the standard I(0), the standard concentration (in mg/ml), and the standard filename (only for reference), it is more natural to change these settings by loading the standard scattering profile into RAW and using the Use as MW Standard option.

Molecular Weight Estimation From Volume of Correlation

This subpanel corresponds to the parameters used for the volume of correlation method of estimating molecular weight. This method is the method used for calculating MW in the SEC panel ). The protein and RNA coefficients correspond to the A and B coefficients. The default type selection selects if the MW calculation defaults to Protein or RNA. Changing this option will change whether the MW calculated in the SEC panel is for protein or RNA.

Molecular Weight Estimation From Corrected Porod Volume

This subpanel corresponds to the parameters for the MW calculation by corrected Porod volume. For this method, the only parmater that can be changed is the protein density in kDa/Å:sup:3.

Molecular Weight Estimation From Absolute Intensity Calibration

These parameters correspond to the parameters necessary for calculating the molecular weight when a scattering profile is on an absolute scale.

Reset MW Parameters To Defaults

If you have customized the MW parameters for a particular sample, you can restore the parameters to the RAW defaults (which are the defaults from the relevant papers for each method). There are no default settings for the estimation using a standard.

Artifact removal panel¶

Zingers are pixel values on the detector that are unusually high due to either cosmic radiation or readout errors. RAW includes three methods that can be used for zinger removal.

Zinger removal by smoothing¶

A window of “Window Length” data points can be run across the data and discard values that are more than “Std” standard deviations away from the average of the points in the window. A starting index is given to specify where on the data curve the window should start.

Zinger removal when averaging¶

If three or more exposures of the same sample are available, then these can be used to eliminate zingers by comparing the intensity values of each data set to the others. An intensity value in a data-set that is larger than x standard deviations (Sensitivity) from third quintile of all related data points in the rest of the data sets is removed and replaced by the average of the third quintile.

Zinger removal after radial averaging¶

This method is the most effective method for removing zingers. Pixel intensities in the image for the same q value are compared and should be fairly constant. Values that are more than “Sensitivity” standard deviations away from the median are discarded.

IFT panel¶

RAW currently supports one built-in method for determining the inverse fourier transform (IFT) of a scattering profile, the Bayesian IFT (BIFT) method. In the future we anticipate supporting a python based implementation of the GNOM algorithm called pyGNOM, but currently that is not available.

BIFT¶

The BIFT panel allows you to set the BIFT Grid-Search parameters. These define the large grid that the BIFT algorithm searches over before doing the fine search near the best value on the grid.

Dmax Upper Bound

Sets the largest maximum dimension (Dmax) value that will be used in the coarse grid search, in Å.

Dmax Lower Bound

Sets the smallest Dmax value that will be used in the coarse grid search, in Å.

Dmax Search Points

Total number of Dmax values in the coarse grid search that. These are evenly distributed between the lower and upper bounds.

Alpha Upper Bound

Sets the largest alpha value that will be used in the coarse grid search.

Alpha Lower Bound

Sets the smallest alpha value that will be used in the coarse grid search.

Alpha search points

Sets the total number of alpha values in the coarse grid search. These are distributed logarithmically between the lower and upper bound.

P(r) Points

Sets the number of points in the calculated P(r) curve.

Save Directories panel¶

This panel controls the settings for automated saving of data.

Auto Save

In this subpanel, the checkboxes control whether or not RAW automatically saves Processed image files, Averaged data files, and Subtracted data files. When the boxes are checked, that file type will be automatically saved, when they are unchecked, it will not.

Save Directories

This panel allows you to selected the directories into which files will be saved for each of the automated saving file types (Processed, Averaged, Subtracted). To pick a directory, click the “Set” button and use the window that opens to select a folder. Click “Open” once the appropriate folder is selected. To clear a directory click the “ Clear” button.

Note: A save directory must be selected before an Auto Save checkbox can be enabled.

Online Mode panel¶

This panel includes settings for the online mode. This lets you enable online filtering and set up the filter list.

Enabling/disabling online filtering¶

Online filtering filters files by filename, so that you can control which files are loaded into RAW automatically. To enable this mode, check the “Enable Online Filtering” checkbox. To disable this mode, uncheck that checkbox.

Reordering and removing filter items¶

The order in which the filtering is carried out can be changed by selecting the item in the filter list and using the Move Up and Move Down buttons. Items can be removed by selecting the operation in the list and clicking the Delete button. All filter items can be removed using the “Clear all” button.

SEC-SAXS panel¶

The SEC-SAXS panel controls settings related to the SEC-SAXS data processing.

Intensity ratio (to background) threshold for calculationg Rg, MW, I0

In order to speed up the calculation of Rg, MW, and I0 as a function of frame for SEC-SAXS data, a ratio of the frame intensity to the background intensity is taken. If that value is less than the threshold set here, the frame is skipped. To attempt to calculate structural parameters for all frames, set this threshold to -1.

ATSAS panel¶

The top level ATSAS panel allows you to control where the ATSAS bin location is (the folder with all of the ATSAS programs in it). By default, RAW will attempt to automatically find the ATSAS installation. If you wish to set the location yourself, uncheck the “Automatically find the ATSAS bin location” checkbox, and either type the location into the ATSAS bin location field or use the “Select Directory” button to select the appropriate directory.

Note: If you uncheck and then check the Automatically find checkbox, RAW will attempt to find the ATSAS directory again. This can be useful if, for example, you install ATSAS and want RAW to find the new installation without restarting RAW.

GNOM panel¶

The top level GNOM panel allows you to set the commonly used advanced settings for the ATSAS software GNOM, which is run from the GNOM window. All of these options correspond in name and allowable values to those of GNOM as described in the GNOM manual: http://www.embl-hamburg.de/biosaxs/manuals/gnom.html

Settings can be rest to their defaults (which correspond to the GNOM defaults) by clicking the “Reset to default” button. This resets the settings in this panel and in the GNOM Advanced panel.

GNOM Advanced panel¶

This GNOM panel allows setting GNOM settings which are not as commonly used in GNOM. Again, all of the options correspond in name and allowable values to those of GNOM as described in the GNOM manual: http://www.embl-hamburg.de/biosaxs/manuals/gnom.html

Settings can be rest to their defaults (which correspond to the GNOM defaults) by clicking the “Reset to default” button in the GNOM panel. This resets the settings in this panel and in the GNOM panel.

DAMMIF panel¶

The top level DAMMIF panel allows setting two things: First, the default settings for DAMMIF that are set when the DAMMIF window is opened. Second, standard settings that are available in Fast and Slow mode can be set in the “Standard Settings” subpanel. All of the settings correspond in name and allowable values to those in the DAMMIF manual: http://www.embl-hamburg.de/biosaxs/manuals/dammif.html

Settings can be rest to their defaults (which generally correspond to the DAMMIF defaults) by clicking the “Reset to default” button. This resets the settings in this panel and in the DAMMIF Advanced panel.

DAMMIF Advanced panel¶

The settings in the DAMMIF advanced panel are only used when the “Custom” mode is selected in the DAMMIF panel. This is equivalent to the interactive DAMMIF mode at the command line. Unless otherwise noted, a value of -1 for a field indicates that it will use the default setting. The settings correspond in name and allowable values to those in the DAMMIF manual: http://www.embl-hamburg.de/biosaxs/manuals/dammif.html

Settings can be rest to their defaults (which generally correspond to the DAMMIF defaults) by clicking the “Reset to default” button in the DAMMIF panel. This resets the settings in this panel and in the DAMMIF panel.

The Centering/Calibration panel¶

The Centering/Calibration panel allows you to set the beam center, x-ray wavelength or energy, sample to detector distance, and detector pixel size. This allows RAW to calibrate the q position for each pixel, and integrate the image into a one dimensional scattering profile.

Opening the Centering/Calibration panel¶

Before opening the Centering/Calibration panel, you should make sure that an appropriate image for calibration (such as the scattering from silver behenate) is open in the Image panel. To open the Centering/Calibration panel, go to the Tools menu and click on “Centering/Calibration”. The Centering/Calibration panel will be placed (temporarily) over the control panel.

Setting calibration parameters¶

The Wavelength, Energy, Sample-Detector Distance, and Detector Pixel Size can all be set in the Manual Centering/Calibration Adjustments panel of the Centering/Calibration panel. For each, they can be set in one of two ways. First, a value can be directly typed into the appropriate field. Second, the spin controls can be used to adjust the value in the field. The spin controls only change the last digit of the value, so if the sample detector distance is set at 1000, the up spin control will change it to 1001, whereas if it is set at 1000.0, the up spin control will change it to 1000.1.

If a centering pattern is displayed on the image, changing these calibration will change the pattern. In order for the pattern to update when a value is typed into a field, you must either hit the enter key, or click out of the field.

Note: Setting the wavelength will automatically set the appropriate energy, and vice versa. You only need to set one of these values.

Manually setting the beam center¶

The beam center is displayed in x and y coordinates in the Manual Center/Calibration Adjustments panel. These coordinates correspond with the x and y coordinates shown on the image. If a centering pattern is selected in the Pattern drop down menu, the eam center is also displayed on the image as a red dot 6 pixels in diameter.

The location of the beam center can be changed in three ways. First, values can be directly entered into the X center and Y center fields. As with the calibration parameters, in order to update the pattern display after entering a value in these fields, you must either click out of the field or hit the enter key.

Second, the step size of the beam center in pixels can be set using the drop-down “Steps” menu. The Steps menu is also a field you can type into, so you can set it to any numerical value if the options available in the list are insufficient. Once you have set this, the large red centering arrows on the right side of the Manual Center/Calibration Adjustments box can be used to move the beam center <steps> pixels in the direction indicated by the arrow, where <steps> is the value of the “Steps” field/menu. The arrows can be held down for repeated motion in the same direction with the same step size.

Third, the target button in the center of the centering arrows can be clicked. You then click on the image, and the beam center will be set to that position on the image, to the nearest pixel.

Automatic centering and distance calibration¶

There are two different automated centering methods available in RAW. Which one is available in your version depends on whether or not you have the pyFAI library installed. With either of these methods, you will have the best chance to obtain accurate results if you manually calibrate the center and distance first, so that the automated centering can refine on that starting position.

With pyFAI (recommended)

10000201000001890000009175CE1B8F47D2AF8A_png

In the Manual Centering/Calibration Adjustments panel:

Select the appropriate Standard.

In the Automatic Centering/Calibration panel:

Select the parameters to hold constant by checking boxes in the Fix section.
Select the detector type to be used. If your detector is not in the list, select Other.
Set the Ring # to the index of the first ring visible on the detector. The ring index starts at zero for the largest d-spacing ring (nearest the beam) and increments by one for each ring thereafter. IMPORTANT: The first ring visible on your detector image may not be ring 0!
Click the Start button. Then click on the first ring in the image. Points in that ring will be automatically selected. Click on other parts of the ring as necessary to fill in points.
Increment the ring number as appropriate for the next ring visible on the image (usually increment by 1, for example from 0 to 1), and click on the next ring on the image to select points there. Repeat for all visible rings.
(If necessary) To remove points in a ring, set the Ring # to that ring, and click the Clear All Points In Ring button.
Click the Done button once you have selected points in all of the visible standard rings. At this point, automatic centering and calibration will be carried out.

Note: Ring points cannot be selected if the Pan or Zoom tool are selected.

Without pyFAI (not recommended)

10000201000001880000004957972A08CDA2E5BF_png

This automated centering function attempts to position the center and find the sample-detector distance based on the innermost ring of a Silver Behenate sample:

Click the start button in the “Centering/Calibration Panel”
Select at least 3 points just outside the innermost ring in the Silver Behenate image.
Hit the “Done” button.

This function attempts to find the center of the circle that all of the points defined by your clicks make. If that is found, it then uses the known wavelength, pixel size, and q value of the first silver behenate ring to calculate the sample-detector distance.

Centering patterns¶

With pyFAI (recommended)

If the pyFAI library is installed, more than twenty different calibrants are available, all of those supported by the pyFAI package ( https://github.com/silx-kit/pyFAI/tree/master/pyFAI/resources/calibration ), including Silver-Behenate (AgBh). Rings are displayed as red dashed lines at all appropriate q values.

Without pyFAI (not recommended)

Only one centering pattern is available in RAW. This is the Silver-Behenate pattern, which displays red dashed rings on the detector corresponding to the expected rings from Silver-Behenate. RAW displays the first 5 rings, which at q = 0.1076, 0.2152, 0.3229, 0.4305, and 0.538 Å:sup:-1. The q values are labeled in RAW, but only show at the bottom of the rings, and so you typically have to zoom out of the image to show the label.

The Masking panel¶

The masking panel allows you to create various types of masks that will be used when the image is integrated into a one dimensional scattering profiles.

Types of masks¶

Beamstop mask¶

Unwanted regions such as the beamstop, bad pixels, or gaps between detector panels need to be excluded from the detector image before processing the image. Creating a beamstop mask allows you to do so. For a beamstop mask, when a normal mask is drawn (shown as red on the image), the area inside the mask is excluded from integration. When an inverted mask is used (green on the image), only the area within the mask is included during integration. These types of masks can be combined: normal masked regions will be excluded from integration, even if some or all of the same region is included in the inverse mask region. This allows you to create an inverse mask, and mask out undesired features within that inverse mask.

ROI counter mask¶

RAW allows you to define a region of interest (ROI) mask. If you have an ROI mask defined, every time an image is integrated to a scattering profile, all of the pixel intensities in the ROI mask will be summed, and the value will be available as a counter. In this case, the normal mask (red color) is an include only mask, while the inverted mask (green color) is an exclusion mask. Thus, any areas with a normal mask will be counted in the ROI, while any areas with an inverted mask will be ignored by the ROI.

Typically this is used for normalization purposes, for example to normalize by the transmitted beam through a semi-transparent beamstop. The counter name that this value goes to is roi_counter.

SAXSLAB Beamstop mask¶

The SAXSLAB home source instruments embed the beamstop mask within the image header. Selecting this beamstop mask option will allow you to show this mask on the image. This mask cannot be changed within RAW.

Note: Both a beamstop mask and a SAXSLAB beamstop mask can be simultaneously defined in RAW. In that case, they are both used to mask the image.

Readout-dark mask¶

The area that is selected by a readout-dark mask is averaged for each image, and that average value is subtracted off of every pixel in the image. The standard deviation in the pixel counts is added in quadrature with the noise in the scattering profile calculated during integration (the standard deviation of the counts in each q bin). In this case, the normal mask (red color) is an include only mask, while the inverted mask (green color) is an exclusion mask. Thus, any areas with a normal mask will be averaged to get the readout-dark counts, while any areas with an inverted mask will be ignored by the average.

Setting a readout-dark mask is intended to allow RAW to compensate for dark counts on the detector (most common with CCD detectors). Typically it is used when a fixed area of the detector is permanently masked, such as by lead tape applied to the detector face.

Creating a mask¶

In order to creating a mask, an appropriate image must first be l:ref:open <showimage> in the Image panel. Mask creation is then done as follows:

Select the appropriate mask type from the drop down masking menu in the Mask Creation panel.
Use the mask drawing tools to draw a mask on the image.
For the circle and rectangle tools, left click once to start drawing, move the mouse until you have the desired shape, and click again to stop drawing.
For a polygon, left click once to start drawing, and continue left clicking to add vertices. To finish drawing and connect the final point to the initial point, right click.
To invert a mask section, right click on the section and select “Inverted mask”.
To remove a mask section, left click on the section to select it and click the delete key.
You may draw as many mask sections as you want.
Once done, click the “Set” button to save the mask.

Note: Clicking “Set” overwrites the existing mask in memory. To add new regions to an existing mask, see below.

Viewing and modifying a mask¶

At times you may wish to view an existing mask, and possibly modify it. To do so:

Select the mask type in the Mask Creation panel drop down menu.
Click “Show” to show the existing mask.
Draw new masked regions and/or remove old masked regions as above.
Once you have finished modifying the mask, click “Set” to save your changes.

Removing a mask¶

To completely remove a mask, select the appropriate type of mask in the drop down menu in the Mask Creation panel, and click the “Remove” button.

Saving/Loading a mask to/from disk¶

A mask may be saved to disk by itself (saving the settings saves the mask as well), using the “Save” button in the mask drawing panel, and loaded into RAW using the “Load” button. Once a mask is loaded, select the appropriate mask type from the drop down menu in the Mask Creation panel and click “Set” to set the loaded mask as that mask type.

Note: The “Save” button does not save the mask in program memory! You must use the “Set” button to set a mask as the current mask for use. Save and load are exclusively for saving/loading to/from disk.

Clearing a mask¶

The “Clear” button clears any mask from the image. However, to save these changes, you must click “Set” for the mask type of interest. To remove a mask from use in RAW, see above.

Mask drawing options¶

The check box “Show Beam Center” puts a red circle of diameter 6 pixels on the image where the beam center is (as set in the Centering/Calibration panel).

The Files tab in the Control Panel¶

Changing file directories¶

There are three ways to change the directory whose contents are displayed in the Files tab:

Click on the folder icon next to the bar to open a directory window. This allows you to select the directory you want to display.
Type a path into the bar at the top of the Files tab and hit enter to go to a directory.
Double click the up arrow in the File list (blue arrow with a filename of ‘..’) to navigate up a directory level. Double click on any displayed directory to open it and show the contents in the Files tab.

Note: The Files tab simply displays the files in your system. None of these files are loaded into RAW until you take some action on them (for example, plotting them)

Updating the file list¶

The file list in the Files tab does not automatically update when new files are placed in the folder. In order to update the file list to show new files, you must click the Refresh button (two green arrows in a circle) to the right of the Folder button near the top right of the Files tab.

Plotting files¶

Plotting is done from the Files tab in the Control Panel. It can be done in two ways:

Simply double-click an image, compatible ASCII file, or RAW sec file to plot it.
Select one or more files to be plotted at the same time and click the “Plot” button

The Calibration, Normalization, Masking, and other Advanced Options will be used to convert images into one dimensional scattering profiles when an image is selected and plotted. The Normalization settings, including absolute scale factor, counter norms, and solid angle correction status, the configuration file used, and the image load path are saved as part of the manipulation history for a scattering profile in RAW created from an image.

Note 1: Selecting files is as simple as left clicking on them. Multiple files can be selected by shift or ctrl (command on OS X) left clicking on the files.

Note 2: Some image files can only be loaded if the image type in the Advanced Options is set appropriately.

Filtering and searching the file list¶

Below the file list is a filter/search bar. You can either select filters from a list or create your own using wildcard characters. The filter bar accepts ‘*’ as a wildcard matching any number of characters and ‘?’ as a wildcard matching a single character.

Examples:

If you were looking for all files with ‘BSA’ anywhere in the filename you would write: *BSA* in the filter/search bar. The two * wildcard matches any characters before and after the ‘BSA’.
If you wanted all files with ‘BSA’ at the start of the filename you would write BSA* in the filter/search bar. The single * wildcard will match any characters after the ‘BSA’.
If you wanted all of the files with ‘BSA’ at the start of the filename and ‘.tiff’ at the end of the filename you would write BSA*.tiff. The single * wildcard will match any characters between BSA and .tiff in the filename.
If you wanted all of the files with ‘BSA_000x.tiff’ where x is a single character, you would write BSA_000?.tiff. The ? wildcard will match any single alphanumeric character in that position.

Sorting the file list¶

The columns at the top of the file list, Name, Ext, Modified and Size, show the file name, the file extension, the file modified time, and the file size respectively. By clicking on the column heading, you can sort the file list by those attributes. By default, the file list is sorted by name, ascending (low to high, A to Z). This is indicated by the upward pointing green arrow in the Name column heading. Clicking on the Name column heading will switch the order of the sort to descending (high to low, Z to A) sort. Clicking again will return it to the ascending sort. Clicking on another column heading will sort the file list by that column, and again can be switched between ascending and descending.

Files tab buttons¶

Quick reduce¶

Processes selected images and save them to ASCII files without plotting the scattering profiles using the integration parameters (calibration, masking, etc) current set in RAW.

System Viewer¶

Attempts to open/run the selected file using whatever software is set up in the operating system to handle the file format. You can use the ‘E’ keyboard key as a shortcut.

Plot¶

This plots the selected files in RAW, assuming the formats can be read by RAW. If the files are images, they will be integrated into one dimensional scattering profiles using the integration parameters (calibration, masking, etc) set in RAW. Files are shown in the following locations:

Images (which RAW integrates) and 1D scattering profiles (typically “.dat” files) opened by RAW display in the Main Plot tab of the Plot panel. The Manipulation panel shows the items in the Main Plot and lets change and process the curves individually.
IFT files (.out and .ift extensions) are loaded into the IFT Plot Panel of RAW. The IFT panel in the Control Panel shows the items in the IFT Plot and lets you change and process the curves individually.
SEC files (.sec extensions) are loaded into the SEC Plot Panel of RAW. The SEC panel in the Control Panel shows the items in the SEC plot and lets you change and process the curves individually.

Note: If a file has previously been saved from RAW, all of the saved information (see later sections) will be loaded when the file is reloaded. For example, if a subtracted scattering profile had a Guinier analysis performed on it and was saved as a “.dat” file, when that saved file is loaded back into RAW, the Rg and I(0) values will be loaded as well.

Show image¶

Shows the image selected in the file list in RAW’s image plot. You can use the ‘S’ keyboard key as a shortcut.

Note: If multiple files are selected, the image shown is from the first file in the list.

Clear all¶

Resets and clears all plots and also clears the item lists in the Manipulation, IFT, and SEC tab.

Plot SEC¶

The “Plot SEC” button plots the selected images and/or 1D scattering profiles as a SEC curve. The images are integrated into 1D scattering profiles.

Note: This button can also be used to plot files with the .sec extension that have previously been saved by RAW.

Manipulating files and folders¶

Files and folders can be manipulated in the Files tab by right clicking on a filename in the file list. The pop-up menu has the options to create a new folder, rename a file, copy/cut/paste file(s) or delete file(s).

Note: These options change the files on disk, not just in RAW! So if you delete a file here, it will be deleted from your disk. If you want to work with the files you have loaded into RAW, see the sections on the Manipulation, IFT, and SEC Control Panels.

The Manipulation Panel and Main Plot Window¶

When individual scattering profiles are loaded, they are placed in the Manipulation tab of the Control Panel, and displayed in the Main Plot window. This section will cover the features of the Manipulation panel and the Main Plot Panel.

The Manipulation Panel¶

The Manipulation Panel refers to the panel that is shown in the Control Panel when the Manipulation Tab is selected.

The panel consists of two parts. The top part is where individual scattering profiles that are loaded into RAW are shown. The bottom part consists of buttons that allow you to manipulation these scattering profiles. Further manipulation of the scattering profiles can be done from the right click (context) menu and from the Tools menu.

An individual item in the Manipulation Panel list is called a manipulation data item.

All items loaded into the manipulation panel are initially displayed in the top plot of the Main Plot.

Manipulation Data Items¶

When a new item is plotted, a data item is added to the manipulation list in the manipulation tab. This allows you to control the properties of the data item, and perform manipulations and analysis on it.

The name of the data item is displayed at the top of each item. If an item is given a different name for the plot legend, this legend name is displayed in [square brackets] next to the item name. On the same line as the item name, on the right side of the data item, there are a series of buttons that can be used for further manipulation of the data item.

Note: If there is a * to the left of the item name (between the checkbox and the item name), it indicates there are unsaved changes to the item. This can occur if you change the scale, offset, or q range of the item, if the item is a newly created scattering profile (such as integrated from an image, or an averaged or subtracted item), if there is new analysis information associated with the item (such as Rg and I0), and from binning and interpolating the scattering profile.

Offsetting, scaling and adjusting the q-range of a data item¶

Each data item lets the user interactively adjust the scaling, offset and q-range by using the spin-buttons or entering values in the associated text field and hitting enter. These changes will be updated on the scattering profile displayed on the main plot.

Offset and Scale

The spin-buttons in the scaling and offset boxes will adjust the last digit of the displayed number by 1. So if the scale is 1.0, using the up spin button will adjust it 1.1, while the down spin button will adjust it to 0.9. If the scale was set to 1.15, the up button would adjust it to 1.16, while the down button would adjust it to 1.14.

The scale can never be set to zero by the spin buttons, so if it is adjusted down in a way that would do so, it starts adjusting the next significant figure. For example, if you use the down button on a scale of 0.1, the new scale will be 0.09. A negative scale factor is treated like a positive scale factor (the absolute value of the scale factor is applied), thought it can cause unexpected behavior from the spin buttons.

Q-range

Adjusting q Min and q Max can be done in two ways. Each q limit has two boxes associated with it. The left box for each displays the actual q value, and a q value can be typed into those boxes. As there are a finite set of points in q space, RAW will automatically determine the nearest q point, and set the q value to this nearest point. The box on the right is the (0 indexed) point in the q vector. That is, if for q Min the value reads 15, then the scattering profile is being displayed starting at point number 15 in the q vector (note that this is the 16th point of the q vector, because it is zero indexed). The spin buttons adjust these point numbers, so if you use the up spinner on q Min, it will start at the next point in the q vector.

Showing/Hiding data items on the Main plot¶

To show/hide the scattering profile associated with a data item on the plot, click the checkbox next to the sample filename. If the checkbox is checked, the item is currently shown on the Main plot, if the checkbox is unchecked, the item is currently hidden on the main plot.

Data item buttons¶

On the right of the top line of the data item (the same line as the item name and show/hide checkbox) are a series of buttons for controlling the data item. In order of left to right, these buttons have the following properties.

Collapse/Expand item controls

In order to see more data items in the list, or to prevent accidental changes to the q Min, q Max, Scale, or Offset values, the controls for a data item can be hidden using the triangle button in the upper right of the data item. Note that the direction of the triangle switches when it is toggled. If the triangle is pointing up, the controls are expanded, if the triangle is pointing down the controls are collapsed.

Extended Info

By hovering the mouse over the i in the blue circle button, a tooltip will appear which shows more information about the item, including Rg and MW.

Locate Line

The target button is used to highlight the scattering profile on the graph that is associated with the data item. When the target is pressed, it ‘bolds’ the line of the scattering profile (increases the line width by several points). When the target is pressed again, the line width is set back to normal. You can tell if a line is currently bolded, as the target will be orange instead of grey.

Line Properties

The colored line button has two purposes. First, the color matches the current color of the scattering profile in the Main Plot. Second, when pressed it opens a line properties dialog which allows you to set the legend label; the line style, width, and color; the data point marker style, size, line color, and fill color; and the error bar line style, width, and color.

Mark

The star button marks an item. This is used when doing operations such as subtraction, syncing, superimposing or merging. In those cases, the marked (also referred to as starred) item has a special significance.

Selecting data items¶

A single data item can be selected by clicking on the item name in the Manipulation list (similar to how you would select files in your system file browser). When an item is selected, the color of the item background changes from white to gray. If the item is currently selected, clicking on it will cause it to be unselected. Note that for a regular click, all other selected items will be unselected when a new item is selected.

Multiple items may be selected in two ways. If the Control key (Command key on Macs) is held down while clicking on items, each item that is clicked on will be added to the set of selected items. If a single item is first selected and then the Shift key is held down and another item is selected, all of the items in the list between the two items will be selected (including the second item that is clicked on).

All of the items in the list can be selected in two ways. The first is using the select all button, the second is pressing Ctrl-A (Cmd-A), the Control (Command) key and the A key at the same time when you are in the Manipulation panel. All items can be unselected by clicking in a empty spot of the Manipulation list (but not above or below the list), or by clicking on an already selected item.

Note: If you have a set of selected items and wish to remove some, holding down the Control (Command) key and clicking on selected items will deselect them without affecting the other selected items.

The top buttons of the Manipulation Panel¶

The Manipulation Panel has a set of five buttons at the top of the panel. These buttons have the following effects, listed from left to right.

Show All

Clicking on the button that looks like an eye will show all scattering profiles. This is the same as if you manually set all of the show/hide checkboxes in the data items to on.

Hide All

Clicking on the button that looks like an eye with a red x through it will hide all scattering profiles. This is the same as if you manually set all of the show/hide checkboxes in the data items to off.

Select All

Clicking on the button that looks like a spreadsheet with selected cells will select all of the Manipulation items.

Collapse All

Clicking on the button that looks like an upward pointing arrow with a box under it will collapse all of the data item controls. This is the same as if you manually toggled the Expand/Collapse button for each data item to the collapse position.

Expand All

Clicking on the button that looks like a downward pointing arrow with a box over it will expand all of the data item controls. This is the same as if you manually toggled the Expand/Collapse button for each data item to the expand position.

Synchronizing settings for several data items¶

Settings for several data items can be easily synchronized.

Click the star icon on the data item you want the other items synchronized to.
Select one or more data items, you wish to synchronize with the starred item.
Click the “Sync” button
Select what parameters you want synchronized
Hit the “OK” button

Parameters that can be synchronized in this way are: q min, q max, n min, n max, scale, offset, line style, line width, and line marker.

Renaming a data item¶

Data items can be renamed by selecting the data item of interest and selecting “Rename” in the right click popup menu.

Note: While no characters are expressly forbidden in the filename, RAW does not sanitize file names before saving, and thus special characters such as ‘/’ and ‘\’ are likely to cause problems when the file is saved.

Saving data items¶

All scattering profiles can be saved to a standard 3 column ASCII format with a header. To save:

Select the item(s) to be saved.
Click the “Save” button or select “Save selected file(s)” from the right click menu.
In the window that pops up, navigate to the directory in which you want to save the files.
If you are saving a single item, the window will give you an opportunity to rename your file if desired. Click “Save” when ready.
If you are saving multiple items, you simply need to select the folder for the items to be saved in, and click “Open”. The items will be saved with the same names displayed in the Manipulation Panel, in the folder that you chose.

Items are saved in a standard format with three columns of data. These columns have listed headers, and are Q, I, and Error in I respectively. Depending on your settings, at the bottom (default) or top of the file there will be a “Header”. This header contains any information loaded as ‘Header’ i nformation from the image or separate header file, the normalization factor, any notes you made in the Notes section of the information box, any analysis results (such as Rg, MW), and the manipulation history saved by the averaging, subtracting, merging, rebinning, and interpolating functions (see following sections).

The files are saved with a “.dat” extension, which is the standard for scattering profiles and can be read in by most standard SAXS processing software (including Scatter, Primus, and programs in the ATSAS package). The files are simply text files, and can be opened and viewed in any standard text editor. The header is saved in the JSON format.

Removing data items from the manipulation list¶

To remove one or more data items, select them and do one of the following:

Press the “Delete” key on the keyboard
Click the “Remove” button
Select “Remove” from the right click menu

Averaging data¶

Averaging data is an important part of processing SAXS data. At the moment in RAW, data can only be averaged if the q vectors match exactly. To average data:

Select two or more data items that should be averaged.
Either click the “Average” button or right-click on a selected item and choose “Average” from the popup menu.

After averaging, a new data item will be displayed in the manipulation panel with the filename in green and an ‘A_’ prefix. All averaged items are initially displayed on the top plot of the Main plot, regardless of where the items being averaged are displayed.

A record of which profiles were averaged is saved in the metadata for the averaged profile. This can be viewed within RAW by right clicking on the item and selecting “Show history”, or externally if the scattering profile is saved as a .dat file and opened by an external text editor. This history contains all of the history of the individual files, so if you average a set of averaged (or subtracted) files, you can dig down through the metadata to see what files were used at each step.

Subtracting data¶

Subtracting data is an important part of processing SAXS data. RAW can handle subtracting data that has different q vectors, as long as there is a region of overlap between the q vectors. Data subtraction is done in the following way:

Click the star icon on the data item to mark the scattering profile that should be used for subtraction (typically the background or buffer scattering profile).
Select another data item, or multiple items.
Either click the “Subtract” button or right-click and select “Subtract” from the popup menu. Note: this subtracts the marked item (step 1) from the selected items (step 2).

After subtraction, a new data item will be displayed in the Manipulation panel with the filename in red and a ‘S_’ prefix. All subtracted items are initially displayed on the bottom plot of the main plot, regardless of where the items being subtracted are displayed.

A record of which profiles were involved in the subtraction is saved in the metadata for the subtracted profile This can be viewed within RAW by right clicking on the item and selecting “Show history”, or externally if the scattering profile is saved as a .dat file and opened by an external text editor. This history contains all of the history of the individual files, so if you subtract a set of averaged (or subtracted) files, you can dig down through the metadata to see what files were used at each processing step.

Data with different q vectors is handled as follows. First, the range of overlap is established. Second, RAW checks whether these overlapping regions have matching q vectors, if so, RAW carries out the subtraction. If not, RAW bins both curves so that the q vectors in the overlapping region match, and then carries out the subtraction.

Superimposing data¶

Scattering profiles can be superimposed on each other automatically. This function tries to calculate a scale and an offset that minimizes difference between two or more curves using a least-squares method. To superimpose:

Click the star icon on the data item you want the other items superimposed to.
Select one or more data items, that you wish to superimpose onto the starred item.
Click the “Superimpose” button

Merging data¶

Merging scattering profiles is typically done if you have measured multiple q ranges using different detectors. RAW can merge these profiles into one single profile. To merge profiles in RAW:

Click the star icon on one of the curves. This items name will be used for the name of the merged item, otherwise it does not matter.
Select one or more data items, that you wish to merge together with the starred item
Click the “Merge” button, select “Merge” from the right click menu, or select “Merge” from the Tools->Operations menu.

After merging, a new data item will be displayed in the Manipulation panel with a filename with a ‘M_’ prefix. All merged items are initially displayed on the top plot of the Main plot, regardless of where the items being merged are displayed.

Merging will average together the overlapping regions, if any exist. It can be done on an arbitrary number of data items, and is done in serial. So first the two items with the two lowest q vectors are merged, then that item is merged with the data item with the next lowest q vector, and so on. For example, if you had selected items with a minimum q vector of 0.01, 0.25, and 0.75, it would first merge the 0.01 and 0.25 items, and then merge that resulting item with the 0.75 item.

A record of which profiles were merged is saved in the metadata for the averaged profile. This can be viewed within RAW by right clicking on the item and selecting “Show history”, or externally if the scattering profile is saved as a .dat file and opened by an external text editor.

Note: If items are merged without an overlap region, there will simply be no data in those intervening points.

Rebinning data¶

Rebinning scattering profiles is typically done to improve signal to noise of individual points. As most scattering profiles are significantly oversampled (compared to the Shannon sampling frequency), there is no loss of information from doing this. To rebin curves in RAW:

Select one or more data items, that you wish to rebin with the same binning factor.
Select “Rebin” from the right click menu, or select “Rebin” from the Tools->Operations menu.
Select the bin reduction factor, and if you want logarithmic binning check the logarithmic box. Note: the log binning is log base 10.
Click “OK”.

After rebinning, a new data item will be displayed in the Manipulation panel with a filename with an ‘R_’ prefix. All rebinned items are initially displayed on the top plot of the Main plot, regardless of where the items being rebinned are displayed.

A record of which profiles were averaged is saved in the metadata for the rebinned profile. This can be viewed within RAW by right clicking on the item and selecting “Show history”, or externally if the scattering profile is saved as a .dat file and opened by an external text editor.

Note: The bin reduction factor can be thought of as how many q points will go into a single bin in the new profile. So a bin reduction factor of 2 will result in half as many q points in the new profile, with each q point being the average of two q points from the previous profile (note that this may not quite be true for q points at the start and end of the profile). Rebinning will slightly reduce the q range of the scattering profile, as the new q value will be less than the extreme values in each bin.

Interpolating data¶

RAW can interpolate a scattering profile to find values at all points in a reference q vector. This can be useful for creating scattering profiles with perfectly matched q-vectors. To interpolate curves in RAW:

Click the star icon on the scattering profile you wish to interpolate to. The q vector for this profile will be used as the interpolation points for the other profile(s).
Select one or more data items, that you wish to interpolate to match the q vector of the starred item
Select “Interpolate” from the right click menu, or select “Interpolate” from the Tools->Operations menu.

After interpolating, a new data item will be displayed in the Manipulation panel with a filename with an ‘I_’ prefix. All interpolated items are initially displayed on the top plot of the Main plot, regardless of where the items being merged are displayed.

Interpolation works as follows: The overlapping q region is found for the two curves. Within this region, for every point in the q vector of the reference (starred) scattering profile an intensity value is found from the item being interpolated via linear interpolation (using the scipy.interpolate.interp1d function). A new scattering profile is then created from these interpolated values and the reference q vector.

A record of which profiles were averaged is saved in the metadata for the averaged profile. This can be viewed within RAW by right clicking on the item and selecting “Show history”, or externally if the scattering profile is saved as a .dat file and opened by an external text editor.

Converting q-scale¶

Q-values may sometimes be in inverse angstroms (1/Å), while others use inverse nanometers (1/nm). The q vector can be multiplied or divided by 10 to accommodate for this difference when comparing two curves on different q-scales.

Select the data item(s) to be scaled and right-click on them.
Select the “Convert q-scale” option and choose whether to adjust the q-scale upwards or downwards by a factor of 10.

Note: All of the processing in RAW assumes that the scattering profiles are in inverse angstroms.

Normalizing by concentration¶

The overall intensity of the scattering profile should be proportional to the concentration, so it can be useful to normalize scattering profiles by the measured concentration value. To do so:

Set the concentration of the item in the Information panel or the MW window.
Select the item to be normalized.
Right click on the item and select “Normalize by conc”

Note: This function simply calculates 1/conc and sets that value as the scale factor. This is equivalent to doing the same calculation and manually setting the scale factor of the scattering profile.

Saving analysis results¶

After analyzing the data using the Guinier, MW, BIFT, or GNOM panels, the resulting parameters such as Rg, I(0), MW, Dmax, and others can be saved to a comma separated format that easily loads into a spreadsheet program such as Excel. To do so:

Select the data items of interest.
Right-click on a selected data item and select “Save all analysis info”
Choose a name and a location for the .csv file
Load the .csv file into a spreadsheet program.

This saves a spreadsheet where each row is an individual scattering profile, and each column an analysis parameter.

Saving select data item information¶

Information about data items including any analysis parameters, header file information, image header information, scale factor, offset, concentration, and notes can be selectively saved to a comma separated format that easily loads into a spreadsheet program such as Excel. To do so:

Select the data items of interest
Right-click on a selected data item and select “Save item info”
A window will pop up with two panels. The panel on the left is all of the information available about the selected items, sorted into appropriate categories (such as Image Header and Guinier Analysis).
In the window, select the information of interest in the left hand panel by clicking (including Ctrl and Shift clicking) on it.
Once all of the variables of interest are selected (blue), click the “->” button to put them in the list of data that will be saved. They should appear in the right hand panel.
If you accidentally selected an item, select the item and click the “<-“ button to remove it from the list of data to be saved.
You can add additional items to the list by repeating steps 4 and 5 until everything you want to save is selected.
Click the “OK” button.
Choose a name and a location for the .csv file
Load the .csv file into a spreadsheet program.

This saves a spreadsheet where each row is an individual scattering profile, and each column one of the selected variables.

Note: This differs from the save function described above in two ways. First, it allows you select which information you want to save. Second, it can save header information and other parameters about the scattering profile beyond the analysis information.

Data point browsing¶

The q, I, and I error values for each individual point in a scattering profile on the data curve can be inspected using the data browser. To do so:

Right-click on the data item of interest.
Select “Show data” in the popup menu.

Displaying the detector image¶

If the plotted data was loaded from an image, the image can be shown in the Image tab of RAW. To do so, right-click on the data item of interest and click “Show image” in the popup menu.

Displaying the header information¶

If the plotted data has header information, either from an image header or separate header file, this can be displayed in RAW. To do so, right-click on the data item of interest and click “Show header” in the popup menu.

Displaying the history information¶

If the plotted data has history information, either from integration of an image or from manipulation inside RAW such averaging or subtraction, this can be displayed in RAW. To do so, right-click on the data item of interest and click “Show history” in the popup menu.

The Manipulation data item right click menu options¶

When you right click on a data item, a popup menu is shown. This section describes what each item on the menu does.

Subtract

Carries out subtraction.

Average

Carries out averaging.

Merge

Carries out merging.

Rebin

Carries out rebinning.

Interpolate

Carries out interpolation.

Use as MW Standard

Sets the selected item to be the MW standard used in the I(0) reference MW calculation method. To do this the concentration must be set, for example in the Information Panel (it can also be set in the MW panel). A Guinier fit must also have been done, as this requires a known I(0) value for the reference scattering profile. In the popup window that opens, enter the known protein MW in units of kDa.

Normalize by conc

Normalizes the profile by the concentration.

Remove

Removes the item.

Guinier fit

Opens the Guinier fit analysis panel for the selected scattering profile.

Molecular Weight

Opens the molecular weight analysis panel for the selected scattering profile.

BIFT

Opens the Bayesian indirect Fourier transform (BIFT) analysis panel for the selected scattering profile.

GNOM (ATSAS)

Opens the GNOM analysis panel for the selected scattering profile. Note: this option is only available if RAW has detected a valid ATSAS package installation.

SVD

Opens the singular value decomposition (SVD) analysis panel for the selected scattering profiles (must have at least 2 profiles selected).

EFA

Opens the evolving factor analysis (EFA) panel for the selected scattering profiles (must have at least 2 profiles selected).

Convert q-scale

Has a submenu with two options: q * 10 which multiples the scattering profile q vector by a factor of 10, and q / 10 which divides the scattering profile q vector by a factor of 10.

Show image

If the scattering profile was integrated from an image, and the image is still in the same location as when the profile was first integrated, it displays the image in the Image Plot Panel.

Show data

Shows the q, I(q) and I_error(q) data.

Show header

Shows the header information.

Show history

This shows the history of the scattering profile.

Move to top plot

Moves the scattering profile to the top subplot of the Main plot (no effect if the scattering profile is already plotted on the top plot).

Move to bottom plot

Moves the scattering profile to the bottom subplot of the Main plot (no effect if the scattering profile is already plotted on the top plot).

Save all analysis info

Saves all of the analysis info of the selected data item(s).

Save item info

Saves select item info.

Save selected file(s)

Saves the selected data item(s).

The Manipulation panel bottom buttons¶

There are six buttons at the bottom of the Manipulation panel. They are:

Save

This button saves the selected data item(s).

Sync

This button syncs the settings of the selected data item(s) to the starred data item.

Remove

This button removes the selected data item(s) from the Manipulation panel.

Superimpose

This button superimposes the selected data item(s) onto the starred data item.

Average

This button averages the selected data item(s).

Subtract

This button subtracts the selected data item(s).

The Main Plot window¶

The Main Plot window displays individual scattering profiles associated with items in the Manipulation panel. It has two subplots, which by default are titled “Main Plot” (top) and “Subtracted” (bottom). Any item that is initially loaded is plotted on the top plot.

The features that are general between all of the plots are described elsewhere. This section will describe features unique to this plot.

Changing axes and plot types¶

Right-click in the plot window to view a pop-up menu with different axis settings.

The available plot modes are:

Lin-Lin
Log-Lin
Log-Log
Lin-Log
Guinier plot (ln(I(q)) vs. q²)
Porod plot (q⁴I(q) vs. q)
Kratky plot (q²I(q) vs. q)

Moving profiles between plots¶

Scattering profiles can be moved between the top and bottom plots. To do so:

Select the data item(s) of interest in the Manipulation panel.
Right-click on a selected data item and select “Move to top plot” or “Move to bottom plot” from the popup menu.

Setting the legend label¶

Instead of the filename showing in the legend, a special label can be created for that data item.

Click on the Line Properties button of the data item.
In the Line Properties window that pops up, enter the desired legend label in the Legend Label box.
Click “OK”.

The new legend label will be shown in [square brackets] next to the data item name. To revert to t he standard label (the filename), erase the custom label.

Locating data items¶

The data item associated with a plotted scattering profile can be quickly found by clicking plotted curve. When a curve is clicked on, it will flash ‘bold’ (increased line weight), and the associated data item will be selected in the Manipulation panel.

Note: If there are other selected items in the Manipulation panel, they will be unselected.

The IFT Control Panel and IFT Plot Panel¶

When inverse Fourier transforms (P(r) functions) are loaded into RAW or generated by BIFT or GNOM in RAW, they are placed in the IFT tab of the Control Panel, and displayed in the IFT Plot window. This section will cover the features of the IFT control panel and IFT plot panel.

The IFT control panel¶

The IFT control panel refers to the panel that is shown in the Control panel when the IFT tab is selected.

The panel consists of two parts. The top part is where individual IFT items loaded into RAW are shown. The bottom part consists of buttons that allow you to save or remove those IFT items. Further manipulation of the IFT items can be done from the right click (context) menu and from the Tools menu.

An individual item in the IFT list is called an IFT data item. It is associated with a P(r) function, the scattering profile used to generate that P(r) function, and the scattering profile generated from the P(r) function.

All items loaded into the IFT panel have a P(r) function displayed in the top plot of the IFT Plot window, and an experimental scattering profile and a scattering profile from the P(r) function displayed in the bottom plot of the IFT Plot window.

IFT Data Items¶

When a new item is plotted, a data item is added to the IFT list in the IFT tab. This allows you to control the properties of the data item, and perform analysis on it.

The name of the data item is displayed for each item. If an item is given a different name for the plot legend, this legend name is displayed in [square brackets] next to the item name. On the same line as the item name, on the right side of the data item, there are several buttons that can be used for further manipulation of the data item.

Note: If there is a * to the left of the item name (between the checkbox and the item name), it indicates there are unsaved changes to the item. This can occur if the item is newly created IFT data (such as from the BIFT or GNOM panels), or if item properties such as the name have been changed.

Showing/Hiding data items on the IFT plot¶

To show/hide all of the curves (P(r), experimental data, scattering profile from P(r) curve) associated with a data item on the plot, click the checkbox next to the filename. If the checkbox is checked, the item is currently shown on the IFT plot, if the checkbox is unchecked, the item is currently hidden on the IFT plot.

Data item buttons¶

On the right of the data are a series of buttons for controlling the data item. In order of left to right, these buttons have the following properties.

Locate Line

The target button is used to highlight the scattering profile on the graph that is associated with the data item. When the target is pressed, it ‘bolds’ the line each plotted curve associated with the data item (increases the line width by several points). When the target is pressed again, the line width is set back to normal. You can tell if a line is currently bolded, as the target will be orange instead of grey.

Line Properties

The colored line button has two purposes. First, the color matches the current color of the P(r) function in the IFT Plot. Second, when pressed it opens a line properties dialog which allows you to set the legend label; the line style, width, and color; the data point marker style, size, line color, and fill color; and the error bar line style, width, and color for each line associated with the IFT data item.

Selecting data items¶

A single data item can be selected by clicking on the item name in the IFT list (similar to how you would select files in your system file browser). When an item is selected, the color of the item background changes from white to gray. If the item is currently selected, clicking on it will cause it to be unselected. Note that for a regular click, all other selected items will be unselected when a new item is selected.

Multiple items may be selected in two ways. If the Control key (Command key on Macs) is held down while clicking on items, each item that is clicked on will be added to the set of selected items. If a single item is first selected and then the Shift key is held down and another item is selected, all of the items in the list between the two items will be selected (including the second item that is clicked on).

All of the items in the list can be selected in two ways. The first is using the select all button, the second is pressing Ctrl-A (Cmd-A), the Control (Command) key and the A key at the same time when you are in the IFT panel. All items can be unselected by clicking in an empty spot of the IFT list (but not above or below the list), or by clicking on an already selected item.

Note: If you have a set of selected items and wish to remove some, holding down the Control (Command) key and clicking on selected items will deselect them without affecting the other selected items.

The top buttons of the IFT Panel¶

The IFT Panel has a set of three buttons at the top of the panel. These buttons have the following effects, listed from left to right.

Show All

Clicking on the button that looks like an eye will show all IFT items. This is the same as if you manually set all of the show/hide checkboxes in the data items to on.

Hide All

Clicking on the button that looks like an eye with a red x through it will hide all IFT items. This is the same as if you manually set all of the show/hide checkboxes in the data items to off.

Select All

Clicking on the button that looks like a spreadsheet with selected cells will select all of the IFT data items.

Renaming a data item¶

Data items can be renamed by selecting the data item of interest and selecting “Rename” in the right click popup menu.

Note: While no characters are expressly forbidden in the filename, RAW does not sanitize file names before saving, and thus special characters such as ‘/’ and ‘\’ are likely to cause problems when the file is saved.

Saving data items¶

IFT items are saved in two different formats, depending on whether the item was generated by BIFT (“.ift”) or GNOM (“.out”). The procedure to save either is the same. To save:

Select the item(s) to be saved.
Click the “Save” button or select “Save selected file(s)” from the right click menu.
In the window that pops up, navigate to the directory in which you want to save the files.
If you are saving a single item, the window will give you an opportunity to rename your file if desired. Click “Save” when ready.
If you are saving multiple items, you simply need to select the folder for the items to be saved in, and click “Open”. The items will be saved with the same names displayed in the IFT Panel, in the folder that you chose.

BIFT items are saved as “.ift” files, which is a text file with RAW specific formatting of the text. The first two lines are “BIFT” and the “Filename: <filename>”. After that, the P(r) function is saved as 3 column data. The first column is “R”, the second column is “P(R)” and the third column is “Error”, these headers are included as the third line of the file. After the P(R) function there are two blank lines, followed by the scattering data. This is saved in four columns, the first three are “Q”, “I(q)” and “Error” which correspond to the experimental data, while the fourth column is “Fit” which is the scattering profile from the P(r) function. After this data is written there is a “header” written, which consists of the Chisquared, algorithm used, I(0) value, log base 10 of the alpha value, Dmax, Rg, and the filename, all saved in JSON format. The files are simply text files, and can be opened and viewed in any standard text editor.

GNOM items are saved in the standard “.out” format. This is described in the ATSAS manual for GNOM. These files can be directly input into any ATSAS (or other) program that requires a GNOM .out file as input.

Removing data items from the IFT list¶

To remove one or more data items, select them and do one of the following:

Press the “Delete” key on the keyboard
Click the “Remove” button
Select “Remove” from the right click menu

Sending data to the main plot¶

If you wish to examine the experimental scattering profile and fit to this profile from the P(r) function more closely, the data can be sent to the main plot. To do this, right click on the IFT data item and select “To Main Plot”. This will plot items on the Main plot and add them to the Manipulation list.

For a “.ift” file generated by BIFT, the following items will be added to the Manipulation list and Main plot. In all cases, <filename> corresponds to the filename of the IFT data item without the extension (so “my_ift”.ift would have a filename of “my_ift”).

<filename>_data – This is the experimental data that was used to generate the P(r) curve.

<filename>_fit – This is the scattering profile generated from the P(r) curve.

For a “.out” file generated by GNOM, the same two curves as for a BIFT item (above) are sent to the main plot, there is also one additional file.

<filename>_extrap – This is the scattering profile generated from the P(r) curve, extrapolated to q=0.

Running DAMMIF on the P(r) function¶

RAW allows you to run DAMMIF on a P(r) function from within RAW. Currently, this can only be done on P(r) items generated by GNOM (these can be generated in RAW, or loaded in after being generated outside of RAW). To run DAMMIF, select an appropriate IFT data item (a “.out” item), and either right click and select the “Run DAMMIF” option or from the Tools->ATSAS menu select “DAMMIF”. This opens the DAMMIF window.

Running AMBIMETER on the P(r) function¶

RAW allows you to run AMBIMETER on a P(r) function from within RAW. Currently, this can only be done on P(r) items generated by GNOM (these can be generated in RAW, or loaded in after being generated outside of RAW). To run AMBIMETER, select an appropriate IFT data item (a “.out” item), and either right click and select the “Run AMBIMETER” option or from the Tools->ATSAS menu select “AMBIMETER”. This opens the AMBIMETER window.

Data point browsing¶

Each individual point of the r; P(r); error in P(r); experimental q, I(q), and error; I(q) from the P(r) function; and, for GNOM generated IFT data items, the q and I(q) values extrapolated to q=0 vectors; can be inspected using the data browser. To do so:

Right-click on the data item of interest.
Select “Show data” in the popup menu.

The IFT data item right click menu options¶

When you right click on a data item, a popup menu is shown. This section describes what each item on the menu does.

Remove

Removes the item.

To Main Plot

This sends the scattering profile data to the main plot.

Run DAMMIF

This item is only available for “.out” files generated by GNOM. It opens the DAMMIF window.

Run AMBIMETER

This item is only available for “.out” files generated by GNOM. It opens the AMBIMETER window.

SVD

Opens the singular value decomposition (SVD) analysis panel for the selected scattering profiles (must have at least 2 items selected).

EFA

Opens the evolving factor analysis (EFA) panel for the selected scattering profiles (must have at least 2 items selected).

Show data

Shows the individual data points.

Rename

Renames the data item,

Save selected file(s)

Saves the selected data item(s).

The IFT panel bottom buttons¶

There are three buttons at the bottom of the IFT control panel. They are:

Save

This button saves the selected data item(s).

Remove

This button removes the selected data item(s) from the IFT panel.

Clear IFT Data

This button clears all loaded IFT data. It works the same as if you had selected all of the IFT data items and then removed them.

The IFT Plot window¶

The IFT Plot window displays P(r) data (top plot), the scattering profiles generated from the P(r) data (bottom plot), and the experimental scattering profile used to generate the P(r) data (bottom plot). Each set of three curves is associated with a single IFT data item in the IFT control panel. The two subplots are the Pair Distance Distribution Function (top) and Data/Fit (bottom) plots.

The features that are general between all of the plots are described elsewhere. This section will describe features unique to this plot.

Changing axes and plot types¶

Right-click in the Data/Fit (top) plot to view a pop-up menu with different axis settings.

The available plot modes are:

Lin-Lin
Log-Lin
Log-Log
Lin-Log
Guinier plot (ln(I(q)) vs. q²)
Porod plot (q⁴I(q) vs. q)
Kratky plot (q²I(q) vs. q)

The axes cannot be changed for the P(r) (top) plot.

The SEC Control Panel and SEC Plot Panel¶

When SEC-SAXS data is loaded into RAW, it is placed in the SEC tab of the Control Panel, and displayed in the SEC Plot window. This section will cover the features of the SEC control panel and the SEC plot panel.

SEC-SAXS data is assumed to come from continuous collection of images while sample is being eluted from an FPLC column. Depending on the frame rate of the detector and the total elution time, this can generate thousands of images. The basic approach RAW uses to deal with this data is to load every scattering profile (or integrate every image into a scattering profile), and associate it with a single SEC data item. For each SEC item, the total (or average) intensity in each scattering profile is plotted vs. frame number, where frame number is simply what position the item was loaded in (so if 10 items are loaded, the frame number for the first item loaded would be 0, for the next item loaded 1, etc). For standard SEC-SAXS data, this intensity vs. frame number should look similar to the UV chromatograph of the sample after it passes through the column. Some analysis can be carried out directly on this SEC data item, and the scattering profiles of interest can be extracted for further analysis.

The SEC control panel¶

The SEC control panel refers to the panel that is shown in the Control panel when the SEC tab is selected.

The panel consists of three parts. The top part is a set of controls for loading SEC items, controlling the SEC online mode, and analyzing SEC items. The middle part is where individual SEC items loaded into RAW are shown. The bottom part consists of buttons that allow you to save or remove the SEC items. Further manipulation of the SEC items can be done from the right click (context) menu.

An individual item in the SEC list is the called a SEC data item. It is associated with two curves on the SEC plot, one plotted on the right y axis and one on the left y axis.

SEC Data Items¶

When a new item is plotted, a data item is added to the SEC list in the SEC control panel. This allows you to control the properties of the data item, and perform analysis on it.

The name of the data item is displayed for each item. If an item is given a different name for the plot legend, this legend name is displayed in [square brackets] next to the item name. On the same line as the item name, on the right side of the data item, there are several buttons that can be used for further manipulation of the data item.

SEC data items are different from Manipulation and IFT data items as they have many (often hundreds or thousands) of scattering profiles associated with a single data item, and they can have new profiles appended to them after they are loaded.

Note: If there is a * to the left of the item name (between the checkbox and the item name), it indicates there are unsaved changes to the item. This can occur if the item is newly created IFT data (such as from the BIFT or GNOM panels), or if item properties such as the name have been changed.

Showing/Hiding data items on the SEC plot¶

To show/hide the curves on the plot associated with a data item, click the checkbox next to the filename. If the checkbox is checked, the item is currently shown on the SEC plot, if the checkbox is unchecked, the item is currently hidden on the SEC plot.

Data item buttons¶

On the right of the data item are several buttons for controlling the data item. In order of left to right, these buttons have the following properties.

Extended Info

By hovering the mouse over the i in the blue circle button, a tooltip will appear which shows more information about the item. It shows the buffer range and window size used (if any) to calculate the structural parameters of the SEC data item as a function of frame number.

Locate Line

The target button is used to highlight the scattering profile on the graph that is associated with the data item. When the target is pressed, it ‘bolds’ the line each plotted curve associated with the data item (increases the line width by several points). When the target is pressed again, the line width is set back to normal. You can tell if a line is currently bolded, as the target will be orange instead of grey.

Line Properties

The colored line button has two purposes. First, the color matches the current color of the Intensity curve in the SEC Plot. Second, when pressed it opens a line properties dialog which allows you to set the legend label; the line style, width, and color; the data point marker style, size, line color, andfill color; and the error bar line style, width, and color for each line associated with the SEC data item.

Mark

The star button marks an item. This is used when doing operations such as sending data frames to plot or calculating structural parameters. In those cases, the marked (also referred to as starred) item has a special significance.

Loading data items¶

SEC data items can be loaded from the files tab. There is an additional way to load files available in the SEC control panel, this method is intended to be used in conjunction with the SEC online mode. This method is described in the rest of this section.

At the top of the SEC control panel, there is a section for loading files. It has a button “Select file in SEC run”, and then descriptive information about the file: the image prefix, the run number, and the frame numbers of the file. Note that none of these descriptive fields can be edited. Below the descriptive fields is an “Update” button and an “Automatically Update” checkbox.

To load a SEC curve using this method, click the “Select file in SEC run” button. In the file browser that pops up, select any file in the SEC data set. The data set will automatically be loaded into RAW as a SEC item, and the descriptive information will be displayed in the appropriate boxes. This loaded file can then be used with the SEC automatic update mode described below.

Note: Due to how files are loaded from the SEC control panel, it only works for certain header file types. RAW must be able to automatically determine what files are associated with the SEC run, from any file in the run. At the moment, this can only be done with the G1 and G1 WAXS header file formats. If you want to have a particular beamline’s file format added to this loading (and thus able to use the online mode) please contact the RAW developers.

Updating a SEC data item¶

If a SEC data item is loaded via the SEC control panel, it can be updated if additional files are added to the folder that are part of the data collection. This can only be done for the most recent file loaded via the SEC control panel, for which the descriptive information is shown at the top of the SEC control panel.

To do this update, hit the “Update” button on the SEC control panel. RAW will automatically determine all of the files associated with the SEC run, based on the file you selected with the “Select file in SEC run” button, determine if any of those files have not yet been loaded, and if so, load the files and add them to the SEC item as appropriate.

This is useful when working at the beamline while data is being collected, as you may want to start analysis of the SEC curve before all of the images are taken.

SEC automatic update (online mode)¶

The SEC control panel can be used in an online mode, where a SEC curve is automatically updated as data comes in. All of the conditions for manually updating a curve, described above, must be met. Instead of using the “Update” button as in that section, check the “Automatically Update” box in the SEC control panel.

Note: The automatic update will stay on even if you load a new file into the SEC panel using the “Select file in SEC run” button. It will switch to updating this newly loaded file. As with the “Update” button described above, the automatic update only applies to the SEC data item most recently loaded in the SEC panel.

Selecting data items¶

A single data item can be selected by clicking on the item name in the SEC list (similar to how you would select files in your system file browser). When an item is selected, the color of the item background changes from white to gray. If the item is currently selected, clicking on it will cause it to be unselected. Note that for a regular click, all other selected items will be unselected when a new item is selected.

Multiple items may be selected in two ways. If the Control key (Command key on Macs) is held down while clicking on items, each item that is clicked on will be added to the set of selected items. If a single item is first selected and then the Shift key is held down and another item is selected, all of the items in the list between the two items will be selected (including the second item that is clicked on).

All of the items in the list can be selected in two ways. The first is using the select all button, the second is pressing Ctrl-A (Cmd-A), the Control (Command) key and the A key at the same time when you are in the SEC panel. All items can be unselected by clicking in an empty spot of the SEC list (but not above or below the list), or by clicking on an already selected item.

Note: If you have a set of selected items and wish to remove some, holding down the Control (Command) key and clicking on selected items will deselect them without affecting the other selected items.

The top buttons of the SEC item list¶

The SEC item list has a set of three buttons at the top of the panel. These buttons have the following effects, listed from left to right.

Show All

Clicking on the button that looks like an eye will show all SEC items. This is the same as if you manually set all of the show/hide checkboxes in the data items to on.

Hide All

Clicking on the button that looks like an eye with a red x through it will hide all SEC items. This is the same as if you manually set all of the show/hide checkboxes in the data items to off.

Select All

Clicking on the button that looks like a spreadsheet with selected cells will select all of the SEC data items.

Renaming a data item¶

Data items can be renamed by selecting the data item of interest and selecting “Rename” in the right click popup menu.

Note: While no characters are expressly forbidden in the filename, RAW does not sanitize file names before saving, and thus special characters such as ‘/’ and ‘\’ are likely to cause problems when the file is saved.

Saving data items¶

SEC items are saved as “.sec” files, and is the only data that RAW does not save in a human readable format. To save:

Select the item(s) to be saved.
Click the “Save” button or select “Save selected file(s)” from the right click menu.
In the window that pops up, navigate to the directory in which you want to save the files.
If you are saving a single item, the window will give you an opportunity to rename your file if desired. Click “Save” when ready.
If you are saving multiple items, you simply need to select the folder for the items to be saved in, and click “Open”. The items will be saved with the same names displayed in the SEC Panel, in the folder that you chose.

SEC items often contain hundreds or thousands of scattering profiles, so they are not saved in a human readable format. The “.sec” files that RAW saves can only be read by RAW.

Removing data items from the SEC list¶

To remove one or more data items, select them and do one of the following:

Press the “Delete” key on the keyboard
Click the “Remove” button
Select “Remove” from the right click menu

Exporting SEC data¶

The following data can be exported in a spreadsheet ready format: frame number, integrated intensity, mean intensity, Rg, Rg error, I(0), I(0) error, MW, filename, and, if available intensity a q=<#> where <#> is a user selected value, for each individual point.

To do so:

Select the item(s) to be saved.
Select “Export data” from the right click menu.
In the window that pops up, navigate to the directory in which you want to save the file(s).
If you are saving a single item, the window will give you an opportunity to rename your file if desired. Click “Save” when ready.
If you are saving multiple items, you simply need to select the folder for the items to be saved in, and click “Open”. The items will be saved with the same names displayed in the SEC Panel, in the folder that you chose.

The data is saved as a comma separated value (“.csv”) file. This can be opened directly into most spreadsheet programs, such as Excel.

Saving all SEC scattering profiles¶

If you want to save every individual scattering profile loaded into a SEC data item, you can do so by:

Select the item(s) to be saved.
Select “Save all profiles as .dats” from the right click menu.
In the window that pops up, navigate to the directory in which you want to save the file(s).
Select the folder for the items to be saved in, and click “Open”. The items will be saved with the same filenames displayed in the data browser.

Sending data to the main plot¶

Individual scattering profiles can be sent to the main plot for further analysis. This utilizes the middle section of the controls at the top of the SEC Control panel.

To do so:

Star the SEC data item containing the scattering profiles of interest (note: if only one SEC data item is loaded, it does not have to be starred).
Enter the data frames of interest in the “Select Data Frames:” boxes. The box on the left is the first frame of interest, the box on the right is the last frame of interest. All of the frames between those two endpoints (inclusively) are selected.
Either click the “Frames To Main Plot” button, which will send each individual frame selected in part 2 to the Main plot, or click “Average To Main Plot” which will send the average of the selected frames to the Main plot.
Click on the Manipulation panel and Main plot panel to view the scattering profiles.

Calculate structural parameters¶

You can calculate the Rg, MW, and I(0) as a function of frame number for a SEC profile. This is done using the “Calculate/Plot Structural Parameters” section of the SEC control panel (the bottom section of the controls at the top of the panel).

It is important to have a big picture for what is happening when this is done. First you set a buffer range. All of the scattering profiles in that range will be averaged, and then subtracted from every loaded scattering profile. Next you set a window size. This window will be slid across the curve and all of the frames within it averaged (note: this average is of the buffer subtracted scattering profiles). This Rg, MW, and I(0) for this averaged profile is then calculated. Those values are assigned to the center frame of the window. The window is then slid down the curve one frame, and the process is repeated until the window reaches the end of the SEC data. For example, if the window size is 5, the first 5 frames, frames 0, 1, 2, 3, and 4, are averaged, and have the Rg, MW, and I(0) calculated. Then window is moved over by one, and frames 1, 2, 3, 4, and 5 are averaged and have the Rg, MW, and I(0) calculated, and so on.

To do this:

Star the SEC data item containing the scattering profiles of interest (note: if only one SEC data item is loaded, it does not have to be starred).
Enter the buffer range in the “Buffer Range” boxes. The box on the left is the starting buffer frame and the box on the right is the final buffer frame.
Enter the window size.
Select the appropriate molecule type (Protein or RNA) for the molecular weight calculation.
Click the “Set/Update Parameters” button.
The structural parameters will be plotted as a function of frame number in the SEC plot. If RAW was unable to determine the parameters for a particular window, then all parameters in that window are set to -1.

The Rg and I(0) calculations are done using RAW’s automatic rg function. The molecular weight calculation is done using the volume of correlation method.

Note: See below for how to show different structural parameters on the SEC plot.

Data point browsing¶

The Frame number, integrated intensity, mean intensity, Rg, Rg error, I(0), I(0) error, MW, filename, and, if available intensity a q=<#> where <#> is a user selected value, for each individual point can be inspected using the data browser. To do so:

Right-click on the data item of interest.
Select “Show data” in the popup menu.

The SEC data item right click menu options¶

When you right click on a data item, a popup menu is shown. This section describes what each item on the menu does.

Remove

Removes the item.

Export data

Exports the SEC data item to a spreadsheet.

Save all profiles as .dats

Saves all of the scattering profiles loaded into the SEC item as individual .dat files.

Save

Saves the selected data item(s).

SVD

Opens the singular value decomposition (SVD) analysis panel for the selected SEC item.

EFA

Opens the evolving factor analysis (EFA) panel for the selected SEC item.

Show data

Shows the individual data points.

Rename

Renames the data item.

The SEC panel bottom buttons¶

There are three buttons at the bottom of the SEC control panel. They are:

Save

This button saves the selected data item(s).

Remove

This button removes the selected data item(s) from the SEC panel.

Clear SEC Data

This button clears all loaded SEC data. It works the same as if you had selected all of the SEC data items and then removed them.

The SEC Plot window¶

The SEC plot window has only one plot, however data can be plotted on both the left and right y axes. The left axis plots intensity from scattering profiles, while the right axis plots structural parameters (Rg, MW, I(0)). The items associated with the plotted curves are shown in the SEC control panel list.

The features that are general between all of the plots are described elsewhere. This section will describe features unique to this plot.

Changing axes and plot types¶

Right-click in the plot to view a pop-up menu with different axis settings. In this plot, you can change what data is plotted on each axis.

Y Data (Left Axis)

This allows you to change intensity plotted on the left y axis. The methods are:

Integrated intensity (default), which is

\[I_{tot} = \int_{q_{min}}^{q_{max}} I(q) dq\]

Mean intensity, which is the average intensity across the whole scattering profile
Intensity at q=…, which allows the user to specify a q value, and displays the intensity at the nearest q point to that specified value.

Y Data (Right Axis)

This allows you to change which calculated structural parameter is plotted on the right y axis.

RG (default) which plots the Rg
MW which plots the molecular weight
I(0) which plots the I(0)
None, which turns off the right y axis.

X Data

This allows you to change what is plotted on the x axis.

Frame Number, which plots the intensity and structural parameter as a function of frame number.
Time, which puts the x axis in terms of experimental time.

Note: The time display on the x axis is only available for certain types of header files. Currently only G1 and G1 WAXS header files will have associated time values.

The Image plot panel¶

The Image plot panel displays any detector image that RAW is able to read. The image name is shown at the top of the plot, while the x and y axes are in dimensions of pixels. There is no right click menu for this plot, but scrolling with a mouse scroll wheel will zoom in and out. It is possible to display regions where there is no image data, these will show up as a white background (for example, if you zoom out and show negative pixel ranges.

In addition to the plot toolbar buttons shared by all of the plots, the Image plot has the following buttons:

100002010000001800000018B603AD208545D77F_png

Displays the image header. This does not display any information from separate header files, only what is in the image itself.

1000020100000018000000187E6D3770EAA7994B_png

Opens the image display settings dialog described in below.

10000201000000500000001EACBFAAFAC41D2CC1_png

Shows the previous/next image in the same directory as the current image (as sorted by filename, A-Z).

The Image Display Settings dialog¶

The Image Display Settings dialog lets you change how the image is displayed on the screen. It has three sections, Image parameters, Image scaling, and Colormaps.

Image parameters

There are three slider bars here. The Upper and Lower limit sliders let you set what pixel values in the image correspond to the most extreme values of the color scale chosen. The Brightness color bar does not work at the moment. By default, the upper and lower limit values are automatically set each time an image is loaded into the Image plot panel. However, checking the Lock values checkbox will prevent the current set values from changing. This allows you to scroll through images maintaining constant upper and lower limits.

Image scaling

Image scaling allows you to use a linear or logarithmic (base 10) color scale. The logarithmic scale uses the matplotlib.colors.SymLogNorm function (more available here: http://matplotlib.org/api/colors_api.html), with a linear threshold set at 1. So the image is logarithmic in color display above 1, and linear in color display below 1 (this is so 0 and negative value pixels can be handled).

Colormaps

The colormaps option allows you to change the color scale used to display intensity values in the Image plot. The color maps are those available through matplotlib ( http://matplotlib.org/users/colormaps.html), a limited selection of which are available in RAW: Gray, Heat, Rainbow, Jet (default), Spectral.

Note: Currently there is also a brightness setting, but it does not work properly.

Analysis windows¶

There are a variety of analysis windows in RAW: the Guinier Fit window, the Molecular Weight window, the BIFT window, the GNOM window, the AMBIMETER window, the DAMMIF window, the SVD window, and the EFA window. In this chapter we will cover how to open these windows and carry out the analysis.

Guinier Fit window¶

Data that has been properly background subtracted can be analyzed in the standard fashion via a Guinier Fit to yield the radius of gyration and the scattering intensity at zero angle. The Guinier Fit window allows you to interactively do the fit.

Opening the Guinier fit window¶

The Guinier fit window can be opened by selecting a data item in the Manipulation Panel and either right clicking and selecting “Guinier fit” or selecting “Guinier fit” from the “Tools” menu. Whichever data item is selected will be the one that is being analyzed. If multiple items are selected, the first (top item in the list) will be sent to the Guinier Fit panel.

Once the window opens you will see two panels. The left panel has the filename of the data loaded into the Guinier Fit panel, the fit parameters, and the fit controls. The right panel has two plots that show the fit and residual. If no previous Guinier fit analysis has been saved for the data item, the fit range defaults to the whole scattering profile. If a previous Guinier fit analysis was saved for the data item, the fit range will be the range of that analysis.

Once you have completed a satisfactory Guinier fit, clicking the “OK” button will save the fit for the data item selected. That will allow you to save the Guinier analysis parameters when saving all results, view the results in the information window, and save it with the scattering profile if that is saved as a “.dat” file. Clicking the “Cancel” button (or the close window button from the operating system, typically in the upper left or right corner) will exit the window without saving the analysis results.

Note 1: Only one Guinier Fit window can be open at once. If a window is already open and you try to open a new one, it will close the current window and open a new one.

Note 2: RAW will automatically attempt to automatically determine the Rg when the Guinier window is opened, if no previous Guinier fit has been done. If a previous Guinier fit has been done, RAW will set the q min and q max to the previous values.

Guinier Fit Controls¶

The Guinier Fit Controls allow you to manually set the q_min and q_max values over which the Guinier fit is done. You can do this in two ways. First, if you type a value into the q_min or q_max box and hit enter, RAW will find the nearest q point, and set the q value to that nearest point. Second, you can adjust the n_min and n_max values either by typing an integer value in the appropriate box or using the spin controls. The n_min and n_max controls change the minimum and maximum points of the q vector used (zero indexed), and the q_min and q_max are updated to match. For example if n_min is set to 5 and n_max is set to 130, the 5th indexed point through the 130th indexed point in the q vector (so points 6-131, because of the z ero indexing) will be used in the Guinier fit.

The AutoRG button calls a function that attempts to automatically determine the best range of data to use for the Guinier fit. It considers the following criteria: q_min*Rg as small as possible, q_max*Rg as close to 1.3 as possible, smallest fit error in the Rg, smallest fit error in the I(0) possible, r^2 of the fit as close to one as possible, and the largest range of q values used as possible.

Guinier Fit Plots¶

There are two plots in the Guinier fit window. The top plot is the standard Guinier plot, ln(I(q)) vs. q², shown in the range of n_min -20 to n_max +3 (if possible) as set in the Guinier Control panel. The scattering profile is marked by the blue points, the fit is shown as the solid straight red line, the fit extrapolated to q=0 is shown by the green dashed line, and the range of the fit, q_min and q_max, are shown by vertical red dashed lines.

The bottom plot has the residual between each point in the scattering profile and the fit, plotted as a Δln(I(q)) vs. q². The red line is a line at zero. Note that the x range of this plot may be slightly reduced compared to the top plot, as it only displays the fit, while the top plot can display points beyond the limits of the fit.

Note: There are no controls available on right click for these plots, but there is a standard plot control bar.

Guinier Fit Parameters¶

The fit parameters available in the Guinier Fit window are:

I(0), the extrapolated scattering intensity at zero angle
Rg, the radius of gyration
r² (fit), the r² value of the fit
qRg, which is q_min*Rg and q_max*Rg for the left and right boxes respectively.

Molecular weight¶

Data that has been background subtracted and had a Guinier fit carried out can be analyzed to find the molecular weight of the macromolecule. RAW provides four different ways of calculating the molecular weight, which are described below.

Opening the Molecular Weight Window¶

The Molecular Weight window can be opened by selecting a data item in the Manipulation Panel and either right clicking and selecting “Molecular Weight” or selecting “Molecular Weight” from the “Tools” menu. Whichever data item is selected will be the one that is being analyzed. If multiple items are selected, the first (top item in the list) will be the one analyzed.

The window has two parts. The top part is a description of the MW methods and a panel with the Guinier fit parameters. The bottom part is four panels providing the calculated MW from each method, and the ability to see more details and learn more about each method. All of the MW methods require a Guinier fit to have been done, as they rely on the I(0) value. Two of the methods require knowing the sample concentration, and the same two methods also depend on (different) calibration of the scattering profile.

Once you are satisfied with the molecular weight analysis, clicking the “OK” button will save the analysis for the data item selected. That will allow you to save the molecular weight analysis parameters when saving all results, view the results in the information window, and save the results with the scattering profile if that is saved as a “.dat” file. Clicking the “Cancel” button (or the close window button from the operating system, typically in the upper left or right corner) will exit the window without saving the analysis results.

Note: Only one molecular weight window can be open at once. If a window is already open and you try to open a new one, it will close the current window and open a new one.

The molecular weight controls¶

There are several controls available in the molecular weight panel. In the top panel, the Guinier Fit button can be used to open the Guinier panel and (re)analyze the scattering profile using that method, so that an I(0) value is available for the molecular weight panel.

Each of the four lower panels has a “Show Details” and “More Info” button. The “More Info” button simply provides an extended description of the method used, and, when relevant, a citation. The “Show Details” button provides extended information on the parameters used to calculate the MW for a given method. None of these parameters are editable in the molecular weight panel. When the “Show Details” button is clicked, it becomes a “Hide Details” button, which hides the extra parameters.

The concentration box in both the first and fourth panels can have the sample concentration entered, if it has not already been (for example, in the Information panel). These are linked, so that any change to one also changes the other. Sample concentration should be in mg/ml.

The volume of correlation method uses different parameters for Proteins and RNA, and the drop down menu can be used to toggle between those two sets of parameters. The default setting can be changed in the Options window.

If RAW is set to normalize scattering profiles to an absolute scale, the “Intensity on Absolute Scale” box will be checked in the fourth panel. If RAW is not normalizing to an absolute scale, it will be unchecked. If that is set incorrectly for a particular scattering profile for some reason, it can be manually toggled to the correct position.

At the bottom of the window there are three buttons. The “OK” and “Cancel” buttons have been described above. The “Change Advanced Parameters” button opens the Options window, with the Molecular Weight section shown. This allows you to change the parameters used to calculate the molecular weight for each method of calculation.

The molecular weight parameters¶

There are a number of parameters listed in the molecular weight panel. In the top panel there are:

Filename

Gives the filename of the data item being analyzed.

Guinier parameters

Gives the I(0) and Rg from the Guinier fit.

In the I(0) Ref. MW panel there are:

Concentration

The sample concentration in mg/ml, this field accepts input.

MW

The sample MW calculated by this method, in kDa.

Ref. I(0)

The I(0) of the molecular weight standard set for RAW. This field corresponds to the I(0) field in the “Molecular Weight Estimation Using a Standard” box in the Molecular Weight section of the Options window, and can be changed there or by setting the MW standard.

Ref. MW

The molecular weight of the molecular weight standard set for RAW. This field corresponds to the MW field in the “Molecular Weight Estimation Using a Standard” box in the Molecular Weight section of the Options window, and can be changed there or by setting the MW standard.

Ref. Concentration

The concentration of the molecular weight standard set for RAW. This field corresponds to the Conc. field in the “Molecular Weight Estimation Using a Standard” box in the Molecular Weight section of the Options window, and can be changed there or by setting the MW standard.

File – The data item name of the molecular weight standard set for RAW. This field corresponds to the Filename field in the “Molecular Weight Estimation Using a Standard” box in the Molecular Weight section of the Options window, and can be changed there or by setting the MW standard.

In the Vc MW panel there are:

MW

The sample molecular weight calculated by this method, in kDa.

Vc

The volume of correlation calculated by this method, in Å².

Qr

The Qr parameter calculated by this method, in Å³.

a

The macromolecular type (protein/RNA) dependent “a” parameter used for the calculation. This field corresponds to the “Protein (RNA) Coef. A” field in the “Molecular Weight Estimation From Volume of Correlation” box in the Molecular Weight section of the Options window, and can be changed there. The value depends on whether Protein or RNA is selected in the drop down menu at the top of this panel.

b

The macromolecular type (protein/RNA) dependent “b” parameter used for the calculation. This field corresponds to the “Protein (RNA) Coef. B” field in the “Molecular Weight Estimation From Volume of Correlation” box in the Molecular Weight section of the Options window, and can be changed there. The value depends on whether Protein or RNA is selected in the drop down menu at the top of this panel.

This panel also has a plot which shows $\int q I(q) dq$ vs. q, over the q-range of the scattering profile. For this method to be accurate, the integral value needs to have converged at high q (the graph needs to be flat at high q).

In the Vp MW panel there are:

MW

The sample molecular weight calculated by this method, in kDa.

Vp

The Porod volume calculated by direct integration of the scattering profile, in Å³.

Corrected Vp

The corrected Porod volume based on the method described below, in Å³.

Macromolecule Density

The density of the macromolecule, used to calculate the molecular weight. This field corresponds to the “Density” field in the “Molecular Weight Estimation from Corrected Porod Volume” box in the Molecular Weight section of the Options window, and can be changed there.

In the Abs. MW panel there are:

Concentration

The sample concentration in mg/ml, this field accepts input.

MW

The sample MW calculated by this method, in kDa.

# electrons per mass dry macromolecule

The dry mass number density of electrons for the macromolecule, in e-/g. This field corresponds to the “Electrons per dry mass of macromolecule” field in the “Molecular Weight Estimation from Absolute Intensity Calibration” box in the Molecular Weight section of the Options window, and can be changed there.

# electrons per volume of buffer

The number density of electrons for the protein buffer/solvent, in e-/cm³. This field corresponds to the “Electrons per volume of aqueous solvent” field in the “Molecular Weight Estimation from Absolute Intensity Calibration” box in the Molecular Weight section of the Options window, and can be changed there.

Protein partial specific volume

The partial specific volume of the macromolecule, in cm³/g. This field corresponds to the “Partial specific volume of the macromolecule” field in the “Molecular Weight Estimation from Absolute Intensity Calibration” box in the Molecular Weight section of the Options window, and can be changed there.

Scattering length of an electron

The scattering length of an electron in cm. This field corresponds to the “Scattering length of an electron” field in the “Molecular Weight Estimation from Absolute Intensity Calibration” box in the Molecular Weight section of the Options window, and can be changed there.

Scattering contrast per mass

The calculated scattering contrast per mass. This is calculated from the other parameters as $r_0(\rho_{Mmac}-\rho_{solv}\bar{\nu})$ where $r_0$ is the scattering length of an electron, $\rho_{Mmac}$ is the electrons per dry mass of macromolecule, $\rho_{solv}$ is the electrons per volume of aqueous solvent, and $\bar{\nu}$ is the partial specific volume of the protein.

The molecular weight methods¶

Four different methods are used to calculate the molecular weight of the macromolecule from the background subtracted scattering profile.

I(0) Referenced molecular weight calculation (I(0) Ref. MW panel)

The scattering at zero angle, I(0) is proportional to the molecular weight of the macromolecule, and the concentration and contrast of the macromolecule in solution. If a reference sample of known molecular weight and concentration is measured, it can be used to calibrate the molecular weight of any other scattering profile with known concentration (assuming constant contrast between reference and sample, and a monodisperse sample). Molecular weight is calculated as:

\[MW_m=\left(\frac{I(0)_m}{c_m}\right)\left(\frac{MM_{st}}{I(0)_{st}/c_{st}}\right)\]

where MW is the molecular weight, c is the concentration, and the m and st subscripts denote quantities from the macromolecule of interest and the standard respectively. For a reference see, among many, Mylonas, E. & Svergun, D. I. (2007). J. Appl. Crystallogr. 40, s245-s249.

This method can yield inaccurate results if the reference is not properly calibrated, I(0) is not well estimated from the Guinier fit, or the contrast between the macromolecule and buffer is significantly different between the reference and sample.

Volume of correlation based molecular weight calculation (Vc MW panel)

This method uses the approach described in: Rambo, R. P. & Tainer, J. A. (2013). Nature. 496, 477-481. First, the volume of correlation, V_c, is calculated as

\[V_c=\frac{I(0)}{\int qI(q)dq}\]

Unlike the Porod volume, V_c is expected to converge for both compact and flexible macromolecules. Physically, V_c can be interpreted as the particle volume per self-correlation length, and has units of Å². V_c and the radius of gyration, Rg, are then used to calculate a parameter $Q_r=V_c^2/R_g$. The molecular weight is then calculated as:

\[MW=\left(\frac{Q_r}{b}\right)^a\]

where a and b are empirically determined constants that depend upon the type of macromolecule. More details on the calculation are in the reference. The authors claim the error in MW determination is ~5-10%.

This method can yield inaccurate results if the integral $\int qI(q)dq$ doesn’t converge, which may indicate the scattering profile is not measured to high enough q or that there is a bad buffer match. It also requires accurate determination of I(0) and Rg. It doesn’t work for protein-nucleic acid complexes.

Corrected Porod Volume method (Vp MW panel)

This method uses the approach described in: Fischer, H., de Oliveira Neto, M., Napolitano, H. B., Polikarpov, I., & Craievich, A. F. (2009). J. Appl. Crystallogr. 43, 101-109. First, the Porod volume, V_p, is determined. True determination of the Porod volume requires the scattering profile measured to infinite q. A correction is applied to V_p to account for the limited range of the measurement. The authors report a maximum of 10% uncertainty for calculated molecular weight from globular proteins.

This method can yield inaccurate results if the molecule is not globular. It requires accurate determination of I(0). It also requires an accurate protein density. It only works for proteins.

Note: To do the integration, RAW extrapolates the scattering profile to I(0) using the Guinier fit. The authors of the original paper used smoothed and extrapolated scattering profiles generated by GNOM. This may cause discrepancy. To use this method on GNOM profiles, use the online SAXS MoW calculator located at: http://www.if.sc.usp.br/~saxs/

Absolute calibrated intensity method (Abs. MW panel)

This uses the absolute calibration of the scattering profile to determine the molecular weight, as described in Orthaber, D., Bergmann, A., & Glatter, O. (2000). J. Appl. Crystallogr. 33, 218-225. By determining the absolute scattering at I(0), if the sample concentration is also known, the molecular weight is calculated as:

\[MW=\frac{N_AI(0)/c}{\Delta \rho^2_M}\]

where N_A is the Avagadro number, c is the concentration, and $\Delta \rho_M^2$ is the scattering contrast per mass described above. The accuracy of this method was assessed in Mylonas, E. & Svergun, D. I. (2007). J. Appl. Crystallogr. 40, s245-s249, and for most proteins is <~10%.

This method can yield inaccurate results if the absolute calibration is off, or if the partial specific volume of the macromolecule in solution is incorrect. I(0) and the concentration in solution must be well determined. Unless the scattering contrast is adjusted, this method will only work for proteins.

BIFT¶

The BIFT window allows you to run a Bayesian Indirect Fourier Transform (BIFT) method on background subtracted scattering profiles to find the P(r) function. The advantage to this method over the method implemented by GNOM is that once the search parameters are set, there is no subjective input required from the user, a single “best” solution is provided by the algorithm. The BIFT algorithm being used to find the P(r) is that of: Hansen, S. (2000). J. Appl. Crystallogr. 33, 1415-1421.

Opening the BIFT Window¶

The BIFT window can be opened by selecting a data item in the Manipulation Panel and either right clicking and selecting “BIFT” or selecting “BIFT” from the “Tools” menu. Whichever data item is selected will be the one that is being analyzed. If multiple items are selected, the first (top item in the list) will be the one analyzed.

The window has two parts. The left part shows the file being analyzed, and the controls, parameter outputs, and status for the BIFT. The right is two plots, the top showing the P(r) function found by the BIFT and the bottom showing the experimental data and the scattering profile generated from the P(r) function. The space allotted to each side can be adjusted by clicking and dragging the separator bar. The whole window can be resized by clicking and dragging an edge or corner.

When the window is first opened, it runs a BIFT analysis to find the P(r) function of the data, based on the current settings. These settings can be altered from the BIFT panel in the Options window.

Once you are satisfied with the BIFT results, clicking the “OK” button will save the Dmax, real space Rg, real space I(0), the χ² for the fit, and $\log_{10}\alpha$ for the data item selected. That will allow you to save the BIFT analysis parameters when saving all results and save the results with the scattering profile if that is saved as a “.dat” file. Additionally, a new IFT data item will be created, which will be shown in the IFT Control and Plot panels. Clicking the “Cancel” button (or the close window button from the operating system, typically in the upper left or right corner) will exit the window without saving the analysis results or new IFT item. IFT items created by BIFT will have an extension “.ift”.

Note: Only one BIFT window can be open at once. If a window is already open and you try to open a new one, it will close the current window and open the new one.

The BIFT Controls¶

A BIFT analysis of the scattering profile is automatically run when the window is opened. BIFT has very few controls available to the user, though settings can be customized. The controls consist of three buttons:

Run

Reruns the BIFT analysis. Needs to be done if the settings are changed after the BIFT panel is opened.

Abort

Aborts the BIFT analysis if it is currently running.

Settings

Opens the Options window and shows the settings for BIFT. If settings are changed, the Run button: must be used to generate a new P(r) function with the changed settings.

The BIFT parameters¶

The BIFT panel displays the following parameters:

Dmax

The maximum dimension of the P(r) function found by the BIFT algorithm. This is in units of 1/q, which RAW assumes to be Å.

Log(Alpha)

The log base 10 of the alpha value found as optimal by the BIFT search.

Rg (A)

The radius of gyration in Å (assumed, actual units of 1/q). This is shown from the Guinier fit (if available) and the P(r) function. The value from the P(r) function is the value calculated in real space by

\[R_g=\frac{\int_0^{D_{max}}r^2 P(r) dr}{2\int_0^{D_{max}} P(r)dr}\]

I(0)

The scattering at zero angle. This is shown from the Guinier fit (if available) and the P(r) function. The value from the P(r) function is the value calculated in real space by

\[I(0)=4\pi\int_0^{D_{max}} P(r) dr\]

chi^2 (fit)

The χ² value of the scattering profile from the P(r) function to the experimental data.

The BIFT status¶

The status box for the BIFT search shows parameters that update as the BIFT search is performed. Once the search is over, they show the parameters of the final solution. The status items displayed are:

Status

An overall status, which can be: Performing search grid, Performing Fine Search, BIFT done, or BIFT canceled.

Evidence

The evidence value for a given search point.

Chi

The χ² value of a given search point.

Alpha

The log base 10 of the alpha value of a given search point.

Dmax

The maximum dimension of the current search point.

Current Search Point

The current search point (numbered along the search grid, essentially arbitrary).

Total Search Points

The total number of search points, equal to the number of Dmax search points multiplied by the number of alpha search points.

Note: The status window does not update the evidence, chi, alpha, or dmax values during the fine search, only at the end of the fine search.

The BIFT plots¶

There are two plots in the BIFT window. The top plot shows the P(r) function in red. The units for the r (bottom) axis of this plot are 1/q, which RAW assumes to be Å. A black line is displayed at zero on the plot for reference. The bottom plot shows the measured scattering profile data as blue points, and the scattering profile generated from the P(r) function in red.

Note: There are no controls available on right click for these plots, but there is a standard plot control bar.

The BIFT algorithm¶

The algorithm used is described in Hansen, S. (2000). J. Appl. Crystallogr. 33, 1415-1421. In RAW, a coarse grid is used for an initial search, and then a fine optimization is performed from the best point in that search space. The limits of the coarse grid and the number of points in the coarse grid can be set in the Options window.

GNOM (ATSAS)¶

RAW allows you to run certain analyses using the ATSAS software package from within RAW. One of the programs that can be run from RAW is GNOM, which performs an IFT to find the P(r) function. Using the ATSAS package programs requires a separate installation and (possibly) some additional configuration of RAW.

Opening the GNOM Window¶

The GNOM window can be opened by selecting a data item in the Manipulation Panel and either right clicking and selecting “GNOM (ATSAS)” or selecting “GNOM” from the “Tools”->“ATSAS” menu. Whichever data item is selected will be the one that is being analyzed. If multiple items are selected, the first (top item in the list) will be the one analyzed.

The window has two parts. The left part shows the filename being analyzed, and the controls and parameter outputs for GNOM. The right part has two plots, the top showing the P(r) function found by the GNOM, and the bottom showing the experimental data and the scattering profile generated from the P(r) function. The space allotted to each side can be adjusted by clicking and dragging the separator bar. The whole window can be resized by clicking and dragging an edge or corner.

When the window is first opened, if no previous GNOM analysis is available for the data item, RAW runs DATGNOM from the ATSAS package analysis to find a P(r) function of the data. Generally, better results are obtained from DATGNOM when an Rg value is available from the Guinier fit. If GNOM analysis has previously be done on the data item, RAW will display the P(r) function corresponding to the Dmax value found by that analysis.

Once you are satisfied with the GNOM results, clicking the “OK” button will save the Dmax, Total Estimate, real space Rg, real space I(0), and the starting and ending q values for the data item selected. That will allow you to save the GNOM analysis parameters when saving all results and save the results with the scattering profile if that is saved as a “.dat” file. Additionally, a new IFT data item will be created, which will be shown in the IFT Control and Plot panels. Clicking the “Cancel” button (or the close window button from the operating system, typically in the upper left or right corner) will exit the window without saving the analysis results or new IFT item. IFT items created by GNOM will have an extension “.out”.

Note 1: Only one GNOM window can be open at once. If a window is already open and you try to open a new one, it will close the current window and open the new one.

Note 2: If the GNOM option is unavailable in the right click menu for a data item, it indicates that RAW is unable to find the ATSAS programs on your computer.

The GNOM Controls¶

The GNOM Controls allow you to manually set the q_min and q_max values GNOM uses. You can do this in two ways. First, if you type a value into the q_min or q_max box and hit enter, RAW will find the nearest q point, and set the q value to that nearest point. Second, you can adjust the n_min and n_max values either by typing an integer value in the appropriate box or using the spin controls. The n_min and n_max controls change the minimum and maximum points of the q vector used (zero indexed), and the q_min and q_max are updated to match. For example if n_min is set to 5 and n_max is set to 130, the 5th indexed point through the 130th indexed point in the q vector (so points 6-131, because of the zero indexing) will be used by GNOM. The Dmax value can be either by typing an integer value in the appropriate box or using the spin controls. Changing any of these values (q_min, q_max, and Dmax) automatically updates the P(r) function.

The “Change Advanced Parameters” button opens the Options panel and shows the options for GNOM. This allows you to change advanced parameters for your GNOM analysis.

The “DATGNOM” button runs the DATGNOM program from the ATSAS software package. The Dmax it finds is rounded to the nearest integer, and GNOM is run with that Dmax value to generate the P(r) function.

The GNOM parameters¶

The GNOM panel displays the following parameters from the GNOM fit:

Rg (A)

The radius of gyration in Å (assumed, actual units of 1/q). This is shown from the Guinier fit (if available) and the P(r) function. The value from the P(r) function is the value calculated in real space by

\[R_g=\frac{\int_0^{D_{max}}r^2 P(r) dr}{2\int_0^{D_{max}} P(r)dr}\]

I(0)

The scattering at zero angle. This is shown from the Guinier fit (if available) and the P(r) function. The value from the P(r) function is the value calculated in real space by

\[I(0)=4\pi\int_0^{D_{max}} P(r) dr\]

Total Estimate

The “Total Estimate” produced by GNOM. A value close to 1 is good.

chi^2 (fit)

The χ² value of the scattering profile from the P(r) function to the experimental data.

GNOM says

The subject interpretation of the quality of the P(r) function provided by GNOM.

The GNOM plots¶

There are two plots in the GNOM window. The top plot shows the P(r) function in red. The units for the r (bottom) axis of this plot are 1/q, which RAW assumes to be Å. A black line is displayed at zero on the plot for reference. The bottom plot shows the measured scattering profile data as blue points, and the scattering profile generated from the P(r) function in red.

Note: There are no controls available on right click for these plots, but there is a standard plot control bar.

AMBIMETER (ATSAS)¶

RAW allows you to run certain analyses using the ATSAS software package from within RAW. One of the programs that can be run from RAW is AMBIMETER, which provides an estimate of the ambiguity a 3D shape reconstruction will have, based on the scattering profile generated from the P(r) function. Using the ATSAS package programs requires equires a separate installation and (possibly) some additional configuration of RAW.

Opening the AMBIMETER Window¶

The AMBIMETER window can be opened by selecting a data item in the IFT Panel and either right clicking and selecting “Run AMBIMETER” or selecting “AMBIMETER” from the “Tools” -> “ATSAS” menu. Whichever data item is selected will be the one that is being analyzed. If multiple items are selected, the first (top item in the list) will be the one analyzed. Currently, AMBIMETER only works on IFT items generated by GNOM (“.out” files in the IFT panel). The AMBIMETER window shows the file it is being run on, the Rg (real space form the P(r) function), and controls and results.

When the window is first opened, AMBIMETER is run on the data. Once you are satisfied with the GNOM results, clicking the “OK” or “Cancel” will close the window. Because of the strict save format required for “.out” files to be used by the ATSAS package, the AMBIMETER results are not saved anywhere, and must be manually saved by the user (such as writing it down).

Note 1: Only one AMBIMETER window can be open at once. If a window is already open and you try to open a new one, it will close the current window and open the new one.

Note 2: If the AMBIMETER option is unavailable in the right click menu for a data item, it indicates that RAW is unable to find the ATSAS programs on your computer or that you do not have a recent enough version of the ATSAS package installed (version 2.7.1 or greater required for AMBIMETER).

The AMBIMETER Controls¶

The AMBIMETER controls allow you to adjust the maximum q value used by AMBIMETER, by adjusting the upper q*Rg limit between 3 and 7. Note that if the maximum q value of the scattering profile times the Rg is less than the limit set, the whole curve is used.

The AMBIMETER program can also save output shapes. For more information about this, see the AMBIMETER manual available with the ATSAS software. In the window, you can select which shapes to save, None (default), Best (one shape, the best fit), or All (all of the shapes that fit). If you are saving shapes, you should then select the output directory to save them in, and provide an output prefix. The shapes will with the prefix value provided in the Output prefix box, as described in the AMBIMETER manual. Clicking the Run button is necessary to rerun AMBIMETER after any settings have been changed.

The AMBIMETER Results¶

The results section shows the output from AMBIMETER. It reports:

Number of compatible shape categories

The number of compatible shape categories, as described in the AMBIMETER manual.

Ambiguity score

Log base 10 of the number of compatible shape categories.

AMBIMETER says

The subjective interpretation of the ambiguity score provided by AMBIMETER.

DAMMIF (ATSAS)¶

RAW allows you to run certain analyses using the ATSAS software package from within RAW. One of the programs that can be run from RAW is DAMMIF and the accompanying programs DAMAVER and DAMCLUST, which carry out 3D shape reconstructions based on the P(r) function and scattering profile. Using the ATSAS programs requires equires a separate installation and (possibly) some additional configuration of RAW.

Opening the DAMMIF Window¶

The DAMMIF window can be opened by selecting a data item in the IFT Panel and either right clicking and selecting “Run DAMMIF” or selecting “DAMMIF” from the “Tools”->“ATSAS” menu. Whichever data item is selected will be the one that is being analyzed. If multiple items are selected, the first (top item in the list) will be the one analyzed. Currently, DAMMIF only works on IFT items generated by GNOM (“.out” files in the IFT panel).

The DAMMIF window shows current DAMMIF settings, controls, the log output for each separate DAMMIF and DAMAVER run, and the overall status of the processing. The “Close” button closes the window. If this is done before the DAMMIF processing is finished, it will abort the processing.

Note 1: Only one DAMMIF window can be open at once. If a window is already open and you try to open a new one, it will close the current window and open the new one.

Note 2: If the DAMMIF option is unavailable in the right click menu for a data item, it indicates that RAW is unable to find the ATSAS programs on your computer.

Note 3: The DAMMIF processing can be run in the background while further data processing is done in RAW.

The DAMMIF settings¶

The DAMMIF window settings section allows you to change the most commonly used DAMMIF settings. The “Change Advanced Settings” button allows you to change all of the advanced settings of DAMMIF.

Output directory

This sets the output directory for DAMMIF and DAMAVER results, and can be set either by typing a directory into the box and hitting enter, or using the Select/Change Directory button. The directory defaults to the directory showing the Files tab.

Output prefix

The output prefix will be appended to each DAMMIF and DAMAVER file. It defaults to the filename being analyzed. This should contain no spaces.

Number of reconstructions

The total number of DAMMIF models to construct.

Number of simultaneous runs

The number of DAMMIF models to reconstruct simultaneously. DAMMIF runs on a single core, and typically takes 100% of the resources of that core. On multicore machines, the number of simultaneous runs can be set between 1 and the total number of cores. The default is the number of cores minus one.

Mode

Sets the Mode used by DAMMIF for the reconstructions. The “Fast” and “Slow” modes are as described in the DAMMIF manual. The “Custom” mode is equivalent to the “Interactive” mode for DAMMIF, and allows the use of the more advanced settings, as set in the Options panel in the “DAMMIF Advanced” section.

Symmetry

Allows the symmetry to be set, if known, as described in the DAMMIF manual.

Anisometry

Allows the anisometry to be set, if known, as described in the DAMMIF manual.

Change Advanced Settings

This opens the Options panel and shows the options for DAMMIF. This allows you to change advanced parameters for your DAMMIF analysis.

Align and average envelopes (damaver)

If this is selected and two or more reconstructions are generated using DAMMIF, then once all reconstructions are finished DAMAVER is automatically run on the DAMMIF reconstructions. This runs in a mode equivalent to “damaver –a” at the command line.

Align and cluster envelopes (damclust)

If this is selected and two or more reconstructions are generated using DAMMIF, then once all reconstructions are finished DAMCLUST is automatically run on the DAMMIF reconstructions.

Note: The damaver and damclust options are mutually exclusive, you can select DAMAVER or DAMCLUST but not both.

The DAMMIF controls¶

There are only two control buttons, “Start”, which starts the DAMMIF reconstructions, and “Abort”, which aborts the DAMMIF reconstructions. Start is only available if DAMMIF reconstructions are not currently running. Abort is only available if DAMMIF reconstructions are running.

Note: DAMMIF requires that the P(r) function be written to disk as a “.out” file. RAW will check whether there is an existing “.out” file with the same name that will be overwritten before running. It will also check whether the DAMMIF files generated will overwrite any existing files. In either case, it will provide a warning to let you know that is happening.

Note 2: DAMMIF gets the starting random seed value from the computer clock time in seconds. In order to produce different reconstructions, the start of each reconstruction must occur at least 1s after the previous one. This introduces a noticeable delay when starting up a large number simultaneous reconstructions.

The DAMMIF status¶

The status panel provides an overview of the current status of the DAMMIF runs. It updates with the following status:

Starting Processing

Indicates that the initial processing has started.

Starting DAMMIF run <#>

Here the <#> corresponds to the run number (1 up to the total number of reconstructions), and this corresponds to the numbers in the Log panel. This indicates that the given DAMMIF run has started.

Finished DAMMIF run <#>

Here the <#> corresponds to the run number (1 up to the total number of reconstructions), and this corresponds to the numbers in the Log panel. This indicates that the given DAMMIF run has finished.

Starting DAMVER

Indicates that the reconstructions are now being aligned and averaged by DAMAVER.

Finished DAMAVER

Indicates that the reconstructions have finished being aligned and averaged by DAMAVER.

Finished Processing

Indicates that all processing has finished.

Processing Aborted!

Indicates that the processing was aborted before everything finished.

If necessary, the status window is scrollable.

The DAMMIF log¶

The log area provides details of the current and finished DAMMIF and DAMAVER processing. This is the output that would be displayed in the console window if DAMMIF or DAMAVER were run from the command line.

When the DAMMIF window is first opened, the log window will be empty. Once DAMMIF processing is started, a set of different panels accessible via the tabs at the top will be opened. Any tab with a number corresponds with a DAMMIF run (output from DAMMIF run 1 will display in tab “1”, and so on). The runs are simply numbered sequentially starting with 1 and ending with the total number of reconstructions. If DAMAVER will also be run, the last tab is “Damaver”, which shows the output of that processing.

Before a given run is started, the window associated with the tab will be empty. Once it starts, it will be updated with the output from DAMMIF or DAMAVER that would normally display in the console. Once it finishes, it will no longer update, but the log can still be viewed. The long windows are scrollable. You can change which output you are viewing by clicking on the tabs.

Note: The output in the log window is not saved, however by default DAMMIF writes this output to a log file, and the DAMAVER output is available in a set of output files. See the ATSAS manuals for each program for more details.

SVD¶

Singular value decomposition (SVD) is a mathematical technique that is a model independent approach that provides information on the number of unique elements in a data set. Formally, singular value decomposition of a mxn matrix M is a factorization of into three matrices such that

\[M=U\Sigma V^*\]

where U is an mxm unitary matrix, called the left singular values; $\Sigma$ is a diagonal mxn matrix, where the diagonal values are the singular values, and $V^*$ is the conjugate transpose of an nxn unitary matrix V, the right singular vectors. A typical interpretation of singular value decompositions is that the number of singular values significantly above the baseline level represents the number of significant distinct components in the data set.

RAW allows the user to use either scattering profiles or P(r) functions as the data set. This is typically applied to scattering profiles in a SEC-SAXS data set, and the number of significant singular values corresponds to the number of distinct scatterers in the data set. For SVD done across a single well-separated peak from the chromatograph, there would be two significant components: one from the buffer and one from the protein. For SVD done on a poorly separated monomer-dimer peak, there would be three significant components: buffer, monomer, and dimer. RAW allows users to select a range of scattering profiles for SVD, and displays the singular values $\sigma_i$ and the autocorrelation of the left and right singular vectors $R_i$ for each ith singular value defined as

(1)¶\[R_i=\Sigma_n X_{i,n}X_{i,n+1}\]

where X is the U or V singular vector matrix.

Opening the SVD Window¶

The SVD window can be opened by selecting a single data item in the SEC Panel or two or more data items in the Manipulation or IFT Panels and either right clicking and selecting “SVD” or selecting “SVD” from the “Tools” menu.

The SVD window shows current SVD settings, controls, and results. The “Okay” button closes the window and saves the parameters used, the “Cancel” button closes the window and saves nothing.

Note: Only one SVD window can be open at once. If a window is already open and you try to open a new one, it will close the current window and open the new one.

The SVD Controls¶

The SVD controls give the user (up to) three items to control.

Use

This option allows the user to select whether they use Unsubtracted or Subtracted data. Note that this choice is only available for SEC curves, where both unsubtracted and subtracted data can exist within the same data item. Subtracted data can only be selected if the SEC curve has had structural parameters calculated for it.

Use Frames

This option changes the range of frames used for the SVD. If a SEC data item is selected, the frame number corresponds to the frame number in the SEC Plot. If Manipulation or IFT data items are selected in, the frame number is the same as how it would be displayed if the data were plotted on the SEC plot. The plot in the controls window shows the SEC curve of the selected. The red points correspond to all of the data in the data set, the blue points correspond to the data being used for SVD.

Normalize by uncertainty

If this option is selected, the intensity at a given q (SEC and Manipulation data) or r (P(r) data) value is divided by the average uncertainty across all frames at that q or r value. When doing EFA analysis, the data is normalized in this way, so this option allows corresponding SVD analysis.

The SVD Results¶

The SVD results are plotted in the two plots on the right side of the SVD panel. The top plot shows the singular values as a function of index (by default singular values are ordered from largest to smallest). The bottom plot shows the autocorrelation, defined in equation (1), of the left (red) and right (blue) singular vectors as a function of index.

The results control panel allows you to control which singular value indices are plotted. It has two further controls:

Save Plotted Values

This saves the singular values and autocorrelation values as a function of index that are plotted in the plots on the right. The data is saved as a comma separated value (.csv) file.

Save All

This saves all of the singular value information, which is to say, the full U, $\Sigma$, and V^* matrices. It also saves the autocorrelation values for all indices. The data is saved as a comma separated value (.csv) file.

EFA¶

Evolving factor analysis (EFA) is a model independent approach that extends SVD to allow separation of scattering profiles from mixed solutions, particularly overlapping chromatographic peaks from different species. This method was recently applied to SEC-SAXS data (see: Meisburger, S. P., Taylor, A. B., Khan, C. A., Zhang, S., Fitzpatrick, P. F., & Ando, N. (2016). J. Am. Chem. Soc. jacs.6b01563). An improved version of the method described by Meisburger et al. has been implemented in RAW. EFA in RAW starts with SVD, then proceeds by finding the component start and end points in the EFA plot and then rotating the significant singular value vectors into real scattering profiles. RAW implements two new methods for rotation of the singular vectors besides the iterative approached described by Meisburger et al. The first new method is the explicit calculation method described in: Maeder, M. (1987). Anal. Chem. 59, 530–533, while the second is a hybrid method that uses the explicit calculation as a seed for the iterative approach, allowing much faster convergence of the rotation.

Because EFA analysis is relative complex, it consists of three distinct screens in RAW, which are reached by clicking the “Next” and “Back” buttons in the EFA window. These screens step you through the EFA analysis. The window can be closed at any time with the “Cancel” button. If the EFA analysis succeeds, the window can be closed with the “Done” button, which sends the extracted scattering profiles to the Main plot, and saves information about the parameters used in the SEC item (if a SEC item was selected).

Opening the EFA Window¶

The EFA window can be opened by selecting a single data item in the SEC Panel or two or more data items in the Manipulation or IFT Panels and either right clicking and selecting “EFA” or selecting “EFA” from the “Tools” menu.

The EFA window shows current EFA settings, controls, and results. The “Okay” button closes the window and saves the parameters used, the “Cancel” button closes the window and saves nothing.

Note: Only one EFA window can be open at once. If a window is already open and you try to open a new one, it will close the current window and open the new one.

EFA Page 1 – SVD¶

The first EFA page essentially reproduces the SVD window described above. The difference is that there is a panel for User Input, “# Significant SVs” where the user inputs the number of significant singular values/vectors in the data set. RAW will automatically attempt to determine that, but the user can adjust it. If the user changes any of the SVD controls, RAW will not refine the guess for the number of significant singular values.

Once the user is happy with the data range and has determined number of significant singular vectors in the data, they click the “Next” button to move to Page 2.

EFA Page 2 – Evolving Factors¶

The second EFA page presents the results of the Evolving Factor Analysis. The details are described in the above referenced papers. In short, for Forward EFA, SVD is done on pieces of the data set, starting with just the first two frames, then the first three frames, and so on, until the entire data set is used. The singular values are then plotted as a function of final frame index used in that particular SVD, the top plot in this second page. This lets the user determine where certain components start in the data set, when there is a strong increase in a singular value is when a component starts in the data set. For Backward EFA, the same thing is done, except using the last two frames, then the last three frames, etc. The Backward EFA plot, the bottom plot on this page, shows the user where components exit the data set. Both the Forward and Backward EFA plots show one more value than the user marked as significant in Page 1, so that the user can judge where the value diverges from the baseline.

The User Input panel on the left side of the page consists of “Forward” and “Backward” sections each with a number of values equal to the user input of significant singular values from Page 1. These controls allow the user to set where each value diverges from the baseline.

Once the starting points for the values are set for both Forward and Backward EFA, clicking the “Next” button will take the user to Page 3.

Clicking the “Back” button allows the user to go back to Page 1 and adjust the results there.

EFA Page 3 – Rotation¶

The third EFA page allows the user to control the rotation of the singular vectors into scattering profiles and view the result of that rotation. There are several different control boxes:

Component Range Controls

These allow the user to adjust the ranges of the components in the scattering profile. There are a number of controls equal to the number of significant singular vectors selected on Page 1. Each control allows you to set the start and end point of the range, and each control has a “C>=0” checkbox. When checked, that forces the concentration profile for that component to be never negative (a physical constraint that can help with rotation by may hide mathematical errors). When the ranges are adjusted, the plot above this control is updated. The arrows and dashed lines of this plot shown the component ranges, with colors that correspond to the colors in the legend of the Scattering Profiles plot in the top right.

Rotation Controls

Method

This controls the rotation method. There are three choices. Iterative applies the approach of Meisburger et al. (2016). Explicit applies the approach of Maeder (1987). Hybrid uses the explicit approach to seed the iterative approach.

Number of iterations

This can only be set for the Hybrid and Iterative approaches, and sets the maximum number of iterations the algorithm will attempt. If convergence is not reached by the end of these iterations, the method will fail.

Convergence Threshold

This sets the threshold at which the iterative and hybrid solutions are said to be converged. This threshold is defined in Meisburger et al. (2016), and for the nth rotation is the absolute difference between the concentration profiles of the n-1 and n solutions.

Status

The status box displays the status of the rotation. If the rotation has succeeded it will say “Rotation Successful.” If the rotation is in progress it will say “Rotation in progress.” If the rotation has failed it will provide a message with some on the failure, that starts with “Rotation failed.”

Results

Save EFA Data (not profiles)

This button saves the EFA data, including the SVD data, the number of significant values, the ranges Forward and Backward EFA data, the ranges and concentration contraints for each component, the rotation method and other parameters used, and the mean error weighted χ² and concentration data. This is all saved as a .csv file.

In addition to the control boxes, there are also three plots. The top plot shows the scattering profiles obtained via the rotation (if the rotation is successful). The bottom plots show: Left – the mean error weighted χ² as a function of frame, which is a measure difference between the scattering profiles in a given frame as measured and as produced from the rotated scattering profiles and concentration profiles; Right – the concentration profiles for each component, which are color as the scattering profile (top) plot. The concentration profiles are normalized to an area of 1, and so are on an arbitrary scale.

Menus¶

RAW has 5 program wide menus: File, Options, View, Tools, and Help.

The File menu¶

The File menu has five choices:

Load Settings

This loads RAW settings from a .cfg file. This can also be done by double clicking on the .cfg file in the Files tab of the Control Panel.

Save Settings

This saves all of the RAW settings, including image calibration and the image mask, to a .cfg file so they can be loaded again later.

Load Workspace

This loads a RAW workspace.

Save Workspace

This saves a RAW workspace.

Quit

This quits the RAW program. If you have unsaved data or are running DAMMIF reconstructions you will be asked to confirm that you wish to quite.

The View menu¶

The view menu has three top level options and five submenus. The three top level options are: “Show image”, “Show data”, and “Show header”. All of these require that a single manipulation item be selected, and otherwise behave as the options of the same name in the Manipulation data item right click menu.

The view menus named “Top Main Plot Axis” and “Bottom Main Plot Axis” change the plot axes/scale of the top and bottom main plots respective. Each option in the two menus corresponds to the options in the main plot right click menu.

The view menus named “SEC Plot Left Y Axis”, “SEC Plot Right Y Axis”, and “SEC Plot X Axis” change the data plotted on the SEC Plot axes as the options in the SEC plot right click menu.

The Tools menu¶

The Tools menu is split into three parts. In all of the items in the top part of the menu apply to Manipulation data items and all of them require one or more Manipulation data items to be selected. The first submenu is “Operations”. This contains a set of operations that can be run on Manipulation data items, and have identical effects to items of the same name in the manipulation right click menu. The second submenu is “Convert q-scale” and the options in the menu have identical effects to items of the same name in the Manipulation data items, and have identical effects to items of the same name in the manipulation right click menu. The third item is “Use as MW Standard” and has an identical effect as the item of the same name in the manipulation right click menu.

The middle part of the Tools menu consists of analysis tools. The “Guinier Fit”, “Molecular Weight” and “BIFT” options, and the “ATSAS” submenu option “GNOM” all require a single Manipulation data item to be selected. These all open the corresponding analysis windows. The “DAMMIF” and “AMBIMETER” options require a single IFT data item to be selected, and open the corresponding analysis windows. The “SVD” and “EFA” require either that multiple Manipulation or IFT data items be selected for that a single SEC data item be selected, and open the corresponding analysis windows.

The lower part of the Tools menu consists of two options. The “Centering/Calibration” option requires that an appropriate image for centering/calibration be loaded in the Image plot panel. It then opens the Centering/Calibration panel. The “Masking” option requires that an appropriate image for masking be loaded in the Image plot panel. It then opens the Masking panel.

Line properties dialog¶

The Manipulation, IFT, and SEC data items can all open a line properties dialog. This dialog will let you control the following properties for each line associated with the object:

Line: Style (solid, dashed, dotted), width, color

Error bars: Style (solid, dashed, dotted), width, color

Data point marker: Shape (the marker shapes are those used by matplotlib, which are defined here: http://matplotlib.org/api/markers_api.html), size, edge line color, fill color, and whether or not the markers are hollow.

The line properties dialog also lets you set the legend label for the object, if any is needed.

If the dialog is exited with “OK”, all changes take effect. If the dialog is exited with “Cancel”, the old settings are restored.

The Information panel¶

The information panel displays information about a selected data item in the Manipulation panel. It shows the name of the item at the top. It also shows the following analysis, if available, the Rg in Å, I(0), and MW in kDa (from reference to a set MW standard). There is a box into which sample concentration can be entered in mg/mL. Notes or a description of the sample can also be entered, and will be saved as part of the “header” when a .dat file is saved. Finally, the drop down list in the header browser can be used to display and quickly view any header value available for the data item.

If the current Control panel is changed from the Manipulation panel to the IFT or SEC panel, the Information panel is cleared. If the panel is changed back to the Manipulation panel, the data in the Information panel are restored.

Workspaces¶

RAW allows you to save and load sets of files that you are analyzing, called “workspaces”. To save a workspace, go the File menu and select “Save Workspace”. To load a workspace, go to the File menu and select “Load Workspace”.

When you save a workspace, all of the items in the Manipulation, IFT, and SEC panel are saved. When you load a workspace, all of those saved items are loaded back up, with all of the analysis information and line settings they had previously. Any files already open in RAW remain open when a workspace is loaded.

Note: The settings are not saved when a workspace is saved.

Online mode¶

RAW has an online mode that will watch a folder, and automatically load any files placed in the folder that RAW can read. This is typically used to automatically load in data as it is being collected at the beamline. This is distinct from the automatic update feature of the SEC Control panel, and both features can be active at once.

Turning on/off online mode¶

In order to turn on online mode, go to the Options->Online Mode menu and click the “Online” option. When you go online, a file browser will open and you will pick a folder. This folder (and only this folder, online mode does not recurse into subfolders) will be monitored by RAW. Every time a file is added (or updated) in the folder, RAW checks to see whether it can load the file. If it can, it will be loaded into RAW (for example, detector images will be integrated and loaded into the Manipulation panel and Main plot).

To disable online mode, go to the Options->Online Mode menu and click the “Offline” option.

Note: The online status of RAW is indicated in the status bar at the bottom of the main RAW window. On the right side, it says “Mode: <status>” where <status> is OFFLINE or ONLINE. This is only toggled by this online mode, not the automatic update feature of the SEC panel.

Changing directories in online mode¶

If you wish to change the directory that RAW is monitoring in online mode, go to the Options->Online Mode menu and click the “Change Directory” option. A file browser will appear, and you will use that to select the new folder for the Online mode.

Online mode filter¶

In some cases, you may not want every file that enters the directory (that can be read by RAW) to be loaded into RAW. In this case, it is possible to set a filter so that only certainly files can be loaded into RAW. The filter can consist of any number of individual items, which are set as follows. First, chose whether an item is include (open only if) or exclude (don’t open if). Then provide a string for RAW to match in the file name. Then specify where in the filename RAW should match the string (start, end, anywhere). This filter can be set in the options window.

File types¶

Output file types¶

RAW outputs the following file types:

.cfg – These are files containing all of RAW’s settings (configuration files). This file type is not human readable.

.dat – These are files containing three column scattering profile data with a “header” at the start or end of the file for additional information. This file type is human readable.

.ift – These are files containing three column BIFT data followed by four column scattering profile data, with a “header” at the start or end of the file for additional information. This file type is human readable.

.msk – These are files containing a RAW mask. This file type is not human readable.

.out – These are files containing GNOM P(r) data and are written in the standard format described in the ATSAS manual for GNOM. This file is human readable.

.sec – These are files containing SEC-SAXS curves, which can be loaded back into RAW and contain all of the relevant scattering profiles and structural parameters. This file type is not human readable.

.wsp – These are files for saved workspaces. This file type is not human readable.

Input file types¶

In addition to the output file types described above, all of which can be read in by RAW, RAW can load the following types of input files.

ASCII files:

.csv – If data is in a 3 column csv format, it is loaded assuming the columns are q, I, Error.

.dat (2 column) – If a .dat file has exactly 2 columns separated by whitespace, it is loaded as q I.

.dat (4 column) – Produced by FoXs when fitting a theoretical curve to a scattering profile. Loads the experimental and simulated (_FIT) scattering profiles.

.fir – Output by various ATSAS programs such as DAMMIF. Loads the experimental and simulated (_FIT) scattering profiles.

.fit – Output by various ATSAS programs such as DAMMIF. Loads the experimental (often smoothed) and simulated (_FIT) scattering profiles.

.int – Output by CRYSOL if no fitting is used. Loads the scattering intensity in solution.

.rad – Depreciated file format for RAW saved scattering profiles.

.txt - 2 or 3 column data can be read in as if it was a .dat file.

Image files:

Image files are often distinguished not just by their extension, but by their header format. For example, many detectors produce tif files, but require different methods to read in the appropriate header data. RAW uses the fabIO python module to load images. The complete list o f images and extensions supported can be found here: http://pythonhosted.org/fabio/getting_started.html#list-of-file-formats-that-fabio-can-read-and-write

We reproduce that here with the detector/format followed by the file extension in parenthesis: * ADSC Quantum (.img)

Bruker (.sfrm)
CIF binary files (.cbf)
EDNA-XML (.xml)
ESRF EDF (.edf)
Eiger (.h5)
FReLoN
Fit2D spreadsheet (spr)
Gatan Digital Micrograph (.dm3)
General Electric (.No?)
HDF5 (Hierarchical Data Format 5) (.h5)
Hamamatsu CCD (.tif)
MarCCD/Mar165 (.mccd)
Mar345 image plate (.mar3450)
Nonius KappaCCD (.kccd)
Numpy 2D array (.npy)
Oxford Diffraction (.img)
Pixi (.?)
Pilatus Tiff (.tif or .tiff)
Portable aNy Map (.pnm)
Rigaku SAXS format (.img)
Tagged Image File Format (.tif)

In addition, we have written custom methods to support the following detector/file formats:

32 bit TIF (.tif)
FLICAM (.tif)
ILL SANS D11
Medoptics (.tif)
Multiwire (.mpa)
SAXSLab300

Automated saving¶

RAW can automatically saving the following types of files:

Processed image files (after integration)
Averaged data files
Subtracted data files

To turn on any of these capabilities:

Open the Save Directories panel of the Options window.
For the appropriate file type (Processed, Averaged, or Subtracted), select a save directory by clicking the “Set” button.
Check the appropriate Auto Save checkbox.
Click “OK” to exit the Options window and keep the changes to the settings.