Photon-HDF5 file format documentation¶

Overview¶

This document contains specifications of the Photon-HDF5 format. This format allows saving single-molecule spectroscopy experiment data in which at least one stream of photon timestamps is present. It has been designed as a standard container format for a broad range of experiments involving confocal microscopy. Examples are confocal smFRET experiments performed with a single or multiple excitation spots. Both μs-ALEX and ns-ALEX data are supported.

What problems are we trying to solve?¶

• Ensuring long term data persistence.
• A disk space and read speed efficient file format for repeated use as well as for archiving.
• Facilitating data sharing and interoperability between analysis programs and research groups.

Features of HDF5¶

• Open-standard: language and platform independent, self-describing format with open source implementations.
• Efficient: the HDF5 format is a binary format that allows compression and fast read/write operations.
• Flexible: data arrays can be stored in “groups” (hierarchical format). Metadata can be attached to each data entry (attributes).
• No limit in data size.
• Support for a variety of numeric and non-numeric data types.

Photon-HDF5: Design principles¶

The main design principles are:

• Simplicity
• Flexibility
• Compatibility

We aim to define a format that has a minimal set of specifications and therefore is easy to implement. At the same time, it is important that the format can be expanded to accommodate new use cases while maintaining backward compatibility.

To achieve simplicity, the only required file characteristics are a general file layout and the presence of a few basic attributes and parameters. The remaining (small set of) fields defined in this document will be present only when needed by a particular measurement.

We retain flexibility by allowing the user to save any arbitrary data outside the specs of this document. To assure that future versions of this format will not conflict with user-defined fields, we require that all user-defined fields be contained in groups called user.

Overview¶

A Photon-HDF5 is a HDF5 file with a predefined structure for timestamp-based data.

A screen-shot of a typical Photon-HDF5 file opened in HDFView is shown here:

The previous figure shows the 5 main groups contained in a Photon-HDF5 file. Of these, /photon_data and /setup contains the raw data and all the information needed for the analysis. A schematic overview is shown in the next figure:

The remaining 3 groups (/sample, /identity, /provenance) provides additional metadata that make the Photon-HDF5 files self-contained and suitable long-term on-line archival. An overview of there groups is shown below:

As a quick reference, here we list links to the documentation for each group:

• /photon_data: RAW data and measurement type specifications.
• /setup: Description of the experimental setup.
• /identity: Information about the data file.
• /provenance: Information about the original data file (when the Photon-HDF5 file has been converted from another format).
• /sample: Description of the measured sample.

The following sections describe the Photon-HDF5 groups and fields.

Root-level parameters¶

The root node (“/”) in a Photon-HDF5 file contains the following fields:

• /acquisition_time: (float) the measurement duration in seconds.
• /comment: (string) a user-defined comment.

In addition, the root node has the following attributes that can be used to distinguish Photon-HDF5 files from other HDF5 files:

• format_name: must contain the string “Photon-HDF5”
• format_version: (string) the Photon-HDF5 format version.

Photon-data group¶

This section describes the layout and fields in the /photon_data group. Note that only the kind of data is specified (i.e. scalar, integer array, float array), but no data type size is mandated. For arrays, the most commonly used data-type is indicated.

Mandatory fields:

• timestamps: (array) photon timestamps. Typical data-type int64.

• timestamps_specs/

  • timestamps_unit: (float) timestamp units in seconds.

Optional if there is only 1 detector, otherwise mandatory:

• detectors: (array of integers) detector ID for each timestamp. Typical data-type uint8.

When the dataset contains nanotime information (i.e. arrival time of each photon with respect to a laser pulse), the following fields must be present:

• nanotimes:(array of integers) TCSPC nanotimes. Typical data-type uint16.

• nanotimes_specs/

  • tcspc_unit: (float) TAC/TDC bin size (in seconds).
  • tcspc_range:(float) full-scale range of the TAC/TDC (in seconds).
  • tcspc_num_bins: (integer) number of TAC/TDC bins.
  • time_reversed: (boolean) True if nanotimes contains the time elapsed between a photon and the next laser pulse. False if it contains the time elapsed between a laser pulse and a photon.

Finally, if the data come from a simulation, /photon_data may contain:

• particles: (array of integers) a particle ID (integer) for each timestamp. Typical data-type uint8.

Setup group¶

The /setup group contains information about the measurement setup:

• num_pixels: (integer) total number of detector pixels. For example, for a single-spot 2-color smFRET measurement using 2 single-pixel SPADs as detectors this field is 2.
• num_spots: (integer) the number of excitation (or detection) “spots” in the sample. This field is 1 for all the measurements using a single confocal excitation volume. When not applicable, for example under widefield illumination with 2-D imaging detectors, this field is omitted.
• num_spectral_ch: (integer) number of distinct detection spectral channels. For example, in a 2-color smFRET experiment there are 2 detection spectral channels (donor and acceptor), therefore its value is 2. When there is a single detection channel or all channels detect the same spectral band, its value is 1.
• num_polarization_ch: (integer) number of distinct detection polarization channels. For example, in polarization anisotropy measurements, its value is 2. When there is a single detection channel or all channels detect the same polarization (including when no polarization selection is performed) its value is 1.
• num_split_ch: (integer) number of distinct detection channels detecting the same spectral band and polarization state. For example, when a non-polarizing beam-splitter is employed in the detection path, its value is 2. When no splitting is performed, its value is 1.
• modulated_excitation: (boolean) True (or 1) if there is any form of excitation modulation either in the wavelength space (as in μs-ALEX or PAX) or in the polarization space. This field is also True for pulse-interleaved excitation (PIE) or ns-ALEX measurements.
• lifetime: (boolean) True (or 1) if the measurements includes a nanotimes array of (usually sub-ns resolution) photon arrival times with respect to a laser pulse (as in TCSPC measurements).
• excitation_wavelengths: (array of floats) list of excitation wavelengths (center wavelength if broad-band) in increasing order (unit: meter).
• excitation_cw: (array of booleans) for each excitation source, this field indicates whether excitation is continuous wave (CW), True, or pulsed, False. The order of excitation sources is the same as that in excitation_wavelengths and is in increasing order of wavelengths.

The following fields are optional and not necessarily relevant for all experiments. If the associated information is irrelevant or not available, these fields are omitted.

• excitation_polarizations: (arrays of floats) list of polarization angles (in degrees) for each excitation source. The order of excitation sources is the same as in excitation_wavelengths and is in increasing order of wavelengths.
• excitation_input_powers: (array of floats) excitation power in Watts for each excitation source. This is the excitation power entering the optical system.
• excitation_intensity: (array of floats) excitation intensity in the sample for each excitation source (units: Watts/meters²). In the case of confocal excitation this is the peak PSF intensity.
• detection_wavelengths: (arrays of floats) reference wavelengths (in meters) for each detection spectral band. This array is ordered in increasing order of wavelengths. The first element refers to detectors_specs/spectral_ch1, the second to detectors_specs/spectral_ch2 and so on.
• detection_polarizations: (arrays of floats) polarization angles for each detection polarization band. The first element refers to detectors_specs/polarization_ch1, the second to detectors_specs/polarization_ch2 and so on. This field is not relevant if no polarization selection is performed.
• detection_split_ch_ratios: (array of floats) power fraction detected by each “beam-split” channel (i.e. independent detection channels obtained through a non-polarizing beam splitter). For 2 beam-split channels that receive the same power this array should be [0.5, 0.5]. The first element refers to detectors_specs/split_ch1, the second to detectors_specs/split_ch2 and so on. This field is not relevant when no polarization- and spectral-insensitive splitting is performed.

Identity group¶

The identity/ group contains information about the specific Photon-HDF5 file. If some information is not available the relative field may be omitted.

• author: (string) the author of the measurement (or simulation).
• author_affiliation: (string) the company or institution the author is affiliated with.
• creator: (string) the Photon-HDF5 file creator. Used when the data was previously stored in another format and the conversion is performed by a different person than the author.
• creator_affiliation: (string) the company or institution the creator is affiliated with.
• url: (string) URL that allow to download the Photon-HDF5 data file.
• doi: (string) Digital Object Identifier (DOI) for the Photon-HDF5 data file.
• filename: (string) Photon-HDF5 file name at creation time. This field saves the original file name even if the file is later on renamed on disk.
• filename_full: (string) Photon-HDF5 file name (including the full path) at creation time.
• creation_time: (string) the Photon-HDF5 file creation time with the following format: “YYYY-MM-DD HH:MM:SS”.
• software: (string) name of the software used to create the Photon-HDF5 file.
• software_version: (string) version of the software used to create the Photon-HDF5 file.
• format_name: (string) this must always be “Photon-HDF5”
• format_version: (string) for the current version it must be “0.3”
• format_url: (string) A URL pointing to the Photon-HDF5 specification document.

Provenance group¶

The provenance/ group contains info about the original file that has been converted into a Photon-HDF5 file. If some information is not available the relative field may be omitted.

• filename: (string)
• filename_full: (string)
• creation_time: (string)
• modification_time: (string)
• software: (string)
• software_version: (string)

Sample group¶

The /sample group contains information related to the measured sample. This group is optional.

• num_dyes: (integer) number of different dyes present in the samples.
• dye_names: (array of string) list of dye names (for example: ['ATTO550', 'ATTO647N'])
• buffer_name: (string) a user defined description for the buffer.
• sample_name: (string) a user defined description for the sample.

Additional notes and definitions¶

Definition of alternation periods¶

Note for μs-ALEX¶

The alex_period_spectral_ch1 and alex_period_spectral_ch2 fields allow defining photons detected during donor or acceptor excitation. As an example, let’s define the array

A = timestamps MODULO alex_period

as the array of timestamps modulo the μs-ALEX alternation period. Photons emitted during the donor period (respectively, acceptor period) are obtained by applying one of these two conditions:

• (A > start) and (A < stop) when start < stop (internal range)
• (A > start) or  (A < stop) when start > stop (external range).

Alternation histogram showing selection for the donor and acceptor periods. In this case the donor period is defined as an “external range” (2850, 580) while the acceptor period is defined as an “internal range” (900, 2580). This situation is due to the ALEX period being out of phase with respect to the time stamping clock.

photon_data0
photon_data1
...
photon_data10
...
photon_data100

Timestamps with rollover¶

In Photon-HDF5 timestamps are always signed 64 bit integers. Thanks to compression, there is no size penalty compared to 32 bit integers. Most timestamping hardware produce a timestamp with 24 or 32 bits and a rollover flag in order to compute the full “unwrapped” timestamp. Saving timestamps with a separate rollover information is not currently supported, therefore the rollover correction must be computed before saving data in a Photon-HDF5 file.

Timestamps with rollover may be supported in a future version of Photon-HDF5.

Contributions Acknowledgement¶

Any contributions to this document will be listed in the front page, just below the authors.

Contributions to the examples repository need to be under the MIT license and will retain the copyright notice of the original contributor.

Contributions to phconvert, currently, need to be under the GNU GPL v2 license. We are discussing to relicense phconvert to MIT license, so stay tuned.