Welcome to RH 1.5D’s documentation!¶

What is RH 1.5D¶

RH 1.5D is a modified version of the RH radiative transfer code that runs through a 3D/2D/1D atmosphere column-by-column, in the same fashion of rhf1d. It was developed as a way to efficiently run RH column-by-column (1.5D) over large atmospheres, simplifying the output and being able to run in supercomputers. It is MPI-parallel and scales well to at least 4096 processes.

While initially developed as another geometry on the RH tree, the requirements of the parallel version required changes to the RH core tree. Thus, it is more than a wrapper over RH and is distributed with a complete modified version of RH (see below for differences from RH distributed by Han Uitenbroek).

What this manual covers¶

Because there is much in common with RH, this manual should be seen as an incremental documentation of the 1.5D parallel side. This manual focuses on what is different from RH. Users should refer to the RH documentation by Han Uitenbroek for more detailed information on RH.

Comparison with RH¶

RH 1.5D inherits most of the code base from RH, but some features are new. The code is organised in the same was as the RH source tree, with the routines specific to the 1.5D version residing on a subdirectory rh15d of the rh source tree. In this way, it works similarly to the subdirectories in rh for different geometries. The compilation and linking proceeds as for the other geometries: first the general librh.a library should be compiled, and then the code in rh15d will be compiled and linked to it. The run directory is very similar to that of a given geometry in RH: most of the *.input files are used in the same way.

The lists below show a comparison between RH 1.5D and RH for a 1D plane-parallel geometry:

Getting the code¶

The code is available on a git repository, hosted on github: https://github.com/ita-solar/rh. If you don’t have git installed and just want to get started, the easiest way is to download a zip file with the latest revision: https://github.com/ita-solar/rh/archive/master.zip. If you have git installed and would like to be up-to-date with the repository, you can do a git clone:

git clone https://github.com/ita-solar/rh.git

or using SSH (only for contributors):

git clone git@github.com:ita-solar/rh.git

Whether you unpack the zip file or do one of the above it will create a directory called rh in your current path. This directory will have the following subdirectories:

Dependencies¶

RH 1.5D makes use of the HDF5 library to read the atmosphere files and write the output. It is not possible to run the code without this library. RH 1.5D requires HDF5 version 1.8.1 or newer (versions from branch 1.10.x do not currently work).

Because HDF5 is commonly used in high-performance computing, many supercomputers already have them available. In Hexagon, they can be loaded as:

module load cray-hdf5-parallel

in Pleiades:

module load hdf5/1.8.7/gcc/mpt

and at ITA’s Linux system:

module load hdf5/Intel/openmpi/1.8.18

Compilation¶

Compilation of RH 1.5D consists of two steps:

1. Compilation of the geometry-independent main libraries (librh.a and librh_f90.a)
2. Compilation of the rh15d_mpi tree and main binaries

RH 1.5D has been compiled in a variety of architectures and compilers, including gcc, the Intel compilers, and clang. As for MPI implementations, it has been tested with SGI’s mpt, OpenMPI, mpich, mvapich, and Intel’s MPI.

export CPU=`uname -m`
export OS=`uname -s`

OTHER_LIBRARY_DIR = /path/to/library
HDF5_DIR = /path/to/hdf5
LDFLAGS = -L../ -L$(OTHER_LIBRARY_DIR)/lib/  -L$(HDF5_DIR)/lib/
LIBS = -lrh -lrh_f90 $(ARCHLIBS) -lhdf5_hl -lhdf5 -lz -lm

Run directory¶

Once compiled, you can copy or link the binaries to a run directory. This directory will contain all the necessary input files, and it should contain two subdirectories called output and scratch.

The run directory can be located anywhere, but it must have a directory called Atoms two levels below (i.e. ../../Atoms/) with the Barklem_*data.dat files. This is because these relative paths are hardcoded in barklem.c. The input files in the run directory must obviously point to existing path names.

Configuration files¶

The configuration of an RH 1.5D is made primarily through several text files that reside in the run directory. The main file is keyword.input. All the other files and their locations are specified in keyword.input. The source tree contains a sample rh/rh15d/run/ directory with the following typically used configuration files:

The kurucz.input and the molecules.input files are identical under RH, so we refer to the RH manual for more information about them. Most of the other files behave very similarly in RH and RH 1.5D, with a few differences.

The atoms.input file is identical in RH, but it can also have a new starting solution, ESCAPE_PROBABILITY.

The keyword.input file functions in very much the same manner under RH and RH 1.5D. The main difference is that there are new options for the 1.5D version, and some options should not be used.

The new keyword.input options for the 1.5D version are:

The X_START, X_END, and X_STEP keywords (and the equivalent for the y direction) define which columns of the atmosphere file are going to be run. They can be used to calculate only a specific region. RH 1.5D chooses the columns to calculate using the (start, end, step) parameters as in the range() function in Python: the result is [start, start + step, start + 2 * step, ...]. The last element is the largest start + i * step less than end. This means that the numbers given by X_END and Y_END are not inclusive (e.g. if nx = 50 and X_END = 49, the column with the index 49 will not be calculated). One must set X_END = nx to calculate all the columns.

The following options have a different meaning under RH 1.5D:

And the following options are valid for RH but may not work with RH 1.5D:

The ray.input has the same structure in RH1D and RH 1.5D. In RH it is used as input for the solveray program, but in RH 1.5D it is used for the main program. It should contain the following:

1.00
Nsource

The first line is the mu angle for the output ray, and it should always be 1.00. The second line is Nsource, the number of wavelengths for which detailed output (typically source function, opacity, and emissivities) will be written. If Nsource > 0, it should be followed in the same line by the indices of the wavelengths (e.g. 0 2 10 20).

Atom and molecule files¶

The atom and molecule files have the same format as in RH. In the rh/Atoms and rh/Molecules directories there are a few sample files. They are read by the procedures in readatom.c and readmolecule.c. The atom files have the following basic structure:

Atmosphere files¶

The atmosphere files for RH 1.5D are a significant departure from RH. They are written in the flexible and self-describing HDF5 format. They can be written with any version, except the 1.10.x development branch.

The atmosphere files contain all the atmospheric variables necessary for RH 1.5D, and they may contain one or more simulation snapshots. The basic dimensions of the file are:

While strictly 3D atmosphere files, 2D and 1D snapshots can also be used provided that one or both of nx and ny are equal to 1.

The atmosphere file can contain the following variables:

Any other variable in the file will not be used. In addition, the atmosphere file must have a global attribute called has_B. This attribute should be 1 when the magnetic field variables are present, and 0 otherwise. Also recommended, but optional, is a global attribute called description with a brief description of the atmosphere file (e.g. how and from they were generated).

As HDF5 files, the contents of the atmosphere files can be examined with the h5dump utility. To see a summary of what’s inside a given file, one can do:

h5dump -H atmosfile

Here is the output of the above for a sample file:

HDF5 "example.hdf5" {
GROUP "/" {
   ATTRIBUTE "boundary_bottom" {
      DATATYPE  H5T_STD_I64LE
      DATASPACE  SCALAR
   }
   ATTRIBUTE "boundary_top" {
      DATATYPE  H5T_STD_I64LE
      DATASPACE  SCALAR
   }
   ATTRIBUTE "description" {
      DATATYPE  H5T_STRING {
         STRSIZE H5T_VARIABLE;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_UTF8;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
   }
   ATTRIBUTE "has_B" {
      DATATYPE  H5T_STD_I64LE
      DATASPACE  SCALAR
   }
   ATTRIBUTE "nhydr" {
      DATATYPE  H5T_STD_I64LE
      DATASPACE  SCALAR
   }
   ATTRIBUTE "nx" {
      DATATYPE  H5T_STD_I64LE
      DATASPACE  SCALAR
   }
   ATTRIBUTE "ny" {
      DATATYPE  H5T_STD_I64LE
      DATASPACE  SCALAR
   }
   ATTRIBUTE "nz" {
      DATATYPE  H5T_STD_I64LE
      DATASPACE  SCALAR
   }
   DATASET "electron_density" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SIMPLE { ( 1, 512, 512, 425 ) / ( H5S_UNLIMITED, 512, 512, 425 ) }
   }
   DATASET "hydrogen_populations" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 1, 6, 512, 512, 425 ) / ( H5S_UNLIMITED, 6, 512, 512, 425 ) }
   }
   DATASET "snapshot_number" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
   }
   DATASET "temperature" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 1, 512, 512, 425 ) / ( H5S_UNLIMITED, 512, 512, 425 ) }
   }
   DATASET "velocity_z" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 1, 512, 512, 425 ) / ( H5S_UNLIMITED, 512, 512, 425 ) }
   }
   DATASET "x" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 512 ) / ( 512 ) }
   }
   DATASET "y" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 512 ) / ( 512 ) }
   }
   DATASET "z" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 1, 425 ) / ( H5S_UNLIMITED, 425 ) }
   }
}
}

All the floating point variables can be either double or single precision.

Line lists and wavelength files¶

Other auxiliary files that can be used are line lists files and wavelength files.

The line list files are used to include additional lines not included in the different atoms. These lines will be treated in LTE. The line lists are specified in the kurucz.input file (one per line), and have the Kurucz line list format (link).

Just adding new transitions doesn’t mean that they will be included in the synthetic spectra. The extra lines will only be included in the existing wavelength grid, which depends on the active atoms used. The calculation of additional wavelengths can be forced by using a wavelength file. This file is specified in keyword.input using the keyword WAVETABLE. The format is a binary XDR file. Its contents are, in order: the number of new wavelengths (1 XDR int), vacuum wavelength values (XDR doubles).

Binaries and execution¶

Compilation should produce three executables: rh15d_ray_pool, rh15d_ray, and rh15d_lteray. The latter is a special case for running only in LTE. The other two are the main programs. They represent two different modes of job distribution: normal and pool.

In the pool mode there is a process that works as overlord: its function is to distribute the work to other processes. The other processes (drones) ask the overlord for a work unit. When they finish their unit, they go back and ask for more, until all tasks are completed. Because of race conditions and because different columns will run at different speeds, it is not possible to know which columns a given process will run beforehand. Due to the overlord, rh15d_ray_pool needs to run with two or more processes. The advantage of the pool mode is that the dynamic load allocation ensures the most efficient use of the resources. With the normal mode it may happen that some processors will work on columns that take longer to converge (especially as they are adjacent), and in the end the execution will have to wait for the process that takes longer. In some cases (especially with PRD) the pool mode can be 2-3 times faster than the normal mode. When one runs with a large number of processes (> 2000) and each column takes little time to calculate, the pool mode can suffer from communication bottlenecks and may be slower because a single overlord cannot distribute the tasks fast enough. The only disadvantage of the pool mode (so far) is that not all output is currently supported with this mode.

The `normal` mode is deprecated and will be removed in a later revision. Use only if you know what you’re doing! In the normal mode the jobs (all the atmosphere columns for which one wants to calculate) are divided by the number of processes at the start of the execution. There is no communication between processes, and each process knows from the start all the columns it is going to run. These columns are adjacent. If the number of columns is not a multiple of the number of processes, there will be some processes with larger workloads. There is no minimum number of processes to run, and rh15d_ray can also be run in a single process. Regions of an atmosphere can take a lot longer to run than others, and the processes that work on those will take longer to finish. In the normal mode this means that the slowest process will set the overall running time, and therefore in practice it can take more than 10x longer than the pool mode (and is therefore not recommended).

As an MPI program, the binaries should be launched with the appropriate command. Some examples:

mpirun -np N ./rh15d_ray_pool
mpiexec ./rh15d_ray_pool    # use in Pleiades
aprun -B ./rh15d_ray        # use in Hexagon or other Cray

The run directory¶

The run directory contains the configuration files, the binaries, and the scratch and output directories. As the names imply, temporary files will be placed under scratch and the final output files in output. No files under scratch will be used after the run is finished (they are not read for re-runs).

The scratch directory contains different types of files. Most of them are binary files write by RH 1.5D to save memory. Example files are the background_p*.dat with the background opacities, files with PRD weights, and the rh_p*.log log files. Each process creates one of those files, and they will have the suffix _pN.*, where N is the process number. The log files have the same format as in RH. The writing of each process’s log file is buffered by line. Because these are updated often, when running with many processes this can be a drag on some systems. Therefore, it is possible to run full buffering (meaning log files are only written when the main program finishes). This option is not exposed in the configuration files, so one needs to change the file parallel.c in the following part:

/* _IOFBF for full buffering, _IOLBF for line buffering */
setvbuf(mpi.logfile, NULL, _IOLBF, BUFSIZ_MPILOG);

One should replace _IOLBF by _IOFBF to change from line buffering to full buffering.

The output will contain the three output files: output_aux.hdf5, output_indata.hdf5, and output_ray.hdf5. See Output file structure for more details on the structure of these files. If doing a re-run, these files must already exist; they will be updated with the new results. Otherwise, if these files are already in output before the execution, they will be overwritten. The way that HDF5 work means that these files are created and filled with special (masked) values at the start of the execution. This means that the disk space for the full output must be available at the start of the run, and no CPU time will be wasted if at the end of the run there is not enough disk space. The files are usually written every time a process finishes work on a given column. The masked values are overwritten with the data. One advantage of this method is that even if the system crashes or the program stops, it is possible to recover the results already written (and a re-run can be performed for just the missing columns).

All the processes write asynchronously to all the output files. In some cases this can cause contention in the filesystem, with many processes trying to access the same data at the same time. In the worst case scenario, the contention can create bottlenecks which practically stop the execution. Therefore, it is highly recommended that the users tune their filesystem for the typical loads of RH. Many supercomputers make use of Lustre, a parallel filesystem. With Lustre, resources such as files can be divided in different stripes that can be placed in several different machines (OSTs). For running RH with more than 500 processes, one should use as many OSTs as available in the system, and select the lustre stripe size to the typical amount of data written to a file per simulation column. The stripe can set with the lfs setstripe command:

lfs setstripe -s stripe_size -c stripe_count -o stripe_offset directory|filename

It can be run per file (e.g. output_ray.hdf5), or for the whole output directory. Using a stripe count of -1 will ensure that the maximum number of OSTs is used. For the typical files RH 1.5D produces, it is usually ok to apply the same Lustre settings to the whole output directory, and the following settings seem to reasonable:

lfs setstripe -s 4m -c -1 output/

Similarly, the scratch directory can also benefit from Lustre striping. Because most files there are small, it is recommended to use a stripe count of 1 for scratch.

Logs and messages¶

In addition to the logs per process saved to scratch, a much smaller log will be printed in stdout. This log is a smaller summary of what each process is doing. Here is an example of typical messages:

Process   1: --- START task   1, (xi,yi) = (  0,156)
Process 232: --- START task   1, (xi,yi) = (  0,159)
Process  36: --- START task   1, (xi,yi) = (  0,162)
Process  12: --- START task   1, (xi,yi) = (  0,171)
(...)
Process  12: *** END   task   1 iter, iterations = 121, CONVERGED
Process   3: *** END   task   1 iter, iterations = 200, NO Convergence
Process   4: *** SKIP  task   1 (crashed after 81 iterations)
Process   3: --- START task   2, (xi,yi) = ( 23, 64)
Process  12: --- START task   2, (xi,yi) = ( 23, 65)
Process   4: --- START task   2, (xi,yi) = ( 23, 65)
(...)
*** Job ending. Total 262144 1-D columns: 262142 converged, 1 did not converge, 1 crashed.
*** RH finished gracefully.

In this example one can see the three possible outputs for a single-column calculation: convergence, non-convergence (meaning the target ITER_LIMIT was not met in N_MAX_ITER iterations), or a crash (many reasons). If there are singular matrices or other causes for a column to crash, RH 1.5D will skip that column and proceed to the next work unit. Such cases can be re-run with different parameters. In some cases (e.g. inexistent files) it is not possible to prevent a crash, and RH 1.5D will finish non-gracefully.

Helper script¶

There is a Python script called runtools.py designed to make it easier to run RH 1.5D for large projects. It resides in rh/python/runtools.py. It requires Python with the numpy and h5py (or netCDF4) modules. It was made to run a given RH 1.5D setup over many simulation snapshots, spanning several atmosphere files. It supports a progressive re-run of a given problem, and allows the of use different keyword.input parameters for different columns, tackling columns harder to converge.

The first part of runtools.py should be modified for a users’s need. It typically contains:

atmos_dir = '/mydata_dir'
seq_file = 'RH_SEQUENCE'
outsuff = 'output/output_ray_mysim_CaII_PRD_s%03i.hdf5'
mpicmd = 'mpiexec'
bin = './rh15d_ray_pool'
defkey = 'keyword.save'
log = 'rh_running.log'
tm = 40
rerun = True
rerun_opt = [ {'NG_DELAY': 60, 'NG_PERIOD': 40, '15D_DEPTH_REFINE': 'FALSE',
               '15D_ZCUT': 'TRUE', 'N_MAX_ITER': 250, 'PRD_SWITCH': 0.002 },
              {'NG_DELAY': 120, 'NG_PERIOD': 100, '15D_DEPTH_REFINE': 'TRUE',
               'PRD_SWITCH': 0.001 } ]

The different options are:

Output file structure¶

The output is written to three files: output_aux.hdf5, output_indata.hdf5, and output_ray.hdf5. This is a big departure from RH, which contained several more output files. In particular, RH 1.5D will not write all the information that was written by RH, due to the sheer size it would take for large 3D simulations. The files are written in the machine-independent, self-describing HDF5 format. The contents of the files are organised in groups, variables, and attributes. Groups and variables can be imagined as directories and files in a filesystem. Inside groups, different variables and dimensions can be organised. The content of the output files can vary: some runs will have more detailed information and therefore more variables written to the files.

The structure of the three files is given below.

Reading the output files¶

HDF5 is an open, platform-independent format, and therefore interfaces to many programming languages are available. The main interface libraries are available in C, C++, Fortran, and Java. But there are also interfaces for Python (h5py), Julia, IDL (from version 6.2), MATLAB , Octave, Perl, and R.

The RH 1.5D output files can be read with standard HDF5 or NetCDF 4 readers: in most cases one needs to specify only the variable or group name. The HDF5 library provides useful command line tools, which can be used to gather information about the RH 1.5D files or extract data. Additionally, there is a more complete set of tools written in Python to read and analyse these files. Interfaces in other languages are also planned.

>>> from rh15d import *
>>> rr = rh15d.Rh15dout()
--- Read ./output_aux.hdf5 file.
--- Read ./output_indata.hdf5 file.
--- Read ./output_ray.hdf5 file.

>>> [a for a in dir(rr.ray) if a[0] != '_']
['intensity',
 'params',
 'stokes_Q',
 'stokes_U',
 'stokes_V',
 'chi',
 'source_function',
 'scattering',
 'Jlambda',
 'tau_one_height',
 'wavelength',
 'wavelength_indices']
>>> rr.ray.intensity
<HDF5 dataset "intensity": shape (256, 256, 1626), type "<f4">

>>> rr.mpi.convergence
<HDF5 dataset "convergence": shape (256, 256), type "<i8">
>>> rr.atmos.temperature
<HDF5 dataset "temperature": shape (256, 256, 400), type "<f4">

>>> [a for a in dir(rr.atom_CA) if a[0] != '_']
['Rij_continuum',
 'Rij_line',
 'Rji_continuum',
 'Rji_line',
 'params',
 'populations',
 'populations_LTE']

>>> from rh15d import *
>>> atm = rh15d.HDF5Atmos('my_atmos.hdf5')

>>> [a for a in dir(atm) if a[0] != '_']
['B_x',
 'B_y',
 'B_z',
 'close',
 'closed',
 'electron_density',
 'file',
 'hydrogen_populations',
 'params',
 'read',
 'snapshot_number',
 'temperature',
 'velocity_z',
 'write_multi',
 'write_multi_3d',
 'x',
 'y',
 'z']
>>> atm.params     # Contains basic properties
{'boundary_bottom': 1,
 'boundary_top': 0,
 'description': 'Created with make_hdf5_atmos.on 2017-04-03 17:32:06.831811',
 'has_B': 1,
 'nhydr': 6,
 'nt' : 4,
 'nx': 512,
 'ny': 512,
 'nz': 400}
>>> atm.velocity_z
<HDF5 dataset "velocity_z": shape (4, 512, 512, 400), type "<f4">

def make_hdf5_atmos(outfile, T, vz, nH, z, x=None, y=None, Bz=None, By=None,
              Bx=None, rho=None, ne=None, vx=None, vy=None, desc=None,
              snap=None, boundary=[1, 0], comp=None, complev=None,
              append=False):
  """
  Creates HDF5 input file for RH 1.5D.

  Parameters
  ----------
  outfile : string
      Name of destination. If file exists it will be wiped.
  T : n-D array
      Temperature in K. Its shape will determine the output
      dimensions (can be 1D, 2D, or 3D).
  vz : n-D array
      Line of sight velocity in m/s. Same shape as T.
  nH : n-D array
      Hydrogen populations in m^-3. Shape is [nhydr, shape.T] where
      nydr can be 1 (total number of protons) or more (level populations).
  z : n-D array
      Height in m. Can have same shape as T (different height scale
      for each column) or be only 1D (same height for all columns).
  ne : n-D array, optional
      Electron density in m^-3. Same shape as T.
  rho : n-D array, optional
      Density in kg / m^-3. Same shape as T.
  vx : n-D array, optional
      x velocity in m/s. Same shape as T. Not in use by RH 1.5D.
  vy : n-D array, optional
      y velocity in m/s. Same shape as T. Not in use by RH 1.5D.
  Bx : n-D array, optional
      Magnetic field in x dimension, in Tesla. Same shape as T.
  By : n-D array, optional
      Magnetic field in y dimension, in Tesla. Same shape as T.
  Bz : n-D array, optional
      Magnetic field in z dimension, in Tesla. Same shape as T.
  x : 1-D array, optional
      Grid distances in m. Same shape as first index of T.
  y : 1-D array, optional
      Grid distances in m. Same shape as second index of T.
  x : 1-D array, optional
      Grid distances in m. Same shape as first index of T.
  snap : array-like, optional
      Snapshot number(s).
  desc : string, optional
      Description of file
  boundary : Tuple, optional
      Tuple with [bottom, top] boundary conditions. Options are:
      0: Zero, 1: Thermalised, 2: Reflective.
  append : boolean, optional
      If True, will append to existing file (if any).
  comp : string, optional
      Options are: None (default), 'gzip', 'szip', 'lzf'.
  complev : integer or tuple, optional
      Compression level. Integer for 'gzip', 2-tuple for szip.
  """

def make_wave_file(outfile, start=None, end=None, step=None, new_wave=None,
                   ewave=None, air=True):
   """
   Writes RH wave file (in xdr format). All wavelengths should be in nm.

   Parameters
   ----------
   start: number
       Starting wavelength.
   end: number
       Ending wavelength (non-inclusive)
   step: number
       Wavelength separation
   outfile: string
       Name of file to write.
   ewave: 1-D array, optional
       Array of existing wavelengths. Program will make discard points
       to make sure no step is enforced using these points too.
   air: boolean, optional
       If true, will at the end convert the wavelengths into vacuum
       wavelengths.
   """

IDL> data = read_ncdf_var("output_ray.hdf5", "intensity")
IDL> help, data
DATA            FLOAT     = Array[902, 512, 512]
IDL> pops = read_ncdf_var("output_aux.hdf5", "populations", groupname="atom_CA")
IDL> help, pops
POPS            FLOAT     = Array[400, 512, 512, 5]

Analysis tools¶

Current issues¶

• Check the github RH issues page for an updated list.
• If the scratch or output directories are not present, the code will crash. The error message is not very clear.
• In keyword.input, if one sets a SNAPSHOT value to be more than what is in the atmosphere file, the code will stop with an error message: Index exceeds dimension bound. This error should be made more clear.
• The atom files must not end with a blank line, otherwise gencol will fail and the program stops.
• Line buffered or full buffered log options still require the user to change the source code.
• Depth refinement fails in some cases due to problems caused by cubic interpolation artefacts.
• Using more than 4000 cores and writing full output may cause I/O slowdowns and Lustre contention in some systems.

Planned features¶

• Support for multiple snapshots in the output files.
• pool mode be more flexible, with the possibility of several overlord nodes, useful for running with more than 4000 processes.
• More flexible control of what output is written.

A. Compiling netCDF¶

By far the easiest way to use RH is when the proper HDF5 library is already installed in the system. This is the case at most supercomputers, but not usually in user workstations or laptops. Even if your operative system can provide HDF5 through its packaging system (e.g. macports in macOS, rpm in linux distributions), this is often unsuitable for running RH. The reason is that there needs to be consistency between the compiler, MPI libraries used, and the HDF5 library must be compiled with parallel support.

Compiling all of this from scratch is burdensome and prone to fail. Care must be taken to ensure that each library is built properly, and that MPI functions well. Errors with MPI libraries are particularly difficult to diagnose.

The recommended approach is to use an MPI library already in the system. If this is not possible or it is not working properly, then MPI must also be compiled (see below for instructions for Open MPI). The following instructions go through the different steps of compiling the libraries. They assume a bash shell is used, so please adapt if you need csh or tcsh.

export CC=mpicc     # or gcc, clang, icc,
export FC=gfortran  # or mpif90, ifort

export MYPREFIX=/home/user/local