Installation¶

This chapter describes the manual installation of EDGE and its dependencies from scratch.

Examples¶

EDGE’s entire installation process is continuously tested. If you get stuck with this user guide, you might be able to find additional installation hints in the respective configurations:

Getting the Code¶

EDGE’s sources are hosted at GitHub. The repository has two branches master and develop. master is the most recent stable version of EDGE. Periodically EDGE also provides tags, which are simply snapshots of the master branch. Typically tags passed additional, manual testing. We recommend to use the most recent tag to get started with EDGE. develop is the bleeding edge version of EDGE. Changes in develop are intended to be merged into master. However, develop is for ongoing development and broken from time to time.

The procedure for obtaining the code is as follows:

Clone the git-repository and navigate to the root-directory through:
```
git clone https://github.com/3343/edge.git
cd edge
```
This gives you the master-branch. If this is what you want, jump over the next step.
Checkout the desired tag:
```
git checkout YY.MM.VV
```
Remark: YY.MM.VV is a place holder. You have to replace this with your actual tag. Available tags are shown at the GitHub-homepage or directly through git tag.

Initialize and update the submodules:

git submodule init
git submodule update

External Dependencies¶

EDGE relies on a collection of open-source software:

LIBXSMM is the single core backend of EDGE’s high-performance kernels. LIBXSMM is optional, but highly recommended due to severe performance-limitations of the vanilla kernels.
zlib is a requirement for the HDF5 library.
HDF5 is a requirement for point source descriptions and for EDGE-V, EDGE’s mesh interface.
Gmsh is used to read and write meshes.
METIS is used for partitioning in EDGE-V.

Scripts which install the external dependencies are available in the tools/build/deps-directory.

EDGE-V¶

At runtime EDGE-V interfaces the mesh and respective annotations as a library. Further details on preprocessing use-cases, e.g., the derivation of local time stepping schemes or mesh partitioning, are given in Sec. EDGE-V. Assuming that all external dependencies are installed in ./deps do the following to install EDGE-V:

Navigate to EDGE-V’s source directory:

cd tools/edge_v

Run the build script:

scons parallel=omp zlib=../../deps hdf5=../../deps gmsh=../../deps metis=../../deps install_dir=../../deps

EDGE¶

EDGE (and EDGE-V) use SCons as build tool. scons --help returns all of EDGE’s build-options. All build options are given in the respective sub-section of Sec. Configuration. You can enable the libraries in EDGE either by passing their installation directory explicitly (recommended) or by setting the environment variables CPLUS_INCLUDE_PATH and LIBRARY_PATH. For example, let’s assume that you installed LIBXSMM in the directory $(pwd)/deps. Than we could either enable LIBXSMM by passing xsmm=$(pwd)/deps to EDGE’s SCons-script or by using CPLUS_INCLUDE_PATH=$(pwd)/deps/include LIBRARY_PATH=$(pwd)/deps/lib scons [...] xsmm=yes.

If something goes wrong with finding a library, EDGE will tell you so. For example, if we did not install LIBXSMM in /tmp, but tell EDGE so anyways, we get:

scons equations=elastic order=4 cfr=1 element_type=tet4 xsmm=/tmp
[...]
Checking for C++ static library libxsmmnoblas..no
  Warning: Could not find libxsmm, continuing without.

Further information on what went wrong is logged in the file config.log, which, in this case, shows that the compiler could not find the LIBXSMM-header:

[...]
scons: Configure: Checking for C++ static library libxsmmnoblas..
.sconf_temp/conftest_2.cpp <-
  |#include <libxsmm.h>
  |int main(int i_argc, char **i_argv) { return 0; }
g++ -o .sconf_temp/conftest_2.o -c -std=c++11 -Wall -Wextra -Wno-unknown-pragmas -Wno-unused-parameter -Werror -pedantic -Wshadow -Wundef -O2 -ftree-vectorize -DPP_N_CRUNS=1 -DPP_T_EQUATIONS_ELASTIC -DPP_T_ELEMENTS_TET4 -DPP_ORDER=4 -DPP_PRECISION=64 -I. -Isrc -I/tmp/include .sconf_temp/conftest_2.cpp
.sconf_temp/conftest_2.cpp:1:21: fatal error: libxsmm.h: No such file or directory
compilation terminated.
scons: Configure: no

Assets¶

EDGE provides assets, for example, binary input, in a separate repository Due to limitations of Git, when it comes to large files, we use Git Large File Storage (Git LFS) in this repository. To use EDGE’s assets repository, install the Git LFS command-line client by following the linked instructions. Futher background information and documentation on Git LFS is available from https://www.atlassian.com/git/tutorials/git-lfs.

Obtaining Data¶

Be aware that the entire repository contains LFS pointers to data exceeding 100GB. If you are not planning on storing all of this, follow the instructions below.
Best performance when cloning EDGE’s assets repository, is obtained by using git lfs clone. This will download all, possibly large, files stored in the Git LFS store.
If you are only interested in certain files or directories, you can, by using git lfs clone --exclude=*, initialize the assets repository with non-LFS files and Git LFS pointers only. Now, to obtain only a certain file or directories, use git lfs fetch with the arguments -I and -X. For example git lfs fetch -I test/* would download all files and directories in the directory test. After downloading the files from the remote Git LFS store, you can replace the Git LFS pointers in your local Git repository with the actual files through git lfs checkout test/*.

Configuration¶

EDGE uses XML-files for build- and runtime-configurations. Using XML-input for building EDGE through --xml=[...] is optional. All build arguments might also be passed to SCons directly. In contrast, passing XML-input for the runtime configuration through -x [...] is mandatory. Except for the verbose flag -v, no other command-line arguments are accepted.

XML-tree¶

The following shows an overview of all XML-nodes in EDGE. Dependent on the build-type, only a subset of the nodes is active. For example, a given velocity model (<velocity_model>) is only parsed for seismic problems. The comment  indicates, that a node is allowed to appear multiple times in the XML-tree. .. code-block:

<edge>
  <build>
    <cfr/>
    <equations/>
    <element_type/>
    <order/>
    <mode/>
    <arch/>
    <precision/>
    <parallel/>
    <cov/>
    <tests/>
    <build_dir/>
    <xsmm/>
    <zlib/>
    <hdf5/>
    <gmsh/>
    <inst/>
  </build>

  <cfr>
    <mesh>
      <in>
        <base/>
        <extension/>
      </in>
      <boundary>
        <free_surface/>
        <outflow/>
        <periodic/>
      </boundary>
    </mesh>

    <velocity_model>
      <domain>
        <half_space>
          <origin>
            <x/>
            <y/>
            <z/>
          </origin>
          <normal>
            <x/>
            <y/>
            <z/>
          </normal>
        </half_space>
        <!-- [...] -->

        <rho/>
        <lambda/>
        <mu/>
        <!-- quality factors for viscoelastic builds only -->
        <qp/>
        <qs/>
      </domain>
      <!-- [...] -->
    </velocity_model>

    <setups>
      <!-- attenuation config for viscoelastic builds only -->
      <attenuation>
        <central_frequency/>
        <frequency_ratio/>
      </attenuation>

      <point_sources>
        <file/>
        <!-- [...] -->
      </point_sources>

      <initial_values/>
      <!-- [...] -->

      <end_time/>
    </setups>

    <output>
      <receivers>
        <path_to_dir/>
        <freq/>

        <receiver>
          <name/>
          <coords>
            <x/>
            <y/>
            <z/>
          </coords>
        </receiver>
        <!-- [...] -->
      </receivers>

      <wave_field>
        <type/>
        <file/>
        <int/>
        <sparse_type/>
      </wave_field>

      <error_norms>
        <type/>
        <file/>
        <reference_values/>
        <!-- [...] -->
      </error_norms>
    </output>
  </cfr>
</edge>

<edge>¶

The node <edge> is the root of both, the runtime- and the build-configuration.

<build>¶

The node <build> describes the build-configuration and is only used by SCons. EDGE also parses <build> at runtime, however the information is only logged and does not influence runtime behavior.

Attribute	Allowed Values	Description
cfr	1, 2, 4, 8, 12, 16	Number of concurrent/fused forward runs. 1, 4, 8, and 16 are typically used.
equations	advection, elastic, viscoelastic3, viscoelastic4, viscoelastic5, swe	Equations solved. advection: advection equation, elastic: elastic wave equations, viscoelastic3-5: elastic wave equations with frequency-independent quality factors and 3, 4 or 5 relaxation mechanisms, swe: shallow water equations.
element_type	line, quad4r, tria3, hex8r, tet4	Element type used for spatial discretization. line: line elements, quad4r: 4-node, rectangular quads, tria3: 3-node triangles, hex8r: 8-node, rectangular hexes, tet4: 4-node tets.
order	1, 2, 3, 4, 5, ..	Convergence rate of the solver. 1: Finite volume solver (P0 elements), 2-9: ADER-DG solver (P1-P8 elements).
mode	release, debug, release+san, debug+san	Compile mode. release: fastest option, debug: debug flags and disabled optimizations, debug/release+san (gnu and clang): same as debug/release, but with sanitizers.
arch	host, snb, hsw, knl, skx, knm, avx512, aarch64	Targeted architecture. host: uses the architecture of the machine compiling the code, snb: Sandy Bridge, hsw: Haswell, knl: Knights Landing, skx: Skylake, knm: Knights Mill, aarch64: Arm Neoverse N1
precision	32, 64	Floating point precision in bit. 32: single precision arithmetic (recommended), 64: double precision arithmetic.
parallel	none, omp, mpi, mpi+omp	Shared and distributed memory parallelization. none: disabled, omp: OpenMP only, mpi: MPI only, mpi+omp: hybrid parallelization with MPI and OpenMP.
cov	yes, no	Support for code coverage reports.
tests	yes, no	Unit tests. yes: builds unit tests in the separate binary tests.
build_dir	/path/to/build_dir	Path to the build-directory. Temporary files and the final executable(s) are stored in the build-directory.
xsmm	yes, no, path/to/xsmm	LIBXSMM support. Available only for ADER-DG and seismic settings.
zlib	yes, path/to/zlib	zlib support.
hdf5	yes, path/to/hdf5	hdf5 support.
gmsh	yes, path/to/gmsh	Gmsh support.
inst	yes, no	EDGE’s high-level code instrumentation through the Score-P library.

<cfr>¶

The node <cfr> describes the runtime configuration of the forward simulations. <cfr> does not hold any attributes. In the case of fused simulations, children of <cfr> are either shared among all forward simulations or describe varying configurations from run to run. An example of a shared configuration is the child <mesh>.

<mesh>¶

<mesh> describes the used mesh of all, possibly fused, simulations.

Node	Attributes	Description
in	`<base/>`, `<extension/>`	`<base/>`: base-path to the input-mesh. This excludes the partition-id. `<extension/>`: file-extension of the mesh files.
boundary	`<free_surface/>`, `<outflow/>`, `<periodic/>`	`<free_surface/>`: id of free-surface boundary conditions (typically 101), `<free_surface/>`: id of outflow boundary conditions (typically 105), `<periodic/>`: id of periodic boundary conditions (typically 106)

Example: Assume our mesh has two partitions stored at my_path/mesh_1.msh and my_path/mesh_2.msh and corresponding mesh-annotations at my_path/mesh_1.h5 and my_path/mesh_2.h5. Further, we’d like to assign faces with id 101 as free-surface boundaries and those with id 105 as outflow boundaries, then we would use:

<mesh>
  <in>
    <base>my_path/mesh</base>
    <extension>.msh</extension>
  </in>
  <boundary>
      <free_surface>101</free_surface>
      <outflow>106</outflow>
  </boundary>
</mesh>

<velocity_model>¶

The node <velocity_model> describes the used velocity model, currently limited to the three elastic material parameters, given by the mass density $\rho$ and the two Lame parameters $\mu$ and $\lambda$. Alternatively, the velocity model can be defined as part of the mesh’s annotations (see Sec. EDGE-V) We utilize EDGE’s generic domain approach for the velocity model. Here, we define one or more domains for a velocity model, each of which allows for a constant set of material parameters.

Node	Nodes/Attributes	Description
domain	`<half_space/>`, `<rho/>`, `<lambda/>`, `<mu/>`	`<half_space>`: One or more half-spaces building the domain (see separate description)., `<rho/>`: mass density $\rho$, `<lambda/>`: Lame parameter $\lambda$, `<mu/>`: Lame parameter $\mu$, `<qp/>`: qualityfactor $Q_P$ (only for viscoelastic settings), `<qs/>`: qualityfactor $Q_S$ (only for viscoelastic settings),

<domain>¶

The node <domain> describes EDGE’s generic domain approach and might be used to describe different geometric settings. A domain is defined by a set of geometric entities, currently limited to half-spaces. When searching for the corresponding domain of an element in the mesh, EDGE iterates from top to bottom through the defined domains. The first domain, which contains the element, is the one from which the respective parameters are read. E.g., if domains are used to describe the velocity model in a fully elastic setting, we would store the mass density $\rho$ and the two Lame parameters $\mu$ and $\lambda$ for every domain.

For a single domain itself, the domain contains the element if and only if all geometric entities of the domain contain the element.

Node	Description
half_space	One or more descriptions of a half-space.

<half_space>¶

The node <half_space> describes a half-space as geometric entity of a domain. Each half-space consist of an origin and a normal. The origin shifts the hyperplane in space, while the normal gives the orientation of the hyperplane. Points on the side of the hyperplane, to which the normal points, are considered to be in the half-space. Points on the hyperplane itself and within a small error margin are typically considered to be inside the half-space. This, however, might depend on where the <half_space>-node is used. All other points are outside.

Node	Attributes	Description
origin	`<x/>`, `<y/>`, `<z/>`	x-, y- and z- coordinates of the origin. For two-dimensional setups `<z/>` is ignored. For one dimensional setups, both, `<z/>` and `<y/>`, are ignored.
normal	`<x/>`, `<y/>`, `<z/>`	x-, y- and z- coordinates of the normal. For two-dimensional setups `<z/>` is ignored. For one dimensional setups, both, `<z/>` and `<y/>`, are ignored.

<setups>¶

The node <setups> describes the setups of the fused simulations. A setup is given by initial values or source terms, and the shared end time of all fused simulations.

Node	Attributes	Description
attenuation	`<central_frequency/>` `<frequency_ratio/>`	Central frequency and frequency ratio of the attenuation frequency band.
point_sources	`<file/>`	One or more HDF5-files, each containing a point source description for a single fused simulation.
end_time		End time of the fused simulations.
initial_values		Initial values of the degrees of freedom (just-in-time generated code). Examples are available from EDGE’s assets repository. This is an optional parameter, default behavior sets all DOFs to zero.

<output>¶

EDGE supports three types of simulation output, summarized by the node <output/>. The different types can be activated separately:

Wave field output writes all quantities for the constants modes of all fused simulations and elements. A fixed sampling interval determines the frequency of the wave field output. As a side-effect, wave field output enforces synchronization of the entire simulation. Thus, if enabled, the last time step before each output point is adjusted to match the desired time exactly.
Receivers write point-wise output of all quantities for all fused simulations. The polynomial basis is evaluated accordingly. To match output points between two time steps in time, an ADER time prediction is evaluated.
Convergence settings might write errors in the L1-, L2-, and $\text{L}^\infty$ norm. The errors are computed at the end of simulation by comparing the obtained result to the analytical reference solution through quadrature rules. Here, we oversample the error computation by using a quadrature rule one order above the DG-solution. As usual, errors for all quantities and all fused simulations are written.

<wave_field>¶

Attribute	Allowed Values	Description
type	vtk_ascii, vtk_binary	ASCII or binary (recommended) VTK output.
file		Path to the output files. Each rank writes in its own directory. For example: `loh3_uns_200/wf` will write the output of the third synchronization point to `loh3_uns_200/0/wf_0_3.vtk` for the first MPI-rank and to `loh3_uns_200/1/wf_1_3.vtk` for the second MPI-rank.
int		Interval of the wave field. For example 0.75, will write the output at 0, 0.75, 1.5, […].
sparse_type		Sparse type of the elements, which are written. For example, for seismic workloads, 101 would only write elements at the free-surface. All elements are written, if not set.

<receivers>¶

Attribute	Allowed Values	Description
path_to_dir		Path to the output directory of the receivers, will be created if it doesn’t exist.
freq		Sampling of the written data, e.g., using ``<freq>0.1</freq> would write output at 0.0, 0.1, 0.2, …
receiver		One more more receivers for which data is written.

<receiver>¶

Attribute	Allowed Values	Description
name		Name of the receiver.
coords	`<x/>`, `<y/>`, `<z/>`	x-, y- and z-coordinates of the receiver.
receiver		One more more receivers for which data is written.

<error_norms>¶

Attribute	Allowed Values	Description
type	`sout`, `file`, `sout_file`	Writes the error-norms either to standard-out, file or to both.
file		Path of the file to which the error-norms are written
reference_values		Reference values (just-in-time generated code) after completion of the simulation. Examples are available from EDGE’s assets repository.

EDGE-V¶

EDGE-V is a tool prepare meshes for use in the core solver EDGE. It includes the following key features:

Parsing of velocity models and performing respective mesh annotations.
Derivation of background fields with target-edge lengths for problem-aware meshing.
Derivation of optimal Local Time Stepping (LTS) groups.
LTS-aware partitioning.
Mesh reordering based on the partitioning and time step groups.
Splitting of the mesh into separate per-partition chunks and derivation of respective annotations, e.g., the communication structure, for use in EDGE.

Configuration¶

Similar to the core-solver, EDGE-V is controlled by XML-configurations. This part of the documentation has to be written still. The La Habra mesh setups might serve as an example for seismic settings.

EDGEcut¶

EDGEcut is a tool for generating surface meshes for a computational domain with topography. It takes as input a regular sampling from a topographical profile and generates a simple triangular surface mesh using a six-sided box. The topography is sampled at the same resolution as the input whereas the other five sides can be coarsened. The topography mesh and boundary mesh are then output in OFF format as separate files. The tool meshio can be used for mesh-format conversions. EDGEcut’s output is the input for volume mesher, e.g., Gmsh.

Meshed box for the La Habra region.

Zoom showing the triangulation and side-coarsening for the box above.

Configuration¶

Documentation for EDGEcut’s runtime configuration and DEM-input hasn’t been written yet. You may checkout the surface meshes of the La Habra setups for an example.

Sources¶

EDGE implements seismic sources through simple binary HDF5-input. Each of the fused simulations has an independent HDF5-input file, which contains a collection of point sources. This collection is completely decoupled from EDGE’s fused simulation approach. Thus, each of the fused simulations can have an arbitrary source description.

If a point source is outside of the modeling domain, it will be mapped to the closest point of the mesh boundary. For example, assume that we specify a point force at the peak of a mountain for a flat tetrahedral mesh. In our example, the peak is outside the mesh, meaning that the source would be mapped to the closest-by point of the respective surface triangle.

HDF5 Input¶

We use five arrays to discretize the point sources, where #pt is the number of point sources and #ts the number of samples of all time series. All time series are integrated numerically (linear interpolation) before they are applied to Degrees of Freedom in EDGE.

Dataset	Type	2D Size	3D Size	Description
points	fp32	(#pt, 2)	(#pt, 3)	Cartesian coordinates of the point sources.
scalings	fp32	(#pt, 5)	(#pt, 9)	Scalings of each time series for each elastic quantity.
time_parameters	fp32	(#pt, 2)	(#pt, 2)	1st: Offset of each time series in time. 2nd: Time step of each time series.
time_pointers	uint64	#pt+1	#pt+1	0 to #ts-1: Pointers to the first entries of the 1D time series array. #ts: total number of time samples over all time series (#ts).
time_series	fp32	#ts	#ts	Samples of all time series.

The file point_forces_2d_0.h5 contains an exemplary source description. We obtain an ASCII-representation by using the tool h5dump:

> h5dump point_sources_2d_0.h5

  HDF5 "point_sources_2d_0.h5" {
  GROUP "/" {
  DATASET "points" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 2, 2 ) / ( 2, 2 ) }
      DATA {
      (0,0): 0.2, 2.1,
      (1,0): 1.3, 1
      }
  }
  DATASET "scalings" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 2, 5 ) / ( 2, 5 ) }
      DATA {
      (0,0): 0, 0, 0, 0, 2.3,
      (1,0): 0, 0, 0, 1, 3.3
      }
  }
  DATASET "time_parameters" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 2, 2 ) / ( 2, 2 ) }
      DATA {
      (0,0): 1, 0.001,
      (1,0): 2.3, 0.002
      }
  }
  DATASET "time_pointers" {
      DATATYPE  H5T_STD_U64LE
      DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
      DATA {
      (0): 0, 5, 8
      }
  }
  DATASET "time_series" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 8 ) / ( 8 ) }
      DATA {
      (0): 0, 0.1, 0.2, 0.1, 0.15, 0, 0.3, 0.2
      }
  }
  }
  }

In this two-dimensional example, we have two point sources, located at $(0.2\text{m},2.1\text{m})$ and $(1.3\text{m},1.0\text{m})$. The first source acts on the 5th quantity $v$ and has a time series, scaled by $2.3 \times$. The second source acts on the 4th quantity $u$ and on $v$, but only scales $v$ by $3.3 \times$. The first source is active after $1\text{s}$, the second source after $2.3\text{s}$. The first source has a time step of $0.001\text{s}$, while the second one has a time step of $0.002\text{s}$. We have a total of eight samples for all time series, where the first source starts at position $0$ and the second one at $5$. The sampled, unscaled time series for the first source has the values $(0, 0.1, 0.2, 0.1, 0.15)$. The second source has the three values $0, 0.3, 0.2$ and does not scale the time series.

NRF (deprecated)¶

Previous versions of EDGE implemented an intermediate netCDF format (NRF) of the Standard Rupture Format (SRF) for kinematic sources. This implementation performs the moment tensor computation, based on rupture parameters, in the setup of the solver. For higher flexibility, in particular the support of point forces at the surface, this procedure has been replaced by the format above. If using SRF-input and a (deprecated) NRF-enabled EDGE-version, the ASCII-SRF has to converted to an intermediate binary netCDF-format. You can use the tool rconv for the conversion from SRF to netCDF.

Benchmarks¶

EDGE provides input and output data for a series wave propagation benchmarks. Currently, these are the following benchmarks:

Name	Description
conv	2D/3D convergence benchmarks, periodic boundary conditions.
garvin	Garvin’s problem (2D), acoustic/elastic, explosive point source.
ghill_2d	2D Gaussian hill topography, acoustic/elastic, explosive point source.
ghcan_2d	2D Gaussian hill-canyon topography, acoustic/elastic, explosive point source.
sanjacinto_2d	2D Mount San Jacinto topography, elastic, explosive point source.
hsp1a	Homogeneous space, elastic, near receivers, single point source.
hsp2a	Homogeneous space, viscoelastic, near receivers, single point source.
hhs1	Homogeneous halfspace, elastic, single point source.
loh1	Layer over homogeneous halfspace, elastic, single point source.
loh2	Layer over homogeneous halfspace, elastic, kinematic rupture.
loh3	Layer over homogeneous halfspace, viscoelastic, single point source.
can4	Three thin layers in a halfspace, reaching the surface at a low angle, elastic, single point source.
ssp0	Layered velocity model, elastic, single point source.
ssef0	Layered velocity model, elastic, kinematic rupture with variable parameters.
ghill	3D Gaussian hill topography, elastic, double couple source.
sgt	3D mountain topography, homogeneous, elastic, point forces for reciprocal solution.
la_habra	2014 M5.1 La Habra, CA earthquake setting of SCEC’s High Frequency project.

Input: Kinematic Sources¶

EDGE provides pre-generated kinematic source descriptions as assets. However, most are easily generated, since the moment rate time history of many wave propagation benchmarks is simply given by an analytic function. You might generate the sources through the script kinematic_bench.py.

HSP1a (regular) example:

python tools/processing/kinematic_bench.py --xml examples/bench/elastic/wp1/hsp1a/reg_src.xml

The template for the wave propagation bechmarks is located at tools/processing/kinematic_bench.cdl. Sources for the other wave propagation setups can be generated by calling the respective XML-configurations, provided as assets.

LOH.2¶

The Layer Over Halfspace benchmark (LOH.2) is a purely elastic setup, consisting of a 1000m thick layer, having different material parameters ($v_s=2000 \frac{\text{m}}{\text{s}}$, $v_p=4000 \frac{\text{m}}{\text{s}}$, $\rho = 2600 \frac{\text{kg}}{\text{m}^3}$) than the half-space below ($v_s=3464 \frac{\text{m}}{\text{s}}$, $v_p=6000 \frac{\text{m}}{\text{s}}$, $\rho = 2700 \frac{\text{kg}}{\text{m}^3}$). Free-surface boundary conditions at the top of the layer are used, and outflow boundaries everywhere else. Source is a right-lateral strike-slip finite fault with a constant rupture velocity of $3000\frac{\text{m}}{\text{s}}$. The source-time function is similar everywhere, since only the onset time is variable. A detailed description of the LOH.2 problem is given in the “Final Report to Pacific Earthquake Engineering Research Center, Lifelines Program Task 1A01, Tests of 3D Elastodynamic Codes”.

EDGE provides two setups solving the LOH.2 problem. This allows us to check consistency of the results, when using different features of the code.

The first setup, our “reference” solution, simply uses 40 isolated configurations, each with a single point source. The 40 point sources are located at the epicenter in y-direction and their associated patches cover the z-dimension of the entire finite fault. Each of the square patches has a size of $100\text{m}\times 100\text{m}$. The effect of the remaining fault, with respect to the seismic receivers, is computed by simply shifting and adding the obtained solutions accordingly.

Illustration of the reference setup solving the LOH.2 benchmark. The 1000m thick layer is shown in gray, the finite fault in red, and the location of the hypocenter as a yellow star. 40 point sources (black and blue squares) with a finite fault size of 100m in y- and z-direction are used to derive EDGE’s “reference” solution. The patches are centered at the epicenter in y-direction, and cover the entire fault in z-direction.
The second setup uses a single simulation with a single kinematic source description. The kinematic source consists of a total of $80 \times 40 = 3,200$ patches covering the entire finite fault. Each of the square patches has a size of $100\text{m}\times 100\text{m}$.

Illustration of the kinematic source description for the LOH.2 benchmark. The 1000m thick layer is shown in gray and the location of the hypocenter as a yellow star. 3200 point sources (black and blue squares) with a finite-fault size of 100m in y- and z-direction are used to derive EDGE’s kinematic solution. The patches cover the entire finite fault.

Illustration showing the receiver-placement at the surface in gray, which is required to obtain the LOH.2 solution from isolated simulations. The fault is shown in red, the epicenter s1, indicating the location of the 40 point sources, is located at the yellow star. Three additional sources s2-s4 (extending in depth) are illustrated through yellow squares. We obtain the solution at receiver r1 with respect to the waves of s2-s4 by adding r2-r4 with respect to the waves generated by s1. Analogue, m1 is given by adding m2-m4.

By using identical meshes and convergence rates for both setup, we obtain almost identical numerical setups. Small differences, however, exist:

The seismic waves, originating from the point sources in the second setup, propagate through different elements before reaching the considered receivers.

The 40 point sources of the reference setup do not exist in the kinematic setup: The centers of the 40 fault patches are located on the boundaries of the second setup’s patches.

We only executed EDGE five times for the provided solution of the first setup, and thus fused eight simulations per run.

For the second setup, only one non-fused forward simulation was used. Since EDGE uses different kernels for the seismic wave propagation component in the two cases, errors, resulting from machine precision, are present.

Can4¶

The Can4 benchmark is purely elastic and consists of a simple basin model with three layers, embedded in a half-space. A detailed description of the benchmark is given in SISMOWINE’s description. Discussions of benchmark results are presented in Earthquake Ground Motion in the Mygdonian Basin, Greece: The E2VP Verification and Validation of 3D Numerical Simulation up to 4 Hz - E. Maufroy et al. and “19 - Modelling of earthquake motion: Mygdonian basin” of the book The Finite-Difference Modelling of Earthquake Motions - P. Moczo, J. Kristek, M. Gális.

The layers of the benchmark are shallow and reach the surface at a low dipping angle (wedge). This poses a modeling challenge to numerical software. We model the layers explicitly by using a tetrahedral mesh and aligning the faces to the material contrasts. Further, we avoid ill-shaped elements in the spatial discretization, by vertically cutting off the last dipping part of the layers. Here, the cut-off is chosen, such that the resulting height of the first layer is not smaller than the characteristic length of the elements in the wedge. Despite not explicitly meshing the remainder of the wedge, we still used appropiate material parameters for elements after the cut-off. This results in an increased scattering of the seismic waves, since the material interface now follows the unstructured mesh.

Illustration showing the three layers of the Can4 benchmark. The red, dashed line shows the cut-off in EDGE’s assumed geometry, avoiding ill-shaped elements in the mesh. The result is a minimum thickness (blue) of the first layer, equal to the characteristic length of the elements in the wedge.

We mitigate the extreme ratio of the computional domain with respect to the depth of the layers, by using a problem-adapted mesh-refinement. Here, we use the highest refinement in the wedge of the three layers, which reduces the negative impact of the normalization through the cut-off. The remainder of the three layers and our region of interest, given by $[-5000\,\text{m},5000\,\text{m}]\times[-5000\,\text{m},5000\,\text{m}]\times[0,5000\,\text{m}]$ use tetetrahedral element sizes, matching the desired frequency content. The location of the point source is additionally refined by an attractor. This allows for sharper a discretization of discontinuities, and thus reduces errors, which might be introduced by insufficient source discretization through large element-sizes in the region of interest. In x-direction (south-north) and z-direction (depth), our region of interest is surrounded by a sponge layer with a coarse resolution.

Illustration showing the problem-adapted mesh refinement of our Can4 setup. The highest resolution is used for the dipping parts of the layers (red), followed by decreasing resolution in the three layers (darker to lighter gray). Further, the point source (yellow star) is refined with a distance-dependent, linear gradient of decreasing refinement (blue sphere), reaching the coarsest resolution at the boundary of the sphere. The resolution in the region of interest (light gray) is chosen to match our desired frequency content, while the remainder (white) is coarse and acts as a sponge layer.

La Habra¶

2014 Mw5.1 a Habra, CA earthquake setting of SCEC’s High-F project. The verification benchmark uses frequency-independent Q, a planar free surface, kinematic sources, and the velocity model CVM-S4.26.M01.

Study area of the Southern California Earthquake Center’s High Frequency project simulating the 2014 M5.1 La Habra, California earthquake. Shown are the “small domain” through the inner box and partially the “large domain” through the outer box. Additionally the locations of the epicenter and three stations are given.

Visualization of a velocity-aware tetrahedral mesh for the small simulation region shown above. The example shows a highly increased mesh resolution in the near- surface parts of the Los Angeles basin. Some partitions in the North-West part of the computational domain are excluded. The epicenter is located in the center of the computational domain. The user also defined a cylindrical high-resolution region around the epicenter in addition to the velocity-derived target edge- lengths.

Visualization of the seismic wave field for a simulation of the 2014 Mw 5.1 La Habra Earthquake. Shown are the amplitudes of the horizontal particle velocities after seven seconds of simulated time. In addition to the High-F verification efforts, this simulation included mountain topograhy.

Comparison of EDGE’s South-North velocity component (red) to another solver of the High Frequency project (black). Shown are synthetic seismograms for the three stations depicted in above. The seismograms were low-pass filtered at 5Hz. EDGE’s respective ground motion simulation harnessed 1,536 nodes of the Frontera machine for a total of 48 hours to advance the used 2.1 billion tetrahedral element-mesh in time.

Resource Requirements¶

This chapter summarizes EDGE’s resource requirements for seismic simulations.

Memory¶

EDGE’s memory requirements depend on the chosen convergence rate and the number of fused runs.

In this section we only consider required memory for 4-node tetrahedral elements, the elastic wave equations (9 quantities), double precision arithmetic (64-bit per value) and the following data structures in every element:

Degrees Of Freedom (DOFs)

Time Integrated DOFs

Each of the eight Riemann solvers (sometimes called flux solvers)

Each of the three Jacobians (sometimes called star matrices)

Therefore, the memory requirements for the mesh, kinematic sources, internal dynamic rupture boundaries, etc. are neglected.

Increasing the convergence rate, increases the number of modes per element. An increase in the number of fused runs, also increases the memory footprint of every element. However, data is shared for fused simulations, which reduces the relative memory footprint per run. For example, a second order simulation without fused runs (C1) requires 6,336 bytes in theory. By fusing eight runs, the per-element footprint increases to 10,368 bytes. This is equivalent to an increase by $\frac{10.368}{6,336} \approx 1.64$ in required memory. However, the memory footprint per element and forward run decreases to $\frac{10,368}{8} = 1,296$. The corresponding improvement per forward run is therefore: $\frac{6,336}{1,296} \approx 4.9$.

The following table gives the memory footprint per element in dependency of the order for a non-fused run (C1), four (C4), and eight fused runs (C8):

Order	Modes	C1-Bytes	C4-Bytes	C8-Bytes
1	1	5,904	6,336	6,912
2	4	6,336	8,064	10,368
3	10	7,200	11,520	17,280
4	20	8,640	17,280	28,800
5	35	10,800	25,920	46,080
6	56	13,824	38,016	70,272
7	84	17,856	54,144	102,528

Element Throughput¶

Analogue, to the discussed memory requirements, all considerations in this section are limited to 4-node tetrahedral elements, the elastic wave equations (9 quantities), and double precision arithmetic (64-bit per value). Further, the reported times per element and time step were measured simulating the LOH.1 benchmark with a total of 350,264 tetrahedral elements. Architecture was a single node of Cori Phase II (Intel Xeon Phi 7250 68-core processors at 1.4 GHz with Intel Turbo Boost enabled) and all data allocated in High Bandwidth Memory (HBM/MCDRAM).

The following table shows the required time per element and per time step in dependency of the order for non-fused configurations (C1) and eight fused runs (C8):

Order	C1-Seconds	C8-Seconds
2	6.06E-08	1.30E-07
3	1.14E-07	2.88E-07
4	2.08E-07	7.30E-07
5	4.41E-07
6	6.93E-07

Introduction¶

Hey there, you have reached the user guide of the Extreme-scale Discontinuous Galerkin Environment (EDGE). EDGE uses the Discontinuous Galerkin (DG-) Finite Element Method (FEM) to solve hyperbolic partial differential equations. EDGE supports different equations and element types.

Currently, the application’s focus is on seismic simulations using unstructured tetrahedral meshes. EDGE targets seismic model setups with high geometric complexity and extreme-scale ensemble simulations. The entire software stack is uniquely tailored to “fused” simulations. Fused simulations allow for different model setups within one execution of the forward solver. For example, you could share the mesh and velocity model in the fused runs, but alter the kinematic source from run to run. This approach allows the code to exploit inter-simulation parallelism and reach significantly higher simulation throughput. Typically, the speedup is around 2-5 times, depending on the configuration. In short: Fusing simulations makes the code faster.

EDGE is distributed across different resources. First of all, there is this user guide you are reading currently. The main purpose of the user guide is to guide you through the installation of the code and the setup of simulations. If you run into trouble, don’t hesitate to get in touch. This user guide’s issue tracker is available at https://github.com/3343/edge_usr/issues, the issue tracker of the code at https://github.com/3343/edge/issues.

The next resource is the software itself. EDGE only provides source code, which is nice because you can always look at the guts of the software. However, it requires you to go through the compilation yourself. Then there are the assets, which are example setups, and supporting scripts and data. For example, this is the place where setups for benchmarks and unit tests are hosted. Finally, EDGE’s homepage provides up-to-date information on the Extreme-scale Discontinuous Galerkin Environment. To reach any of EDGE’s other resources, consider using the dispatcher

BSD 3-Clause and CC0¶

Disclaimer: We do not provide legal advice. The provided information is incomplete and is not a substitute for legal advice.

EDGE’s core is licensed under the BSD 3-Clause license. This does not apply to dependencies and used libraries. Libraries are either directly located in the directory submodules or included through .gitmodules. For the master branch the directory is located here and .gitmodules located here. EDGE’s automated FOSSA-reports might be helpful for further details on the licenses:

The sources of this user guide, and the sources of EDGE’s developer guide are CC0’d. An FAQ on CC0 is provided by Creative Commons.

EDGE’s assets follow a mixed approach. Software-components, e.g., configurations for EDGE’s core, or scripts, are BSD Clause-3. Other, non-software files, e.g., generated meshes, or source-input are CC0’d to a large extend.