Warning

This is the documentation for the development version of DTK. There may be significant differences from the latest stable release. Please follow this link if you are looking for DTK 2.0

DataTransferKit Documentation

Contents:

Warning

This is the documentation for the development version of DTK. There may be significant differences from the latest stable release. Please follow this link if you are looking for DTK 2.0

Getting started with DTK

Overview

DataTransferKit is an open-source software library of parallel solution transfer services for multiphysics simulations. DTK uses a general operator design to provide scalable algorithms for solution transfer between shared volumes and surfaces.

DTK was originally developed at the University of Wisconsin - Madison as part of the Computational Nuclear Engineering Group (CNERG) and is now actively developed at the Oak Ridge National Laboratory as part of the Computational Engineering and Energy Sciences (CEES) group.

DTK is supported and used by the following projects and programs:

  • Oak Ridge National Laboratory (ORNL) Laboratory Directed Research and Development (LDRD)
  • Consortium for Advanced Simulation of Light Water Reactors (CASL)
  • Nuclear Energy Advanced Modeling and Simulation (NEAMS)
  • National Highway Traffic Safety Administration (NHTSA)

DataTransferKit Development Team

DTK is developed and maintained by:

DataTransferKit Packages

DTK has the following packages:

Utils
General utilities for software development including exception handling, MPI-based tools, and functional programming tools.
Interface
Implement interface with user applications.
Search
Implement the search algorithms.

Questions, Bug Reporting, and Issue Tracking

Questions, bug reporting and issue tracking are provided by GitHub. Please report all bugs by creating a new issue. You can ask questions by creating a new issue with the question tag.

Publications

Publications to date related to DataTransferKit:

  • S. Slattery, “Mesh-Free Data Transfer Algorithms for Partitioned Multiphysics Problems: Conservation, Accuracy, and Parallelism”, Journal of Computational Physics, vol. 307, pp. 164-188, 2016.
  • S. Slattery, S. Hamilton, T. Evans, “A Modified Moving Least Square Algorithm for Solution Transfer on a Spacer Grid Surface”, ANS MC2015 - Joint International Conference on Mathematics and Computation (M&C), Supercomputing in Nuclear Applications (SNA) and the Monte Carlo (MC) Method, Nashville, Tennessee · April 19–23, 2015, on CD-ROM, American Nuclear Society, LaGrange Park, IL (2015).
  • R. Schmidt, K. Belcourt, R. Hooper, R. Pawlowski, K. Clarno, S. Simunovic, S. Slattery, J. Turner, S. Palmtag, “An Approach for Coupled-Code Multiphysics Core Simulations from a Common *Input”, Annals of Nuclear Energy, Volume 84, pp. 140-152, 2014.
  • S. Slattery, P.P.H. Wilson, R. Pawlowski, “The Data Transfer Kit: A Geometric Rendezvous-Based Tool for Multiphysics Data Transfer”, International Conference on Mathematics and Computational Methods Applied to Nuclear Science & Engineering (M&C 2013), American Nuclear Society, Sun Valley, ID, May 5-9, 2013.

Warning

This is the documentation for the development version of DTK. There may be significant differences from the latest stable release. Please follow this link if you are looking for DTK 2.0

Installation

This section provide guidelines for installing DataTransferKit and its TPLs.

Install third-party libraries

The following third party libraries (TPLs) are used by DTK:

Packages Dependency Version
Boost Required 1.59.0
BLAS/LAPACK Required N/A
MPI Optional N/A
Trilinos Required 12.X

Note

DTK is built as an external package of Trilinos. Thus, DTK needs the source of the Trilinos library rather than an installed version.

The dependencies of DataTransferKit may be built using Spack package manager. You need to install the following packages:

$ spack install openblas
$ spack install boost
$ spack install mpi

Once installed, the module files for the packages must be loaded into the environment by doing

$ spack load openblas
$ spack load boost
$ spack load openmpi

DTKData repository

The DTKData repository contains mesh files used in DTK examples. To build the examples and include the mesh files include the git submodule with your cloned repository:

$ git submodule init
$ git submodule update

Another way to achieve this is to pass the --recursive option to the git clone command which will automatically initialize and update DTKData in the DataTransferKit repository.

Building DTK

DTK is configured and built using TriBITS. DTK builds within Trilinos effectively as an extension package. First, link DTK into the Trilinos main directory:

$ cd $TRILINOS_DIR
$ ln -s $DTK_DIR

Create a do-configure script such as:

EXTRA_ARGS=$@

cmake \
    -D CMAKE_INSTALL_PREFIX:PATH=$INSTALL_DIR \
    -D CMAKE_BUILD_TYPE:STRING=DEBUG \
    -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF \
    -D BUILD_SHARED_LIBS:BOOL=OFF \
    -D TPL_ENABLE_MPI:BOOL=ON \
    -D TPL_ENABLE_Boost:BOOL=ON \
    -D Boost_INCLUDE_DIRS:PATH=$BOOST_INCLUDE_DIR \
    -D TPL_ENABLE_Libmesh:BOOL=OFF \
    -D TPL_ENABLE_Netcdf:BOOL=OFF \
    -D TPL_ENABLE_MOAB:BOOL=OFF \
    -D TPL_ENABLE_BinUtils:BOOL=OFF \
    -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF \
    -D Trilinos_EXTRA_REPOSITORIES="DataTransferKit" \
    -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON \
    -D Trilinos_ENABLE_CXX11:BOOL=ON \
    -D Trilinos_ENABLE_DataTransferKit:BOOL=ON \
    -D DataTransferKit_ENABLE_DBC:BOOL=ON \
    -D DataTransferKit_ENABLE_TESTS:BOOL=ON \
    -D DataTransferKit_ENABLE_EXAMPLES:BOOL=ON \
    $EXTRA_ARGS \
    $TRILINOS_DIR

and run it from your build directory:

$ mkdir build && cd build
$ ../do-configure

More install scripts can be found in scripts/.

Build this documentation

Building documentation requires sphinx. (Re)configure with -D DataTransferKit_ENABLE_ReadTheDocs=ON and run:

$ make docs

Open the index.html in the directory DataTransferKit/docs/html.

Generate Doxygen documentation

Configure with -D DataTransferKit_ENABLE_Doxygen=ON and run:

$ make doxygen

Checkout DataTransferKit/docs/doxygen/html/index.html.

Warning

This is the documentation for the development version of DTK. There may be significant differences from the latest stable release. Please follow this link if you are looking for DTK 2.0

DataTransferKit API

C++ API

User Function Registry

template <class Scalar>
class DataTransferKit::UserFunctionRegistry

Registry for user functions.

The registry is the mechanism by which user functions and the data to be called with those user functions are registered with DTK.

Template Parameters
  • Scalar: Scalar type of the fields this user function implementation will provide.

Unnamed Group

void setNodeListSizeFunction(NodeListSizeFunction &&func, std::shared_ptr<void> user_data = nullptr)

Set geometry.

void setNodeListDataFunction(NodeListDataFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setBoundingVolumeListSizeFunction(BoundingVolumeListSizeFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setBoundingVolumeListDataFunction(BoundingVolumeListDataFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setPolyhedronListSizeFunction(PolyhedronListSizeFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setPolyhedronListDataFunction(PolyhedronListDataFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setCellListSizeFunction(CellListSizeFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setCellListDataFunction(CellListDataFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setMixedTopologyCellListSizeFunction(MixedTopologyCellListSizeFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setMixedTopologyCellListDataFunction(MixedTopologyCellListDataFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setBoundarySizeFunction(BoundarySizeFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setBoundaryDataFunction(BoundaryDataFunction &&func, std::shared_ptr<void> user_data = nullptr)

Unnamed Group

void setDOFMapSizeFunction(DOFMapSizeFunction &&func, std::shared_ptr<void> user_data = nullptr)

Set degree-of-freedom maps.

void setDOFMapDataFunction(DOFMapDataFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setMixedTopologyDOFMapSizeFunction(MixedTopologyDOFMapSizeFunction &&func, std::shared_ptr<void> user_data = nullptr)
void setMixedTopologyDOFMapDataFunction(MixedTopologyDOFMapDataFunction &&func, std::shared_ptr<void> user_data = nullptr)

Unnamed Group

void setFieldSizeFunction(FieldSizeFunction<Scalar> &&func, std::shared_ptr<void> user_data = nullptr)

Set fields.

void setPullFieldDataFunction(PullFieldDataFunction<Scalar> &&func, std::shared_ptr<void> user_data = nullptr)
void setPushFieldDataFunction(PushFieldDataFunction<Scalar> &&func, std::shared_ptr<void> user_data = nullptr)
void setEvaluateFieldFunction(EvaluateFieldFunction<Scalar> &&func, std::shared_ptr<void> user_data = nullptr)

Public Types

template<>
using UserImpl = std::pair<CallableObject, std::shared_ptr<void>>

User implementation.

User Application

template <class Scalar, class ParallelModel>
class DataTransferKit::UserApplication

High-level interface to user applications.

The user application provides a high-level interface to compose DTK input data structures and push and pull field data to and from the application through sequences of user function calls.

Template Parameters
  • Scalar: Scalar type of the fields this user application will provide.
  • ParallelModel: The DTK parallel model indicating the on-node parallelism of the user application. Indicates where data will be allocated.

Unnamed Group

template<>
using ExecutionSpace = typename ParallelTraits::ExecutionSpace

Type aliases.

Public Functions

UserApplication(const std::shared_ptr<UserFunctionRegistry<Scalar>> &user_functions)

Constructor.

auto getNodeList()
auto getBoundingVolumeList()
auto getPolyhedronList()
auto getCellList(std::vector<std::string> &cell_topologies)
template <class ListType>
void getBoundary(const std::string &boundary_name, ListType &list)
auto getDOFMap(std::string &discretization_type)
auto getField(const std::string &field_name)
void pullField(const std::string &field_name, Field<Scalar, Kokkos::LayoutLeft, ExecutionSpace> field)
void pushField(const std::string &field_name, const Field<Scalar, Kokkos::LayoutLeft, ExecutionSpace> field)
void evaluateField(const std::string &field_name, const EvaluationSet<Kokkos::LayoutLeft, ExecutionSpace> eval_set, Field<Scalar, Kokkos::LayoutLeft, ExecutionSpace> field)

C API

C interface header.

Typedefs

typedef struct _DTK_UserApplicationHandle *DTK_UserApplicationHandle

DTK user application handle.

Must be created using DTK_create() to be a valid handle.

The handle essentially hides C++ implementation details from the user.

typedef void( * DTK_NodeListSizeFunction) (void *user_data, unsigned *space_dim, size_t *local_num_nodes, bool *has_ghosts)

Prototype function to get the size parameters for building a node list.

Register with a user application using DTK_set_function() by passing DTK_NODE_LIST_SIZE_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • space_dim: Spatial dimension.
  • local_num_nodes: Number of nodes DTK will allocate memory for.
  • has_ghosts: Whether some of the nodes that will be passed are ghosted (i.e. belong to another process)

typedef void( * DTK_NodeListDataFunction) (void *user_data, Coordinate *coordinates, bool *is_ghost_node)

Prototype function to get the data for a node list.

Register with a user application using DTK_set_function() by passing DTK_NODE_LIST_DATA_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • coordinates: Node coordinates.
  • is_ghost_node: Indicates whether a given node is ghosted.

typedef void( * DTK_BoundingVolumeListSizeFunction) (void *user_data, unsigned *space_dim, size_t *local_num_volumes, bool *has_ghosts)

Prototype function to get the size parameters for building a bounding volume list.

Register with a user application using DTK_set_function() by passing DTK_BOUNDING_VOLUME_LIST_SIZE_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • space_dim: Spatial dimension.
  • local_num_volumes: Number of volumes DTK will allocate memory for.
  • has_ghosts: Whether some of the bounding volumes that will be passed are ghosted (i.e. belong to another process)

typedef void( * DTK_BoundingVolumeListDataFunction) (void *user_data, Coordinate *bounding_volumes, bool *is_ghost_volume)

Prototype function to get the data for a bounding volume list.

Register with a user application using DTK_set_function() by passing DTK_BOUNDING_VOLUME_LIST_DATA_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • bounding_volumes: Bounding volumes.
  • is_ghost_node: Indicates whether a given volume is ghosted.

typedef void( * DTK_PolyhedronListSizeFunction) (void *user_data, unsigned *space_dim, size_t *local_num_nodes, size_t *local_num_faces, size_t *total_nodes_per_face, size_t *local_num_cells, size_t *total_faces_per_cell, bool *has_ghosts)

Prototype function to get the size parameters for building a polyhedron list.

Register with a user application using DTK_set_function() by passing DTK_POLYHEDRON_LIST_SIZE_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • space_dim: Spatial dimension.
  • local_num_nodes: Number of nodes DTK will allocate memory for.
  • local_num_faces: Number of faces DTK will allocate memory for.
  • total_nodes_per_face: Total degrees of freedom per face.
  • local_num_cells: Number of cells DTK will allocate memory for.
  • total_faces_per_cell: Total number of faces per cell.
  • has_ghosts: Whether some of the cells that will be passed are ghosted (i.e. belong to another process)

typedef void( * DTK_PolyhedronListDataFunction) (void *user_data, Coordinate *coordinates, LocalOrdinal *faces, unsigned *nodes_per_face, LocalOrdinal *cells, unsigned *faces_per_cell, int *face_orientation, bool *is_ghost_cell)

Prototype function to get the data for a polyhedron list.

Register with a user application using DTK_set_function() by passing DTK_POLYHEDRON_LIST_DATA_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • coordinates: Node coordinates.
  • faces: Connectivity list of faces.
  • nodes_per_face: Number of nodes per face.
  • cells: Connectivity list of polyhedrons.
  • faces_per_cell: Number of faces per cell.
  • face_orientation: Orientation of the faces.
  • is_ghost_cell: Indicates whether a given cell is ghosted.

typedef void( * DTK_CellListSizeFunction) (void *user_data, unsigned *space_dim, size_t *local_num_nodes, size_t *local_num_cells, unsigned *nodes_per_cell, bool *has_ghosts)

Prototype function to get the size parameters for building a cell list with a single topology.

Register with a user application using DTK_set_function() by passing DTK_CELL_LIST_SIZE_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • space_dim: Spatial dimension.
  • local_num_nodes: Number of nodes DTK will allocate memory for.
  • local_num_cells: Number of cells DTK will allocate memory for.
  • nodes_per_cell: Number of nodes per cell.
  • has_ghosts: Whether some of the cells that will be passed are ghosted (i.e. belong to another process)

typedef void( * DTK_CellListDataFunction) (void *user_data, Coordinate *coordinates, LocalOrdinal *cells, bool *is_ghost_cell, char *cell_topology)

Prototype function to get the data for a single topology cell list.

Register with a user application using DTK_set_function() by passing DTK_CELL_LIST_DATA_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • coordinates: Node coordinates.
  • cells: List of cells.
  • is_ghost_cell: Indicates whether a given cell is ghosted.
  • cell_topology: Topology of the cells.

typedef void( * DTK_MixedTopologyCellListSizeFunction) (void *user_data, unsigned *space_dim, size_t *local_num_nodes, size_t *local_num_cells, size_t *total_nodes_per_cell, bool *has_ghosts)

Prototype function to get the size parameters for building a cell list with a mixed topology.

Register with a user application using DTK_set_function() by passing DTK_MIXED_TOPOLOGY_CELL_LIST_SIZE_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • space_dim: Spatial dimension.
  • local_num_nodes: Number of nodes DTK will allocate memory for.
  • local_num_cells: Number of cells DTK will allocate memory for.
  • total_nodes_per_cell: Total number of nodes per cell.
  • has_ghosts: Whether some of the cells that will be passed are ghosted (i.e. belong to another process)

typedef void( * DTK_MixedTopologyCellListDataFunction) (void *user_data, Coordinate *coordinates, LocalOrdinal *cells, unsigned *cell_topology_ids, bool *is_ghost_cell, char **cell_topologies)

Prototype function to get the data for a mixed topology cell list.

Register with a user application using DTK_set_function() by passing DTK_MIXED_TOPOLOGY_CELL_LIST_DATA_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • coordinates: Node coordinates.
  • cells: List of cells.
  • cell_topology_ids: Topology id for each cell.
  • is_ghost_cell: Indicates whether a given cell is ghosted.
  • cell_topologies: Topologies of the cells.

typedef void( * DTK_BoundarySizeFunction) (void *user_data, const char *boundary_name, size_t *local_num_faces)

Prototype function to get the size parameters for a boundary.

Register with a user application using DTK_set_function() by passing DTK_BOUNDARY_SIZE_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • boundary_name: Name of the boundary.
  • local_num_faces: Number of faces owned by this process.

typedef void( * DTK_BoundaryDataFunction) (void *user_data, const char *boundary_name, LocalOrdinal *boundary_cells, unsigned *cell_faces_on_boundary)

Prototype function to get the data for a boundary.

Register with a user application using DTK_set_function() by passing DTK_BOUNDARY_DATA_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • boundary_name: Name of the boundary.
  • boundary_cells: Indices of the cells on the boundary.
  • cell_faces_on_boundary: Indices of the faces within a given cell that is on the boundary.

typedef void( * DTK_DOFMapSizeFunction) (void *user_data, size_t *local_num_dofs, size_t *local_num_objects, unsigned *dofs_per_object)

Prototype function to get the size parameters for a degree-of-freedom id map with a single number of dofs per object.

Register with a user application using DTK_set_function() by passing DTK_DOF_MAP_SIZE_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • local_num_dofs: Number of degrees of freedom owned by this process.
  • local_num_objects: Number of objects on this process.
  • dofs_per_objects: Degrees of freedom per object.

typedef void( * DTK_DOFMapDataFunction) (void *user_data, GlobalOrdinal *global_dof_ids, LocalOrdinal *object_dof_ids, char *discretization_type)

Prototype function to get the size data for a degree-of-freedom id map with a single number of dofs per object.

Register with a user application using DTK_set_function() by passing DTK_DOF_MAP_DATA_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • global_dof_ids: Globally unique ids for DOFs on this process.
  • object_dof_ids: For every object of the given type in the object list give the local dof ids for that object. The local dof ids correspond to the index of the entry in the global dof id view.
  • discretization_type: Type of discretization.

typedef void( * DTK_MixedTopologyDofMapSizeFunction) (void *user_data, size_t *local_num_dofs, size_t *local_num_objects, size_t *total_dofs_per_object)

Prototype function to get the size parameters for a degree-of-freedom id map with each object having a potentially different number of dofs (e.g. mixed topology cell lists or polyhedron lists).

Register with a user application using DTK_set_function() by passing DTK_MIXED_TOPOLOGY_DOF_MAP_SIZE_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • local_num_dofs: Number of degrees of freedom owned by this process.
  • local_num_objects: Number of objects on this process.
  • total_dofs_per_objects: Total degrees of freedom per objects.

typedef void( * DTK_MixedTopologyDofMapDataFunction) (void *user_data, GlobalOrdinal *global_dof_ids, LocalOrdinal *object_dof_ids, unsigned *dofs_per_object, char *discretization_type)

Prototype function to get the data for a multiple object degree-of-freedom id map (e.g. mixed topology cell lists or polyhedron lists).

Register with a user application using DTK_set_function() by passing DTK_MIXED_TOPOLOGY_DOF_MAP_DATA_FUNCTION as the type argument.

Parameters
  • user_data: Pointer to custom user data.
  • global_dof_ids: Globally unique ids for DOFs on this process.
  • object_dof_ids: Local object IDs.
  • dofs_per_object: Degrees of freedom per object.
  • discretization_type: Type of discretization.

typedef void( * DTK_FieldSizeFunction) (void *user_data, const char *field_name, unsigned *field_dimension, size_t *local_num_dofs)

Prototype function to get the size parameters for a field.

Register with a user application using DTK_set_function() by passing DTK_FIELD_SIZE_FUNCTION as the type argument.

Field must be of size local_num_dofs in the associated dof_id_map.

Parameters
  • user_data: Custom user data.
  • field_name: Name of the field.
  • field_dimension: Dimension of the field (i.e. 1 for the pressure, or 3 for the velocity in 3-D)
  • local_num_dofs: Number of degrees of freedom owned by this process.

typedef void( * DTK_PullFieldDataFunction) (void *user_data, const char *field_name, double *field_dofs)

Prototype function to pull data from the application into a field.

Register with a user application using DTK_set_function() by passing DTK_PULL_FIELD_DATA_FUNCTION as the type argument.

Parameters
  • user_data: Custom user data.
  • field_name: Name of the field to pull.
  • field_dofs: Degrees of freedom for that field.

typedef void( * DTK_PushFieldDataFunction) (void *user_data, const char *field_name, const double *field_dofs)

Prototype function to push data from a field into the application.

Register with a user application using DTK_set_function() by passing DTK_PUSH_FIELD_DATA_FUNCTION as the type argument.

Parameters
  • user_data: Custom user data.
  • field_name: Name of the field to push.
  • field_dofs: Degrees of freedom for that field.

typedef void( * DTK_EvaluateFieldFunction) (void *user_data, const char *field_name, const Coordinate *evaluation_points, const LocalOrdinal *object_ids, double *values)

Prototype function to evaluate a field at a given set of points in a given set of objects.

Register with a user application using DTK_set_function() by passing DTK_EVALUATE_FIELD_FUNCTION as the type argument.

Parameters
  • user_data: Custom user data.
  • field_name: Name of the field to evaluate.
  • evaluate_points: Coordinates of the points at which to evaluate the field.
  • objects_ids: ID of the cell/face with repect of which the coordinates are expressed.
  • values: Field values.

Enums

enum DTK_ExecutionSpace

Execution space (where functions execute)

Values:

DTK_SERIAL
DTK_OPENMP
DTK_CUDA
enum DTK_FunctionType

Passed as the type argument to DTK_set_function() in order to indicate what callback function is being registered with the user application.

Note
Callback functions are passed as pointers to functions that take no arguments and return nothing (void(*)()) so the value of the DTK_FunctionType enum is necessary to indicate what is being registered with the user application and how to cast the function pointer back to the appropriate signature.

Values:

DTK_NODE_LIST_SIZE_FUNCTION

See DTK_NodeListSizeFunction()

DTK_NODE_LIST_DATA_FUNCTION

See DTK_NodeListDataFunction()

DTK_BOUNDING_VOLUME_LIST_SIZE_FUNCTION

See DTK_BoundingVolumeListSizeFunction()

DTK_BOUNDING_VOLUME_LIST_DATA_FUNCTION

See DTK_BoundingVolumeListDataFunction()

DTK_POLYHEDRON_LIST_SIZE_FUNCTION

See DTK_PolyhedronListSizeFunction()

DTK_POLYHEDRON_LIST_DATA_FUNCTION

See DTK_PolyhedronListDataFunction()

DTK_CELL_LIST_SIZE_FUNCTION

See DTK_CellListSizeFunction()

DTK_CELL_LIST_DATA_FUNCTION

See DTK_CellListDataFunction()

DTK_MIXED_TOPOLOGY_CELL_LIST_SIZE_FUNCTION

See DTK_MixedTopologyCellListSizeFunction()

DTK_MIXED_TOPOLOGY_CELL_LIST_DATA_FUNCTION

See DTK_MixedTopologyCellListDataFunction()

DTK_BOUNDARY_SIZE_FUNCTION

See DTK_BoundarySizeFunction()

DTK_BOUNDARY_DATA_FUNCTION

See DTK_BoundaryDataFunction()

DTK_DOF_MAP_SIZE_FUNCTION

See DTK_DOFMapSizeFunction()

DTK_DOF_MAP_DATA_FUNCTION

See DTK_DOFMapDataFunction()

DTK_MIXED_TOPOLOGY_DOF_MAP_SIZE_FUNCTION

See DTK_MixedTopologyDofMapSizeFunction()

DTK_MIXED_TOPOLOGY_DOF_MAP_DATA_FUNCTION

See DTK_MixedTopologyDofMapDataFunction()

DTK_FIELD_SIZE_FUNCTION

See DTK_FieldSizeFunction()

DTK_PULL_FIELD_DATA_FUNCTION

See DTK_PullFieldDataFunction()

DTK_PUSH_FIELD_DATA_FUNCTION

See DTK_PushFieldDataFunction()

DTK_EVALUATE_FIELD_FUNCTION

See DTK_EvaluateFieldFunction()

Functions

const char *DTK_version()

Get the current version of DTK.

Return
Returns a string containing the version number for DTK.

const char *DTK_git_commit_hash()

Get the current repository hash.

Note
If the source code is not under revision control (e.g. downloaded as a tarball), this functions returns an error string indicating that it is not a GIT repository.
Return
Returns a string containing the revision number.

DTK_UserApplicationHandle DTK_create(DTK_ExecutionSpace space)

Create a DTK handle.

Return
DTK_create returns a handle for the user application.
Parameters
  • space: Execution space for the callback functions that are to be registered using DTK_set_function().

bool DTK_is_valid(DTK_UserApplicationHandle handle)

Indicates whether a DTK handle is valid.

A handle is valid if it was created by DTK_create() and has not yet been deleted by DTK_destroy().

Return
true if the given user application handle is valid; false otherwise.
Parameters
  • handle: The DTK user application handle to check.

void DTK_destroy(DTK_UserApplicationHandle handle)

Destroy a DTK handle.

Parameters
  • handle: User application handle.

void DTK_initialize()

Initializes the DTK execution environment.

This initializes Kokkos if it has not already been initialized.

void DTK_initialize_cmd(int *argc, char ***argv)

Initialize DTK.

This initializes Kokkos if it has not already been initialized.

This version of initialize() effectively calls Kokkos::initialize( *argc, *argv ). Pointers to argc and argv arguments are passed in order to match MPI_Init’s interface. This function name was suffixed with _cmd because, unlike C++, C does not allow to overload functions.

Parameters
  • argc: Pointer to the number of argument.
  • argv: Pointer to the argument vector.

bool DTK_is_initialized()

Indicates whether DTK has been initialized.

This function may be used to determine whether DTK has been initialized.

void DTK_finalize()

Finalizes DTK.

This function terminates the DTK execution environment. If DTK initialized Kokkos, finalize Kokkos. However, if Kokkos was initialized before DTK, then this function does NOT finalize Kokkos.

void DTK_set_function(DTK_UserApplicationHandle handle, DTK_FunctionType type, void (*f)()void *user_data, )

Register a function as a callback.

This registers a custom function as a callback for DTK to communicate with the user application.

Parameters
  • handle: User application handle.
  • type: Type of callback function.
  • f: Pointer to user defined callback function.
  • user_data: Pointer to the user data that will be passed to the callback function when executing it.

Warning

This is the documentation for the development version of DTK. There may be significant differences from the latest stable release. Please follow this link if you are looking for DTK 2.0

Developer Tools

Run DTK development environment in a Docker container

To start a container from the DTK pre-built Docker image that is used in the automated build on Jenkins, run:

[host]$ cd docker
[host]$ echo COMPOSE_PROJECT_NAME=$USER > .env # [optional] specify a project name
[host]$ docker-compose pull # pull the most up-to-date version of the DTK base image
[host]$ docker-compose up -d # start the container

This will mount the local DTK source directory into the container at ${TRILINOS_DIR}/DataTransferKit. The environment variable TRILINOS_DIR is already defined and contains the path to a release version of Trilinos that has been downloaded into the DTK base image. We recommend you use a .env file to specify an alternate project name (the default being the directory name, i.e. docker). This will let you run multiple isolated environments on a single host. Here the service name will be prefixed by your username which will prevent interferences with other developers on the same system.

Then to launch an interactive Bash session inside that container, do:

[host]$ docker-compose exec dtk_dev bash

Configure, build, and test as you would usually do:

[container]$ cd $TRILINOS_DIR/DataTransferKit
[container]$ mkdir build && cd build
[container]$ ../scripts/docker_cmake
[container]$ make -j<N>
[container]$ ctest -j<N>

Do not forget to cleanup after yourself:

[container]$ exit
[host]$ docker-compose down # stop and remove the container

Make the container GPU-aware

Follow these instructions to launch containers leveraging the NVIDIA GPUs on the host machine:

[host]$ cd docker
[host]$ nvidia/setup_nvidia_docker_compose.py # extend the regular Compose file to leverage GPUs
[host]$ docker-compose build # add the CUDA development tools to the DTK base image
[host]$ docker-compose up -d # as previously described

Do not forget to set the environment for CUDA before you configure:

[container]$ # assuming you are in $TRILINOS_DIR/DataTransferKit/build
[container]$ source ../scripts/set_kokkos_env.sh # set environment for CUDA
[container]$ ../scripts/docker_cuda_cmake # configure
[container]$ # now you may build and test

Code completion for Vim

Configure with -D DataTransferKit_ENABLE_YouCompleteMe=ON to generate a .ycm_extra_conf.py file at the root of your DTK source directory tree for use with YouCompleteMe.

Warning

This is the documentation for the development version of DTK. There may be significant differences from the latest stable release. Please follow this link if you are looking for DTK 2.0

Coding Style Guidelines

DataTransferKit developers follow a set of style guidelines and use the clang format tool to create a consistent appearance to all source code committed to the repository.

ClangFormat

ClangFormat (version 4.0) is used to check the C++ code formatting style in DTK. A pull request that does not comply will be rejected. Configure with -D DataTransferKit_ENABLE_ClangFormat=ON and do make format-cpp to apply the formatting style before your commit. Alternatively, run ctest -V -R check_format_cpp display the diff without applying the changes.

Style Guide

The following conventions are used in the code.

Names of classes, structs, and enumerations are camel case and capitalized:

class ExampleClassName {};

Function names are camel case and not capitalized:

void exampleFunctionName() const;

Variable names are lower case and have underscores to separate words:

double example_double_var;
std::vector<int> example_vec_var;

If a variable is class data prefixed with a _:

class ExampleClass
{
  public:
    int _a_public_var;
  private:
    double _class_double_var;
    std::vector<int> _example_vec_var;
  protected:
    std::string _a_protected_string;
};

Previously, the convention for class data was to prefix the variable name with d_ so this will be seen throughout the code. We will be transitioning to the _ prefix convention in future work and slowly transition existing code.

The clang-format tool described above enforces spacing, line breaks, and other general file formatting requirements. Header files are suffixed with .hpp and non-templated implementation files are suffixed with .cpp. Header guards are needed for all header files following the convention of DTK_CLASSNAME_HPP. For example:

#ifndef DTK_EXAMPLECLASS_HPP
#define DTK_EXAMPLECLASS_HPP

class ExampleClass
{
    // Class definition...
};

#endif

Indices and tables