Skip to content

Latest commit

 

History

History

test

README.md

SUNDIALS Testing

SUNDIALS utilizes CTest to build and run the example programs distributed with SUNDIALS as a test suite. In order to run the tests SUNDIALS examples must be enabled by setting the CMake options EXAMPLES_ENABLE_<language> to ON.

To run the standard C tests use the following commands:

mkdir <build directory>
cd <build directory>
cmake <sundials directory> \
  -DCMAKE_INSTALL_PREFIX=<prefix> \
  -DEXAMPLES_ENABLE_C=ON
make
make test

This will create an out of source build, build SUNDIALS and all the C examples, and run the tests (which are the examples) that return a pass/fail status. All output is saved to the file <build directory>/Testing/Temporary/LastTest.log.

Development Test

Examples that do not return a pass/fail status are considered "development" only tests and are excluded by default when running make test. To enable all examples when testing, set the CMake option SUNDIALS_TEST_DEVTESTS to ON. With this option enabled tests are run using a Python test runner test/testRunner. The test runner determines if a test passes or fails by comparing the test output against saved output files that are deemed correct by the SUNDIALS team.

To run the development C tests use the following commands:

mkdir <build directory>
cd <build directory>
cmake <sundials directory> \
  -DCMAKE_INSTALL_PREFIX=<prefix> \
  -DEXAMPLES_ENABLE_C=ON \
  -DSUNDIALS_TEST_DEVTESTS=ON
make
make test

This will create an out of source build, build SUNDIALS and all the C examples, and run all the tests (which are the examples). The output from each test is saved to <build directory>/Testing/output.

The saved output files are machine specific and may produce erroneous failures when run on different systems. New output files for a different machine can be generated by redirecting the test output to a machine specific answer directory using the SUNDIALS_TEST_OUTPUT_DIR cache variable. This file can then be set as the answer directory when running subsequent tests using the SUNDIALS_TEST_ANSWER_DIR cache variable.

For example the test output can be redirected to <machine output directory> using the following commands:

cd <build directory>
cmake <sundials directory> \
  -DCMAKE_INSTALL_PREFIX=<prefix> \
  -DEXAMPLES_ENABLE_C=ON \
  -DSUNDIALS_TEST_DEVTESTS=ON \
  -DSUNDIALS_TEST_OUTPUT_DIR=<machine output directory>
make
make test

The <machine output directory> can then be set as the test answer directory using the following commands:

cd <build directory>
cmake <sundials directory> \
  -DCMAKE_INSTALL_PREFIX=<prefix> \
  -DEXAMPLES_ENABLE_C=ON \
  -DSUNDIALS_TEST_DEVTESTS=ON \
  -DSUNDIALS_TEST_ANSWER_DIR=<machine output directory>
make
make test

Testing Scripts

The script test_driver.sh is provided to setup and run SUNDIALS tests. The input option --testtype can be used to select from a predefined set of test configurations:

  • BRANCH -- C90 build tests and a small set of build and run tests.

  • PR -- C90 build tests and a larger set of build, run, and install tests using the SUNDIALS release tarball.

  • RELEASE -- C90 build test and an even larger set of build, run, and install tests using the SUNDIALS release tarball.

  • CUSTOM -- run a single user-defined test configuration set using additional input options.

Run test_driver.sh -h for more information on the options available.

Note: At this time the testing scripts only run development tests when SUNDIALS is configured with real type double (either index size can be used).

Testing environment

The env/setup_env.sh script and a machine specific environment script are used to setup the testing environment. The machine specific environment script used (listed in the order checked) is:

#. The script specified by the environment variable SUNDIALS_ENV_FILE #. A user's local environment script: <sunrepo>/test/env/env.sh #. A machine environment script: <sunrepo>/test/env/${HOSTNAME}.sh #. A machine environment script: <sunrepo>/test/env/${HOST}.sh #. The default SUNDIALS environment script: <sunrepo>/test/env/default.sh

Environment scripts must set the following environment variables that are used when configuring SUNDIALS for testing.

CC  = C compiler
CXX = C++ compiler
FC  = Fortran compiler

CFLAGS   = C compiler flags
CXXFLAGS = C++ compiler flags
FFLAGS   = Fortran compiler flags

Note the test scripts will append the C standard flag (-std=c90 or -std=c99) and C++ standard flag (-std=c++11 or -std=c++17) to the compiler flags provided by the environment variables.

An environment script may optionally set additional environment variables to enable or disable third party libraries (TPLs) in the SUNDIALS configuration. The currently supported TPL environment variables are as follows:

# Enable/disable MPI support
SUNDIALS_MPI = ON or OFF
MPICC        = MPI C compiler wrapper
MPICXX       = MPI C++ compiler wrapper
MPIFC        = MPI Fortran compiler wrapper
MPIEXEC      = executable for launching MPI runs

# Enable/disable PThread support
SUNDIALS_PTHREAD = ON or OFF

# Enable/disable OpenMP support
SUNDIALS_OPENMP = ON or OFF

# Enable/disable OpenMP device offloading support
SUNDIALS_OPENMPDEV = ON or OFF

# Enable/disable CUDA support
SUNDIALS_CUDA = ON or OFF

# Enable/disable HIP support
SUNDIALS_HIP = ON or OFF

# Enable/disable RAJA support
SUNDIALS_RAJA = ON or OFF
RAJA_ROOT     = full path to RAJA installation
RAJA_BACKENDS = RAJA backends

# Enable/disable SYCL support
SUNDIALS_SYCL = ON or OFF

# Enable/disable LAPACK linear solvers
SUNDIALS_LAPACK  = ON or OFF
LAPACK_LIBRARIES = full path to LAPACK library

# Enable/disable KLU linear solver
SUNDIALS_KLU             = ON or OFF
SUITE_SPARSE_INCLUDE_DIR = full path to SuiteSparse include directory
SUITE_SPARSE_LIBRARY_DIR = full path to SuiteSparse library directory

# Enable/disable SuperLU_MT linear solver
SUNDIALS_SUPERLU_MT    = ON or OFF
SUPERLU_MT_INCLUDE_DIR = full path to SuperLU_MT include directory
SUPERLU_MT_LIBRARY_DIR = full path to SuperLU_MT library directory

# Enable/disable SuperLU_DIST linear solver
SUNDIALS_SUPERLU_DIST  = ON or OFF
SUPERLU_DIST_INCLUDE_DIR = full path to SuperLU_DIST include directory
SUPERLU_DIST_LIBRARY_DIR = full path to SuperLU_DIST library directory
SUPERLU_DIST_LIBRARIES   = additional link libraries for SuperLU_DIST

# Enable/disable MAGMA linear solver
SUNDIALS_MAGMA = ON or OFF
MAGMA_ROOT     = full path to MAGMA installation
MAGMA_BAKCENDS = MAGMA backend

# Enable/disable hypre support
SUNDIALS_HYPRE    = ON or OFF
HYPRE_INCLUDE_DIR = full path to hypre include directory
HYPRE_LIBRARY_DIR = full path to hypre library directory

# Enable/disable PETSc support
SUNDIALS_PETSC = ON or OFF
PETSC_ROOT     = full path to PETSc installation

# Enable/disable Trilinos support
SUNDIALS_TRILINOS = ON or OFF
TRILINOS_ROOT     = full path to Trilinos installation

# Enable/disable Trilinos support
SUNDIALS_XBRAID = ON or OFF
XBRAID_ROOT     = full path to XBraid installation

To aid in setting the above variables appropriately, env/setup_env.sh will set environment variables SUNDIALS_PRECISION, SUNDIALS_INDEX_SIZE, and SUNDIALS_LIBRARY_TYPE for the real type (single, double, or extended), index size (32 or 64), and library type (shared, static or both) used in the test. Additionally, SUNDIALS_TPLS is set to ON or OFF to indicate if the test will use third-party libraries.

Any additional input options passed to test_driver.sh will be passed through to the environment script for additional setup information. For example, the machine script may accept a compiler spec (e.g., [email protected]) and/or a build type (e.g., opt for an optimized build).

Using Spack to install TPLs

The TPLs needed for a complete build of SUNDIALS can be easily installed with spack and the spack environment included in the SUNDIALS repository. Simply navigate to test/spack and run spack install. For more information on Spack environments is the Spack tutorial.