This directory contains the automated testing framework for Quail. The testing library used is Pytest. To get started, first generate the autogenerated tests, then run pytest in the test directory. After the tests are done running, the generated tests can be cleared.

cd test/end_to_end
python generate_tests.py
cd ..
pytest
cd end_to_end
python clear_tests.py

The test directory is split into two directories, component and end_to_end, which represent component-level and end-to-end tests respectively.

• Component-level tests will test only a small component of Quail. These will generally include things like unit tests and integration tests.
• End-to-end tests will test a full run through the code. This means calling the Quail executable with an input file, performing iterations, and comparing the result stored in the output file.

Some guidelines are established to keep the testing framework uniform and easy to update when changes to the code are made. Straying from these guidelines may cause tests to run unsuccessfully or give errors.

General guidelines for all tests are as follows.

• Add Pytest markers to your test to be able to distinguish tests and run specific sets of tests. For example, running only tests marked as dg can be done with:

   cd test/
   pytest -m "dg"

Additionally, boolean expressions may be used to match markers:

   pytest -m "one_d and dg not splitting"

• When a new marker is defined, it must be added to the pytest.ini file with a brief explanation.
• Creation of test data should be performed in a conftest.py file using a Pytest fixture instead of being performed in each test. This allows code reuse (allowing multiple tests to be performed on the same fixture) and also separates the creation of test data from the function of the test itself, improving modularity and maintainability of test code.

Guidelines for component-level tests are as follows.

• All test files begin with test_, followed by the rest of the file name. For example, the test file for basis.py would be test_basis.py.
• The test files in directory test/component/ should look identical to the src/ directory. For example, to test something in src/numerics/basis/basis.py, this test should be placed in test/component/numerics/basis/test_basis.py.

Guidelines for end-to-end tests are as follows.

• Each test case directory must be added to list_of_cases.py along with a list of markers for the test case.
• Each test case directory should contain only one input file, and it should be named input_file.py.
• When a new test case is added or existing cases are modified, regression_data_generator.py must be run to regenerate the regression test data, and generate_tests.py must be rerun to recreate the test scripts.
• When switching between branches it can be helpful to clean the test directories. This can be done by running clear_tests.py.
• If a future update to Quail changes the solution in any existing test case such that it does not pass the regression test, it is likely that a bug may have been introduced to the code (this is how new bugs are caught by regression tests). However, if you make a change to the code that changes solutions and you are comfortable with this change (for example, fixing a bug in Quail) then the regression test data should be updated by rerunning regression_data_generator.py.