Skip to content

GitLab

Last edited by Gabriel Wlazłowski
Page history

Testsuite

Version>=2021.05.25

W-SLDA Toolkit has implemented an automated tests system. Each version is a subject extensive testing process before release. The latest version (2022.01.05) was subject of more than 250 tests:

--- Summary ---
TESTS: 258
OK: 258
FAIL: 0

Documentation below describes how to define and execute the tests - for developers.

WSLDA test system instruction

The documentation contains instructions, on how to prepare the tests used for the W-SLDA project purposes. The testing system performs all given tests according to the order set by the user, and then generates a report based on the results of the simulations.

Tests can be prepared on the local machine, yet due to requirements of working memory, to be run actual calculations, access to the computing cluster is highly recommended.

The W-SLDA project relies on the wslda repository, which includes the testsuite directory. Testsuite's directory is a working folder and all tests should be placed and executed there.

Preparation of the test list:

  1. In the path testsuite prepare a text file named: tlist.txt.
  2. The tlist.txt file contains information about the directories with tests that User intends to perform.
  3. Comments can be placed in the tlist.txt file but each comment is preceded by a hash sign (#). Hash sign is MANDATORY.
  4. Name of the directory with tests is elective, yet must be a string. Do not use any special character other than dot ".", dash "-" or underscore "_".
  5. Each subsequent name of the directory must start from a new line in the tlist.txt. One line for one directory is MANDATORY.

Sample "tlist.txt" file format:

# User: John
# Date: 01/01/1970

# A directory with tests below is the first directory to process.
first_dir
# Here is the next group of tests to be run.
second_dir
# Directory below will be processed as well:
third_dir # despite the comment left after the name of the test


fourth_dirrr # this directory will also run despite a comment or two blank lines left above
# dir_fifth # these tests will NOT run due to the comment hash sign (#)
# dir_six will fail either

end # this will be considered as a group of tests to run
#end # this will not be considered as a group of tests to run

Preparation of the directory with tests:

  1. When tlist.txt file is ready, User needs to prepare directories with tests in the testsuite directory.
  2. Each subsequent directory may contain unlimited amount of tests.
  3. Each directory name must match the name of the directory given in tlist.txt.
  4. Each directory needs to contain reference file with expected values.
  5. Each dir must contain a text file test.desc. It contains a list of tests and Unix commands to be run upon them. The format of the text file test.desc is shown below.
  6. Test.desc file contains assigned flags depending on the purpose. All flags are MANDATORY:
    • tag - refers to the name of test (also an information for an User)
    • exec - command to be executed
    • test - name of the output file for each test
    • diff - test is over and ready to compare output file values with the reference values.
  7. Test.desc file needs no ending. Placing comments or blank lines are optional.

Example of test.desc file:

tag: test1
exec: make 1d
test: test1.cmp
diff: test1.cmp ref.test1

Start testing:

  1. Switch directory to: testsuite
  2. Make sure that all directories are included, according to the tlist.txt file.
  3. Make sure that all directories contain the necessary files, including description file test.desc and reference file.
  4. If everything has been prepared correctly, run the test script with the command: ./testsuiteRun.sh.
  5. After the script is done, a text report will be received in the format: user_data_hour, so that every user could recognize his test.

Obtaining report:

  1. The report will be performed only at the very end of the script operation. Interrupting the program will provide no results.
  2. The report will be generated regardless of the correctness of the test execution.
  3. The report is in the format of a 5-column text file: | Folder | Tag | Make | Run | Check |:
    • Folder - name of dir
    • Tag - name of test
    • Make - string value: OK or FAIL. OK if W-SLDA toolkit worked successfully.
    • Run - string value: OK or FAIL. 'OK' if output file from test.desc exists.
    • Check - string value: OK or FAIL. 'OK' when ALL output values compared with reference values are below the tolerance specified in the reference file.

FAIL value will be obtained if otherwise (for each flag respectively).

After receiving report, it is recommended to copy it to your local device and remove it from the testsuite directory.

Contribution

The Testsuite was implemented by:

  • Maksymilian Odziemczyk, Warsaw University of Technology, Faculty of Physics, Computational Physics Research Interest Group
  • Bartosz Ruszczak, Warsaw University of Technology, Faculty of Physics, Computational Physics Research Interest Group