Skip to content

GitLab

Results reproducibility · Changes

Page history
Update Results reproducibility authored by Gabriel Wlazłowski's avatar Gabriel Wlazłowski
Show whitespace changes
Inline Side-by-side
Results-reproducibility.md
View page @ 819e7994
[[_TOC_]]
# Introduction
Results reproducibility is a critical issue in science. It has already been noted that in many cases, reproducing your results even after a few months (the typical time scale of the referee process) may be challenging. In most cases, having the same code version is insufficient, but you also need precise knowledge about the input parameters used, and the same input data must be provided. Since the standard methodology in science is based on try-and-fail methods, typically, the researcher ends up with many datasets. Only a few are released for publication, while others serve as experimental runs. Under such conditions, tracking changes introduced to codes in the research process becomes problematic. The W-SLDA Toolkit implements a methodology that does it automatically and allows for reproducing the results (up to machine precision). Namely, the generated **results** are always accompanied by the **reproducibility pack**, where complete information needed to reproduce them is included.
Reproducibility of results is a critical issue in science. It has already been noted that in many cases, reproducing your results even after a few months (the typical time scale of the referee process) may be challenging. In most cases, having the same code version is insufficient; you also need precise knowledge of the input parameters used, and the same input data must be provided. Since the standard methodology in science is based on trial-and-error methods, the researcher typically ends up with many datasets. Only a few are released for publication, while others serve as experimental runs. Under such conditions, tracking changes introduced to code in the research process becomes problematic. The W-SLDA Toolkit implements a methodology that does it automatically and allows for reproducing the results (up to machine precision). Namely, the generated **results** are always accompanied by the **reproducibility pack**, which includes all the information needed to reproduce them.
![reproducibility](uploads/8ee07ac131aa636f0b6e041cc0948cac/reproducibility.png)
For the meaning of each file, see [here](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/wikis/Output%20files).
For the meaning of each file, see [here](Output-files).
# W-SLDA mechanism of results reproducibility
Developers of the W-SLDA Toolkit recognize the need for intrinsically implemented support that will simplify the process of reproducing the results. To comply with this requirement, the following mechanism has been implemented (called a reproducibility pack):
Developers of the W-SLDA Toolkit recognize the need for intrinsic support that simplifies reproducing results. To comply with this requirement, the following mechanism has been implemented (called a reproducibility pack):
1. Each file generated by the W-SLDA Toolkit in the header provides basic info about the code version that has been used; for example, the header of `wlog` file may look like:
```
# CREATION TIME OF THE LOG: Sun Feb 7 15:29:44 2021
......@@ -16,7 +16,7 @@ Developers of the W-SLDA Toolkit recognize the need for intrinsically implemente
# COMPILATION DATE & TIME : Feb 7 2021, 15:19:57
```
2. When executing the code, all user-definable files are recreated and attached to the output-set. For example, if the user sets `outprefix` as `test`, then among output files, there will be:
2. When executing the code, all user-definable files are recreated and attached to the output set. For example, if the user sets `outprefix` as `test`, then among the output files, there will be:
```bash
test_input.txt # input file used for calculations
test_predefines.h # predefines selected at compilation stage
......@@ -28,17 +28,17 @@ test_checkpoint.dat.init # checkpoint file that was used as input (st codes on
test_extra_data.dat # Binary file with the extra_data array (if provided)
test_reprowf.tar # reproducibility pack for restoring wave functions that were used as input (td codes only)
```
This provides the complete information required to reproduce your results (up to machine precision).
This provides all the information required to reproduce your results (to machine precision).
# Good practices
1. For each project, use a separate folder; do not mix results from various projects in the same folder. Use a meaningful name for the folder.
2. Use meaningful `outprefix` names.
3. Do not modify output files, except `wtxt` file. This one is designed to store various metadata information, including your comments. The `wtxt` file is easy to reproduce if you destroy it accidentally, which is not the case with other files. Add your comments/remarks/etc in the form of comments starting with `#`.
4. When copying results to a new location/machine, copy all files assisted with the run. The simplest way is to execute the command (for more info, see [here](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/wikis/Output%20files#copying-results-to-a-new-location)):
3. Do not modify output files, except `wtxt` file. This one is designed to store various metadata information, including your comments. The `wtxt` file is easy to reproduce if you accidentally delete it, unlike other files. Add your comments/remarks/etc in the form of comments starting with `#`.
4. When copying results to a new location/machine, copy all files associated with the run. The simplest way is to execute the command (for more info, see [here](Output-files#copying-results-to-a-new-location)):
```bash
scp outprefix* new_location
```
5. When printing messages to `stdout` use functions:
5. When printing messages to `stdout`, use functions:
```c
// prints to stdout and to file outprefix.stdout
void wprintf( const char * format, ... );
......@@ -46,7 +46,7 @@ void wprintf( const char * format, ... );
// prints to stream (like stdout or stderr) and to file outprefix.stdout
void wfprintf(FILE *stream, const char * format, ... );
```
These are analogs of [printf](http://www.cplusplus.com/reference/cstdio/printf/) and [fprintf](http://www.cplusplus.com/reference/cstdio/fprintf/) with the difference that the message will also be added to `outprefix.stdout`.
These are analogs of [printf](http://www.cplusplus.com/reference/cstdio/printf/) and [fprintf](http://www.cplusplus.com/reference/cstdio/fprintf/), with the difference that the message will also be added to `outprefix.stdout`.
To learn more about good practices related to results reproducibility issues, see:
* [Build a Reproducible and Maintainable Data Science Project](https://khuyentran1401.github.io/reproducible-data-science/README.html)
......@@ -59,10 +59,10 @@ Check one of the provided `*.h` files to determine the version of code that shou
Use template folders (`st-project-template` or `td-project-template`) to create an empty project.
3. _Overwrite user-defined files_:
Overwrite the user-defined files `*.h` and the input file by those provided in the reproducibility pack.
Overwrite the user-defined files `*.h` and the input file with those provided in the reproducibility pack.
# List of reproducibility packs
Since 2022, reproducibility packs have been provided for all our papers where new results generated by the W-SLDA are presented. For the list of our publications and links to the reproducibility packs, see [here](https://wslda.fizyka.pw.edu.pl/index.php/Gallery).
Since 2022, reproducibility packs have been provided for all our papers that present new results generated by the W-SLDA. For the list of our publications and links to the reproducibility packs, see [here](https://wslda.fizyka.pw.edu.pl/index.php/Gallery).
# Known issues with existing reproducibility packs
## New J. Phys. 25, 033013 (2023)
......