|
|
# Introduction
|
|
|
Results reproducibility is a critical issue in science. It has been already noted that in many cases, reproducing your results even after a few months (typical time scale of referee process) may be challenging. In most cases, it is not sufficient to have the same version of the code, but you also need precise knowledge about the input parameters used and the same input data must be provided. Since the standard methodology in science is based on try and fail methodology, typically, the researcher ends up with many datasets. Only a few of them are released to publication finally, while others serve as experimental runs. Under such conditions tracking of changes introduced to codes in the research process becomes problematic. W-SLDA implements a methodology that does it automatically and allows for the reproduction of the results (up to machine precision). Namely, the generated **results** are always accompanied by the **reproducibility pack**, where complete information needed to reproduce them is included.
|
|
|
Results reproducibility is a critical issue in science. It has already been noted that in many cases, reproducing your results even after a few months (the typical time scale of the referee process) may be challenging. In most cases, having the same version of the code is insufficient, but you also need precise knowledge about the input parameters used, and the same input data must be provided. Since the standard methodology in science is based on try-and-fail methodology, typically, the researcher ends up with many datasets. Only a few of them are released for publication, while others serve as experimental runs. Under such conditions, tracking changes introduced to codes in the research process becomes problematic. W-SLDA implements a methodology that does it automatically and allows for the reproduction of the results (up to machine precision). Namely, the generated **results** are always accompanied by the **reproducibility pack**, where complete information needed to reproduce them is included.
|
|
|
![reproducibility](uploads/8ee07ac131aa636f0b6e041cc0948cac/reproducibility.png)
|
|
|
|
|
|
For meaning of each file see [here](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/wikis/Output%20files).
|
|
|
|
|
|
# W-SLDA mechanism of results reproducibility
|
|
|
Developers of W-SLDA Toolkit recognize the need for intrinsically implemented support that will simplify the process of reproducing of the results. To comply with this requirement following mechanism has been implemented (called as reproducibility pack):
|
|
|
1. Each file generated by W-SLDA Toolkit in the header provides basic info about the code version that has been used, for example, the header of `wlog` file may look like:
|
|
|
Developers of W-SLDA Toolkit recognize the need for intrinsically implemented support that will simplify the process of reproducing the results. To comply with this requirement, the following mechanism has been implemented (called a reproducibility pack):
|
|
|
1. Each file generated by W-SLDA Toolkit in the header provides basic info about the code version that has been used; for example, the header of `wlog` file may look like:
|
|
|
```
|
|
|
# CREATION TIME OF THE LOG: Sun Feb 7 15:29:44 2021
|
|
|
# EXECUTION COMMAND : ./st-wslda-2d input.txt
|
... | ... | @@ -28,10 +30,10 @@ test_reprowf.tar # reproducibility pack for restoring wave-functions t |
|
|
This provides the full information required to reproduce your results (up to machine precision).
|
|
|
|
|
|
# Good practices
|
|
|
1. For each project use a separate folder, do not mix results from various projects in the same folder. Use a meaningful name for folder.
|
|
|
1. For each project, use a separate folder; do not mix results from various projects in the same folder. Use a meaningful name for the folder.
|
|
|
2. Use meaningful `outprefix` names.
|
|
|
3. Do not modify output files, except `wtxt` file. This one is designed to store various metadata information, including your comments. `wtxt` file is easy to reproduce in case if you destroy it accidentally, which is not the case with other files. Add your comments/remarks/etc in form of comments starting with `#`.
|
|
|
4. When copying results to a new location/machine, copy all files assisted with the run. The simplest way is to execute the command (for more info see [here](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/wikis/Output%20files#copying-results-to-a-new-location)):
|
|
|
3. Do not modify output files, except `wtxt` file. This one is designed to store various metadata information, including your comments. The `wtxt` file is easy to reproduce if you destroy it accidentally, which is not the case with other files. Add your comments/remarks/etc in the form of comments starting with `#`.
|
|
|
4. When copying results to a new location/machine, copy all files assisted with the run. The simplest way is to execute the command (for more info, see [here](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/wikis/Output%20files#copying-results-to-a-new-location)):
|
|
|
```bash
|
|
|
scp outprefix* new_location
|
|
|
```
|
... | ... | @@ -43,7 +45,7 @@ void wprintf( const char * format, ... ); |
|
|
// prints to stream (like stdout or stderr) and to file outprefix.stdout
|
|
|
void wfprintf(FILE *stream, const char * format, ... );
|
|
|
```
|
|
|
These are analogs of [printf](http://www.cplusplus.com/reference/cstdio/printf/) and [fprintf](http://www.cplusplus.com/reference/cstdio/fprintf/) with the difference that the message will be added also to `outprefix.stdout`.
|
|
|
These are analogs of [printf](http://www.cplusplus.com/reference/cstdio/printf/) and [fprintf](http://www.cplusplus.com/reference/cstdio/fprintf/) with the difference that the message will also be added to `outprefix.stdout`.
|
|
|
|
|
|
To learn more about good practices related to results reproducibility issue see:
|
|
|
To learn more about good practices related to results reproducibility issues see:
|
|
|
* [Creating Reproducible Data Science Projects](https://towardsdatascience.com/creating-reproducible-data-science-projects-1fa446369386) |
|
|
\ No newline at end of file |