|
# Introduction
|
|
# Introduction
|
|
Results reproducibility is a very important issue in science. It has been already noted that in many cases reproducing your own results even after a few months (typical time scale of referee process) may be challenging. It is because in most cases it is not sufficient to have the same version of the code, but you also need precise knowledge about input parameters that were used.
|
|
Results reproducibility is a very important issue in science. It has been already noted that in many cases reproducing your own results even after a few months (typical time scale of referee process) may be challenging. It is because in most cases it is not sufficient to have the same version of the code, but you also need precise knowledge about input parameters that were used.
|
|
Since the standard methodology in science is based on _try and fail_ methodology, typically at the end we end up with many datasets, and only a few of them is released to publication finally, while others serve as _experimental runs_. W-SLDA implements a methodology that allows for results reproduction. Namely, the generated **results** are always accompanied by the **reproducibility pack**:
|
|
Since the standard methodology in science is based on _try and fail_ methodology, typically at the end we end up with many datasets, and only a few of them is released to publication finally, while others serve as _experimental runs_. Then, tracking of settings that were used for various runs becomes a problem. W-SLDA implements an automatic framework that allows for results reproduction. Namely, the generated **results** are always accompanied by the **reproducibility pack**:
|
|
![reproducibility](uploads/8ee07ac131aa636f0b6e041cc0948cac/reproducibility.png)
|
|
![reproducibility](uploads/8ee07ac131aa636f0b6e041cc0948cac/reproducibility.png)
|
|
|
|
|
|
# W-SLDA mechanism of results reproducibility
|
|
# W-SLDA mechanism of results reproducibility
|
|
Developers of W-SLDA Toolkit recognize the need for intrinsically implemented support that will simplify the process of reproducing of the results. To comply with this requirement following mechanism has been implemented (called as reproducibility pack):
|
|
Developers of W-SLDA Toolkit recognize the need for intrinsically implemented support that will simplify the process of reproducing of the results. To comply with this requirement following mechanism has been implemented (called as reproducibility pack):
|
|
1. Each file generated by W-SLDA Toolkit in the header provides basic info about the code version that has been used, for example, header of `wlog` file may look like:
|
|
1. Each file generated by W-SLDA Toolkit in the header provides basic info about the code version that has been used, for example, the header of `wlog` file may look like:
|
|
```
|
|
```
|
|
# CREATION TIME OF THE LOG: Sun Feb 7 15:29:44 2021
|
|
# CREATION TIME OF THE LOG: Sun Feb 7 15:29:44 2021
|
|
# EXECUTION COMMAND : ./st-wslda-2d input.txt
|
|
# EXECUTION COMMAND : ./st-wslda-2d input.txt
|
... | @@ -13,23 +13,24 @@ Developers of W-SLDA Toolkit recognize the need for intrinsically implemented su |
... | @@ -13,23 +13,24 @@ Developers of W-SLDA Toolkit recognize the need for intrinsically implemented su |
|
# VERSION OF THE CODE : 2021.01.27
|
|
# VERSION OF THE CODE : 2021.01.27
|
|
# COMPILATION DATE & TIME : Feb 7 2021, 15:19:57
|
|
# COMPILATION DATE & TIME : Feb 7 2021, 15:19:57
|
|
```
|
|
```
|
|
2. When executing the code, all user-definable files are recreated and attached to the data set. For example, if the user set `outprefix` as `test`, then among output files there will be:
|
|
|
|
|
|
2. When executing the code, all user-definable files are recreated and attached to the output-set. For example, if the user set `outprefix` as `test`, then among output files there will be:
|
|
```bash
|
|
```bash
|
|
test_input.txt # input file used for calculations
|
|
test_input.txt # input file used for calculations
|
|
test_predefines.h # predefines selected at compilation stage
|
|
test_predefines.h # predefines selected at compilation stage
|
|
test_problem-definition.h # user's definition of the problem
|
|
test_problem-definition.h # user's definition of the problem
|
|
test_logger.h # user's logger
|
|
test_logger.h # user's logger
|
|
test.stdout # standard output generated by the code
|
|
test.stdout # standard output generated by the code
|
|
test_checkpoint.dat.init # checkpoint point that was used as input (st codes only)
|
|
test_checkpoint.dat.init # checkpoint file that was used as input (st codes only)
|
|
test_extra_data.dat # Binary file with the extra_data array (if provided)
|
|
test_extra_data.dat # Binary file with the extra_data array (if provided)
|
|
test_reprowf.tar # reproducibility pack for restoring wave-functions that were used as input (td codes only)
|
|
test_reprowf.tar # reproducibility pack for restoring wave-functions that were used as input (td codes only)
|
|
```
|
|
```
|
|
This provides full information (apart from any external data added by user) required to reproduce your results.
|
|
This provides the full information required to reproduce your results (up to machine precision).
|
|
|
|
|
|
# Good practices
|
|
# Good practices
|
|
1. For each project use a separate folder, do not mix results from various project in the same files. Use a meaningful name for folders.
|
|
1. For each project use a separate folder, do not mix results from various projects in the same folder. Use a meaningful name for folder.
|
|
2. Use meaningful `outprefix` names.
|
|
2. Use meaningful `outprefix` names.
|
|
3. Do not modify output files, except `wtxt` file. This one is designed to store various metadata information, including your comments. `wtxt` file is easy to reproduce in case if you destroy it accidentally, which is not the case of other files. Add your comments/remarks/etc in form of comments starting with `#`.
|
|
3. Do not modify output files, except `wtxt` file. This one is designed to store various metadata information, including your comments. `wtxt` file is easy to reproduce in case if you destroy it accidentally, which is not the case with other files. Add your comments/remarks/etc in form of comments starting with `#`.
|
|
4. When copying results to a new location/machine, copy all files assisted with the run. The simplest way is to execute the command (for more info see [here](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/wikis/Output%20files#copying-results-to-a-new-location)):
|
|
4. When copying results to a new location/machine, copy all files assisted with the run. The simplest way is to execute the command (for more info see [here](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/wikis/Output%20files#copying-results-to-a-new-location)):
|
|
```bash
|
|
```bash
|
|
scp outprefix* new_location
|
|
scp outprefix* new_location
|
... | | ... | |