Skip to content

GitLab

Installing the toolkit · Changes

Page history
Update Installing the toolkit authored by Gabriel Wlazłowski's avatar Gabriel Wlazłowski
Show whitespace changes
Inline Side-by-side
Installing-the-toolkit.md
View page @ 8fe93ce2
[[_TOC_]] [[_TOC_]]
# Requirements # Requirements
## Static codes (st-wslda)
* C compiler (eg. `gcc`, intel compiler `icc`, ...)
* MPI C compiler (eg. `mpicc`, intel compiler `mpiicc`, ...)
* python
* [FFTW library](http://www.fftw.org/)
* LAPACK library
* ScaLAPACK library
* [ELPA library](https://elpa.mpcdf.mpg.de/) (optionally)
* [CUDA compiler](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html) (optionally, for ELPA)
*Note 1*: compilers and libraries (FFTW, LAPACK, ScaLAPACK) are installed by default on most of the computing clusters. If the computing system is equipped with GPUs, it is likely that the CUDA Toolkit is also installed. On many systems, ELPA is also installed by default. If not, you can compile it yourself. ## Static codes (`st-wslda`)
*Note 2*: LAPACK and ScaLAPACK libraries are part of Intel® Math Kernel Library. - C compiler (e.g., `gcc`, Intel compiler `icc`, etc.)
- MPI C compiler (e.g., `mpicc`, Intel compiler `mpiicc`, etc.)
- Python
- [FFTW library](http://www.fftw.org/)
- LAPACK library
- ScaLAPACK library
- [ELPA library](https://elpa.mpcdf.mpg.de/) (optional)
- [CUDA
compiler](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html)
(optional, for ELPA)
*Note 3*: It is recommended to use `st-wslda` with ELPA support if possible. It provides a substantial speed-up of the computation process. See [here](ELPA installation guide) for more instructions related to installing ELPA. **Note 1:** Compilers and core libraries (FFTW, LAPACK, ScaLAPACK) are
typically installed by default on most computing clusters. If the system
is equipped with GPUs, the CUDA Toolkit is usually available as well.
ELPA is also installed on many systems; if not, it can be compiled
manually.
## Time-dependent codes (td-wslda) **Note 2:** LAPACK and ScaLAPACK are included in the Intel® Math Kernel
* C compiler (eg. `gcc`, intel compiler `icc`, ...) Library (MKL).
* MPI C compiler (eg. `mpicc`, intel compiler `mpiicc`, ...)
* [CUDA compiler](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html)
* python
*Note 1*: Time-dependent codes **require** systems accelerated by GPUs. `td-wslda` codes cannot run on standard systems based on CPUs only. **Note 3:** It is strongly recommended to use `st-wslda` with ELPA
support whenever possible, as it significantly accelerates
diagonalization. See the [ELPA installation
guide](ELPA-installation-guide) for details.
# Downloading the toolkit ## Time-dependent codes (`td-wslda`)
It is recommended to download the toolkit directly from the repository using git
```bash - C compiler (e.g., `gcc`, Intel compiler `icc`, etc.)
- MPI C compiler (e.g., `mpicc`, Intel compiler `mpiicc`, etc.)
- [CUDA
compiler](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html)
- Python
**Note 1:** Time-dependent codes **require GPU-accelerated systems**.
`td-wslda` cannot run on CPU-only systems.
# Downloading the Toolkit
The recommended way to obtain the toolkit is via `git`:
``` bash
git clone https://gitlab.fizyka.pw.edu.pl/wtools/wslda.git git clone https://gitlab.fizyka.pw.edu.pl/wtools/wslda.git
``` ```
Optionally, you can download it manually from [main webpage](https://gitlab.fizyka.pw.edu.pl/wtools/wslda).
The toolkit is contained in the `wslda` folder; see [here](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/wikis/Folders-structure) for more info about the content.
# Compiling standard libraries and tools (Optionally) Alternatively, you may use mirror repositories:\
The toolkit has various [libs](https://gitlab.fizyka.pw.edu.pl/wtools) tools supporting the computation process. See [here](Auxiliary tools) for available tools. While it is not mandatory, we recommend compiling them during the installation process. You can do it by executing the provided script in the main folder: [GitLab](https://gitlab.com/coldatoms/wslda) or
[GitHub](https://gitlab.com/coldatoms/wslda).
The toolkit resides in the `wslda` directory. See the [folder structure
description](Folders-structure)
for details.
After cloning, define the global environment variable:
``` bash
export WSLDA=path_to_wslda_folder
``` ```
It is recommended to place this command in your shell configuration file
(e.g., `.bashrc`).
# Compiling Standard Libraries and Tools (Optional)
The toolkit includes several auxiliary [libraries and
tools](https://gitlab.fizyka.pw.edu.pl/wtools) that support computation
and data processing. See [Auxiliary tools](Auxiliary-tools) for
details. Although optional, compiling them during installation is
recommended.
To compile, ensure that both a C compiler and an MPI C compiler are
available. Specify them in the header of `install-libs.sh`:
``` bash
export CC="gcc -std=gnu99"
export MPICPP="mpic++ -std=gnu99"
```
Some libraries require FFTW. Ensure it is installed before compilation.
To build all libraries:
``` bash
./install-libs.sh ./install-libs.sh
``` ```
Once the compilation is finished, you will get instructions on how to set system variables in order to be able to use the W-SLDA Toolkit, for example
After completion, the script provides instructions for setting
environment variables, for example:
``` bash
export WSLDA=/path/to/wslda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/wslda/lib/wdata:/path/to/wslda/lib/wderiv:/path/to/wslda/lib/winterp:/path/to/wslda/lib/wbox
export PATH=$PATH:/path/to/wslda/lib/wdata/bin:/path/to/wslda/tools/bin
``` ```
gabrielw@e1135:~/grant_518/project_data/wslda$ ./install-libs.sh
Installing libs...
# Making lib/wderiv..
# ...
make: Leaving directory `/mnt/storage_2/project_data/grant_518/cold-atoms/tools'
# ==========================================================================
# =========================== SET ENVIRONMENT ==============================
# ==========================================================================
# Add to your .bashrc
export WSLDA=/mnt/storage_2/project_data/grant_518/wslda Share these `export` statements with other users if the toolkit is
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/mnt/storage_2/project_data/grant_518/wslda/lib/wdata:/mnt/storage_2/project_data/grant_518/wslda/lib/wderiv:/mnt/storage_2/project_data/grant_518/wslda/lib/winterp installed in a shared location. Ensure that the `wslda` directory is
export PATH=$PATH:/mnt/storage_2/project_data/grant_518/wslda/lib/wdata/bin:/mnt/storage_2/project_data/grant_518/wslda/tools/bin readable by all intended users.
Each library is compiled via `make` in its respective directory. For
example:
``` bash
make -C lib/wdata
``` ```
Share `export` statements with other toolkit users, so they can also use it. You need to make `wslda` folder readable for other members of the group. If compilation fails, enter the relevant directory and adjust the
`Makefile` accordingly.
# Setting up templates # Setting Up Templates
W-SLDA exploits a working model based on templates.
The user starts with a project template where items like external potential, pairing potential, and external velocity must be provided in C language. Templates are localized in folders:
* [st-project-template](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/st-project-template) for static problems (`st-wslda` codes)
* [td-project-template](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/td-project-template) for time-dependent problems (`td-wslda` codes)
Within these folders, `Makefile` and `machine.h` files compatible with your computing system must be provided. W-SLDA follows a template-based workflow.
In the [templates](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/templates) folder, examples of machine-dependent files for various systems are provided. The systems are listed in [templates/README.md](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/templates/README.md). Templates are located in:
To copy templates for the selected machine, you can use the script [install-templates.py](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/blob/public/install-templates.py), for example: - [`st-project-template`](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/st-project-template)
```bash -- static problems (st)
[gabrielw@dell cold-atoms]$ ./install-templates.py eagle - [`td-project-template`](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/td-project-template)
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/st/* /home/gabrielw/MyProjects/cold-atoms/st-project-template -- time-dependent problems (td)
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/st-project-template
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/st/* /home/gabrielw/MyProjects/cold-atoms/st-testcase-uniform A template folder should contain:
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/st-testcase-uniform | Name | Requirement | Dependency| Description |
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/td/* /home/gabrielw/MyProjects/cold-atoms/td-project-template |----- | ----------- | --------- | ----------- |
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/td-project-template | `Makefile` | Mandatory | Machine-dependent | Compiler settings and flags |
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/td/* /home/gabrielw/MyProjects/cold-atoms/td-testcase-uniform | `machine.h` | Mandatory| Machine-dependent | Machine configuration (diagonalization engine, GPUs per node, etc.) |
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/td-testcase-uniform | `README.md` | Optional | Machine-dependent | User instructions |
Done. | `env.sh` | Optional | Machine-dependent | Environment setup script |
``` | `job.sh` | Optional | Machine-dependent | Example submission script |
| `input.txt` | Mandatory | Problem-dependent | Default input file |
# Testing the correctness of templates | `predefines.h` | Mandatory | Problem-dependent | Compiler predefinitions |
## Testing of compilation | `problem-definition.h` |Mandatory | Problem-dependent | Problem definition |
Copy the project folder to a separate location (in the example below, I use `tmp` folder) | `logger.h` | Mandatory | Problem-dependent | Logging configuration |
```bash
At minimum, `Makefile` and `machine.h` must be adapted to your system.
Machine-specific examples are available in the
[`templates`](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/templates)
directory.
If your system is listed, use:
``` bash
./install-templates.py system_name
```
This script copies appropriate files into the template directories.
# Testing Templates and Execution
## Compilation Test
Copy a test project to a temporary directory:
``` bash
cd ~/tmp cd ~/tmp
cp -r $WSLDA/st-testcase-uniform/ . cp -r $WSLDA/st-testcase-uniform/ .
cd st-testcase-uniform/ cd st-testcase-uniform/
make make
``` ```
If the process finishes successfully, executables `st-wslda-?d` will be created. The analogous operation can be applied to `td` part of the toolkit.
*Note 1*: The ELPA Library must be in addition activated in [machine.h](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/blob/public/st-project-template/machine.h) file. To learn more about ELPA see [here](Setting up diagonalization engine). Successful compilation produces executables such as `st-wslda-?d`, where `?` is `1`, `2`, or `3` and indicates the code dimensionality.
*Note 2*: `st-wslda` codes utilize LAPACK and ScaLAPACK libraries. The standard method of calling Fortran functions from C requires supplementing routine names by `_` at the end, i.e, Fortran routine `DGEMM` is called from C as `dgemm_`. However, in cases of some systems/compilers (very rare), it is not needed. If this is the case with your system, then you need to add to `CFLAGS` flag **If compilation fails**: Check environment variables. Verify
`-DFOTRAN_NO_UNDERSCORE`. `Makefile` and `machine.h`
*Note 3*: In some cases, additional information about the architecture of the computing system must be provided in [machine.h](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/blob/public/st-project-template/machine.h) file. For more information, see [here](Configuring GPU machine). **Note 1:** Enable ELPA in `machine.h` if available.
**Note 2:** Some systems require `-DFORTRAN_NO_UNDERSCORE` in `CFLAGS`.
**Note 3:** Additional architecture-specific settings may be required in `machine.h`.
*Recommendation*: Typically, (super)computers utilize the `module load` system to configure the programming environment. We recommend placing a list of modules in a separate file, for example, `env.sh`: It is recommended to use a dedicated `env.sh` file for module loading:
```bash
module load daint-gpu
module swap PrgEnv-cray PrgEnv-gnu/6.0.9
module load cudatoolkit/11.2.0_3.39-2.1__gf93aa1c
module load craype-accel-nvidia60
module load cray-fftw
export LD_LIBRARY_PATH=/project/pr125/share/elpa-2020.11.001/lib:$LD_LIBRARY_PATH
``` bash
module load ...
export LD_LIBRARY_PATH=...
module list module list
cat $WSLDA/VERSION.h
``` ```
```
Then, the compilation process looks like this: Then compile with:
```bash
``` bash
source ./env.sh source ./env.sh
make make
``` ```
## Testing of execution Repeat the procedure analogously for `td` templates.
Execute the binary codes. Below is an example of a test for static 2d code:
```bash ## Execution Test
[gabrielw@dell st-testcase-uniform]$ mpirun -np 4 ./st-wslda-2d input.txt
The toolkit provides two reference test cases that verify both
correctness of implementation and proper configuration of the
computational environment.
- `st-testcase-uniform`: This test solves the *uniform system* in the static (self-consistent)
mode. The self-consistent loop is initialized using analytical expressions
derived from BCS theory for a uniform Fermi gas. The
initial guess corresponds to the exact solution of the uniform problem.
Since the initial state is already the correct solution, the self-consistent procedure should converge immediately.
This means that only a single iteration should be performed, and the code should conclude with `ALGORITHM CONVERGED`.
- `td-testcase-uniform`: This test evolves in time the same *uniform solution*, but now using the time-dependent solver.
The initial state provided to the code is a stationary solution of the
time-dependent equations. Therefore, it should remain unchanged during time evolution. All quantities (e.g., total energy, contributions to the energy, particle number) must remain constant within numerical precision. Observables printed to the screen and to the files should not exhibit time-dependence.
Together, these two tests verify: 1. Correct static self-consistency
implementation. 2. Proper time-dependent propagation. 3. Consistency
between static and time-dependent solvers. 4. Correct linkage of
numerical libraries and GPU backends.
## Example execution
``` bash
mpirun -np 4 ./st-wslda-2d input.txt
``` ```
The complete testing process may look like this: Full test workflow:
```bash
# Testing static part ``` bash
[gabrielw@node2067 tmp]$ cd ~/tmp # Static
[gabrielw@node2067 tmp]$ cp -r $WSLDA/st-testcase-uniform/ . cd st-testcase-uniform
[gabrielw@node2067 tmp]$ cd st-testcase-uniform source env.sh
[gabrielw@node2067 st-testcase-uniform]$ source env.sh make
[gabrielw@node2067 st-testcase-uniform]$ make mpirun -np 4 ./st-wslda-1d input.txt > st-test-1d.txt
[gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-1d input.txt > st-test-1d.txt mpirun -np 4 ./st-wslda-2d input.txt > st-test-2d.txt
[gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-2d input.txt > st-test-2d.txt mpirun -np 4 ./st-wslda-3d input.txt > st-test-3d.txt
[gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-3d input.txt > st-test-3d.txt
cd td-testcase-uniform
# Testing time-dependent part source env.sh
[gabrielw@node2067 tmp]$ cd ~/tmp make
[gabrielw@node2067 tmp]$ cp -r $WSLDA/td-testcase-uniform/ . mpirun -np 4 ./td-wslda-1d input.txt > td-test-1d.txt
[gabrielw@node2067 tmp]$ cd td-testcase-uniform mpirun -np 4 ./td-wslda-2d input.txt > td-test-2d.txt
[gabrielw@node2067 td-testcase-uniform]$ source env.sh mpirun -np 4 ./td-wslda-3d input.txt > td-test-3d.txt
[gabrielw@node2067 td-testcase-uniform]$ make ```
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-1d input.txt > td-test-1d.txt
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-2d input.txt > td-test-2d.txt Examples of outputs are (note they may change with the version change):
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-3d input.txt > td-test-3d.txt
```
The outputs are:
* [st-test-1d.txt](uploads/cd8f4dc231b76388f3ea51ec2b69516e/st-test-1d.txt) * [st-test-1d.txt](uploads/cd8f4dc231b76388f3ea51ec2b69516e/st-test-1d.txt)
* [st-test-2d.txt](uploads/c6251282595915c06aebf6c8b4a4d665/st-test-2d.txt) * [st-test-2d.txt](uploads/c6251282595915c06aebf6c8b4a4d665/st-test-2d.txt)
* [st-test-3d.txt](uploads/dfe7441168f3bbac0737d1d6d73da0db/st-test-3d.txt) * [st-test-3d.txt](uploads/dfe7441168f3bbac0737d1d6d73da0db/st-test-3d.txt)
...@@ -156,28 +243,62 @@ The outputs are: ...@@ -156,28 +243,62 @@ The outputs are:
* [td-test-2d.txt](uploads/b91aef4995b862b28505ef2a4f9cf7da/td-test-2d.txt) * [td-test-2d.txt](uploads/b91aef4995b862b28505ef2a4f9cf7da/td-test-2d.txt)
* [td-test-3d.txt](uploads/f4fc567667c3d510377a0b4d2e803fba/td-test-3d.txt) * [td-test-3d.txt](uploads/f4fc567667c3d510377a0b4d2e803fba/td-test-3d.txt)
# Compiling visit plugin (optional) # Creating Machine Templates
After successful testing, create a machine-specific template:
Visit plugin is located in [lib/wdata/visit-plugin](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/lib/wdata/visit-plugin). ``` bash
Follow instructions from [README.txt](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/lib/wdata/visit-plugin/README.txt) file to install it. mkdir -p $WSLDA/templates/computer_name/st
mkdir -p $WSLDA/templates/computer_name/td
```
Copy relevant files:
``` bash
cp Makefile README.md env.sh job.sh $WSLDA/templates/computer_name/st
cp Makefile README.md env.sh job.sh $WSLDA/templates/computer_name/td
cp machine.h $WSLDA/templates/computer_name/
```
# Creating templates of submission scripts & REAMD.md (optional) `machine.h` must be shared between static and time-dependent
It is recommended to place the submission scripts in template folders suitable for your computer, along with a README.md file that provides instructions for users. Examples of submission scripts and README.md files can be found in the folder [templates](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/templates). configurations.
# Making W-SLDA Toolkit accessible to other users (optional) Go to the main folder of the toolkit and propagate templates to the project template folders
To make the W-SLDA Toolkit accessible to other users, ensure that `wslda` is placed in a location that is readable by all users. Make sure that permissions are set correctly, for example:
```bash ```bash
cd $WSLDA
./install-templates.py computer_name
```
We encourage submitting new machine templates to the W-SLDA developers
for inclusion in the codebase.
# Making the Toolkit Accessible to Other Users (Optional)
Ensure the `wslda` directory is readable:
``` bash
chmod -R a+r wslda chmod -R a+r wslda
``` ```
Provide users with a path to the W-SLDA Toolkit. Users can place it in `.bashrc` file, for example: Provide users with the required environment variables:
``` bash
export WSLDA=/project/path/wslda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/project/path/wslda/lib/wdata:/project/path/wslda/lib/wderiv:/project/path/wslda/lib/winterp:/path/to/wslda/lib/wbox
export PATH=$PATH:/project/path/wslda/lib/wdata/bin:/project/path/wslda/tools/bin
```
# Update the toolkit
If a new version is released, the simplest option is to update your repository
```bash ```bash
export WSLDA=/project/pr91/share/wslda/ cd $WSLDA
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/project/pr91/share/wslda/lib/wdata:/project/pr91/share/wslda/lib/wderiv:/project/pr91/share/wslda//lib/winterp git pull
export PATH=$PATH:/project/pr91/share/wslda/lib/wdata/bin:/project/pr91/share/wslda/tools/bin cat VERSION.h
``` ```
This operation should update the engine files while keeping machine-dependent files (such as `Makefile` or `machine.h`) unchanged.
# Installation examples # Installation Examples
* [Eagle system (PCSS, Poland)](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/wikis/Installing-the-toolkit:-Eagle-example) - [Eagle system (PCSS,
* [Kamiak system (WSU)](Kamiak-system-of-WSU) Poland)](Eagle-system-PCSS)
- [Kamiak system (WSU)](Kamiak-system-of-WSU)