Skip to content

GitLab

Installing the toolkit · Changes

Page history
Update Installing the toolkit authored by Gabriel Wlazłowski's avatar Gabriel Wlazłowski
Hide whitespace changes
Inline Side-by-side
Installing-the-toolkit.md
View page @ 8fe93ce2
[[_TOC_]]
# Requirements
## Static codes (st-wslda)
* C compiler (eg. `gcc`, intel compiler `icc`, ...)
* MPI C compiler (eg. `mpicc`, intel compiler `mpiicc`, ...)
* python
* [FFTW library](http://www.fftw.org/)
* LAPACK library
* ScaLAPACK library
* [ELPA library](https://elpa.mpcdf.mpg.de/) (optionally)
* [CUDA compiler](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html) (optionally, for ELPA)
*Note 1*: compilers and libraries (FFTW, LAPACK, ScaLAPACK) are installed by default on most of the computing clusters. If the computing system is equipped with GPUs, it is likely that the CUDA Toolkit is also installed. On many systems, ELPA is also installed by default. If not, you can compile it yourself.
## Static codes (`st-wslda`)
*Note 2*: LAPACK and ScaLAPACK libraries are part of Intel® Math Kernel Library.
- C compiler (e.g., `gcc`, Intel compiler `icc`, etc.)
- MPI C compiler (e.g., `mpicc`, Intel compiler `mpiicc`, etc.)
- Python
- [FFTW library](http://www.fftw.org/)
- LAPACK library
- ScaLAPACK library
- [ELPA library](https://elpa.mpcdf.mpg.de/) (optional)
- [CUDA
compiler](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html)
(optional, for ELPA)
*Note 3*: It is recommended to use `st-wslda` with ELPA support if possible. It provides a substantial speed-up of the computation process. See [here](ELPA installation guide) for more instructions related to installing ELPA.
**Note 1:** Compilers and core libraries (FFTW, LAPACK, ScaLAPACK) are
typically installed by default on most computing clusters. If the system
is equipped with GPUs, the CUDA Toolkit is usually available as well.
ELPA is also installed on many systems; if not, it can be compiled
manually.
## Time-dependent codes (td-wslda)
* C compiler (eg. `gcc`, intel compiler `icc`, ...)
* MPI C compiler (eg. `mpicc`, intel compiler `mpiicc`, ...)
* [CUDA compiler](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html)
* python
**Note 2:** LAPACK and ScaLAPACK are included in the Intel® Math Kernel
Library (MKL).
*Note 1*: Time-dependent codes **require** systems accelerated by GPUs. `td-wslda` codes cannot run on standard systems based on CPUs only.
**Note 3:** It is strongly recommended to use `st-wslda` with ELPA
support whenever possible, as it significantly accelerates
diagonalization. See the [ELPA installation
guide](ELPA-installation-guide) for details.
# Downloading the toolkit
It is recommended to download the toolkit directly from the repository using git
```bash
## Time-dependent codes (`td-wslda`)
- C compiler (e.g., `gcc`, Intel compiler `icc`, etc.)
- MPI C compiler (e.g., `mpicc`, Intel compiler `mpiicc`, etc.)
- [CUDA
compiler](https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html)
- Python
**Note 1:** Time-dependent codes **require GPU-accelerated systems**.
`td-wslda` cannot run on CPU-only systems.
# Downloading the Toolkit
The recommended way to obtain the toolkit is via `git`:
``` bash
git clone https://gitlab.fizyka.pw.edu.pl/wtools/wslda.git
```
Optionally, you can download it manually from [main webpage](https://gitlab.fizyka.pw.edu.pl/wtools/wslda).
The toolkit is contained in the `wslda` folder; see [here](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/wikis/Folders-structure) for more info about the content.
# Compiling standard libraries and tools (Optionally)
The toolkit has various [libs](https://gitlab.fizyka.pw.edu.pl/wtools) tools supporting the computation process. See [here](Auxiliary tools) for available tools. While it is not mandatory, we recommend compiling them during the installation process. You can do it by executing the provided script in the main folder:
Alternatively, you may use mirror repositories:\
[GitLab](https://gitlab.com/coldatoms/wslda) or
[GitHub](https://gitlab.com/coldatoms/wslda).
The toolkit resides in the `wslda` directory. See the [folder structure
description](Folders-structure)
for details.
After cloning, define the global environment variable:
``` bash
export WSLDA=path_to_wslda_folder
```
It is recommended to place this command in your shell configuration file
(e.g., `.bashrc`).
# Compiling Standard Libraries and Tools (Optional)
The toolkit includes several auxiliary [libraries and
tools](https://gitlab.fizyka.pw.edu.pl/wtools) that support computation
and data processing. See [Auxiliary tools](Auxiliary-tools) for
details. Although optional, compiling them during installation is
recommended.
To compile, ensure that both a C compiler and an MPI C compiler are
available. Specify them in the header of `install-libs.sh`:
``` bash
export CC="gcc -std=gnu99"
export MPICPP="mpic++ -std=gnu99"
```
Some libraries require FFTW. Ensure it is installed before compilation.
To build all libraries:
``` bash
./install-libs.sh
```
Once the compilation is finished, you will get instructions on how to set system variables in order to be able to use the W-SLDA Toolkit, for example
After completion, the script provides instructions for setting
environment variables, for example:
``` bash
export WSLDA=/path/to/wslda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/wslda/lib/wdata:/path/to/wslda/lib/wderiv:/path/to/wslda/lib/winterp:/path/to/wslda/lib/wbox
export PATH=$PATH:/path/to/wslda/lib/wdata/bin:/path/to/wslda/tools/bin
```
gabrielw@e1135:~/grant_518/project_data/wslda$ ./install-libs.sh
Installing libs...
# Making lib/wderiv..
# ...
make: Leaving directory `/mnt/storage_2/project_data/grant_518/cold-atoms/tools'
# ==========================================================================
# =========================== SET ENVIRONMENT ==============================
# ==========================================================================
# Add to your .bashrc
export WSLDA=/mnt/storage_2/project_data/grant_518/wslda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/mnt/storage_2/project_data/grant_518/wslda/lib/wdata:/mnt/storage_2/project_data/grant_518/wslda/lib/wderiv:/mnt/storage_2/project_data/grant_518/wslda/lib/winterp
export PATH=$PATH:/mnt/storage_2/project_data/grant_518/wslda/lib/wdata/bin:/mnt/storage_2/project_data/grant_518/wslda/tools/bin
Share these `export` statements with other users if the toolkit is
installed in a shared location. Ensure that the `wslda` directory is
readable by all intended users.
Each library is compiled via `make` in its respective directory. For
example:
``` bash
make -C lib/wdata
```
Share `export` statements with other toolkit users, so they can also use it. You need to make `wslda` folder readable for other members of the group.
If compilation fails, enter the relevant directory and adjust the
`Makefile` accordingly.
# Setting up templates
W-SLDA exploits a working model based on templates.
The user starts with a project template where items like external potential, pairing potential, and external velocity must be provided in C language. Templates are localized in folders:
* [st-project-template](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/st-project-template) for static problems (`st-wslda` codes)
* [td-project-template](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/td-project-template) for time-dependent problems (`td-wslda` codes)
# Setting Up Templates
Within these folders, `Makefile` and `machine.h` files compatible with your computing system must be provided.
W-SLDA follows a template-based workflow.
In the [templates](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/templates) folder, examples of machine-dependent files for various systems are provided. The systems are listed in [templates/README.md](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/templates/README.md).
Templates are located in:
To copy templates for the selected machine, you can use the script [install-templates.py](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/blob/public/install-templates.py), for example:
```bash
[gabrielw@dell cold-atoms]$ ./install-templates.py eagle
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/st/* /home/gabrielw/MyProjects/cold-atoms/st-project-template
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/st-project-template
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/st/* /home/gabrielw/MyProjects/cold-atoms/st-testcase-uniform
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/st-testcase-uniform
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/td/* /home/gabrielw/MyProjects/cold-atoms/td-project-template
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/td-project-template
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/td/* /home/gabrielw/MyProjects/cold-atoms/td-testcase-uniform
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/td-testcase-uniform
Done.
```
# Testing the correctness of templates
## Testing of compilation
Copy the project folder to a separate location (in the example below, I use `tmp` folder)
```bash
- [`st-project-template`](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/st-project-template)
-- static problems (st)
- [`td-project-template`](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/td-project-template)
-- time-dependent problems (td)
A template folder should contain:
| Name | Requirement | Dependency| Description |
|----- | ----------- | --------- | ----------- |
| `Makefile` | Mandatory | Machine-dependent | Compiler settings and flags |
| `machine.h` | Mandatory| Machine-dependent | Machine configuration (diagonalization engine, GPUs per node, etc.) |
| `README.md` | Optional | Machine-dependent | User instructions |
| `env.sh` | Optional | Machine-dependent | Environment setup script |
| `job.sh` | Optional | Machine-dependent | Example submission script |
| `input.txt` | Mandatory | Problem-dependent | Default input file |
| `predefines.h` | Mandatory | Problem-dependent | Compiler predefinitions |
| `problem-definition.h` |Mandatory | Problem-dependent | Problem definition |
| `logger.h` | Mandatory | Problem-dependent | Logging configuration |
At minimum, `Makefile` and `machine.h` must be adapted to your system.
Machine-specific examples are available in the
[`templates`](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/templates)
directory.
If your system is listed, use:
``` bash
./install-templates.py system_name
```
This script copies appropriate files into the template directories.
# Testing Templates and Execution
## Compilation Test
Copy a test project to a temporary directory:
``` bash
cd ~/tmp
cp -r $WSLDA/st-testcase-uniform/ .
cd st-testcase-uniform/
make
```
If the process finishes successfully, executables `st-wslda-?d` will be created. The analogous operation can be applied to `td` part of the toolkit.
*Note 1*: The ELPA Library must be in addition activated in [machine.h](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/blob/public/st-project-template/machine.h) file. To learn more about ELPA see [here](Setting up diagonalization engine).
Successful compilation produces executables such as `st-wslda-?d`, where `?` is `1`, `2`, or `3` and indicates the code dimensionality.
*Note 2*: `st-wslda` codes utilize LAPACK and ScaLAPACK libraries. The standard method of calling Fortran functions from C requires supplementing routine names by `_` at the end, i.e, Fortran routine `DGEMM` is called from C as `dgemm_`. However, in cases of some systems/compilers (very rare), it is not needed. If this is the case with your system, then you need to add to `CFLAGS` flag
`-DFOTRAN_NO_UNDERSCORE`.
**If compilation fails**: Check environment variables. Verify
`Makefile` and `machine.h`
*Note 3*: In some cases, additional information about the architecture of the computing system must be provided in [machine.h](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/blob/public/st-project-template/machine.h) file. For more information, see [here](Configuring GPU machine).
**Note 1:** Enable ELPA in `machine.h` if available.
**Note 2:** Some systems require `-DFORTRAN_NO_UNDERSCORE` in `CFLAGS`.
**Note 3:** Additional architecture-specific settings may be required in `machine.h`.
*Recommendation*: Typically, (super)computers utilize the `module load` system to configure the programming environment. We recommend placing a list of modules in a separate file, for example, `env.sh`:
```bash
module load daint-gpu
module swap PrgEnv-cray PrgEnv-gnu/6.0.9
module load cudatoolkit/11.2.0_3.39-2.1__gf93aa1c
module load craype-accel-nvidia60
module load cray-fftw
export LD_LIBRARY_PATH=/project/pr125/share/elpa-2020.11.001/lib:$LD_LIBRARY_PATH
It is recommended to use a dedicated `env.sh` file for module loading:
``` bash
module load ...
export LD_LIBRARY_PATH=...
module list
cat $WSLDA/VERSION.h
```
```
Then, the compilation process looks like this:
```bash
Then compile with:
``` bash
source ./env.sh
make
```
make
```
## Testing of execution
Execute the binary codes. Below is an example of a test for static 2d code:
```bash
[gabrielw@dell st-testcase-uniform]$ mpirun -np 4 ./st-wslda-2d input.txt
Repeat the procedure analogously for `td` templates.
## Execution Test
The toolkit provides two reference test cases that verify both
correctness of implementation and proper configuration of the
computational environment.
- `st-testcase-uniform`: This test solves the *uniform system* in the static (self-consistent)
mode. The self-consistent loop is initialized using analytical expressions
derived from BCS theory for a uniform Fermi gas. The
initial guess corresponds to the exact solution of the uniform problem.
Since the initial state is already the correct solution, the self-consistent procedure should converge immediately.
This means that only a single iteration should be performed, and the code should conclude with `ALGORITHM CONVERGED`.
- `td-testcase-uniform`: This test evolves in time the same *uniform solution*, but now using the time-dependent solver.
The initial state provided to the code is a stationary solution of the
time-dependent equations. Therefore, it should remain unchanged during time evolution. All quantities (e.g., total energy, contributions to the energy, particle number) must remain constant within numerical precision. Observables printed to the screen and to the files should not exhibit time-dependence.
Together, these two tests verify: 1. Correct static self-consistency
implementation. 2. Proper time-dependent propagation. 3. Consistency
between static and time-dependent solvers. 4. Correct linkage of
numerical libraries and GPU backends.
## Example execution
``` bash
mpirun -np 4 ./st-wslda-2d input.txt
```
The complete testing process may look like this:
```bash
# Testing static part
[gabrielw@node2067 tmp]$ cd ~/tmp
[gabrielw@node2067 tmp]$ cp -r $WSLDA/st-testcase-uniform/ .
[gabrielw@node2067 tmp]$ cd st-testcase-uniform
[gabrielw@node2067 st-testcase-uniform]$ source env.sh
[gabrielw@node2067 st-testcase-uniform]$ make
[gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-1d input.txt > st-test-1d.txt
[gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-2d input.txt > st-test-2d.txt
[gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-3d input.txt > st-test-3d.txt
# Testing time-dependent part
[gabrielw@node2067 tmp]$ cd ~/tmp
[gabrielw@node2067 tmp]$ cp -r $WSLDA/td-testcase-uniform/ .
[gabrielw@node2067 tmp]$ cd td-testcase-uniform
[gabrielw@node2067 td-testcase-uniform]$ source env.sh
[gabrielw@node2067 td-testcase-uniform]$ make
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-1d input.txt > td-test-1d.txt
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-2d input.txt > td-test-2d.txt
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-3d input.txt > td-test-3d.txt
```
The outputs are:
Full test workflow:
``` bash
# Static
cd st-testcase-uniform
source env.sh
make
mpirun -np 4 ./st-wslda-1d input.txt > st-test-1d.txt
mpirun -np 4 ./st-wslda-2d input.txt > st-test-2d.txt
mpirun -np 4 ./st-wslda-3d input.txt > st-test-3d.txt
cd td-testcase-uniform
source env.sh
make
mpirun -np 4 ./td-wslda-1d input.txt > td-test-1d.txt
mpirun -np 4 ./td-wslda-2d input.txt > td-test-2d.txt
mpirun -np 4 ./td-wslda-3d input.txt > td-test-3d.txt
```
Examples of outputs are (note they may change with the version change):
* [st-test-1d.txt](uploads/cd8f4dc231b76388f3ea51ec2b69516e/st-test-1d.txt)
* [st-test-2d.txt](uploads/c6251282595915c06aebf6c8b4a4d665/st-test-2d.txt)
* [st-test-3d.txt](uploads/dfe7441168f3bbac0737d1d6d73da0db/st-test-3d.txt)
......@@ -156,28 +243,62 @@ The outputs are:
* [td-test-2d.txt](uploads/b91aef4995b862b28505ef2a4f9cf7da/td-test-2d.txt)
* [td-test-3d.txt](uploads/f4fc567667c3d510377a0b4d2e803fba/td-test-3d.txt)
# Compiling visit plugin (optional)
# Creating Machine Templates
After successful testing, create a machine-specific template:
Visit plugin is located in [lib/wdata/visit-plugin](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/lib/wdata/visit-plugin).
Follow instructions from [README.txt](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/lib/wdata/visit-plugin/README.txt) file to install it.
``` bash
mkdir -p $WSLDA/templates/computer_name/st
mkdir -p $WSLDA/templates/computer_name/td
```
Copy relevant files:
``` bash
cp Makefile README.md env.sh job.sh $WSLDA/templates/computer_name/st
cp Makefile README.md env.sh job.sh $WSLDA/templates/computer_name/td
cp machine.h $WSLDA/templates/computer_name/
```
# Creating templates of submission scripts & REAMD.md (optional)
It is recommended to place the submission scripts in template folders suitable for your computer, along with a README.md file that provides instructions for users. Examples of submission scripts and README.md files can be found in the folder [templates](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/templates).
`machine.h` must be shared between static and time-dependent
configurations.
# Making W-SLDA Toolkit accessible to other users (optional)
To make the W-SLDA Toolkit accessible to other users, ensure that `wslda` is placed in a location that is readable by all users. Make sure that permissions are set correctly, for example:
Go to the main folder of the toolkit and propagate templates to the project template folders
```bash
cd $WSLDA
./install-templates.py computer_name
```
We encourage submitting new machine templates to the W-SLDA developers
for inclusion in the codebase.
# Making the Toolkit Accessible to Other Users (Optional)
Ensure the `wslda` directory is readable:
``` bash
chmod -R a+r wslda
```
Provide users with a path to the W-SLDA Toolkit. Users can place it in `.bashrc` file, for example:
Provide users with the required environment variables:
``` bash
export WSLDA=/project/path/wslda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/project/path/wslda/lib/wdata:/project/path/wslda/lib/wderiv:/project/path/wslda/lib/winterp:/path/to/wslda/lib/wbox
export PATH=$PATH:/project/path/wslda/lib/wdata/bin:/project/path/wslda/tools/bin
```
# Update the toolkit
If a new version is released, the simplest option is to update your repository
```bash
export WSLDA=/project/pr91/share/wslda/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/project/pr91/share/wslda/lib/wdata:/project/pr91/share/wslda/lib/wderiv:/project/pr91/share/wslda//lib/winterp
export PATH=$PATH:/project/pr91/share/wslda/lib/wdata/bin:/project/pr91/share/wslda/tools/bin
cd $WSLDA
git pull
cat VERSION.h
```
This operation should update the engine files while keeping machine-dependent files (such as `Makefile` or `machine.h`) unchanged.
# Installation examples
# Installation Examples
* [Eagle system (PCSS, Poland)](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/wikis/Installing-the-toolkit:-Eagle-example)
* [Kamiak system (WSU)](Kamiak-system-of-WSU)
- [Eagle system (PCSS,
Poland)](Eagle-system-PCSS)
- [Kamiak system (WSU)](Kamiak-system-of-WSU)