Skip to content

GitLab

Installing the toolkit · Changes

Page history
Update Installing the toolkit authored by Gabriel Wlazłowski's avatar Gabriel Wlazłowski
Show whitespace changes
Inline Side-by-side
Installing-the-toolkit.md
View page @ 214aaefe
...@@ -57,144 +57,112 @@ export PATH=$PATH:/mnt/storage_2/project_data/grant_518/wslda/lib/wdata/bin:/mnt ...@@ -57,144 +57,112 @@ export PATH=$PATH:/mnt/storage_2/project_data/grant_518/wslda/lib/wdata/bin:/mnt
Share `export` statements with other users of the toolkit, so they can also use it. You need to make `wslda` folder readable for other members of the group. Share `export` statements with other users of the toolkit, so they can also use it. You need to make `wslda` folder readable for other members of the group.
# Creating Makefiles # Setting up templates
W-SLDA exploits a working model based on templates. W-SLDA exploits a working model based on templates.
User starts with the template of a project where items like external potential, external pairing potential, external velocity must be provided in C language. Templates are localized in folders: User starts with the template of a project where items like external potential, external pairing potential, external velocity must be provided in C language. Templates are localized in folders:
* [st-project-template](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/st-project-template) for static problems (`st-wslda` codes) * [st-project-template](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/st-project-template) for static problems (`st-wslda` codes)
* [td-project-template](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/td-project-template) for time-dependent problems (`td-wslda` codes) * [td-project-template](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/td-project-template) for time-dependent problems (`td-wslda` codes)
Within this folders `Makefile` and `machine.h` file compatible with your computing system must be provided. Within these folders `Makefile` and `machine.h` files compatible with your computing system must be provided.
## Makefile for static codes
Operation below apply to folders In the [templates](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/templates) folder there are provided examples of machine-dependent files for various systems. The systems are listed in [templates/README.md](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/templates/README.md).
* [st-project-template](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/st-project-template)
* [st-testcase-uniform](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/st-testcase-uniform)
Edit `Makefile` accordingly to your system. Below we provide an example of `Makefile` targeted to [Piz Daint](https://www.cscs.ch/computers/piz-daint/) supercomputer.
```make
# LOAD BEFORE COMPILATION
#
# module load daint-gpu
# module swap PrgEnv-cray PrgEnv-gnu
# module load cudatoolkit
# module load craype-accel-nvidia60
# module load cray-fftw
# export LD_LIBRARY_PATH=/project/pr91/share/lib:$LD_LIBRARY_PATH
# COMPILER
CXX=cc
# DIRECTORY SETTINGS (must end with /)
WSLDADIR=/project/pr91/share/wslda/
OBJDIR=./obj/
BINDIR=./
# COMPILER FLAGS
CFLAGS= -O3 -I/project/pr91/share/include/elpa-2020.05.001
LIBS=-lfftw3 -L/project/pr91/share/lib -lelpa
# ----- DO NOT MODIFY -----
include $(WSLDADIR)hpc-engine/mk.st
```
It is recommended to provide absolute path to `wslda` folder, in this example:
```make
WSLDADIR=/project/pr91/share/wslda/
```
In the example, we also assume that ELPA lib will be utilized, and paths to header files and lib files are provided in `CFLAGS` and `LIBS`. Various examples of makefiles are located in [Makefiles-templates](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/Makefiles-templates). In order to check the correctness of `Makefile` try to compile codes by typing:
```bash
make
```
To clear your compilation use: To copy templates for the selected machine you can use script [install-templates.py](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/blob/public/install-templates.py), for example:
```bash ```bash
make clean [gabrielw@dell cold-atoms]$ ./install-templates.py eagle
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/st/* /home/gabrielw/MyProjects/cold-atoms/st-project-template
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/st-project-template
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/st/* /home/gabrielw/MyProjects/cold-atoms/st-testcase-uniform
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/st-testcase-uniform
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/td/* /home/gabrielw/MyProjects/cold-atoms/td-project-template
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/td-project-template
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/td/* /home/gabrielw/MyProjects/cold-atoms/td-testcase-uniform
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/td-testcase-uniform
Done.
```
# Testing correctness of templates
## Testing of compilation
Copy project folder to separate location (in the example below I use `tmp` folder)
```bash
cd ~/tmp
cp -r $WSLDA/st-testcase-uniform/ .
cd st-testcase-uniform/
make
``` ```
If the process finishes successfully, executables `st-wslda-?d` will be created. The analogous operation can be applied to `td` part of the toolkit.
*Note 1*: The ELPA Library must be in addition activated in [predefines.h](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/blob/public/st-project-template/predefines.h) file. To learn more about ELPA see [here](Setting up diagonalization engine). *Note 1*: The ELPA Library must be in addition activated in [machine.h](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/blob/public/st-project-template/machine.h) file. To learn more about ELPA see [here](Setting up diagonalization engine).
*Note 2*: `st-wslda` codes utilizes LAPACK and ScaLAPACK libraries. The standard method of calling Fortran functions from C requires supplementing routine's names by `_` at the end, i.e fortran routine `DGEMM` is called from C as `dgemm_`. However in cases of some systems/compilers (very rare) it is not needed. If this is the case of your system then you need to add to `CFLAGS` flag *Note 2*: `st-wslda` codes utilizes LAPACK and ScaLAPACK libraries. The standard method of calling Fortran functions from C requires supplementing routine's names by `_` at the end, i.e fortran routine `DGEMM` is called from C as `dgemm_`. However in cases of some systems/compilers (very rare) it is not needed. If this is the case of your system then you need to add to `CFLAGS` flag
`-DFOTRAN_NO_UNDERSCORE`. `-DFOTRAN_NO_UNDERSCORE`.
*Note 3*: In some cases, additional information about the architecture of the computing system must be provided in [machine.h](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/blob/public/st-project-template/machine.h) file. For more information see [here](Configuring GPU machine).
*Recommendation*: Typically (super)computers utilize `module load` system for configuring the programming environment. We recommend placing a list of modules in a separate file, for example `env.sh`: *Recommendation*: Typically (super)computers utilize `module load` system for configuring the programming environment. We recommend placing a list of modules in a separate file, for example `env.sh`:
```bash ```bash
# Content of env.sh
module load daint-gpu module load daint-gpu
module swap PrgEnv-cray PrgEnv-gnu module swap PrgEnv-cray PrgEnv-gnu/6.0.9
module load cudatoolkit module load cudatoolkit/11.2.0_3.39-2.1__gf93aa1c
module load craype-accel-nvidia60 module load craype-accel-nvidia60
module load cray-fftw module load cray-fftw
export LD_LIBRARY_PATH=/project/pr91/share/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/project/pr125/share/elpa-2020.11.001/lib:$LD_LIBRARY_PATH
module list module list
cat $WSLDA/VERSION.h
```
``` ```
The compilation process looks like this: Then, the compilation process looks like this:
```bash ```bash
source ./env.sh source ./env.sh
make make
``` ```
## Makefile for time dependent codes ## Testing of execution
Operation below apply to folders Execute the binary codes. Below example of a test for static 2d code is presented:
* [td-project-template](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/td-project-template)
* [td-testcase-uniform](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/td-testcase-uniform)
Edit `Makefile` accordingly to your system. Below we provide an example of `Makefile` targeted to [Piz Daint](https://www.cscs.ch/computers/piz-daint/) supercomputer.
```make
# LOAD BEFORE COMPILATION
#
# module load daint-gpu
# module load cudatoolkit
# module load craype-accel-nvidia60
# COMPILERS
CXX=cc
NVCC = nvcc
# DIRECTORY SETTINGS (must end with /)
WSLDADIR=/project/pr91/share/wslda/
OBJDIR=./obj/
BINDIR=./
# COMPILER FLAGS
CFLAGS= -std=gnu99 -O3
NVCCFLAGS = -arch sm_60 -O3
# LIBRARIES
LIBS=-lcudart -lcufft -lm
# ----- DO NOT MODIFY -----
include $(WSLDADIR)hpc-engine/mk.td
```
It is recommended to provide an absolute path to `wslda` folder, in this example:
```make
WSLDADIR=/project/pr91/share/wslda/
```
Various examples of makefiles are located in [Makefiles-templates](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/Makefiles-templates). In order to check the correctness of `Makefile` try to compile codes by typing:
```bash ```bash
make [gabrielw@dell st-testcase-uniform]$ mpirun -np 4 ./st-wslda-2d input.txt
``` ```
*Note*: In some cases, additional information about the architecture of the computing system must be provided in [predefines.h](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/blob/public/td-project-template/predefines.h) file. For more information see [here](Configuring GPU machine). The complete testing process may look like this:
*Recommendation*: Typically (super)computers utilize `module load` system for configuring the programming environment. We recommend placing list of in a separate file, for example `env.sh`:
```bash ```bash
# Content of env.sh # Testing static part
module load daint-gpu [gabrielw@node2067 tmp]$ cd ~/tmp
module load cudatoolkit [gabrielw@node2067 tmp]$ cp -r $WSLDA/st-testcase-uniform/ .
module load craype-accel-nvidia60 [gabrielw@node2067 tmp]$ cd st-testcase-uniform
``` [gabrielw@node2067 st-testcase-uniform]$ source env.sh
The compilation process looks like: [gabrielw@node2067 st-testcase-uniform]$ make
```bash [gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-1d input.txt > st-test-1d.txt
source ./env.sh [gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-2d input.txt > st-test-2d.txt
make [gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-3d input.txt > st-test-3d.txt
```
# Testing time-dependent part
[gabrielw@node2067 tmp]$ cd ~/tmp
[gabrielw@node2067 tmp]$ cp -r $WSLDA/td-testcase-uniform/ .
[gabrielw@node2067 tmp]$ cd td-testcase-uniform
[gabrielw@node2067 td-testcase-uniform]$ source env.sh
[gabrielw@node2067 td-testcase-uniform]$ make
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-1d input.txt > td-test-1d.txt
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-2d input.txt > td-test-2d.txt
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-3d input.txt > td-test-3d.txt
```
The outputs are:
* [st-test-1d.txt](uploads/cd8f4dc231b76388f3ea51ec2b69516e/st-test-1d.txt)
* [st-test-2d.txt](uploads/c6251282595915c06aebf6c8b4a4d665/st-test-2d.txt)
* [st-test-3d.txt](uploads/dfe7441168f3bbac0737d1d6d73da0db/st-test-3d.txt)
* [td-test-1d.txt](uploads/3d1a9a120c4b12c6600f0a4cb92d72ec/td-test-1d.txt)
* [td-test-2d.txt](uploads/b91aef4995b862b28505ef2a4f9cf7da/td-test-2d.txt)
* [td-test-3d.txt](uploads/f4fc567667c3d510377a0b4d2e803fba/td-test-3d.txt)
# Compiling visit plugin (optional) # Compiling visit plugin (optional)
Visit plugin is located in [lib/wdata/visit-plugin](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/lib/wdata/visit-plugin). Visit plugin is located in [lib/wdata/visit-plugin](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/lib/wdata/visit-plugin).
Follow instruction from [README.txt](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/lib/wdata/visit-plugin/README.txt) file to install it. Follow instructions from [README.txt](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/lib/wdata/visit-plugin/README.txt) file to install it.
# Creating templates of submission scripts (optional) # Creating templates of submission scripts & REAMD.md (optional)
It is recommended to place in template folders submission scripts suitable for your computer. Examples of submission scripts can be found in the folder [job-scripts-templates](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/job-scripts-templates). It is recommended to place in template folders submission scripts suitable for your computer, and README.md file with instructions for users. Examples of submission scripts and README.md files can be found in the folder [templates](https://gitlab.fizyka.pw.edu.pl/wtools/wslda/-/tree/public/templates).
# Making W-SLDA Toolkit accessible to other users (optional) # Making W-SLDA Toolkit accessible to other users (optional)
To make W-SLDA Toolkit accessible to other users assure that `wslda` is placed in a location that is readable for all users. Make sure that permissions are set correctly, for example: To make W-SLDA Toolkit accessible to other users assure that `wslda` is placed in a location that is readable for all users. Make sure that permissions are set correctly, for example:
...@@ -202,7 +170,7 @@ To make W-SLDA Toolkit accessible to other users assure that `wslda` is placed i ...@@ -202,7 +170,7 @@ To make W-SLDA Toolkit accessible to other users assure that `wslda` is placed i
chmod -R a+r wslda chmod -R a+r wslda
``` ```
Provide to users path to W-SLDA Toolkit. Recommend for users to place it in `.bashrc` file, example: Provide to users path to W-SLDA Toolkit. Users can place it in `.bashrc` file, for example:
```bash ```bash
export WSLDA=/project/pr91/share/wslda/ export WSLDA=/project/pr91/share/wslda/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/project/pr91/share/wslda/lib/wdata:/project/pr91/share/wslda/lib/wderiv:/project/pr91/share/wslda//lib/winterp export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/project/pr91/share/wslda/lib/wdata:/project/pr91/share/wslda/lib/wderiv:/project/pr91/share/wslda//lib/winterp
......