Skip to content

GitLab

Last edited by Gabriel Wlazłowski
Page history

Installing the toolkit

Requirements

Static codes (st-wslda)

  • C compiler (eg. gcc, intel compiler icc, ...)
  • MPI C compiler (eg. mpicc, intel compiler mpiicc, ...)
  • python
  • FFTW library
  • LAPACK library
  • ScaLAPACK library
  • ELPA library (optionally)
  • CUDA compiler (optionally, for ELPA)

Note 1: compilers and libraries (FFTW, LAPACK, ScaLAPACK) are installed by default on most of the computing clusters. If the computing system is equipped with GPUs most likely CUDA Toolkit is also installed. On many systems, ELPA is also installed by default. If not you can compile it yourself.

Note 2: LAPACK and ScaLAPACK libraries are part of Intel® Math Kernel Library.

Note 3: It is recommended to use st-wslda with ELPA support if possible. It provides substantial speed-up of the computation process. See here for more instructions related to installing ELPA.

Time-dependent codes (td-wslda)

  • C compiler (eg. gcc, intel compiler icc, ...)
  • MPI C compiler (eg. mpicc, intel compiler mpiicc, ...)
  • CUDA compiler
  • python

Note 1: Time-dependent codes require systems accelerated by GPUs. td-wslda codes cannot run on standard systems based on CPUs only.

Downloading the toolkit

It is recommended to download the toolkit directly from the repository using git

git clone https://gitlab.fizyka.pw.edu.pl/wtools/wslda.git

Optionally you can download it manually from main webpage.
The toolkit is contained in wslda folder; see here for more info about content.

Compaling standard libraries and tools (Optionally)

The toolkit has various libs tools supporting the computation process. See here for available tools. While it is not mandatory, we recommend compiling them at the stage of the installation process. You can do it by executing the provided script in the main folder:

./install-libs.sh

Once the compilation is finished, you will get instructions on how to set system variables in order to be able to use W-SLDA Toolkit, for example

gabrielw@e1135:~/grant_518/project_data/wslda$ ./install-libs.sh 
Installing libs...
# Making lib/wderiv..
# ...
make: Leaving directory `/mnt/storage_2/project_data/grant_518/cold-atoms/tools'
# ==========================================================================
# =========================== SET ENVIRONMENT ==============================
# ==========================================================================
#  Add to your .bashrc
 
export WSLDA=/mnt/storage_2/project_data/grant_518/wslda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/mnt/storage_2/project_data/grant_518/wslda/lib/wdata:/mnt/storage_2/project_data/grant_518/wslda/lib/wderiv:/mnt/storage_2/project_data/grant_518/wslda/lib/winterp
export PATH=$PATH:/mnt/storage_2/project_data/grant_518/wslda/lib/wdata/bin:/mnt/storage_2/project_data/grant_518/wslda/tools/bin

Share export statements with other toolkit users, so they can also use it. You need to make wslda folder readable for other members of the group.

Setting up templates

W-SLDA exploits a working model based on templates. The user starts with a project template where items like external potential, pairing potential, and external velocity must be provided in C language. Templates are localized in folders:

Within these folders, Makefile and machine.h files compatible with your computing system must be provided.

In the templates folder, there are provided examples of machine-dependent files for various systems. The systems are listed in templates/README.md.

To copy templates for the selected machine, you can use script install-templates.py, for example:

[gabrielw@dell cold-atoms]$ ./install-templates.py eagle
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/st/* /home/gabrielw/MyProjects/cold-atoms/st-project-template
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/st-project-template
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/st/* /home/gabrielw/MyProjects/cold-atoms/st-testcase-uniform
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/st-testcase-uniform
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/td/* /home/gabrielw/MyProjects/cold-atoms/td-project-template
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/td-project-template
cp -r /home/gabrielw/MyProjects/cold-atoms/templates/eagle/td/* /home/gabrielw/MyProjects/cold-atoms/td-testcase-uniform
cp /home/gabrielw/MyProjects/cold-atoms/templates/eagle/machine.h /home/gabrielw/MyProjects/cold-atoms/td-testcase-uniform
Done.

Testing the correctness of templates

Testing of compilation

Copy the project folder to a separate location (in the example below, I use tmp folder)

cd ~/tmp
cp -r $WSLDA/st-testcase-uniform/ .
cd st-testcase-uniform/

make

If the process finishes successfully, executables st-wslda-?d will be created. The analogous operation can be applied to td part of the toolkit.

Note 1: The ELPA Library must be in addition activated in machine.h file. To learn more about ELPA see here.

Note 2: st-wslda codes utilizes LAPACK and ScaLAPACK libraries. The standard method of calling Fortran functions from C requires supplementing routine's names by _ at the end, i.e fortran routine DGEMM is called from C as dgemm_. However, in cases of some systems/compilers (very rare) it is not needed. If this is the case of your system then you need to add to CFLAGS flag -DFOTRAN_NO_UNDERSCORE.

Note 3: In some cases, additional information about the architecture of the computing system must be provided in machine.h file. For more information see here.

Recommendation: Typically (super)computers utilize the module load system to configure the programming environment. We recommend placing a list of modules in a separate file, for example env.sh:

module load daint-gpu
module swap PrgEnv-cray PrgEnv-gnu/6.0.9
module load cudatoolkit/11.2.0_3.39-2.1__gf93aa1c
module load craype-accel-nvidia60
module load cray-fftw
export LD_LIBRARY_PATH=/project/pr125/share/elpa-2020.11.001/lib:$LD_LIBRARY_PATH

module list
cat $WSLDA/VERSION.h
Then, the compilation process looks like this:
```bash
source ./env.sh
make

Testing of execution

Execute the binary codes. Below an example of a test for static 2d code is presented:

[gabrielw@dell st-testcase-uniform]$ mpirun -np 4 ./st-wslda-2d input.txt

The complete testing process may look like this:

#  Testing static part
[gabrielw@node2067 tmp]$ cd ~/tmp
[gabrielw@node2067 tmp]$ cp -r $WSLDA/st-testcase-uniform/ .
[gabrielw@node2067 tmp]$ cd st-testcase-uniform
[gabrielw@node2067 st-testcase-uniform]$ source env.sh 
[gabrielw@node2067 st-testcase-uniform]$ make
[gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-1d input.txt > st-test-1d.txt
[gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-2d input.txt > st-test-2d.txt
[gabrielw@node2067 st-testcase-uniform]$ mpirun -np 4 ./st-wslda-3d input.txt > st-test-3d.txt

# Testing time-dependent part
[gabrielw@node2067 tmp]$ cd ~/tmp
[gabrielw@node2067 tmp]$ cp -r $WSLDA/td-testcase-uniform/ .
[gabrielw@node2067 tmp]$ cd td-testcase-uniform
[gabrielw@node2067 td-testcase-uniform]$ source env.sh
[gabrielw@node2067 td-testcase-uniform]$ make
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-1d input.txt > td-test-1d.txt
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-2d input.txt > td-test-2d.txt
[gabrielw@node2067 td-testcase-uniform]$ mpirun -np 4 ./td-wslda-3d input.txt > td-test-3d.txt

The outputs are:

Compiling visit plugin (optional)

Visit plugin is located in lib/wdata/visit-plugin. Follow instructions from README.txt file to install it.

Creating templates of submission scripts & REAMD.md (optional)

It is recommended to place in template folders submission scripts suitable for your computer, and README.md file with instructions for users. Examples of submission scripts and README.md files can be found in the folder templates.

Making W-SLDA Toolkit accessible to other users (optional)

To make W-SLDA Toolkit accessible to other users, ensure that wslda is placed in a readable location for all users. Make sure that permissions are set correctly, for example:

chmod -R a+r wslda

Provide to users path to W-SLDA Toolkit. Users can place it in .bashrc file, for example:

export WSLDA=/project/pr91/share/wslda/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/project/pr91/share/wslda/lib/wdata:/project/pr91/share/wslda/lib/wderiv:/project/pr91/share/wslda//lib/winterp
export PATH=$PATH:/project/pr91/share/wslda/lib/wdata/bin:/project/pr91/share/wslda/tools/bin

Installation examples