This is an old version of this page. You can view the most recent version or browse the history.

Installing the toolkit

Requirements

Static codes (st-wslda)

C compiler (eg. gcc, intel compiler icc, ...)
MPI C compiler (eg. mpicc, intel compiler mpiicc, ...)
python
FFTW library
LAPACK library
ScaLAPACK library
ELPA library (optionally)
CUDA compiler (optionally, for ELPA)

Note 1: compilers and libraries (FFTW, LAPACK, ScaLAPACK) are installed by default on most of the computing clusters. If the computing system is equipped with GPUs most likely CUDA Toolkit is also installed. On many systems, ELPA is also installed by default. If not you can compile it yourself.

Note 2: LAPACK and ScaLAPACK libraries are part of Intel® Math Kernel Library.

Note 3: It is recommended to use st-wslda with ELPA support if possible. It provides substantial speed-up of the computation process. See here for more instructions related to installing ELPA.

Time-dependent codes (td-wslda)

C compiler (eg. gcc, intel compiler icc, ...)
MPI C compiler (eg. mpicc, intel compiler mpiicc, ...)
CUDA compiler
python

Note 1: Time-dependent codes require systems accelerated by GPUs. td-wslda codes cannot run on standard systems based on CPUs only.

Downloading the toolkit

It is recommended to download the toolkit directly from the repository using git

git clone https://gitlab.fizyka.pw.edu.pl/wtools/wslda.git

Optionally you can download it manually from main webpage.
The toolkit is contained in wslda folder, see here for more info about content.

Compaling standard libraries and tools

The toolkit is equipped with various libs and tools that support the computation process. See here for a list of presently available tools. While it is not mandatory we recommend compiling them at the stage of the installation process. You can do it by executing the provided script in the main folder:

./install-libs.sh

Once the compilation is finished you will get instruction how to set system variable in order to be able use W-SLDA Toolkit, example

gabrielw@e1135:~/grant_518/project_data/wslda$ ./install-libs.sh 
Installing libs...
# Making lib/wderiv..
# ...
make: Leaving directory `/mnt/storage_2/project_data/grant_518/cold-atoms/tools'
# ==========================================================================
# =========================== SET ENVIRONMENT ==============================
# ==========================================================================
#  Add to your .bashrc
 
export WSLDA=/mnt/storage_2/project_data/grant_518/wslda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/mnt/storage_2/project_data/grant_518/wslda/lib/wdata:/mnt/storage_2/project_data/grant_518/wslda/lib/wderiv:/mnt/storage_2/project_data/grant_518/wslda/lib/winterp
export PATH=$PATH:/mnt/storage_2/project_data/grant_518/wslda/lib/wdata/bin:/mnt/storage_2/project_data/grant_518/wslda/tools/bin

Share export statements with other users of the toolkit, so they can also use it. You need to make wslda folder readable for other members of the group.

Creating Makefiles

W-SLDA exploits a working model based on templates. User starts with the template of a project where items like external potential, external pairing potential, external velocity must be provided in C language. Templates are localized in folders:

st-project-template for static problems (st-wslda codes)
td-project-template for time-dependent problems (td-wslda codes)

Within this folders Makefile compatible with your computing system must be provided.

Makefile for static codes

Operation below apply to folders

Edit Makefile accordingly to your system. Below we provide an example of Makefile targeted to Piz Daint supercomputer.

# LOAD BEFORE COMPILATION
#
# module load daint-gpu
# module swap PrgEnv-cray PrgEnv-gnu
# module load cudatoolkit
# module load craype-accel-nvidia60
# module load cray-fftw
# export LD_LIBRARY_PATH=/project/pr91/share/lib:$LD_LIBRARY_PATH

# COMPILER
CXX=cc

# DIRECTORY SETTINGS (must end with /)
WSLDADIR=/project/pr91/share/wslda/
OBJDIR=./obj/
BINDIR=./

# COMPILER FLAGS
CFLAGS= -O3  -I/project/pr91/share/include/elpa-2020.05.001
LIBS=-lfftw3  -L/project/pr91/share/lib -lelpa

# ----- DO NOT MODIFY -----
include $(WSLDADIR)hpc-engine/mk.st

It is recommended to provide absolute path to wslda folder, in this example:

WSLDADIR=/project/pr91/share/wslda/

In the example, we also assume that ELPA lib will be utilized, and paths to header files and lib files are provided in CFLAGS and LIBS. Various examples of makefiles are located in Makefiles-templates. In order to check the correctness of Makefile try to compile codes by typing:

make

To clear your compilation use:

make clean

Note 1: The ELPA Library must be in addition activated in predefines.h file. To learn more about ELPA see here.

Note 2: st-wslda codes utilizes LAPACK and ScaLAPACK libraries. The standard method of calling Fortran functions from C requires supplementing routine's names by _ at the end, i.e fortran routine DGEMM is called from C as dgemm_. However in cases of some systems/compilers (very rare) it is not needed. If this is the case of your system then you need to add to CFLAGS flag -DFOTRAN_NO_UNDERSCORE.

Recommendation: Typically (super)computers utilize module load system for configuring the programming environment. We recommend placing a list of modules in a separate file, for example env.sh:

# Content of env.sh
module load daint-gpu
module swap PrgEnv-cray PrgEnv-gnu
module load cudatoolkit
module load craype-accel-nvidia60
module load cray-fftw
export LD_LIBRARY_PATH=/project/pr91/share/lib:$LD_LIBRARY_PATH
module list

The compilation process looks like this:

source ./env.sh
make

Makefile for time dependent codes

Operation below apply to folders

Edit Makefile accordingly to your system. Below we provide an example of Makefile targeted to Piz Daint supercomputer.

# LOAD BEFORE COMPILATION
# 
# module load daint-gpu
# module load cudatoolkit
# module load craype-accel-nvidia60

# COMPILERS
CXX=cc
NVCC = nvcc 

# DIRECTORY SETTINGS (must end with /)
WSLDADIR=/project/pr91/share/wslda/
OBJDIR=./obj/
BINDIR=./

# COMPILER FLAGS
CFLAGS= -std=gnu99 -O3
NVCCFLAGS = -arch sm_60 -O3 

# LIBRARIES
LIBS=-lcudart -lcufft -lm


# ----- DO NOT MODIFY -----
include $(WSLDADIR)hpc-engine/mk.td

It is recommended to provide an absolute path to wslda folder, in this example:

WSLDADIR=/project/pr91/share/wslda/

Various examples of makefiles are located in Makefiles-templates. In order to check the correctness of Makefile try to compile codes by typing:

make

Note: In some cases, additional information about the architecture of the computing system must be provided in predefines.h file. For more information see here.

Recommendation: Typically (super)computers utilize module load system for configuring the programming environment. We recommend placing list of in a separate file, for example env.sh:

# Content of env.sh
module load daint-gpu
module load cudatoolkit
module load craype-accel-nvidia60

The compilation process looks like:

source ./env.sh
make

Compiling visit plugin (optional)

Visit plugin is located in lib/wdata/visit-plugin. Follow instruction from README.txt file to install it.

Creating templates of submission scripts (optional)

It is recommended to place in template folders submission scripts suitable for your computer. Examples of submission scripts can be found in the folder job-scripts-templates.

Making W-SLDA Toolkit accessible to other users (optional)

To make W-SLDA Toolkit accessible to other users assure that wslda is placed in a location that is readable for all users. Make sure that permissions are set correctly, for example:

chmod -R a+r wslda

Provide to users path to W-SLDA Toolkit. Recommend for users to place it in .bashrc file, example:

export WSLDA=/project/pr91/share/wslda/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/project/pr91/share/wslda/lib/wdata:/project/pr91/share/wslda/lib/wderiv:/project/pr91/share/wslda//lib/winterp
export PATH=$PATH:/project/pr91/share/wslda/lib/wdata/bin:/project/pr91/share/wslda/tools/bin