This is an old version of this page. You can view the most recent version or browse the history.

Setting up diagonalization engine

ELPA Library

For static calculations it is recommend to use ELPA Library, which has better performance than ScaLapack. In particularly, ELPA allows for utilization of GPUs which provide significant boost for calculations. In order to activate ELPA lib in predefines.h set:

// select diagonalization routine
#define DIAGONALIZATION_ROUTINE ELPA

Moreover, you need to inspect carefully part:

// ---------------------- ELPA SETTINGS ---------------------------
// Fill this part only if ELPA library is used for diagonnalization

// uncomment it if you want to activate GPU for diagonalizations 
#define ELPA_USE_GPU

// Select ELPA kernels
#define ELPS_USE_SOLVER ELPA_SOLVER_1STAGE
#define ELPA_USE_COMPLEX_KERNEL ELPA_2STAGE_COMPLEX_DEFAULT
#define ELPA_USE_REAL_KERNEL ELPA_2STAGE_REAL_DEFAULT

// Fraction of eigenvectors to be extracted in each cycle.
// 1.0 corresponds to extraction if all eigenvectors (USE IT IF YOU YOU ARE NOT SURE)
// NOTE: value of this parameter should assure that all eigenstates below requested Ec are extracted.  
// NOTE: For 3D case this value typically can be set to 0.78
#define ELPA_NEV_FRACTION 1.0

Documentation

Publications about ELPA performance

GPU-Acceleration of the ELPA2 Distributed Eigensolver for Dense Symmetric and Hermitian Eigenproblems

ScaLapack library

If target system does not provide ELPA libray user can use (standard) diagonalization library: ScaLAPACK. W-SLDA Toolkit can utilize following ScaLapack diagonalization engines:

#define DIAGONALIZATION_ROUTINE PZHEEVR

#define DIAGONALIZATION_ROUTINE PZHEEVD

It is recommended to use PZHEEVR. This engine takes advantage from fact that typically we extract only fraction of eigenstates. However, we find that in some rare cases (system dependent) this routine does not work correctly. In such can PZHEEVD should be used.

Benchmarks

matrix size	p	q	mb	nb	prec.	routine	system	time [sec]	cost
128,000 = 2x40^3	20	20	32	32	complex	ELPA (1-GPU)	Daint	220	24.4 nh
128,000	54	64	32	32	complex	ELPA (2-CPU)	Daint	677	54.1 nh
128,000	54	64	32	32	complex	PZHEEVR	Daint	945	75.6 nh
147,456 = 4x64x24^2	24	25	32	32	complex	ELPA (1-GPU)	Daint	375	62.5 nh
147,456 = 2x768x96	18	18	16	16	double	ELPA (1-GPU)	Daint	395	35.6 nh
221,184 = 2x48^3	46	84	16	16	complex	ELPA (1-GPU)	Summit	736	18.8 nh
221,184	46	84	16	16	complex	ELPA (2-GPU)	Summit	3098	79.2 nh
221,184	46	84	16	16	complex	PZHEEVD	Summit	5995	153.2 nh
524,288 = 2x64^3	96	112	16	16	complex	ELPA (1-GPU)	Summit	2,217	157.7 nh
746,496 = 2x72^3	112	192	16	16	complex	ELPA (1-GPU)	Summit	3,436	488.7 nh
1,769,472 = 2x96^3	300	560	32	32	complex	ELPA (1-GPU)	Summit	52,024	57,804 nh

(1-GPU): ELPA_SOLVER_1STAGE, ELPA_2STAGE_COMPLEX_GPU or ELPA_2STAGE_REAL_GPU
(2-GPU): ELPA_SOLVER_2STAGE, ELPA_2STAGE_COMPLEX_GPU or ELPA_2STAGE_REAL_GPU
(2-CPU): ELPA_SOLVER_2STAGE, ELPA_2STAGE_COMPLEX_DEFAULT or ELPA_2STAGE_REAL_DEFAULT