Skip to content

GitLab

W data format · Changes

Page history
Update W data format authored by Gabriel Wlazłowski's avatar Gabriel Wlazłowski
Show whitespace changes
Inline Side-by-side
W-data-format.md
View page @ d0ee5476
### W-data format
#### Introduction
W-data was designed to satisfy following requirements:
# Introduction
W-data was designed to satisfy the following requirements:
* Binary data is stored in a **conceptually easy format** that allows a variety of tools/languages to be used.
* Format provides storage for data with time stepping (frames / measurements / cycles).
* Binary data is stored in conceptually easy format that allows variety of tools/languages to be used.
* Data format is suitable for parallel processing (preferably with MPI I/O).
* Data is easy for processing via VisIt.
* It provides extensible framework - new variables can be created and easily added to existing dataset.
* It provides the extensible framework - new variables can be created and easily added to the existing dataset.
* Data is convenient for copy between computing systems.
* It allows for easy extraction / copy of selected variables
#### W-data format concept
# W-data format concept
Data set will consist of set of files, example:
```bash
test.wtxt # metadata file, this one should be indicated when opening in VisIt
......@@ -52,12 +50,12 @@ const eF 0.5
const kF 1
```
According our experience three types of variables (`real`, `complex`, `vector`) are sufficient and cover more 90% of applications.
According to our experience, three types of variables (`real`, `complex`, `vector`) are sufficient and cover more 90% of applications.
Binary files store data as row arrays, called datablocks:
![datablocks](http://git2.if.pw.edu.pl/gabrielw/cold-atoms/uploads/0c3f3e8cf8d2d2a4630da138c390166c/datablocks.png)
![datablocks](uploads/41c52d993d9e6cda930a309e0e16d9ac/datablocks.png)
Size of datablock depends on variable type and data dimensonality.
The size of datablock depends on variable type and data dimensionality.
* *real*: `blocksize=blocklength*8 Bytes`
* *complex*: `blocksize=blocklength*16 Bytes`
* *vector*: `blocksize=blocklength*8*3 Bytes`
......@@ -68,31 +66,30 @@ where `blocklength` is
* for datadim=1: `blocklength=NX`
Note that for vector variables we use following storage pattern:
![vecvar](http://git2.if.pw.edu.pl/gabrielw/cold-atoms/uploads/28a263ab139358c5160d8fe38890a234/vecvar.png)
![vecvar](uploads/64a47839f226cb58a60f81edabdf01eb/vecvar.png)
To compute time associated with given cycle use: `time=t0+cycleid*dt`
W-data format allows for representation of following elements:
W-data format allows for the representation of the following elements:
* *variable*:
Each variable is represented by binary file of name `prefix_varname.format`. Variable description has follwing format:
Each variable is represented by the binary file of name `prefix_varname.format`. The variable description has following format:
`var name type unit format`
Following formats are allowed:
* `wdat`: default format for WSLDA codes. Binary files contain row data (no header).
* `dpca`: (*deprecated*) previous format of cold atomic codes. Binary file contain header of size 68B where additional info about file content is stored. For this format *wdata* lib provides only reading functionality.
* `dpca`: (*deprecated*) previous format of cold atomic codes. Binary file contains header of size 68B where additional info about file content is stored. For this format *wdata* lib provides only reading functionality.
* `npy`: binary files are *numpy* arrays. **Functionality under construction**
In order to switch WSLDA codes to writing in this format add to input file:
`dataformat npy`
* *link*:
It is alternative name for given variable. In case of WSLDA codes in many cases we do computation for systems that exhibit some symmetries, like spin symmetry. Then densities for spin-a and spin-b particles are exactly the same. In order to save disk space we can save only one of them:
It is an alternative name for a given variable. In the case of WSLDA codes in many cases, we do the computation for systems that exhibit some symmetries, like spin symmetry. Then densities for spin-a and spin-b particles are exactly the same. In order to save disk space we can save only one of them:
`var density_a real none wdat`
and for another one set link:
`link density_b density_a`
* *constant*:
Typically beside variables we have some constants that are useful during the data analysis process. For example, when making plots it is convenient to express variables in dimensionless form, like delta/eF. To provide user info what are values of selected constants we use `const` field:
Typically besides variables, we have some constants that are useful during the data analysis process. For example, when making plots it is convenient to express variables in dimensionless form, like delta/eF. To provide user info what are values of selected constants we use `const` field:
`const eF 0.5`
#### Low level reading and writing of data
# Low-level reading and writing of data
Code snippet demonstrating how to load datablock from `wdat` file:
```c
......@@ -143,7 +140,7 @@ if(test_ele != 1) printf("ERROR: Cannot add datablock\n");
fclose(pFile); // close file
```
#### W-data C library
# W-data C library
Folder [lib-wdata](https://gitlab.fizyka.pw.edu.pl/gabrielw/wslda/-/tree/public/lib-wdata) contains library that provides support for w-data processing.
List of examples demonstrating how to use this library.
* [example-write.c](https://gitlab.fizyka.pw.edu.pl/gabrielw/wslda/-/tree/public/lib-wdata/example-write.c): code creates artificial set of variables and writes them do wdat files.
......@@ -155,3 +152,6 @@ In order to compile examples (optionally you will need to modify Makefile):
[gabrielw@dell cold-atoms]$ cd lib-wdata/
[gabrielw@dell lib-wdata]$ make
```
# W-data python library
See [PyPi W-data Format](https://pypi.org/project/wdata/)
\ No newline at end of file