HDF5 data format

In the beginning there was the EDF format...

INFO: Problems caused by the "ESRF data format" are unfortunately far too common. The orginal software available from the ESRF is far too bugged to be usable. It is not available for many of the operating systems on which FIT2D is required to run, and users use. Therefore I (and others) have written their own input routines. However, this is only a partial solution. This is because the "format" is totally inadequately defined, and even where defined there are huge differences between the specification and the files actually produced. It is also an unnecessarily complicated format with certain data compression schemes almost totally undefined. Therefore this code has only been written to input a small subset of the possible file formats which could be produced. In particular all header information must be found before the image data, and data compression is not supported. Message when importing EDF files in Fit2D

...now we have a NeXus standard

NeXus is an effort by an international group of scientists to define a common data exchange and archival format for neutron, X-ray and muon experiments. NeXus is built on top of the scientific data format HDF5 and adds domain-spe- cific rules for organizing data within HDF5 files, in addition to a dictiona- ry of well defined domain-specific field names. The NeXus data format has two purposes. First, it defines a format that can serve as a container for all relevant data associated with a beamline. This is a very important use case. Second, it defines standards in the form of application definitions for the exchange of data between applications. NeXus provides structures for raw experimental data as well as for processed data. J. Appl. Cryst. (2015). 48, 301-305

This sounds promising, but reality is as usual much, much more chaotic. In fact, it is nowadays rather straightforward to read or write data/error arrays together with important metadata in EDF format. Routines are available for Python (fabio library) or Matlab (see EDF_data_format|EDF data format).

NeXus on the other hand is a standard which leaves a lot of room for interpretation. As a result, there are unfortunately no routines available for Python or Matlab to read data/error arrays and metadata from HDF5 files as comfortably as it is the case for EDF files. In the following I list the search path for data and metadata for a selection of known interpretations of the NeXus standard. The main problem in many cases is to programmatically determine the present data type. Once known, it is easy to extract data as high level routines are available in programming languages.

Raw data:

1. Lima written raw data at ESRF from 2020
2. Lima written raw data at ESRF before 2020
3. Dectris written raw data

Reduced data:

4. Written by online datareduction at ESRF (pyFAI/DAHU) - SAXS
6. Written by online datareduction at ESRF (pyFAI/DAHU) - XPCS
7. Written by SAXSutilites2 package
8. Written by saxs programs