yadg objects

yadg dataschema

A dataschema is an object defining the files and folders to be processed by yadg, as well as the types of parsers and the parser options to be applied. One can think of a dataschema as a representation of a single experiment, containing measurements from multiple sources (or devices) and following a succession of experimental steps.

The current version of the dataschema is implemented as a Pydantic model in the class DataSchema of the dgbowl_schemas.yadg module. The following (previous) versions of the dataschema are available in the same repository:

An example is a simple catalytic test with a temperature ramp. The goal of such an experiment may be to measure the catalytic conversion as a function of temperature, and then calculate the activation energy of the catalytic reaction. The monitored devices and their filetypes are:

  • the inlet flow and pressure meters -> csv data in foo.csv

  • the temperature controller -> csv data in bar.csv

  • the gas chromatograph -> Fusion json data in ./GC/ folder

Despite these three devices measuring concurrently, we would have to specify three separate steps in the schema to process all relevant output files:

{
    "metadata": {
        "version": "5.0",
        "provenance": {"type": "manual"}
    },
    "steps": [
        {
            "tag": "flow",
            "parser": "basiccsv",
            "input": {"files": ["foo.csv"]},
            "extractor": {"filetype": "None"},
            "parameters": {"sep": ","}
        },
        {
            "parser": "basiccsv",
            "input": {"files": ["bar.csv"]},
            "extractor": {"filetype": "None"},
            "parameters": {"sep": ","}
        },
        {
            "parser": "chromtrace",
            "input": {"files": ["./GC/"]},
            "extractor": {"filetype": "fusion.json"}
        }
    ]
}

Note

Further information about the dataschema can be found in the documentation of the dgbowl_schemas.yadg module.

yadg datagram

The datagram is a structured and annotated representation of raw data. Here, “raw data” strictly denotes the data present in the parsed files, as they come out of an instrument directly. It may therefore also contain derived data (e.g. data processed using a calibration curve in chromatography, or a more involved transformation in electrochemistry), while referring to them as “raw data”, since they are present in the parsed files.

The datagram is designed to be a FAIR representation of the parsed files, with:

  • uncertainties of measured datapoints;

  • units associated with data;

  • a consistent data structure for timestamped traces;

  • a consistent variable mapping between different filetypes.

Additionally, the datagram is annotated by relevant metadata, including:

  • version information;

  • clear provenance of the data;

  • uniform data timestamping within and between all datagrams.

As of yadg-5.0, the datagram is exported as a NetCDF file. In memory, it is represented by a datatree.DataTree, with individual steps as nodes of that datatree.DataTree containing a xarray.Dataset.

The top level datatree.DataTree contains the following metadata stored in its attributes:

  • the version of yadg and the execution command used to generate the datagram;

  • a copy of the dataschema used to created the datagram;

  • the version of the datagram; and

  • the datagram creation timestamp formatted according to ISO8601.

The contents of the attribute fields for each step will vary depending on the parser used to create the corresponding xarray.Dataset. The following conventions are used:

  • a coord field uts contains a Unix timestamp (float),

  • uncertainties for data_vals are stored using separate entries with names composed as f"{entry}_std_err"

    • the parent f"{entry}" is pointing to its uncertainty by annotation using the ancillary_variables field,

    • the uncertainty links back to the f"{entry}" by annotation using the standard_name field.

  • the use of spaces (and other whitespace characters) in the names of entries is to be avoided,

  • the use of forward slashes (/) in the names of entries is not allowed.

This follows the NetCDF CF Metadata Conventions, see Section 3.4 on Ancillary Data.