yadg package

yadg.run_with_arguments()

Main execution function.

This is the function executed when yadg is launched via its executable. The function has the following subcommands:

  • extract: extracts (meta)data from a given file.

  • process: processes a given dataschema, extracting data into a NetCDF file.

  • update: updates a given dataschema to the current version.

  • preset: creates a dataschema from a preset file and a target folder.

Subpackages

yadg.core module

yadg.core.process_schema(dataschema: DataSchema, strict_merge: bool = False) → DataTree

The main DataSchema processing function of yadg.

Takes in a DataSchema object, updates it to the latest version compatible with the installed version of yadg, processes each step, and returns a single DataTree created from the DataSchema.

Parameters:

  • dataschema – A DataSchema object describing the extraction process.

  • strict_merge – A bool indicating whether metadata of the files processed in a single step has to be identical. Defaults to False which means conflicts will be dropped.

yadg.main module

yadg.main.set_loglevel(delta: int)
yadg.main.run_with_arguments()

Main execution function.

This is the function executed when yadg is launched via its executable. The function has the following subcommands:

  • extract: extracts (meta)data from a given file.

  • process: processes a given dataschema, extracting data into a NetCDF file.

  • update: updates a given dataschema to the current version.

  • preset: creates a dataschema from a preset file and a target folder.

yadg.subcommands module

yadg.subcommands.process(*, infile: str, outfile: str, ignore_merge_errors: bool, **kwargs: dict) → None

The process subcommand of yadg.

This function first checks that the supplied infile exists, is a valid dataschema, and if yes, proceeds to process the dataschema into a datatree. If this is successful, the datatree is written out into outfile (which is "datagram.nc" by default).

yadg.subcommands.update(*, infile: str, outfile: str, **kwargs: dict) → None

The update subcommand of yadg.

This function updates the dataschema present in the infile to comply with the newest version of dataschema, and saves the resulting object into outfile (which is the infile with a “.new.json” suffix by default).

yadg.subcommands.preset(*, preset: str, folder: str, outfile: str, process: bool, archive: bool, packwith: str, ignore_merge_errors: bool, **kwargs: dict) → None

The preset subcommand of yadg.

If folder is an existing folder and preset a valid dataschema template file, this function will prepend the specified folder to all relative paths in the preset, converting them to absolute paths.

The resulting dataschema will be saved in the supplied outfile.

Alternatively, if process is specified, the created dataschema will be directly processed into a datatree, which is then saved in outfile.

Additionally, the contents of the folder can be archived (if archive is set), using a compression algorithm of your choice.

yadg.subcommands.extract(*, filetype: str, infile: str, outfile: str, meta_only: bool, **kwargs: dict) → None

The extract subcommand of yadg.

If filetype is known to yadg, it will attempt to extract all data from the provided infile.

The data is returned as a xarray.Dataset or a datatree, and is stored in a NetCDF file. The output location can be configured using the outfile argument, by default this is set to the stem of infile with a .nc suffix.

Optionally, an export of just the metadata can be requested by setting the meta_only argument, in this case the output is a json file.