yadg version history

yadg version next

Developed in the ConCat Lab at Technische Universität Berlin (Berlin, DE).

New features in yadg-next are:

Breaking changes in yadg-next are:

  • For all extractors, the source (positional) argument now must be specified instead of fn or path.

  • In yadg.extractors.agilent.csv, the signal data variable now has elution_time as a proper coordinate. Previously, the elution_time was expanded manually to the length of the largest trace present in the file, with np.nan used as padding for shorter traces. An arbitrary coordinate _ was also present. Now, elution_time is expanded automatically by xarray.concat(), inserting np.nan into the signal data variable as necessary for elution_time which are not present in each trace.

Bug fixes in yadg-next include:

  • The parameter Set I/C in yadg.extractors.eclab.mpr files should be C / N when set to 1, not C.

  • The command line argument --ignore-merge-errors was not being passed to the individual extractors when using the yadg extract syntax.

  • Fixed passing of yadg.extractors.fusion.json files where no species are present.

yadg version 6.2

https://img.shields.io/static/v1?label=yadg&message=v6.2&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v6.2&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2025-08-20&color=red&logo=pypi

Developed in the ConCat Lab at Technische Universität Berlin (Berlin, DE).

Breaking changes in yadg-6.2 are:

  • Some column names in yadg.extractors.eclab.mpr files might have changed, as EC-Lab 11.62 has a new naming convention for derived quantities. In particular:

    • Energy charge is now Energy we charge,

    • Energy discharge is now Energy we discharge,

    • P is now Pwe,

    • R is now Rwe.

    This will also unfortunately affect processing older mpr files. Depending on which version of EC-Lab was used to convert the mpr file to the mpt file, the mpt file will contain the old (i.e. P or Energy charge) or the new (i.e. Pwe or Energy we charge) column names. For yadg internal test-suite, we still attempt an exact match between mpr and mpt columns; if the mpr column is not present in the mpt file, we look for an equivalent column without the we annotation.

  • The control/V/mA column and the mode column in mpr as well as mpr files is now used to create the control_V (units V) and control_I (units mA) columns in both kinds of files:

    • The original control/V and control/mA which are present in mpt and may be present in mpr files are now overwritten by values from the control/V/mA column, which is always present.

    • In any given row, either control_V or control_I is set, with the other (non-controlling) column being np.nan.

    • The currently known mode values correspond to:

      • 1 for constant current (control_I = limiting value)

      • 2 for constant voltage (control_V = limiting value)

      • 3 for open circuit (control_I = forced to zero)

    Thanks to Graham Kimbell from Empa for helping out with these changes.

Bug fixes in yadg-next include:

  • Fixed metadata extraction in yadg.extractors.eclab.mpt so that multiple Comments: lines are concatenated instead of just the last line being kept.

  • Added further Set I/C parameters. Thanks to @Locki3 for providing test files.

  • Fixed param format and data_column in CV files generated with EC-Lab version 11.50 using the yadg.extractors.eclab.mpr module. Thank you to J.N. Hausmann from Helmholtz-Zentrum Berlin für Materialien und Energie for providing the test files.

  • Fixed parsing of various kinds of Modulo Bat files in yadg.extractors.eclab.mpr. Thank you to Johannes Baller for providing test files.

  • Fixed parsing of missing columns in GCPL files in yadg.extractors.eclab.mpr. Thank you to Joachim Laviolette for providing test files.

  • Columns in yadg.extractors.eclab.mpr files may have different meanings based on which other columns are also present in the files, see issue 225. Added a (hopefully extensible) way to tackle such conflicts and clarified the warnings.

  • It looks like there are only 256 columns in mpr files, with higher IDs corresponding to id % 256 in all cases. Thanks to Graham Kimbell for first pointing this out.

yadg version 6.1

https://img.shields.io/static/v1?label=yadg&message=v6.1&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v6.1&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2025-06-03&color=red&logo=pypi

Developed in the ConCat Lab at Technische Universität Berlin (Berlin, DE).

New features in yadg-6.1 are:

  • Added support for extraction from different sources rather than just files.

    • Extraction from a bytes object can be requested using yadg.extractors.extract_from_bytes().

    • Currently, this is only implemented for yadg.extractors.eclab.mpr, which can now extract either Path or bytes objects.

    • Thanks to @carla-terboven <https://github.com/carla-terboven>.

Breaking changes in yadg-6.1 are:

  • The extract() function of each Extractor module now expects a positional source argument instead of a required keyword argument fn. The internal Extractor API has changed in the following way:

    • Passing fn to Extractors is deprecated and will stop working in a future version of yadg.

    • If an extraction from a file is requested, Path must now be passed as source.

    • An extraction from another object type can be implemented by registering a dispatch function for that type in each Extractor.

    Note that the yadg.extractors.extract() function still requests the usual two arguments, i.e. an filename: str and path: Path | str. The new behaviour does not affect this high-level external API.

Bug fixes in yadg-6.1 include:

yadg version 6.0

https://img.shields.io/static/v1?label=yadg&message=v6.0&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v6.0&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2024-11-14&color=red&logo=pypi

Developed in the ConCat Lab at Technische Universität Berlin (Berlin, DE).

New features in yadg-6.0 are:

Breaking changes in yadg-6.0 are:

  • As DataTree is now merged into the xarray module, yadg-6.0 no longer depends on xarray-datatree. For data stored NetCDF files, this shouldn’t be a problem, as they can be read using xarray.open_datatree(). However, if you stored a datatree.DataTree as a pickle, and attempt to load it using xarray.DataTree, this may not work!

  • The parameter specification within the original_metadata entries in mpr should now be consistent with mpt. This means renaming of most parameters in mpr to match mpt, and a much larger parameter map to map from the int values stored in mpr files to their str counterparts. Thanks to Carla Terboven from HZB for the contribution!

  • Support for python <= 3.9 has been dropped. This is a consequence of yadg-6.0 requiring xarray >= 2024.10.0. However, support for python == 3.13 is now included.

Bug fixes in yadg-6.0 include:

yadg version 5.1

https://img.shields.io/static/v1?label=yadg&message=v5.1&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v5.1&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2024-07-01&color=red&logo=pypi

Developed in the ConCat Lab at Technische Universität Berlin (Berlin, DE).

New features since yadg-5.0 are:

Other changes in yadg-5.1 are:

  • The storage of original metadata present in the extracted files as well as yadg-related metadata has been standardised in all extractors.

  • The dataschema has been simplified, eliminating parsers in favour of extractors.

  • The code has been reorganised to highlight the extractor functionality in favour of parsers.

  • Locale-aware functionality now uses babel instead of the built-in locale module. This means the locale argument should now be a str containing at least the 2-letter country code, ideally also a territory (e.g. en_US or de_CH). As of yadg-5.1, no locale.setlocale() is called, making locale procesing in yadg thread-safe.

  • The processing of uncertainties in yadg.extractors.eclab has been changed, with derived quantities now having uncertainties based on the relative error in the measured voltage and / or current.

  • The yadg extract functionality now accepts --locale, --timezone, and --encoding as optional arguments.

  • The yadg.extractors.extract() function now always returns a xarray.DataTree.

  • The meta-only functionality of yadg extract now returns a dict with the node names of the xarray.DataTree as keys and the metadata as values.

Bug fixes in yadg-5.1 include:

  • Fixed incorrect unit assignment when / was substituted to _ in column names.

  • Fixed incorrect annotation of ancillary variables: standard error should be standard_error.

  • Fixed incorrect parsing of units in the yadg.extractors.ezchrom.asc parser. Now, the 25 μV unit will be correctly replaced by just μV (without modifying data), which can be understood by pint.

  • Fixed processing of duplicate column names in yadg.extractors.eclab. Now, duplicate columns will have duplicate prepended to their name and a warning will be raised.

  • Added several new I-range values to yadg.extractors.eclab parsers. Now, I-range values up to 130 are supported.

  • Fixed incorrect column name (Energy to |Energy|) in yadg.extractors.eclab.mpr.

  • Removed column renaming for Analog IN 1 and Analog IN 2 to maintain consistency within yadg.extractors.eclab.

  • Reworked parsing of .mpt file headers in yadg.extractors.eclab.mpt. The parser is now more stable, and the original labels are used without renaming.

yadg version 5.0

https://img.shields.io/static/v1?label=yadg&message=v5.0&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v5.0&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2023-10-15&color=red&logo=pypi

Developed at Technische Universität Berlin (Berlin, DE) and at Empa (Dübendorf, CH).

New features since yadg-4.2 are:

  • Support for DataSchema-5.0.

  • The output format of yadg is now a NetCDF file (.nc), as written by the xarray.DataTree class.

  • The supplied dataschemas are now automatically updated, yielding the latest verison of dataschema prior to parsing.

  • Added yadg extract usage, with optional --meta-only switch, for a quick data or meta data extraction from individual files. This implements the MaRDA Metadata Extractors WG inteface.

Backwards-incompatible changes include:

  • Data post-processing within yadg has been removed, following its deprecation in yadg-4.2. All previously included post-processing functionality should be available in dgpost-2.0. If you find functionality that has been broken since yadg-4.2 and which cannot be implemented in dgpost, please file an issue on GitHub with example files.

  • The yadg update functionality is now only for updating dataschema; the ability to update datagrams has been removed.

  • The parameter transpose from electrochem parser is no longer available; all electrochemistry data is returned as plain timesteps.

  • The valve number in the fusion-json extractor of chromtrace is now stored as data instead of metadata.

Bug fixes include:

  • the electrochem parser now properly parses files with WAIT technique;

  • the electrochem parser understands more versions of the MB technique versions in the biologic.mpr filetype;

  • the electrochem parser can handle localized versions of data in the biologic.mpt filetype;

  • the chromtrace parser now properly unzips data when using the agilent.dx filetype.

yadg version 4.2

https://img.shields.io/static/v1?label=yadg&message=v4.2&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v4.2&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2022-08-29&color=red&logo=pypi

Developed at Empa - Materials Science and Technology, in Dübendorf.

New features since v4.1 are:

  • a new chromdata parser for parsing post-processed chromatography data,

  • the basiccsv parser has an additional parameter strip, allowing the user to strip extra characters (such as " or ') from column headers and data,

  • yadg will now warn you if a newer version is available on PyPI.

Backwards-incompatible changes include:

  • the chromtrace parser now focuses on parsing chromatography traces only, use chromdata for parsing post-processed chromatographic data;

  • the flowdata parser now no longer creates a default "flow" entry in derived data;

  • data post-processing within yadg, including chromatographic trace integration, reflection coefficient processing, and calibration functionality is deprecated in favour of dgpost.

Bug fixes include:

  • the process_row() from the basiccsv parser now handles empty cells by creating sparse datagrams;

  • the drycal filetypes in flowdata should now work for overnight experiments;

  • the metadata in chromdata and chromtrace was modified to only include true metadata as opposed to sample data;

  • the electrochem parser now allows for toggling the transposing of PEIS/GEIS traces.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189. The project is part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future.

yadg version 4.1.0

https://img.shields.io/static/v1?label=yadg&message=v4.1.0&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v4.1.0&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2022-05-13&color=red&logo=pypi

Developed at Empa - Materials Science and Technology, in Dübendorf.

New features since v4.0.0 are:

  • ability to archive raw data and reference the archive from within the datagram, by using yadg preset --process --archive

  • migrated dataschema validation from an in-house validation routine to a Pydantic-based model, see dgbowl_schemas.yadg

  • added the xpstrace parser with support for Panalytical files

  • added support for tomato.json filetypes to the electrochem and dummy parsers

  • added support for fusion.zip filetypes to the chromtrace parser

Bug fixes and other modifications include:

  • yadg.parsers.chromtrace format modified:

    • if peak integration data is present in the raw data file, this is now included in the "raw" key directly. The included quantities are height, area, concentration, and xout for every detected species.

    • if peak integration is to be carried out by yadg, the resulting data is now available in the "derived" key directly. The included quantities are height, area, concentration, in addition to xout which was already part of the spec in v4.0.0.

    • peak integration is now more reliable by applying a threshold around zero for inflection point detection

  • yadg.parsers.electrochem format fixes:

    • the PEIS/GEIS data is split into timesteps, not cycles.

    • NaN and Inf in the metadata of some input formats should now be handled properly, without producing a non-compliant json file.

    • added partial support for mpr files including the ExtDev module

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189. The project is part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future.

yadg version 4.0.1

https://img.shields.io/static/v1?label=yadg&message=v4.0.1&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v4.0.1&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2022-02-14&color=red&logo=pypi

A major rewrite of yadg. Developed at Empa - Materials Science and Technology, in Dübendorf. The code includes documentation, test suite, and is tested using CI. This version has been published in JOSS [Kraus2022b].

Major features are:

  • validation of schemas and datagrams

  • uncertainties and units for all measured quantities stored by default and enforced by the validators

  • raw data is now retained by default and separated from derived data

  • several new parsers, including:

    • electrochemistry: a parser for electrochemistry data, currently supporting BioLogic cyclers via binary .mpr and text .mpt files generated in EC-Lab. Supports standard methods as well as impedance spectroscopy traces (PEIS, GEIS) and complex methods (MB).

    • masstrace: a parser for mass spectroscopy data, supporting binary Quadstar .sac files.

    • basiccsv: a tabulated data parser, with row processing written for modular use in other parsers

    • xpstrace: a parser for x-ray photoelectron spectroscopy data, supporting ULVAC PHI Multipak XPS traces (.spe)

  • large updates to older parsers:

    • chromtrace: an updated, unified chromatography trace parser. The trace integration routine has been completely re-written. The parser now supports multiple formats, including binary Agilent .ch and .dx, and Fusion .json format in addition to the formats previously supported by gctrace.

    • qftrace: updated with several bug fixes and performance improvements, no longer using (np.matrix).

Other, minor features include:

  • a consistent interface for supplying calibration data

  • versioning of datagrams as well as update pathways from older schema files

  • schema preset functionality, for routine applications of yadg to structured raw data archives

  • externaldate functionality for completing and/or replacing deduced timestamps using external sources in a unified fashion

  • validation of units present in the datagrams

New in version 4.0.1:

  • fixed accuracy of phase in electrochemistry files to 1°.

  • fixed splitting of GEIS/PEIS traces in a single file into multiple timesteps

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189. The project is part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future.

yadg version 3.1.0

https://img.shields.io/static/v1?label=yadg&message=v3.1.0&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v3.1.0&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2021-09-11&color=red&logo=pypi

First released version of yadg, containing functionality developed at the Fritz Haber Institute in Berlin. Design ideas, decisions, and workflow described in [Kraus2022a].

Major features are:

  • yadg is now a python package

  • gctrace: support for gas-phase chromatrography, including:

    • trace integration,

    • EZChrom ASCII export format,

    • CHROMTAB ASCII export format,

  • qftrace: support for reflection trace measurements, including:

    • fitting of quality factor using Lorentzian and naive methods

    • fitting of quality factor using Kajfez’s circle fitting method

  • meascsv: support for in-house MCPT logger for flow and temperature data