yadg version history

yadg version 6.0

Warning

This version of yadg is not yet released.

https://img.shields.io/static/v1?label=yadg&message=v6.0&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v6.0&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2024-11-14&color=red&logo=pypi

Developed in the ConCat Lab at Technische Universität Berlin (Berlin, DE).

New features in yadg-6.0 are:

Breaking changes in yadg-6.0 are:

  • As DataTree is now merged into the xarray module, yadg-6.0 no longer depends on xarray-datatree. For data stored NetCDF files, this shouldn’t be a problem, as they can be read using xarray.open_datatree(). However, if you stored a datatree.DataTree as a pickle, and attempt to load it using xarray.DataTree, this may not work!

  • The parameter specification within the original_metadata entries in mpr should now be consistent with mpt. This means renaming of most parameters in mpr to match mpt, and a much larger parameter map to map from the int values stored in mpr files to their str counterparts. Thanks to Carla Terboven from HZB for the contribution!

  • Support for python <= 3.9 has been dropped. This is a consequence of yadg-6.0 requiring xarray >= 2024.10.0. However, support for python == 3.13 is now included.

Bug fixes in yadg-6.0 include:

yadg version 5.1

https://img.shields.io/static/v1?label=yadg&message=v5.1&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v5.1&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2024-07-01&color=red&logo=pypi

Developed in the ConCat Lab at Technische Universität Berlin (Berlin, DE).

New features since yadg-5.0 are:

Other changes in yadg-5.1 are:

  • The storage of original metadata present in the extracted files as well as yadg-related metadata has been standardised in all extractors.

  • The dataschema has been simplified, eliminating parsers in favour of extractors.

  • The code has been reorganised to highlight the extractor functionality in favour of parsers.

  • Locale-aware functionality now uses babel instead of the built-in locale module. This means the locale argument should now be a str containing at least the 2-letter country code, ideally also a territory (e.g. en_US or de_CH). As of yadg-5.1, no locale.setlocale() is called, making locale procesing in yadg thread-safe.

  • The processing of uncertainties in yadg.extractors.eclab has been changed, with derived quantities now having uncertainties based on the relative error in the measured voltage and / or current.

  • The yadg extract functionality now accepts --locale, --timezone, and --encoding as optional arguments.

  • The yadg.extractors.extract() function now always returns a xarray.DataTree.

  • The meta-only functionality of yadg extract now returns a dict with the node names of the xarray.DataTree as keys and the metadata as values.

Bug fixes in yadg-5.1 include:

  • Fixed incorrect unit assignment when / was substituted to _ in column names.

  • Fixed incorrect annotation of ancillary variables: standard error should be standard_error.

  • Fixed incorrect parsing of units in the yadg.extractors.ezchrom.asc parser. Now, the 25 μV unit will be correctly replaced by just μV (without modifying data), which can be understood by pint.

  • Fixed processing of duplicate column names in yadg.extractors.eclab. Now, duplicate columns will have duplicate prepended to their name and a warning will be raised.

  • Added several new I-range values to yadg.extractors.eclab parsers. Now, I-range values up to 130 are supported.

  • Fixed incorrect column name (Energy to |Energy|) in yadg.extractors.eclab.mpr.

  • Removed column renaming for Analog IN 1 and Analog IN 2 to maintain consistency within yadg.extractors.eclab.

  • Reworked parsing of .mpt file headers in yadg.extractors.eclab.mpt. The parser is now more stable, and the original labels are used without renaming.

yadg version 5.0

https://img.shields.io/static/v1?label=yadg&message=v5.0&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v5.0&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2023-10-15&color=red&logo=pypi

Developed at Technische Universität Berlin (Berlin, DE) and at Empa (Dübendorf, CH).

New features since yadg-4.2 are:

  • Support for DataSchema-5.0.

  • The output format of yadg is now a NetCDF file (.nc), as written by the xarray.DataTree class.

  • The supplied dataschemas are now automatically updated, yielding the latest verison of dataschema prior to parsing.

  • Added yadg extract usage, with optional --meta-only switch, for a quick data or meta data extraction from individual files. This implements the MaRDA Metadata Extractors WG inteface.

Backwards-incompatible changes include:

  • Data post-processing within yadg has been removed, following its deprecation in yadg-4.2. All previously included post-processing functionality should be available in dgpost-2.0. If you find functionality that has been broken since yadg-4.2 and which cannot be implemented in dgpost, please file an issue on GitHub with example files.

  • The yadg update functionality is now only for updating dataschema; the ability to update datagrams has been removed.

  • The parameter transpose from electrochem parser is no longer available; all electrochemistry data is returned as plain timesteps.

  • The valve number in the fusion-json extractor of chromtrace is now stored as data instead of metadata.

Bug fixes include:

  • the electrochem parser now properly parses files with WAIT technique;

  • the electrochem parser understands more versions of the MB technique versions in the biologic.mpr filetype;

  • the electrochem parser can handle localized versions of data in the biologic.mpt filetype;

  • the chromtrace parser now properly unzips data when using the agilent.dx filetype.

yadg version 4.2

https://img.shields.io/static/v1?label=yadg&message=v4.2&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v4.2&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2022-08-29&color=red&logo=pypi

Developed at Empa - Materials Science and Technology, in Dübendorf.

New features since v4.1 are:

  • a new chromdata parser for parsing post-processed chromatography data,

  • the basiccsv parser has an additional parameter strip, allowing the user to strip extra characters (such as " or ') from column headers and data,

  • yadg will now warn you if a newer version is available on PyPI.

Backwards-incompatible changes include:

  • the chromtrace parser now focuses on parsing chromatography traces only, use chromdata for parsing post-processed chromatographic data;

  • the flowdata parser now no longer creates a default "flow" entry in derived data;

  • data post-processing within yadg, including chromatographic trace integration, reflection coefficient processing, and calibration functionality is deprecated in favour of dgpost.

Bug fixes include:

  • the process_row() from the basiccsv parser now handles empty cells by creating sparse datagrams;

  • the drycal filetypes in flowdata should now work for overnight experiments;

  • the metadata in chromdata and chromtrace was modified to only include true metadata as opposed to sample data;

  • the electrochem parser now allows for toggling the transposing of PEIS/GEIS traces.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189. The project is part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future.

yadg version 4.1.0

https://img.shields.io/static/v1?label=yadg&message=v4.1.0&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v4.1.0&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2022-05-13&color=red&logo=pypi

Developed at Empa - Materials Science and Technology, in Dübendorf.

New features since v4.0.0 are:

  • ability to archive raw data and reference the archive from within the datagram, by using yadg preset --process --archive

  • migrated dataschema validation from an in-house validation routine to a Pydantic-based model, see dgbowl_schemas.yadg

  • added the xpstrace parser with support for Panalytical files

  • added support for tomato.json filetypes to the electrochem and dummy parsers

  • added support for fusion.zip filetypes to the chromtrace parser

Bug fixes and other modifications include:

  • yadg.parsers.chromtrace format modified:

    • if peak integration data is present in the raw data file, this is now included in the "raw" key directly. The included quantities are height, area, concentration, and xout for every detected species.

    • if peak integration is to be carried out by yadg, the resulting data is now available in the "derived" key directly. The included quantities are height, area, concentration, in addition to xout which was already part of the spec in v4.0.0.

    • peak integration is now more reliable by applying a threshold around zero for inflection point detection

  • yadg.parsers.electrochem format fixes:

    • the PEIS/GEIS data is split into timesteps, not cycles.

    • NaN and Inf in the metadata of some input formats should now be handled properly, without producing a non-compliant json file.

    • added partial support for mpr files including the ExtDev module

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189. The project is part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future.

yadg version 4.0.1

https://img.shields.io/static/v1?label=yadg&message=v4.0.1&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v4.0.1&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2022-02-14&color=red&logo=pypi

A major rewrite of yadg. Developed at Empa - Materials Science and Technology, in Dübendorf. The code includes documentation, test suite, and is tested using CI. This version has been published in JOSS [Kraus2022b].

Major features are:

  • validation of schemas and datagrams

  • uncertainties and units for all measured quantities stored by default and enforced by the validators

  • raw data is now retained by default and separated from derived data

  • several new parsers, including:

    • electrochemistry: a parser for electrochemistry data, currently supporting BioLogic cyclers via binary .mpr and text .mpt files generated in EC-Lab. Supports standard methods as well as impedance spectroscopy traces (PEIS, GEIS) and complex methods (MB).

    • masstrace: a parser for mass spectroscopy data, supporting binary Quadstar .sac files.

    • basiccsv: a tabulated data parser, with row processing written for modular use in other parsers

    • xpstrace: a parser for x-ray photoelectron spectroscopy data, supporting ULVAC PHI Multipak XPS traces (.spe)

  • large updates to older parsers:

    • chromtrace: an updated, unified chromatography trace parser. The trace integration routine has been completely re-written. The parser now supports multiple formats, including binary Agilent .ch and .dx, and Fusion .json format in addition to the formats previously supported by gctrace.

    • qftrace: updated with several bug fixes and performance improvements, no longer using (np.matrix).

Other, minor features include:

  • a consistent interface for supplying calibration data

  • versioning of datagrams as well as update pathways from older schema files

  • schema preset functionality, for routine applications of yadg to structured raw data archives

  • externaldate functionality for completing and/or replacing deduced timestamps using external sources in a unified fashion

  • validation of units present in the datagrams

New in version 4.0.1:

  • fixed accuracy of phase in electrochemistry files to 1°.

  • fixed splitting of GEIS/PEIS traces in a single file into multiple timesteps

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189. The project is part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future.

yadg version 3.1.0

https://img.shields.io/static/v1?label=yadg&message=v3.1.0&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v3.1.0&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2021-09-11&color=red&logo=pypi

First released version of yadg, containing functionality developed at the Fritz Haber Institute in Berlin. Design ideas, decisions, and workflow described in [Kraus2022a].

Major features are:

  • yadg is now a python package

  • gctrace: support for gas-phase chromatrography, including:

    • trace integration,

    • EZChrom ASCII export format,

    • CHROMTAB ASCII export format,

  • qftrace: support for reflection trace measurements, including:

    • fitting of quality factor using Lorentzian and naive methods

    • fitting of quality factor using Kajfez’s circle fitting method

  • meascsv: support for in-house MCPT logger for flow and temperature data