yadg version history

yadg version 5.0

https://img.shields.io/static/v1?label=yadg&message=v5.0&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v5.0&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2023-10-15&color=red&logo=pypi

Developed at Technische Universität Berlin (Berlin, DE) and at Empa (Dübendorf, CH).

New features since yadg-4.2 are:

  • Support for DataSchema-5.0.

  • The output format of yadg is now a NetCDF file (.nc), as written by the datatree.DataTree class.

  • The supplied dataschemas are now automatically updated, yielding the latest verison of dataschema prior to parsing.

  • Added yadg extract usage, with optional --meta-only switch, for a quick data or meta data extraction from individual files. This implements the MaRDA Metadata Extractors WG inteface.

Backwards-incompatible changes include:

  • Data post-processing within yadg has been removed, following its deprecation in yadg-4.2. All previously included post-processing functionality should be available in dgpost-2.0. If you find functionality that has been broken since yadg-4.2 and which cannot be implemented in dgpost, please file an issue on GitHub with example files.

  • The yadg update functionality is now only for updating dataschema; the ability to update datagrams has been removed.

  • The parameter transpose from electrochem parser is no longer available; all electrochemistry data is returned as plain timesteps.

  • The valve number in the fusion-json extractor of chromtrace is now stored as data instead of metadata.

Bug fixes include:

  • the electrochem parser now properly parses files with WAIT technique;

  • the electrochem parser understands more versions of the MB technique versions in the biologic.mpr filetype;

  • the electrochem parser can handle localized versions of data in the biologic.mpt filetype;

  • the chromtrace parser now properly unzips data when using the agilent.dx filetype.

yadg version 4.2

https://img.shields.io/static/v1?label=yadg&message=v4.2&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v4.2&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2022-08-29&color=red&logo=pypi

Developed at Empa - Materials Science and Technology, in Dübendorf.

New features since v4.1 are:

  • a new chromdata parser for parsing post-processed chromatography data,

  • the basiccsv parser has an additional parameter strip, allowing the user to strip extra characters (such as " or ') from column headers and data,

  • yadg will now warn you if a newer version is available on PyPI.

Backwards-incompatible changes include:

  • the chromtrace parser now focuses on parsing chromatography traces only, use chromdata for parsing post-processed chromatographic data;

  • the flowdata parser now no longer creates a default "flow" entry in derived data;

  • data post-processing within yadg, including chromatographic trace integration, reflection coefficient processing, and calibration functionality is deprecated in favour of dgpost.

Bug fixes include:

  • the process_row() from the basiccsv parser now handles empty cells by creating sparse datagrams;

  • the drycal filetypes in flowdata should now work for overnight experiments;

  • the metadata in chromdata and chromtrace was modified to only include true metadata as opposed to sample data;

  • the electrochem parser now allows for toggling the transposing of PEIS/GEIS traces.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189. The project is part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future.

yadg version 4.1.0

https://img.shields.io/static/v1?label=yadg&message=v4.1.0&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v4.1.0&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2022-05-13&color=red&logo=pypi

Developed at Empa - Materials Science and Technology, in Dübendorf.

New features since v4.0.0 are:

  • ability to archive raw data and reference the archive from within the datagram, by using yadg preset --process --archive

  • migrated dataschema validation from an in-house validation routine to a Pydantic-based model, see dgbowl_schemas.yadg

  • added the xpstrace parser with support for Panalytical files

  • added support for tomato.json filetypes to the electrochem and dummy parsers

  • added support for fusion.zip filetypes to the chromtrace parser

Bug fixes and other modifications include:

  • yadg.parsers.chromtrace format modified:

    • if peak integration data is present in the raw data file, this is now included in the "raw" key directly. The included quantities are height, area, concentration, and xout for every detected species.

    • if peak integration is to be carried out by yadg, the resulting data is now available in the "derived" key directly. The included quantities are height, area, concentration, in addition to xout which was already part of the spec in v4.0.0.

    • peak integration is now more reliable by applying a threshold around zero for inflection point detection

  • yadg.parsers.electrochem format fixes:

    • the PEIS/GEIS data is split into timesteps, not cycles.

    • NaN and Inf in the metadata of some input formats should now be handled properly, without producing a non-compliant json file.

    • added partial support for mpr files including the ExtDev module

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189. The project is part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future.

yadg version 4.0.1

https://img.shields.io/static/v1?label=yadg&message=v4.0.1&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v4.0.1&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2022-02-14&color=red&logo=pypi

A major rewrite of yadg. Developed at Empa - Materials Science and Technology, in Dübendorf. The code includes documentation, test suite, and is tested using CI. This version has been published in JOSS [Kraus2022b].

Major features are:

  • validation of schemas and datagrams

  • uncertainties and units for all measured quantities stored by default and enforced by the validators

  • raw data is now retained by default and separated from derived data

  • several new parsers, including:

    • electrochemistry: a parser for electrochemistry data, currently supporting BioLogic cyclers via binary .mpr and text .mpt files generated in EC-Lab. Supports standard methods as well as impedance spectroscopy traces (PEIS, GEIS) and complex methods (MB).

    • masstrace: a parser for mass spectroscopy data, supporting binary Quadstar .sac files.

    • basiccsv: a tabulated data parser, with row processing written for modular use in other parsers

    • xpstrace: a parser for x-ray photoelectron spectroscopy data, supporting ULVAC PHI Multipak XPS traces (.spe)

  • large updates to older parsers:

    • chromtrace: an updated, unified chromatography trace parser. The trace integration routine has been completely re-written. The parser now supports multiple formats, including binary Agilent .ch and .dx, and Fusion .json format in addition to the formats previously supported by gctrace.

    • qftrace: updated with several bug fixes and performance improvements, no longer using (np.matrix).

Other, minor features include:

  • a consistent interface for supplying calibration data

  • versioning of datagrams as well as update pathways from older schema files

  • schema preset functionality, for routine applications of yadg to structured raw data archives

  • externaldate functionality for completing and/or replacing deduced timestamps using external sources in a unified fashion

  • validation of units present in the datagrams

New in version 4.0.1:

  • fixed accuracy of phase in electrochemistry files to 1°.

  • fixed splitting of GEIS/PEIS traces in a single file into multiple timesteps

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 957189. The project is part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future.

yadg version 3.1.0

https://img.shields.io/static/v1?label=yadg&message=v3.1.0&color=blue&logo=github https://img.shields.io/static/v1?label=yadg&message=v3.1.0&color=blue&logo=pypi https://img.shields.io/static/v1?label=release%20date&message=2021-09-11&color=red&logo=pypi

First released version of yadg, containing functionality developed at the Fritz Haber Institute in Berlin. Design ideas, decisions, and workflow described in [Kraus2022a].

Major features are:

  • yadg is now a python package

  • gctrace: support for gas-phase chromatrography, including:

    • trace integration,

    • EZChrom ASCII export format,

    • CHROMTAB ASCII export format,

  • qftrace: support for reflection trace measurements, including:

    • fitting of quality factor using Lorentzian and naive methods

    • fitting of quality factor using Kajfez’s circle fitting method

  • meascsv: support for in-house MCPT logger for flow and temperature data