yadg version history

yadg version 4.2

Developed at Empa - Materials Science and Technology, in Dübendorf.

New features since v4.1 are:

• a new chromdata parser for parsing post-processed chromatography data,

• the basiccsv parser has an additional parameter strip, allowing the user to strip extra characters (such as " or ') from column headers and data,

• yadg will now warn you if a newer version is available on PyPI.

Backwards-incompatible changes include:

• the chromtrace parser now focuses on parsing chromatography traces only, use chromdata for parsing post-processed chromatographic data;

• the flowdata parser now no longer creates a default "flow" entry in derived data;

• data post-processing within yadg, including chromatographic trace integration, reflection coefficient processing, and calibration functionality is deprecated in favour of dgpost.

Bug fixes include:

• the process_row() from the basiccsv parser now handles empty cells by creating sparse datagrams;

• the drycal filetypes in flowdata should now work for overnight experiments;

• the metadata in chromdata and chromtrace was modified to only include true metadata as opposed to sample data;

• the electrochem parser now allows for toggling the transposing of PEIS/GEIS traces.

yadg version 4.1.0

Developed at Empa - Materials Science and Technology, in Dübendorf.

New features since v4.0.0 are:

• ability to archive raw data and reference the archive from within the datagram, by using yadg preset --process --archive

• migrated dataschema validation from an in-house validation routine to a Pydantic-based model, see dgbowl_schemas.yadg

• added the xpstrace parser with support for Panalytical files

• added support for tomato.json filetypes to the electrochem and dummy parsers

• added support for fusion.zip filetypes to the chromtrace parser

Bug fixes and other modifications include:

• yadg.parsers.chromtrace format modified:

  • if peak integration data is present in the raw data file, this is now included in the "raw" key directly. The included quantities are height, area, concentration, and xout for every detected species.

  • if peak integration is to be carried out by yadg, the resulting data is now available in the "derived" key directly. The included quantities are height, area, concentration, in addition to xout which was already part of the spec in v4.0.0.

  • peak integration is now more reliable by applying a threshold around zero for inflection point detection

• yadg.parsers.electrochem format fixes:

  • the PEIS/GEIS data is split into timesteps, not cycles.

  • NaN and Inf in the metadata of some input formats should now be handled properly, without producing a non-compliant json file.

  • added partial support for mpr files including the ExtDev module

yadg version 4.0.1

A major rewrite of yadg. Developed at Empa - Materials Science and Technology, in Dübendorf. The code includes documentation, test suite, and is tested using CI. This version has been published in JOSS [Kraus2022b].

Major features are:

• validation of schemas and datagrams

• uncertainties and units for all measured quantities stored by default and enforced by the validators

• raw data is now retained by default and separated from derived data

• several new parsers, including:

  • electrochemistry: a parser for electrochemistry data, currently supporting BioLogic cyclers via binary .mpr and text .mpt files generated in EC-Lab. Supports standard methods as well as impedance spectroscopy traces (PEIS, GEIS) and complex methods (MB).

  • masstrace: a parser for mass spectroscopy data, supporting binary Quadstar .sac files.

  • basiccsv: a tabulated data parser, with row processing written for modular use in other parsers

  • xpstrace: a parser for x-ray photoelectron spectroscopy data, supporting ULVAC PHI Multipak XPS traces (.spe)

• large updates to older parsers:

  • chromtrace: an updated, unified chromatography trace parser. The trace integration routine has been completely re-written. The parser now supports multiple formats, including binary Agilent .ch and .dx, and Fusion .json format in addition to the formats previously supported by gctrace.

  • qftrace: updated with several bug fixes and performance improvements, no longer using (np.matrix).

Other, minor features include:

• a consistent interface for supplying calibration data

• versioning of datagrams as well as update pathways from older schema files

• schema preset functionality, for routine applications of yadg to structured raw data archives

• externaldate functionality for completing and/or replacing deduced timestamps using external sources in a unified fashion

• validation of units present in the datagrams

New in version 4.0.1:

• fixed accuracy of phase in electrochemistry files to 1°.

• fixed splitting of GEIS/PEIS traces in a single file into multiple timesteps

yadg version 3.1.0

First released version of yadg, containing functionality developed at the Fritz Haber Institute in Berlin. Design ideas, decisions, and workflow described in [Kraus2022a].

Major features are:

• yadg is now a python package

• gctrace: support for gas-phase chromatrography, including:

  • trace integration,

  • EZChrom ASCII export format,

  • CHROMTAB ASCII export format,

• qftrace: support for reflection trace measurements, including:

  • fitting of quality factor using Lorentzian and naive methods

  • fitting of quality factor using Kajfez’s circle fitting method [Kajfez1994]

• meascsv: support for in-house MCPT logger for flow and temperature data