yadg.dgutils package

yadg.dgutils.get_yadg_metadata() → dict

Returns current yadg metadata.

yadg.dgutils.now(asstr: bool = False, tz: timezone = datetime.timezone.utc) → float | str

Wrapper around datetime.now()

A convenience function for returning the current time as a ISO 8601 or as a Unix timestamp.

yadg.dgutils.infer_timestamp_from(*, headers: list | None = None, spec: TimestampSpec | None = None, timezone: str) → tuple[list, Callable, bool]

Convenience function for timestamping

Given a set of headers, and an optional specification, return an array containing column indices from which a timestamp in a given row can be computed, as well as the function which will compute the timestamp given the returned array.

Parameters:

  • headers – An array of strings. If spec is not supplied, must contain either “uts” (float) or “timestep” (str) (conforming to ISO 8601).

  • spec – A specification of timestamp elements with associated column indices and optional formats. Currently accepted combinations of keys are: “uts”; “timestamp”; “date” and / or “time”.

  • tz – Timezone to use for conversion. By default, UTC is used.

Returns:

(datecolumns, datefunc, fulldate) – A tuple containing a list of indices of columns, a Callable to which the columns have to be passed to obtain a uts timestamp, and whether the determined timestamp is full or partial.

Return type:

tuple[list, Callable, bool]

yadg.dgutils.str_to_uts(*, timestamp: str, timezone: str, format: str | None = None, strict: bool = True) → float | None

Converts a string to POSIX timestamp.

If the optional format is specified, the timestamp string is processed using the datetime.datetime.strptime() function; if no format is supplied, an ISO 8601 format is assumed and an attempt to parse using dateutil.parser.parse() is made.

Parameters:

  • timestamp – A string containing the timestamp.

  • format – Optional format string for parsing of the timestamp.

  • timezone – Optional timezone of the timestamp. By default, “UTC”.

  • strict – Whether to re-raise any parsing errors.

Returns:

uts – Returns the POSIX timestamp if successful, otherwise None.

Return type:

Union[float, None]

yadg.dgutils.ole_to_uts(ole_timestamp: float, timezone: str) → float

Converts a Microsoft OLE timestamp into a POSIX timestamp.

The OLE automation date format is a floating point value, counting days since midnight 30 December 1899. Hours and minutes are represented as fractional days.

https://devblogs.microsoft.com/oldnewthing/20030905-02/?p=42653

Parameters:

  • ole_timestamp – A timestamp in Microsoft OLE format.

  • timezone – String desribing the timezone.

Returns:

time – The corresponding Unix timestamp.

Return type:

float

yadg.dgutils.complete_timestamps(*, timesteps: list, fn: str, spec: ExternalDate, timezone: str) → list[float]

Timestamp completing function.

This function allows for completing or overriding the uts timestamps determined by the individual parsers. yadg enters this function for any parser which does not return a full timestamp, as well as if the externaldate specification is specified by the user.

The externaldate specification is as follows:

pydantic model dgbowl_schemas.yadg.dataschema_5_0.externaldate.ExternalDate

Supply timestamping information that are external to the processed file.

Config:

  • extra: str = forbid

field using: ExternalDateFile | ExternalDateFilename | ExternalDateISOString | ExternalDateUTSOffset [Required]

Specification of the external date format.

field mode: Literal['add', 'replace'] = 'add'

Whether the external timestamps should be added to or should replace the parsed data.

The using key specifies how an external timestamp is created. Only one entry in using is permitted. By default, this entry is:

using:
  filename:
    format: "%Y-%m-%d-%H-%M-%S"
    len: 19

Which means the code will attempt to deduce the timestamp from the path of the processed file (fn), using the first 19 characters of the base filename according to the above format (eg. “2021-12-31-13-45-00”).

If file is specified, the handling of timestamps is handed off to timestamps_from_file().

The mode key specifies whether the offsets determined in this function are added to the current timestamps (eg. date offset being added to time) or whether they should replace the existing timestamps completely.

As a measure of last resort, the mtime of the fn is used. mtime is preferred to ctime, as the former has a more consistent cross-platform behaviour.

Parameters:

  • timesteps – A list of timesteps generated from a single file, fn.

  • fn – Filename used to create timesteps.

  • spec – externaldate specification part of the schema.

  • timezone – Timezone, defaults to “UTC”.

yadg.dgutils.complete_uts(ds: Dataset, filename: str, externaldate: BaseModel, timezone: str) → Dataset

A helper function ensuring that the Dataset ds contains a dimension "uts", and that the timestamps in "uts" are completed as instructed in the externaldate specification.

yadg.dgutils.update_schema(object: list | dict | BaseModel | BaseModel) → DataSchema

The yadg update worker function.

The main purpose is to allow a simple update pathway from older versions of dataschema files to the current latest and greatest.

Currently supports:

  • updating dataschema version 3.1 to 4.0 using routines in yadg,

  • updating dataschema version 4.0 and above to the latest dataschema using the in-built .update() mechanism.

Parameters:

object – The object to be updated

Returns:

newobj – The updated and validated “datagram” or “schema”.

Return type:

dict

yadg.dgutils.schema_from_preset(preset: DataSchema, folder: str) → DataSchema
yadg.dgutils.read_value(data: bytes, offset: int, dtype: dtype | str, encoding: str = 'windows-1252') → Any

Reads a single value or a set of values from a buffer at a certain offset.

Just a handy wrapper for np.frombuffer(…, count=1) With the added benefit of allowing the ‘pascal’ keyword as an indicator for a length-prefixed string.

The read value is converted to a built-in datatype using np.dtype.item().

Parameters:

  • data – An object that exposes the buffer interface. Here always bytes.

  • offset – Start reading the buffer from this offset (in bytes).

  • dtype – Data-type to read in.

  • encoding – The encoding of the bytes to be converted.

Returns:

The unpacked and converted value from the buffer.

Return type:

Any

yadg.dgutils.sanitize_units(units: str | dict[str, str] | list[str]) → str | dict[str, str] | list[str]

Unit sanitizer.

This sanitizer should be used where user-supplied units are likely to occur, such as in the parsers yadg.extractors.basic.csv. Currently, only two replacements are done:

  • “Bar” is replaced with “bar”

  • “Deg C” is replace with “degC

Use with caution.

Parameters:

units – Object containing string units.

yadg.dgutils.dicts_to_dataset(data: dict[str, list[Any]], meta: dict[str, list[Any]], units: dict[str, str] = {}, fulldate: bool = True) → Dataset
yadg.dgutils.append_dicts(vals: dict[str, Any], devs: dict[str, Any], data: dict[str, list[Any]], meta: dict[str, list[Any]], fn: str | None = None, li: int = 0) → None
yadg.dgutils.merge_dicttrees(vals: dict, fvals: dict, mode: str) → dict

A helper function that merges two DataTree.to_dict() objects by concatenating the new values in fvals to the existing ones in vals.

yadg.dgutils.merge_meta(old: dict, new: dict)

yadg.dgutils.btools module

yadg.dgutils.btools.read_pascal_string(pascal_bytes: bytes, encoding: str = 'windows-1252') → str

Parses a length-prefixed string given some encoding.

Parameters:

  • bytes – The bytes of the string starting at the length-prefix byte.

  • encoding – The encoding of the string to be converted.

Returns:

The string decoded from the input bytes.

Return type:

str

yadg.dgutils.btools.read_value(data: bytes, offset: int, dtype: dtype | str, encoding: str = 'windows-1252') → Any

Reads a single value or a set of values from a buffer at a certain offset.

Just a handy wrapper for np.frombuffer(…, count=1) With the added benefit of allowing the ‘pascal’ keyword as an indicator for a length-prefixed string.

The read value is converted to a built-in datatype using np.dtype.item().

Parameters:

  • data – An object that exposes the buffer interface. Here always bytes.

  • offset – Start reading the buffer from this offset (in bytes).

  • dtype – Data-type to read in.

  • encoding – The encoding of the bytes to be converted.

Returns:

The unpacked and converted value from the buffer.

Return type:

Any

yadg.dgutils.dateutils module

yadg.dgutils.dateutils.now(asstr: bool = False, tz: timezone = datetime.timezone.utc) → float | str

Wrapper around datetime.now()

A convenience function for returning the current time as a ISO 8601 or as a Unix timestamp.

yadg.dgutils.dateutils.ole_to_uts(ole_timestamp: float, timezone: str) → float

Converts a Microsoft OLE timestamp into a POSIX timestamp.

The OLE automation date format is a floating point value, counting days since midnight 30 December 1899. Hours and minutes are represented as fractional days.

https://devblogs.microsoft.com/oldnewthing/20030905-02/?p=42653

Parameters:

  • ole_timestamp – A timestamp in Microsoft OLE format.

  • timezone – String desribing the timezone.

Returns:

time – The corresponding Unix timestamp.

Return type:

float

yadg.dgutils.dateutils.str_to_uts(*, timestamp: str, timezone: str, format: str | None = None, strict: bool = True) → float | None

Converts a string to POSIX timestamp.

If the optional format is specified, the timestamp string is processed using the datetime.datetime.strptime() function; if no format is supplied, an ISO 8601 format is assumed and an attempt to parse using dateutil.parser.parse() is made.

Parameters:

  • timestamp – A string containing the timestamp.

  • format – Optional format string for parsing of the timestamp.

  • timezone – Optional timezone of the timestamp. By default, “UTC”.

  • strict – Whether to re-raise any parsing errors.

Returns:

uts – Returns the POSIX timestamp if successful, otherwise None.

Return type:

Union[float, None]

yadg.dgutils.dateutils.infer_timestamp_from(*, headers: list | None = None, spec: TimestampSpec | None = None, timezone: str) → tuple[list, Callable, bool]

Convenience function for timestamping

Given a set of headers, and an optional specification, return an array containing column indices from which a timestamp in a given row can be computed, as well as the function which will compute the timestamp given the returned array.

Parameters:

  • headers – An array of strings. If spec is not supplied, must contain either “uts” (float) or “timestep” (str) (conforming to ISO 8601).

  • spec – A specification of timestamp elements with associated column indices and optional formats. Currently accepted combinations of keys are: “uts”; “timestamp”; “date” and / or “time”.

  • tz – Timezone to use for conversion. By default, UTC is used.

Returns:

(datecolumns, datefunc, fulldate) – A tuple containing a list of indices of columns, a Callable to which the columns have to be passed to obtain a uts timestamp, and whether the determined timestamp is full or partial.

Return type:

tuple[list, Callable, bool]

yadg.dgutils.dateutils.complete_timestamps(*, timesteps: list, fn: str, spec: ExternalDate, timezone: str) → list[float]

Timestamp completing function.

This function allows for completing or overriding the uts timestamps determined by the individual parsers. yadg enters this function for any parser which does not return a full timestamp, as well as if the externaldate specification is specified by the user.

The externaldate specification is as follows:

pydantic model dgbowl_schemas.yadg.dataschema_5_0.externaldate.ExternalDate

Supply timestamping information that are external to the processed file.

Config:

  • extra: str = forbid

field using: ExternalDateFile | ExternalDateFilename | ExternalDateISOString | ExternalDateUTSOffset [Required]

Specification of the external date format.

field mode: Literal['add', 'replace'] = 'add'

Whether the external timestamps should be added to or should replace the parsed data.

The using key specifies how an external timestamp is created. Only one entry in using is permitted. By default, this entry is:

using:
  filename:
    format: "%Y-%m-%d-%H-%M-%S"
    len: 19

Which means the code will attempt to deduce the timestamp from the path of the processed file (fn), using the first 19 characters of the base filename according to the above format (eg. “2021-12-31-13-45-00”).

If file is specified, the handling of timestamps is handed off to timestamps_from_file().

The mode key specifies whether the offsets determined in this function are added to the current timestamps (eg. date offset being added to time) or whether they should replace the existing timestamps completely.

As a measure of last resort, the mtime of the fn is used. mtime is preferred to ctime, as the former has a more consistent cross-platform behaviour.

Parameters:

  • timesteps – A list of timesteps generated from a single file, fn.

  • fn – Filename used to create timesteps.

  • spec – externaldate specification part of the schema.

  • timezone – Timezone, defaults to “UTC”.

yadg.dgutils.dateutils.timestamps_from_file(path: str, type: str, match: str, timezone: str) → float | list[float]

Load timestamps from file.

This function enables loading timestamps from file specified by the path. The currently supported file formats include json and pkl, which must contain a top-level Mapping with a key that is matched by match, or a top-level Iterable, both containing str or float -like objects that can be processed into an Unix timestamp.

Parameters:

  • path – Location of the external file.

  • type – Type of the external file. Currently, "json", "pkl" are supported.

  • match – An optional key to match if the object in path is a Mapping.

  • timezone – An optional timezone string, defaults to “UTC”

Returns:

parseddata – A single or a list of POSIX timestamps.

Return type:

Union[float, list[float]]

yadg.dgutils.dateutils.complete_uts(ds: Dataset, filename: str, externaldate: BaseModel, timezone: str) → Dataset

A helper function ensuring that the Dataset ds contains a dimension "uts", and that the timestamps in "uts" are completed as instructed in the externaldate specification.

yadg.dgutils.dsutils module

yadg.dgutils.dsutils.append_dicts(vals: dict[str, Any], devs: dict[str, Any], data: dict[str, list[Any]], meta: dict[str, list[Any]], fn: str | None = None, li: int = 0) → None
yadg.dgutils.dsutils.dicts_to_dataset(data: dict[str, list[Any]], meta: dict[str, list[Any]], units: dict[str, str] = {}, fulldate: bool = True) → Dataset
yadg.dgutils.dsutils.merge_dicttrees(vals: dict, fvals: dict, mode: str) → dict

A helper function that merges two DataTree.to_dict() objects by concatenating the new values in fvals to the existing ones in vals.

yadg.dgutils.dsutils.merge_meta(old: dict, new: dict)

yadg.dgutils.helpers module

yadg.dgutils.helpers.get_yadg_metadata() → dict

Returns current yadg metadata.

yadg.dgutils.helpers.deprecated(arg, depin='4.2', depout='5.0') → None

yadg.dgutils.pintutils module

pint compatibility functions in yadg.

This package defines ureg, a pint.UnitRegistry used for validation of datagrams in yadg. The default SI pint.UnitRegistry is extended by definitions of fractional quantities (%, ppm, etc.), standard volumetric quantities (smL/min, sccm), and other dimensionless “units” present in several file types.

yadg.dgutils.pintutils.sanitize_units(units: str | dict[str, str] | list[str]) → str | dict[str, str] | list[str]

Unit sanitizer.

This sanitizer should be used where user-supplied units are likely to occur, such as in the parsers yadg.extractors.basic.csv. Currently, only two replacements are done:

  • “Bar” is replaced with “bar”

  • “Deg C” is replace with “degC

Use with caution.

Parameters:

units – Object containing string units.

yadg.dgutils.schemautils module

yadg.dgutils.schemautils.calib_3to4(oldcal: dict, caltype: str) → dict
yadg.dgutils.schemautils.schema_3to4(oldschema: list) → dict
yadg.dgutils.schemautils.update_schema(object: list | dict | BaseModel | BaseModel) → DataSchema

The yadg update worker function.

The main purpose is to allow a simple update pathway from older versions of dataschema files to the current latest and greatest.

Currently supports:

  • updating dataschema version 3.1 to 4.0 using routines in yadg,

  • updating dataschema version 4.0 and above to the latest dataschema using the in-built .update() mechanism.

Parameters:

object – The object to be updated

Returns:

newobj – The updated and validated “datagram” or “schema”.

Return type:

dict

yadg.dgutils.schemautils.schema_from_preset(preset: DataSchema, folder: str) → DataSchema