yadg.dgutils package

Submodules

yadg.dgutils.btools module

yadg.dgutils.btools.read_from_buffer(buf, offset, dtype, count=1)

Return type: Union[str, ndarray]

yadg.dgutils.btools.read_from_file(f, offset, dtype, count=1)

Return type: Union[str, ndarray]

yadg.dgutils.btools.read_value(object, offset, dtype, count=1)

Return type: Union[str, ndarray, int, float]

yadg.dgutils.calib module

yadg.dgutils.calib.calib_handler(x, calib=None, atol=0.0, rtol=0.0)

Calibration handling function.

Returns y calculated from x and the other supplied arguments.

The "calib" parameter is a (dict) in the following format:

- calib:
  - linear:                # y = slope * x + intercept
    intercept:  !!float
    slope:      !!float
  - inverse:               # y = (x - intercept) / slope
    intercept:  !!float
    slope:      !!float
  - polynomial:            # y = sum(cN * x**N)
    c0:         !!float
    ...
    cN:         !!float
  atol:         !!float    # absolute uncertainty of y
  rtol:         !!float    # relative uncertainty of y

Parameters

x (Union[float, AffineScalarFunc]) – The raw value
calib (Optional[dict]) – Calibration dictionary, specified using the format described above. If empty, corresponds to no transformation (i.e. linear with "slope" = 1) and a rtol = 1e-3.
atol (float) – Optional absolute uncertainty of y, overrides that provided in calib.
rtol (float) – Optional relative uncertainty of y, overrides that provided in calib.

Returns

y – The derived value.

Return type

UFloat

yadg.dgutils.dateutils module

yadg.dgutils.dateutils.complete_timestamps(timesteps, fn, spec={}, timezone='UTC')

Timestamp completing function.

This function allows for completing or overriding the uts timestamps determined by the individual parsers. yadg enters this function for any parser which does not return a full timestamp, as well as if the externaldate specification is specified by the user.

The externaldate specification is as follows:

externaldate:
  from:
    file:
        path:    !!str     # path of the external date file
        type:    !!str     # type of the external date file
        match:   !!str     # string to match timestamps with
    isostring:   !!str     # string offset parsed with dateutil
    utsoffset:   !!float   # float offset
    filename:
        format:  !!str     # strptime-style format
        len:     !!int     # length of filename string to match
  mode:          !!str     # "add" or "replace" timestamp

The "from" key specifies how an external timestamp is created. Only one entry in "from" is permitted. By default, this entry is:

externaldate:
  from:
    filename:
      format: "%Y-%m-%d-%H-%M-%S"
      len: 19

Which means the code will attempt to deduce the timestamp from the path of the processed file (fn), using the first 19 characters of the base filename according to the above format (eg. “2021-12-31-13-45-00”).

If "file" is specified, the handling of timestamps is handed off to timestamps_from_file().

The "mode" key specifies whether the offsets determined in this function are added to the current timestamps (eg. date offset being added to time) or whether they should replace the existing timestamps completely.

As a measure of last resort, the mtime of the fn is used. mtime is preferred to ctime, as the former has a more consistent cross-platform behaviour.

Parameters

timesteps (list) – A list of timesteps generated from a single file, fn.
fn (str) – Filename used to create timesteps.
spec (dict) – externaldate specification part of the schema.
timezone (str) – Timezone, defaults to “UTC”.

Return type

None

yadg.dgutils.dateutils.infer_timestamp_from(headers=None, spec=None, timezone='UTC')

Convenience function for timestamping

Given a set of headers, and an optional specification, return an array containing column indices from which a timestamp in a given row can be computed, as well as the function which will compute the timestamp given the returned array.

Parameters

headers (Optional[list]) – An array of strings. If spec is not supplied, must contain either “uts” (float) or “timestep” (str) (conforming to ISO 8601).
spec (Optional[dict]) – A specification of timestamp elements with associated column indices and optional formats. Currently accepted combinations of keys are: “uts”; “timestamp”; “date” and / or “time”.
tz – Timezone to use for conversion. By default, UTC is used.

Returns

(datecolumns, datefunc, fulldate) – A tuple containing a list of indices of columns, a Callable to which the columns have to be passed to obtain a uts timestamp, and whether the determined timestamp is full or partial.

Return type

tuple[list, Callable, bool]

yadg.dgutils.dateutils.now(asstr=False, tz=datetime.timezone.utc)

Wrapper around datetime.now()

A convenience function for returning the current time as a ISO 8601 or as a unix timestamp.

Return type: Union[float, str]

yadg.dgutils.dateutils.ole_to_uts(ole_timestamp)

Converts a Microsoft OLE timestamp into a POSIX timestamp.

The OLE automation date format is a floating point value, counting days since midnight 30 December 1899. Hours and minutes are represented as fractional days.

https://devblogs.microsoft.com/oldnewthing/20030905-02/?p=42653

Parameters: ole_timestamp (float) – A timestamp in Microsoft OLE format.
Returns: time – The corresponding Unix timestamp.
Return type: float

yadg.dgutils.dateutils.str_to_uts(timestamp, format=None, timezone='UTC', strict=True)

Converts a string to POSIX timestamp.

If the optional format is specified, the timestamp string is processed using the datetime.datetime.strptime() function; if no format is supplied, an ISO 8601 format is assumed and an attempt to parse using dateutil.parser.parse() is made.

Parameters

timestamp (str) – A string containing the timestamp.
format (Optional[str]) – Optional format string for parsing of the timestamp.
timezone (str) – Optional timezone of the timestamp. By default, “UTC”.
strict (bool) – Whether to re-raise any parsing errors.

Returns

uts – Returns the POSIX timestamp if successful, otherwise None.

Return type

Union[float, None]

yadg.dgutils.dateutils.timestamps_from_file(path, type, timezone='UTC')

Load timestamps from file.

This function enables loading timestamps from file. The currently supported file formats include json and pkl, which must contain a top-level (Iterable) of (str) or (float)-like objects.

Parameters

path (str) – Location of the external file.
type (str) – Type of the external file. Currently, "json", "pkl" are supported.
timezone (str) – An optional timezone string, defaults to “UTC”

Returns

parseddata – A list of POSIX timestamps.

Return type

list[float]

yadg.dgutils.dgutils module

yadg.dgutils.dgutils.combine(filelist, **kwargs)

yadg.dgutils.dgutils.pointdata(datagram, pars)

yadg.dgutils.dgutils.printdg(datagram, **kwargs)

yadg.dgutils.dgutils.reduce(datagramfile, **kwargs)

yadg.dgutils.helpers module

yadg.dgutils.helpers.get_yadg_metadata()

Returns current yadg metadata.

Return type: dict

yadg.dgutils.pintutils module

pint compatibility functions in yadg.

This package defines ureg, a pint.UnitRegistry used for validation of datagrams in yadg. The default SI pint.UnitRegistry is extended by definitions of fractional quantities (%, ppm, etc.), standard volumetric quantities (smL/min, sccm), and other dimensionless “units” present in several file types.

yadg.dgutils.pintutils.sanitize_units(units)

Unit sanitizer.

This sanitizer should be used where user-supplied units are likely to occur, such as in the parsers yadg.parsers.basiccsv. Currently, only two replacements are done:

“Bar” is replaced with “bar”

“Deg C” is replace with “degC

Use with caution.

Parameters: units (Union[str, dict[str], list[str]]) – Object containing string units.
Return type: None

yadg.dgutils.utils module

yadg.dgutils.utils.calib_3to4(oldcal, caltype)

Return type: dict

yadg.dgutils.utils.datagram_3to4(olddg)

Return type: dict

yadg.dgutils.utils.schema_3to4(oldschema)

Return type: dict

yadg.dgutils.utils.schema_from_preset(preset, folder)

Return type: dict

yadg.dgutils.utils.update_object(type, object)

Yadg’s update worker function.

This is the main function called when yadg is executed as yadg update. The main idea is to allow a simple update pathway from older versions of schema and datagram files to the current latest and greatest.

Currently supports:

updating schema version 3.1.0 to 4.0.0

updating datagram version 3.1.0 to 4.0.0, although you really should be updating the schema instead and re-processing files.

Parameters

type (str) – The type of the passed object, either “datagram” or “schema”.
object (Union[list, dict]) – The object to be updated

Returns

newobj – The updated and validated “datagram” or “schema”.

Return type

dict