yadg.dgutils package

Submodules

yadg.dgutils.btools.read_from_file(f, offset, dtype, count=1)

Return type

Union[str, ndarray]

yadg.dgutils.btools.read_from_buffer(buf, offset, dtype, count=1)

Return type

Union[str, ndarray]

yadg.dgutils.btools.read_value(object, offset, dtype, count=1)

Return type

Union[str, ndarray, int, float]

yadg.dgutils.calib.calib_handler(x, calib=None, atol=0.0, rtol=0.0)

Calibration handling function.

Returns y calculated from x and the other supplied arguments.

The "calib" parameter is a (dict) in the following format:

- calib:
  - linear:                # y = slope * x + intercept
    intercept:  !!float
    slope:      !!float
  - inverse:               # y = (x - intercept) / slope
    intercept:  !!float
    slope:      !!float
  - polynomial:            # y = sum(cN * x**N)
    c0:         !!float
    ...
    cN:         !!float
  atol:         !!float    # absolute uncertainty of y
  rtol:         !!float    # relative uncertainty of y

Parameters

• x (Union[float, AffineScalarFunc]) – The raw value

• calib (Optional[dict]) – Calibration dictionary, specified using the format described above. If empty, corresponds to no transformation (i.e. linear with "slope" = 1) and a rtol = 1e-3.

• atol (float) – Optional absolute uncertainty of y, overrides that provided in calib.

• rtol (float) – Optional relative uncertainty of y, overrides that provided in calib.

Returns

y – The derived value.

Return type

UFloat

yadg.dgutils.dateutils.now(asstr=False, tz=datetime.timezone.utc)

Wrapper around datetime.now()

A convenience function for returning the current time as a ISO 8601 or as a unix timestamp.

Return type

Union[float, str]

yadg.dgutils.dateutils.ole_to_uts(ole_timestamp, timezone='UTC')

Converts a Microsoft OLE timestamp into a POSIX timestamp.

The OLE automation date format is a floating point value, counting days since midnight 30 December 1899. Hours and minutes are represented as fractional days.

https://devblogs.microsoft.com/oldnewthing/20030905-02/?p=42653

Parameters

• ole_timestamp (float) – A timestamp in Microsoft OLE format.

• timezone (str) – String desribing the timezone.

Returns

time – The corresponding Unix timestamp.

Return type

float

yadg.dgutils.dateutils.str_to_uts(timestamp, format=None, timezone='UTC', strict=True)

Converts a string to POSIX timestamp.

If the optional format is specified, the timestamp string is processed using the datetime.datetime.strptime() function; if no format is supplied, an ISO 8601 format is assumed and an attempt to parse using dateutil.parser.parse() is made.

Parameters

• timestamp (str) – A string containing the timestamp.

• format (Optional[str]) – Optional format string for parsing of the timestamp.

• timezone (str) – Optional timezone of the timestamp. By default, “UTC”.

• strict (bool) – Whether to re-raise any parsing errors.

Returns

uts – Returns the POSIX timestamp if successful, otherwise None.

Return type

Union[float, None]

yadg.dgutils.dateutils.infer_timestamp_from(headers=None, spec=None, timezone='UTC')

Convenience function for timestamping

Given a set of headers, and an optional specification, return an array containing column indices from which a timestamp in a given row can be computed, as well as the function which will compute the timestamp given the returned array.

Parameters

• headers (Optional[list]) – An array of strings. If spec is not supplied, must contain either “uts” (float) or “timestep” (str) (conforming to ISO 8601).

• spec (Optional[TimestampSpec]) – A specification of timestamp elements with associated column indices and optional formats. Currently accepted combinations of keys are: “uts”; “timestamp”; “date” and / or “time”.

• tz – Timezone to use for conversion. By default, UTC is used.

Returns

(datecolumns, datefunc, fulldate) – A tuple containing a list of indices of columns, a Callable to which the columns have to be passed to obtain a uts timestamp, and whether the determined timestamp is full or partial.

Return type

tuple[list, Callable, bool]

yadg.dgutils.dateutils.complete_timestamps(timesteps, fn, spec, timezone)

Timestamp completing function.

This function allows for completing or overriding the uts timestamps determined by the individual parsers. yadg enters this function for any parser which does not return a full timestamp, as well as if the externaldate specification is specified by the user.

The externaldate specification is as follows:

pydantic model dgbowl_schemas.yadg.dataschema_4_1.externaldate.ExternalDate

Show JSON schema

{
   "title": "ExternalDate",
   "type": "object",
   "properties": {
      "using": {
         "title": "Using",
         "anyOf": [
            {
               "$ref": "#/definitions/ExternalDateFile"
            },
            {
               "$ref": "#/definitions/ExternalDateFilename"
            },
            {
               "$ref": "#/definitions/ExternalDateISOString"
            },
            {
               "$ref": "#/definitions/ExternalDateUTSOffset"
            }
         ]
      },
      "mode": {
         "title": "Mode",
         "default": "add",
         "enum": [
            "add",
            "replace"
         ],
         "type": "string"
      }
   },
   "required": [
      "using"
   ],
   "additionalProperties": false,
   "definitions": {
      "dgbowl_schemas__yadg__dataschema_4_1__externaldate__ExternalDateFile__Content": {
         "title": "Content",
         "type": "object",
         "properties": {
            "path": {
               "title": "Path",
               "type": "string"
            },
            "type": {
               "title": "Type",
               "type": "string"
            },
            "match": {
               "title": "Match",
               "type": "string"
            }
         },
         "required": [
            "path",
            "type"
         ],
         "additionalProperties": false
      },
      "ExternalDateFile": {
         "title": "ExternalDateFile",
         "type": "object",
         "properties": {
            "file": {
               "$ref": "#/definitions/dgbowl_schemas__yadg__dataschema_4_1__externaldate__ExternalDateFile__Content"
            }
         },
         "required": [
            "file"
         ],
         "additionalProperties": false
      },
      "dgbowl_schemas__yadg__dataschema_4_1__externaldate__ExternalDateFilename__Content": {
         "title": "Content",
         "type": "object",
         "properties": {
            "format": {
               "title": "Format",
               "type": "string"
            },
            "len": {
               "title": "Len",
               "type": "integer"
            }
         },
         "required": [
            "format",
            "len"
         ],
         "additionalProperties": false
      },
      "ExternalDateFilename": {
         "title": "ExternalDateFilename",
         "type": "object",
         "properties": {
            "filename": {
               "$ref": "#/definitions/dgbowl_schemas__yadg__dataschema_4_1__externaldate__ExternalDateFilename__Content"
            }
         },
         "required": [
            "filename"
         ],
         "additionalProperties": false
      },
      "ExternalDateISOString": {
         "title": "ExternalDateISOString",
         "type": "object",
         "properties": {
            "isostring": {
               "title": "Isostring",
               "type": "string"
            }
         },
         "required": [
            "isostring"
         ],
         "additionalProperties": false
      },
      "ExternalDateUTSOffset": {
         "title": "ExternalDateUTSOffset",
         "type": "object",
         "properties": {
            "utsoffset": {
               "title": "Utsoffset",
               "type": "number"
            }
         },
         "required": [
            "utsoffset"
         ],
         "additionalProperties": false
      }
   }
}

field using: Union[dgbowl_schemas.yadg.dataschema_4_1.externaldate.ExternalDateFile, dgbowl_schemas.yadg.dataschema_4_1.externaldate.ExternalDateFilename, dgbowl_schemas.yadg.dataschema_4_1.externaldate.ExternalDateISOString, dgbowl_schemas.yadg.dataschema_4_1.externaldate.ExternalDateUTSOffset] [Required]

field mode: Literal['add', 'replace'] = 'add'

The using key specifies how an external timestamp is created. Only one entry in using is permitted. By default, this entry is:

using:
  filename:
    format: "%Y-%m-%d-%H-%M-%S"
    len: 19

Which means the code will attempt to deduce the timestamp from the path of the processed file (fn), using the first 19 characters of the base filename according to the above format (eg. “2021-12-31-13-45-00”).

If file is specified, the handling of timestamps is handed off to timestamps_from_file().

The mode key specifies whether the offsets determined in this function are added to the current timestamps (eg. date offset being added to time) or whether they should replace the existing timestamps completely.

As a measure of last resort, the mtime of the fn is used. mtime is preferred to ctime, as the former has a more consistent cross-platform behaviour.

Parameters

• timesteps (list) – A list of timesteps generated from a single file, fn.

• fn (str) – Filename used to create timesteps.

• spec (ExternalDate) – externaldate specification part of the schema.

• timezone (str) – Timezone, defaults to “UTC”.

Return type

None

yadg.dgutils.dateutils.timestamps_from_file(path, type, match=None, timezone='UTC')

Load timestamps from file.

This function enables loading timestamps from file specified by the path. The currently supported file formats include json and pkl, which must contain a top-level Mapping with a key that is matched by match, or a top-level Iterable, both containing str or float -like objects that can be processed into an Unix timestamp.

Parameters

• path (str) – Location of the external file.

• type (str) – Type of the external file. Currently, "json", "pkl" are supported.

• match (Optional[str]) – An optional key to match if the object in path is a Mapping.

• timezone (str) – An optional timezone string, defaults to “UTC”

Returns

parseddata – A single or a list of POSIX timestamps.

Return type

Union[float, list[float]]

yadg.dgutils.helpers.get_yadg_metadata()

Returns current yadg metadata.

Return type

dict

yadg.dgutils.helpers.deprecated(arg, depin='4.2', depout='5.0')

Return type

None

pint compatibility functions in yadg.

This package defines ureg, a pint.UnitRegistry used for validation of datagrams in yadg. The default SI pint.UnitRegistry is extended by definitions of fractional quantities (%, ppm, etc.), standard volumetric quantities (smL/min, sccm), and other dimensionless “units” present in several file types.

yadg.dgutils.pintutils.sanitize_units(units)

Unit sanitizer.

This sanitizer should be used where user-supplied units are likely to occur, such as in the parsers yadg.parsers.basiccsv. Currently, only two replacements are done:

• “Bar” is replaced with “bar”

• “Deg C” is replace with “degC

Use with caution.

Parameters

units (Union[str, dict[str, str], list[str]]) – Object containing string units.

Return type

Union[str, dict[str, str], list[str]]

yadg.dgutils.utils.calib_3to4(oldcal, caltype)

Return type

dict

yadg.dgutils.utils.schema_3to4(oldschema)

Return type

dict

yadg.dgutils.utils.update_object(objtype, object)

Yadg’s update worker function.

This is the main function called when yadg is executed as yadg update. The main idea is to allow a simple update pathway from older versions of schema and datagram files to the current latest and greatest.

Currently supports:

• updating schema version 3.1.0 to 4.0.0

• updating datagram version 3.1.0 to 4.0.0, although you really should be updating the schema instead and re-processing files.

Parameters

• objtype (str) – The type of the passed object, either “datagram” or “schema”.

• object (Union[list, dict]) – The object to be updated

Returns

newobj – The updated and validated “datagram” or “schema”.

Return type

dict

yadg.dgutils.utils.schema_from_preset(preset, folder)

Return type

dict