dgpost.utils package

parse: YAML and JSON input handler

Code author: Peter Kraus

dgpost.utils.parse.parse(fn)

Input file parsing function.

Supports loading yaml and json files using the recipe-parsing function and schema provided in the dgbowl_schemas.dgpost_recipe module.

Parameters

fn (str) – Path to the filename to be parsed

Returns

ret – A dictionary representing the recipe.

Return type

dict[str, Any]

load: Datagram and table loading routine

Code author: Ueli Sauter, Peter Kraus

The function dgpost.utils.load.load() processes the below specification in order to load the datagram json file:

pydantic model dgbowl_schemas.dgpost_recipe.load.Load

Show JSON schema
{
   "title": "Load",
   "type": "object",
   "properties": {
      "as": {
         "title": "As",
         "type": "string"
      },
      "path": {
         "title": "Path",
         "type": "string"
      },
      "type": {
         "title": "Type",
         "default": "datagram",
         "enum": [
            "datagram",
            "table"
         ],
         "type": "string"
      },
      "check": {
         "title": "Check",
         "default": true,
         "type": "boolean"
      }
   },
   "required": [
      "as",
      "path"
   ],
   "additionalProperties": false
}

field as_: str [Required] (alias 'as')
field path: str [Required]
field type: Literal['datagram', 'table'] = 'datagram'
field check: bool = True

Note

The key as is not processed by load(), it should be used by its caller to store the returned datagram or pd.DataFrame into the correct variable.

extract: Data extraction and interpolation routine.

Code author: Peter Kraus

The function dgpost.utils.extract.extract() processes the below specification in order to extract the required data from the supplied datagram.

pydantic model dgbowl_schemas.dgpost_recipe.extract.Extract

Show JSON schema
{
   "title": "Extract",
   "type": "object",
   "properties": {
      "into": {
         "title": "Into",
         "type": "string"
      },
      "from": {
         "title": "From",
         "type": "string"
      },
      "at": {
         "$ref": "#/definitions/At"
      },
      "constants": {
         "title": "Constants",
         "type": "array",
         "items": {
            "$ref": "#/definitions/Constant"
         }
      },
      "columns": {
         "title": "Columns",
         "type": "array",
         "items": {
            "$ref": "#/definitions/Column"
         }
      }
   },
   "required": [
      "into"
   ],
   "additionalProperties": false,
   "definitions": {
      "At": {
         "title": "At",
         "type": "object",
         "properties": {
            "steps": {
               "title": "Steps",
               "type": "array",
               "items": {
                  "type": "string"
               }
            },
            "indices": {
               "title": "Indices",
               "type": "array",
               "items": {
                  "type": "integer"
               }
            },
            "timestamps": {
               "title": "Timestamps",
               "type": "array",
               "items": {
                  "type": "number"
               }
            }
         },
         "additionalProperties": false
      },
      "Constant": {
         "title": "Constant",
         "type": "object",
         "properties": {
            "value": {
               "title": "Value"
            },
            "as": {
               "title": "As",
               "type": "string"
            },
            "units": {
               "title": "Units",
               "type": "string"
            }
         },
         "required": [
            "as"
         ],
         "additionalProperties": false
      },
      "Column": {
         "title": "Column",
         "type": "object",
         "properties": {
            "key": {
               "title": "Key",
               "type": "string"
            },
            "as": {
               "title": "As",
               "type": "string"
            }
         },
         "required": [
            "key",
            "as"
         ],
         "additionalProperties": false
      }
   }
}

field into: str [Required]
Validated by
field from_: Optional[str] [Required] (alias 'from')
Validated by
field at: Optional[dgbowl_schemas.dgpost_recipe.extract.At] [Required]
Validated by
field constants: Optional[Sequence[dgbowl_schemas.dgpost_recipe.extract.Constant]] [Required]
Validated by
field columns: Optional[Sequence[dgbowl_schemas.dgpost_recipe.extract.Column]] [Required]
Validated by
validator check_one_input  »  all fields

Note

The keys from and into are not processed by extract(), they should be used by its caller to supply the requested datagram and assign the returned pd.DataFrame into the correct variable.

Handling of sparse data depends on the extraction format specified:

  • for direct extraction, if the value is not present at any of the timesteps specified in at, a NaN is added instead

  • for interpolation, if a value is missing at any of the timesteps specified in at or in the pd.DataFrame index, that timestep is masked and the interpolation is performed from neighbouring points

Interpolation of uc.ufloat is performed separately for the nominal and error component.

Units are added into the attrs dictionary of the pd.DataFrame on a per-column basis.

Data from multiple datagrams can be combined into one pd.DataFrame using a YAML such as the following example:

load:
  - as: norm
    path: normalized.dg.json
  - as: sparse
    path: sparse.dg.json
extract:
  - into: df
    from: norm
    at:
        step: "a"
    columns:
      - key: raw->T_f
        as: rawT
  - into: df
    from: sparse
    at:
        steps: b1, b2, b3
    direct:
      - key: derived->xout->*
        as: xout

In this example, the pd.DataFrame is created with an index corresponding to the timestamps of step: "a" of the datagram. The values specified using columns in the first section are entered directly, after renaming the column names.

The data pulled out of the datagram in the second step using the prescription in at are interpolated onto the index of the existing pd.DataFrame.

transform: Reproducible transformations made simple.

Code author: Peter Kraus

The function dgpost.utils.transform.transform() processes the below specification in order to do data transformation on an extracted (or supplied) pd.DataFrame.

pydantic model dgbowl_schemas.dgpost_recipe.transform.Transform

Show JSON schema
{
   "title": "Transform",
   "type": "object",
   "properties": {
      "table": {
         "title": "Table",
         "type": "string"
      },
      "with": {
         "title": "With",
         "type": "string"
      },
      "using": {
         "title": "Using",
         "type": "array",
         "items": {
            "type": "object"
         }
      }
   },
   "required": [
      "table",
      "with",
      "using"
   ],
   "additionalProperties": false
}

field table: str [Required]
field with_: str [Required] (alias 'with')
field using: Sequence[dict[str, Any]] [Required]

Warning

The arguments passed to the transformation function in the transform.using Sequence have to be dict with str-type keys. The values will be coerced to appropriate types using the transform function decorator: dgpost.transform.helpers.load_data().

plot: Reproducible plots from a table.

Code author: Ueli Sauter, Peter Kraus

The function dgpost.utils.plot.plot() processes the below specification in order to generate a plot:

pydantic model dgbowl_schemas.dgpost_recipe.plot.Plot

Show JSON schema
{
   "title": "Plot",
   "type": "object",
   "properties": {
      "table": {
         "title": "Table",
         "type": "string"
      },
      "ax_args": {
         "title": "Ax Args",
         "type": "array",
         "items": {
            "$ref": "#/definitions/AxArgs"
         }
      },
      "fig_args": {
         "title": "Fig Args",
         "type": "object"
      },
      "style": {
         "title": "Style",
         "type": "object"
      },
      "nrows": {
         "title": "Nrows",
         "default": 1,
         "type": "integer"
      },
      "ncols": {
         "title": "Ncols",
         "default": 1,
         "type": "integer"
      },
      "save": {
         "$ref": "#/definitions/PlotSave"
      }
   },
   "required": [
      "table",
      "ax_args"
   ],
   "additionalProperties": false,
   "definitions": {
      "SeriesIndex": {
         "title": "SeriesIndex",
         "type": "object",
         "properties": {
            "from_zero": {
               "title": "From Zero",
               "default": true,
               "type": "boolean"
            },
            "to_units": {
               "title": "To Units",
               "type": "string"
            }
         },
         "additionalProperties": false
      },
      "Series": {
         "title": "Series",
         "type": "object",
         "properties": {
            "y": {
               "title": "Y",
               "type": "string"
            },
            "x": {
               "title": "X",
               "type": "string"
            },
            "kind": {
               "title": "Kind",
               "default": "scatter",
               "enum": [
                  "scatter",
                  "line",
                  "errorbar"
               ],
               "type": "string"
            },
            "index": {
               "title": "Index",
               "default": {
                  "from_zero": true,
                  "to_units": null
               },
               "allOf": [
                  {
                     "$ref": "#/definitions/SeriesIndex"
                  }
               ]
            }
         },
         "required": [
            "y"
         ]
      },
      "AxArgs": {
         "title": "AxArgs",
         "type": "object",
         "properties": {
            "cols": {
               "title": "Cols",
               "type": "array",
               "minItems": 2,
               "maxItems": 2,
               "items": [
                  {
                     "type": "integer"
                  },
                  {
                     "type": "integer"
                  }
               ]
            },
            "rows": {
               "title": "Rows",
               "type": "array",
               "minItems": 2,
               "maxItems": 2,
               "items": [
                  {
                     "type": "integer"
                  },
                  {
                     "type": "integer"
                  }
               ]
            },
            "series": {
               "title": "Series",
               "type": "array",
               "items": {
                  "$ref": "#/definitions/Series"
               }
            },
            "methods": {
               "title": "Methods",
               "type": "object"
            },
            "legend": {
               "title": "Legend",
               "default": false,
               "type": "boolean"
            }
         },
         "required": [
            "series"
         ]
      },
      "PlotSave": {
         "title": "PlotSave",
         "type": "object",
         "properties": {
            "as": {
               "title": "As",
               "type": "string"
            },
            "tight_layout": {
               "title": "Tight Layout",
               "type": "object"
            }
         },
         "required": [
            "as"
         ]
      }
   }
}

field table: str [Required]
field ax_args: Sequence[dgbowl_schemas.dgpost_recipe.plot.AxArgs] [Required]
field fig_args: Optional[dict[str, Any]] [Required]
field style: Optional[dict[str, Any]] [Required]
field nrows: int = 1
field ncols: int = 1
field save: Optional[dgbowl_schemas.dgpost_recipe.plot.PlotSave] [Required]
dgpost.utils.plot.apply_plot_style(style)

Updates the plot style with the given dictionary. For available kwargs see matplotlib.rcParams. If style is None, applies/resets to the default matplotlib style.

Parameters

style (dict) – A dictionary object containing valid key/value pairs.

Return type

None

dgpost.utils.plot.plt_axes(ax, table, ax_args)

Processes ax_args and plots the data

Parameters

  • ax (Axes) – axes object to be plotted to

  • table (DataFrame) – dataframe containing the data

  • ax_args (dict) – arguments for the axes

Return type

None

save: Save a table into a file.

Code author: Ueli Sauter, Peter Kraus

The function dgpost.utils.save.save() processes the below specification in order to save the given DataFrame:

pydantic model dgbowl_schemas.dgpost_recipe.save.Save

Show JSON schema
{
   "title": "Save",
   "type": "object",
   "properties": {
      "table": {
         "title": "Table",
         "type": "string"
      },
      "as": {
         "title": "As",
         "type": "string"
      },
      "type": {
         "title": "Type",
         "enum": [
            "pkl",
            "json",
            "xlsx",
            "csv"
         ],
         "type": "string"
      },
      "sigma": {
         "title": "Sigma",
         "default": true,
         "type": "boolean"
      }
   },
   "required": [
      "table",
      "as"
   ],
   "additionalProperties": false
}

field table: str [Required]
field as_: str [Required] (alias 'as')
field type: Literal['pkl', 'json', 'xlsx', 'csv'] = None
field sigma: bool = True

Note

Metadata, including the recipe used to create the saved file, as well as provenance information about the version of dgpost used to process the recipe are saved into the df.attrs["meta"] entry and therefore only available in pkl or json exports.