Quick start guide

First time set-up

Note

This section assumes that tomato has been successfully installed.

To prepare tomato before its first run, a settings file containing the basic configuration for the tomato.daemon needs to be placed in tomato’s appdir folder. By default, the appdir path is:

$env:localappdata\dgbowl\tomato\<version> on Windows,
$HOME/.config/tomato/<version> on Linux.

The easiest way to do create this file is using the provided tomato init command:

 kraus@dorje:/home/kraus/$ tomato init
 Success: wrote default settings into /home/kraus/.config/tomato/1.0a1/settings.toml

Where appdir is /home/kraus/.config/tomato/1.0a1/. A custom appdir can be specified using the --appdir argument to tomato.

tomato program flowchart

tomato currently contains two command line user-facing utilities/executables:

tomato, responsible for management of the daemon process, and
ketchup, responsible for job submission and management.

These two executables then internally spawn other processes, including the state daemon process tomato-daemon, a tomato-job process for every running job, and a tomato-driver process for each driver type, managing all devices of a that type.

flowchart TB subgraph daemon[tomato-daemon] cmd{{port:cmd}} ==> st[(state)] st --> jm([job manager]) st --> dm([driver manager]) st <--> io[state io] end subgraph driver[tomato-driver] dcmd{{port:cmd}} ==> dst[(state)] end dst o-.-o h1[hardware] subgraph job[tomato-job] mp([main loop]) --> tp>task process] end t>tomato] -.-> cmd k>ketchup] -.-> cmd jm -.-> job dst -.-> driver dm -..-> driver tp -.-> dcmd

tomato concepts flowchart

The following concepts are used in tomato:

devices, which represent separately addressable (and optionally multichannel) instruments in the lab,
components, which represent the individual channels of each device, describing the role of this component and its associated capabilities,
pipelines, which represent the real world organisation of device components into independent experimental set-ups, containing components from one or more device,
jobs, which are processes that carry out a payload (i.e. a set of experimental instructions) on a pipeline by instructing its components to perform individual tasks, and
drivers, which are separate processes, each managing all defined devices of a certain driver type as well as their components:

flowchart LR c1 --- drv1[driver 123] c2 --- drv1 c3 --- drv1 a100 --- drv2[driver abc] b100 --- drv2 subgraph d1[device 1] ip1[address: 192.168.1.1] c1[channel 1] c2[channel 2] c3[channel 3] end subgraph a[device a] com1[address: COM1] a100[channel 100] end subgraph b[device b] com2[address: COM2] b100[channel 100] end subgraph pipelines pip1 pip2 pip3 end subgraph drivers drv1 drv2 end subgraph devices d1 a b end subgraph jobs j1 j2 end j1[complex] --- |two part payload| pip1 j2[simple] --- |simple payload| pip3 pip1[pipeline a1] -.-> c1 pip1[pipeline a1] -.-> a100 pip2[pipeline b2] -.-> c2 pip2[pipeline b2] -.-> b100 pip3[pipeline 3] -.-> c3

Settings file

The settings file contains the basic information required to start the tomato-daemon. The filename is hard-coded to be a settings.toml file, which has to be located in the appdir. The default file generated by tomato init looks similar to the below example:

 datadir = '/home/kraus/.local/share/tomato/1.0a1'

 [jobs]
 storage = '/home/kraus/.local/share/tomato/1.0a1/Jobs'

 [devices]
 config = '/home/kraus/.config/tomato/1.0a1/devices.yml'

 [drivers]
 example_counter.testpar = 1234

In addition to the appdir, a second path, datadir, is used to specify the location of the data created by tomato. The default datadir is:

$env:localappdata\dgbowl\tomato\<version> on Windows,
$HOME/.local/share/tomato/<version> on Linux.

In the default settings file shown above, the following entries are specified:

jobs.storage which is the directory where the data and logs of tomato jobs will be stored,
devices.config which points to a yaml-formatted devices file, defining the hardware configuration of the devices managed by tomato.

Additional, driver-specific settings may be provided in the [drivers] section, following the example of the drivers.example_counter.testpar entry. These driver-specific settings are passed to each driver when its process is launched and the DriverInterface is initialised, and can therefore contain paths to various libraries or other files necessary for the driver to function.

Devices file

This yaml-formatted file contains information about each device, corresponding to an individual piece of hardware managed by tomato, as well as information about the organisation of the individually-addressable components of those devices into pipelines.

When the devices file is not present, the default file shipped with tomato will be used instead:

devices section

The devices section of the default devices file is shown below:

 devices:
   - name: dev-counter
     driver: "example_counter"
     address: "example-addr"
     channels: ["1"]
     pollrate: 1

Here, we define a single device using the example_counter driver. The definition includes the address of the device (str type) as well as an enumeration of individually-addressable channels the device has (list[str]).

For example, the devices shown in the concepts flowchart above would be defined as:

 devices:
   - name: device 1
     driver: "driver 123"
     address: "192.168.1.1"
     channels: ["1", "2", "3"]
     pollrate: 1
   - name: device a
     driver: "driver abc"
     address: "COM1"
     channels: ["100"]
     pollrate: 5
   - name: device b
     driver: "driver abc"
     address: "COM2"
     channels: ["100"]
     pollrate: 5

Note

The pollrate here is an internal setting for tomato. It is the frequency with which the job interacting with the device driver asks the driver for data, not the frequency with which the driver asks the hardware for data! This latter “sampling” frequency can be defined in each individual task submitted as a payload of a job.

pipelines section

The default pipelines section looks as follows:

 pipelines:
   - name: pip-counter
     components:
       - role: counter
         device: dev-counter
         channel: "1"

Here, a single pipeline called pip-counter is defined to contain the one available channel of the dev-counter device (defined on line 5) shown further above. For multi channel devices, it is also possible to define a pipeline for each channel automatically, e.g. using the following definition:

 pipelines:
   - name: pip-counter-*
     components:
       - role: counter
         device: dev-counter
         channel: each

Here, a set of pipelines would be created using each of the available channels in dev-counter, replacing the * in line 2 using the channel. Therefore, one pipeline with name pip-counter-1 would be created. However, if the dev-counter were to contain more than one channel, a pipeline would be generated for each channel.

Finally, a definition for the pipelines shown in the concepts flowchart above can be defined as:

 pipelines:
   - name: pipeline a1
     components:
       - role: dev 123
         device: device 1
         channel: "1"
       - role: dev abc
         device: device a
         channel: "100"
   - name: pipeline b2
     components:
       - role: dev 123
         device: device 1
         channel: "2"
       - role: dev abc
         device: device b
         channel: "100"
   - name: pipeline 3
     components:
       - role: dev 123
         device: device 1
         channel: "3"

Payload file

The payload file contains all information required to enter a job into the queue and allow its assignment onto a pipeline. The overall schema of the payload is defined in the dgbowl_schemas.tomato module, and is parsed using dgbowl_schemas.tomato.to_payload():

pydantic model dgbowl_schemas.tomato.payload.Payload

Show JSON schema

{
   "title": "Payload",
   "type": "object",
   "properties": {
      "version": {
         "const": "1.0",
         "title": "Version",
         "type": "string"
      },
      "settings": {
         "$ref": "#/$defs/Settings"
      },
      "sample": {
         "$ref": "#/$defs/Sample"
      },
      "method": {
         "items": {
            "$ref": "#/$defs/Task"
         },
         "title": "Method",
         "type": "array"
      }
   },
   "$defs": {
      "Output": {
         "additionalProperties": false,
         "description": "Provide the ``path`` and ``prefix`` for the final FAIR-data archive of the *job*.",
         "properties": {
            "path": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Path"
            },
            "prefix": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Prefix"
            }
         },
         "title": "Output",
         "type": "object"
      },
      "Sample": {
         "additionalProperties": true,
         "description": "Additional attributes for each :class:`Sample` may be required, depending on the\nmethod within the payload.",
         "properties": {
            "name": {
               "title": "Name",
               "type": "string"
            }
         },
         "required": [
            "name"
         ],
         "title": "Sample",
         "type": "object"
      },
      "Settings": {
         "additionalProperties": false,
         "description": "Specification of *job* configuration for tomato.",
         "properties": {
            "unlock_when_done": {
               "default": false,
               "title": "Unlock When Done",
               "type": "boolean"
            },
            "verbosity": {
               "default": "WARNING",
               "enum": [
                  "DEBUG",
                  "INFO",
                  "WARNING",
                  "ERROR",
                  "CRITICAL"
               ],
               "title": "Verbosity",
               "type": "string"
            },
            "output": {
               "$ref": "#/$defs/Output"
            },
            "snapshot": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/Snapshot"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null
            }
         },
         "title": "Settings",
         "type": "object"
      },
      "Snapshot": {
         "additionalProperties": false,
         "description": "Provide the ``frequency``, ``path`` and ``prefix`` to configure the snapshotting\nfunctionality of tomato.",
         "properties": {
            "path": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Path"
            },
            "prefix": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Prefix"
            },
            "frequency": {
               "default": 3600.0,
               "title": "Frequency",
               "type": "number"
            }
         },
         "title": "Snapshot",
         "type": "object"
      },
      "Task": {
         "additionalProperties": false,
         "description": "The :class:`Task` is a driver/device-independent abstraction describing the\nmeasurement steps. The driver-specific information for the :class:`Task` can be\nprovided via the ``technique`` parameter.",
         "properties": {
            "component_tag": {
               "title": "Component Tag",
               "type": "string"
            },
            "max_duration": {
               "title": "Max Duration",
               "type": "number"
            },
            "sampling_interval": {
               "title": "Sampling Interval",
               "type": "number"
            },
            "polling_interval": {
               "anyOf": [
                  {
                     "type": "integer"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Polling Interval"
            },
            "technique_name": {
               "title": "Technique Name",
               "type": "string"
            },
            "technique_params": {
               "anyOf": [
                  {
                     "type": "object"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Technique Params"
            }
         },
         "required": [
            "component_tag",
            "max_duration",
            "sampling_interval",
            "technique_name"
         ],
         "title": "Task",
         "type": "object"
      }
   },
   "additionalProperties": false,
   "required": [
      "version",
      "sample",
      "method"
   ]
}

Config:

extra: str = forbid

Validators:

extract_methodfile » all fields
extract_samplefile » all fields

field version: Literal['1.0'] [Required]

Validated by:

field settings: Settings [Optional]

Additional configuration options for tomato.

Validated by:

field sample: Sample [Required]

Specification of the experimental sample.

Validated by:

field method: Sequence[Task] [Required]

A sequence of the experimental Tasks.

Validated by:

validator extract_samplefile » all fields: If samplefile is provided in values, parse the file as sample.

validator extract_methodfile » all fields: If methodfile is provided in values, parse the file as method.

Note

Of particular importance here is the specification of the individual Tasks. Some general Task parameters are abstracted by tomato, such as the max_duration, sampling_interval and polling_interval. The other are driver or component specific, and can be specified using the technique_params dict.

pydantic model dgbowl_schemas.tomato.payload.Task

The Task is a driver/device-independent abstraction describing the measurement steps. The driver-specific information for the Task can be provided via the technique parameter.

Show JSON schema

{
   "title": "Task",
   "description": "The :class:`Task` is a driver/device-independent abstraction describing the\nmeasurement steps. The driver-specific information for the :class:`Task` can be\nprovided via the ``technique`` parameter.",
   "type": "object",
   "properties": {
      "component_tag": {
         "title": "Component Tag",
         "type": "string"
      },
      "max_duration": {
         "title": "Max Duration",
         "type": "number"
      },
      "sampling_interval": {
         "title": "Sampling Interval",
         "type": "number"
      },
      "polling_interval": {
         "anyOf": [
            {
               "type": "integer"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Polling Interval"
      },
      "technique_name": {
         "title": "Technique Name",
         "type": "string"
      },
      "technique_params": {
         "anyOf": [
            {
               "type": "object"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Technique Params"
      }
   },
   "additionalProperties": false,
   "required": [
      "component_tag",
      "max_duration",
      "sampling_interval",
      "technique_name"
   ]
}

Config:

extra: str = forbid

field component_tag: str [Required]: tag of the pipeline component on which this Method should run

field max_duration: float [Required]: the maximum duration of this Task, in seconds

field sampling_interval: float [Required]: the interval between measurements, in seconds

field polling_interval: int | None = None: the interval between polling for data, in seconds; defaults to the value in driver settings

field technique_name: str [Required]

field technique_params: Dict[str, Any] | None = None: a dict of additional parameters required to specify the experimental technique

Warning

Currently, tomato performs no validation of any technique_params at job submission. It also does not check whether the sampling_interval is realistic. This means your jobs may crash when the technique_params are out of bounds when the Task is being executed.

However, such validation is planned for tomato-2.0, see https://github.com/dgbowl/tomato/issues/99.