Quick start guide

First time set-up

Note

This section assumes that tomato has been successfully installed.

To prepare tomato before its first run, a settings file containing the basic configuration for the tomato.daemon needs to be placed in tomato’s appdir folder. By default, the appdir path is:

• $env:localappdata\dgbowl\tomato\<version> on Windows,

• $HOME/.config/tomato/<version> on Linux.

The easiest way to do create this file is using the provided tomato init command:

kraus@dorje:/home/kraus/$ tomato init
Success: wrote default settings into /home/kraus/.config/tomato/1.0a1/settings.toml

Where appdir is /home/kraus/.config/tomato/1.0a1/. A custom appdir can be specified using the --appdir argument to tomato.

tomato program flowchart

tomato currently contains two command line user-facing utilities/executables:

• tomato, responsible for management of the daemon process, and

• ketchup, responsible for job submission and management.

These two executables then internally spawn other processes, including the state daemon process tomato-daemon, a tomato-job process for every running job, and a tomato-driver process for each driver type, managing all devices of a that type.

flowchart TB subgraph daemon[tomato-daemon] cmd{{port:cmd}} ==> st[(state)] st --> jm([job manager]) st --> dm([driver manager]) st <--> io[state io] end subgraph driver[tomato-driver] dcmd{{port:cmd}} ==> dst[(state)] end dst o-.-o h1[hardware] subgraph job[tomato-job] mp([main loop]) --> tp>task process] end cmd ==> db[(job db\nsqlite)] jm -.-> db mp -.-> db t>tomato] -.-> cmd k>ketchup] -.-> cmd jm -.-> job dst -.-> driver dm -..-> driver tp -.-> dcmd

tomato concepts flowchart

The following concepts are used in tomato:

• devices, which represent separately addressable (and optionally multichannel) instruments in the lab,

• components, which represent the individual channels of each device, describing the role of this component and its associated capabilities,

• pipelines, which represent the real world organisation of device components into independent experimental set-ups, containing components from one or more device,

• jobs, which are processes that carry out a payload (i.e. a set of experimental instructions) on a pipeline by instructing its components to perform individual tasks, and

• drivers, which are separate processes, each managing all defined devices of a certain driver type as well as their components:

flowchart LR c1 --- drv1[driver 123] c2 --- drv1 c3 --- drv1 a100 --- drv2[driver abc] b100 --- drv2 subgraph d1[device 1] ip1[address: 192.168.1.1] c1[channel 1] c2[channel 2] c3[channel 3] end subgraph a[device a] com1[address: COM1] a100[channel 100] end subgraph b[device b] com2[address: COM2] b100[channel 100] end subgraph pipelines pip1 pip2 pip3 end subgraph drivers drv1 drv2 end subgraph devices d1 a b end subgraph jobs j1 j2 end j1[complex] --- |two part payload| pip1 j2[simple] --- |simple payload| pip3 pip1[pipeline a1] -.-> c1 pip1[pipeline a1] -.-> a100 pip2[pipeline b2] -.-> c2 pip2[pipeline b2] -.-> b100 pip3[pipeline 3] -.-> c3

Settings file

The settings file contains the basic information required to start the tomato-daemon. The filename is hard-coded to be a settings.toml file, which has to be located in the appdir. The default file generated by tomato init looks similar to the below example:

datadir = '/home/kraus/.local/share/tomato/1.0a1'
logdir = '/home/kraus/.cache/tomato/1.0a1/log'

[jobs]
storage = '/home/kraus/.local/share/tomato/1.0a1/Jobs'
dbpath = '/home/kraus/.local/share/tomato/1.0a1/Jobs/dbpath.sqlite'

[devices]
config = '/home/kraus/.config/tomato/1.0a1/devices.yml'

[drivers]
example_counter.idle_measurement_interval = 1

In addition to the appdir, a second path, datadir, is used to specify the location of the data created by tomato. The default datadir is:

• $env:localappdata\dgbowl\tomato\<version> on Windows,

• $HOME/.local/share/tomato/<version> on Linux.

Finally, another path, logdir, is used to specify where logs for tomato are placed. By default, logdir is:

• $env:localappdata\dgbowl\tomato\<version> on Windows,

• $HOME/.cache/tomato/<version>/log on Linux.

In the default settings file shown above, the following entries are specified:

• jobs.storage which is the directory where the data of tomato jobs will be stored,

• jobs.dbpath which is the location of the sqlite3 database used to track jobs,

• devices.config which points to a yaml-formatted devices file, defining the hardware configuration of the devices managed by tomato.

Additional, driver-specific settings may be provided in the [drivers] section, following the example of the drivers.example_counter.idle_measurement_interval entry. These driver-specific settings are passed to each driver when its process is launched and the DriverInterface is initialised, and can therefore contain paths to various libraries or other files necessary for the driver to function.

Devices file

This yaml-formatted file contains information about each device, corresponding to an individual piece of hardware managed by tomato, as well as information about the organisation of the individually-addressable components of those devices into pipelines.

When the devices file is not present, the default file shipped with tomato will be used instead:

devices section

The devices section of the default devices file is shown below:

devices:
  - name: dev-counter
    driver: "example_counter"
    address: "example-addr"
    channels: ["1"]
    pollrate: 1

Here, we define a single device using the example_counter driver. The definition includes the address of the device (str type) as well as an enumeration of individually-addressable channels the device has (list[str]).

For example, the devices shown in the concepts flowchart above would be defined as:

devices:
  - name: device 1
    driver: "driver 123"
    address: "192.168.1.1"
    channels: ["1", "2", "3"]
    pollrate: 1
  - name: device a
    driver: "driver abc"
    address: "COM1"
    channels: ["100"]
    pollrate: 5
  - name: device b
    driver: "driver abc"
    address: "COM2"
    channels: ["100"]
    pollrate: 5

Note

The pollrate here is an internal setting for tomato. It is the frequency with which the job interacting with the device driver asks the driver for data, not the frequency with which the driver asks the hardware for data! This latter “sampling” frequency can be defined in each individual task submitted as a payload of a job.

pipelines section

The default pipelines section looks as follows:

pipelines:
  - name: pip-counter
    components:
      - role: counter
        device: dev-counter
        channel: "1"

Here, a single pipeline called pip-counter is defined to contain the one available channel of the dev-counter device (defined on line 5) shown further above. For multi channel devices, it is also possible to define a pipeline for each channel automatically, e.g. using the following definition:

pipelines:
  - name: pip-counter-*
    components:
      - role: counter
        device: dev-counter
        channel: each

Here, a set of pipelines would be created using each of the available channels in dev-counter, replacing the * in line 2 using the channel. Therefore, one pipeline with name pip-counter-1 would be created. However, if the dev-counter were to contain more than one channel, a pipeline would be generated for each channel.

Finally, a definition for the pipelines shown in the concepts flowchart above can be defined as:

pipelines:
  - name: pipeline a1
    components:
      - role: dev 123
        device: device 1
        channel: "1"
      - role: dev abc
        device: device a
        channel: "100"
  - name: pipeline b2
    components:
      - role: dev 123
        device: device 1
        channel: "2"
      - role: dev abc
        device: device b
        channel: "100"
  - name: pipeline 3
    components:
      - role: dev 123
        device: device 1
        channel: "3"

Payload file

The payload file contains all information required to enter a job into the queue and allow its assignment onto a pipeline. The overall schema of the payload is defined in the dgbowl_schemas.tomato module, and is parsed using dgbowl_schemas.tomato.to_payload().

Of particular importance here is the specification of the individual Tasks within the Payload.method field. The Tasks are distributed onto individual components of the pipeline (as matched by the component_role field), and then executed in sequence. Some general Task parameters are abstracted by tomato, such as the max_duration, sampling_interval and polling_interval. The other are driver or component specific, and can be specified using the task_params field.

Warning

Currently, tomato does not check whether the sampling_interval is realistic. However, such validation is planned for a future release, see https://github.com/dgbowl/tomato/issues/127.

As of tomato-2.1, execution of a certain Task can be postponed and triggered to coincide with the start of a different Task; for this, the start_with_task_name attribute of the triggered Task needs to match the task_name of the triggering Task. Similarly, the execution of a Task can be stopped once a different Task starts, by setting stop_with_task_name accordingly.

As of tomato-2.0, the task_params specified in the payload are validated by the device driver. In particular, the values of all entries within task_params are checked for compatibility with the Attr.type and Attr.options (if supplied), as well as against the provided Attr.minimum and Attr.maximum, as specified in the matching Attr.

pydantic model dgbowl_schemas.tomato.payload.Payload

Show JSON schema

{
   "title": "Payload",
   "type": "object",
   "properties": {
      "version": {
         "const": "2.1",
         "title": "Version",
         "type": "string"
      },
      "settings": {
         "$ref": "#/$defs/Settings"
      },
      "sample": {
         "$ref": "#/$defs/Sample"
      },
      "method": {
         "items": {
            "$ref": "#/$defs/Task"
         },
         "title": "Method",
         "type": "array"
      }
   },
   "$defs": {
      "Output": {
         "additionalProperties": false,
         "description": "Provide the ``path`` and ``prefix`` for the final FAIR-data archive of the *job*.",
         "properties": {
            "path": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Path"
            },
            "prefix": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Prefix"
            }
         },
         "title": "Output",
         "type": "object"
      },
      "Sample": {
         "additionalProperties": true,
         "description": "Additional attributes for each :class:`Sample` may be required, depending on the\nmethod within the payload.",
         "properties": {
            "name": {
               "title": "Name",
               "type": "string"
            }
         },
         "required": [
            "name"
         ],
         "title": "Sample",
         "type": "object"
      },
      "Settings": {
         "additionalProperties": false,
         "description": "Specification of *job* configuration for tomato.",
         "properties": {
            "unlock_when_done": {
               "default": false,
               "title": "Unlock When Done",
               "type": "boolean"
            },
            "verbosity": {
               "default": "WARNING",
               "enum": [
                  "DEBUG",
                  "INFO",
                  "WARNING",
                  "ERROR",
                  "CRITICAL"
               ],
               "title": "Verbosity",
               "type": "string"
            },
            "output": {
               "$ref": "#/$defs/Output"
            },
            "snapshot": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/Snapshot"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null
            }
         },
         "title": "Settings",
         "type": "object"
      },
      "Snapshot": {
         "additionalProperties": false,
         "description": "Provide the ``frequency``, ``path`` and ``prefix`` to configure the snapshotting\nfunctionality of tomato.",
         "properties": {
            "path": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Path"
            },
            "prefix": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Prefix"
            },
            "frequency": {
               "default": 3600.0,
               "title": "Frequency",
               "type": "number"
            }
         },
         "title": "Snapshot",
         "type": "object"
      },
      "Task": {
         "additionalProperties": false,
         "description": "The :class:`Task` is a driver/device-independent abstraction describing the\nmeasurement steps. The driver-specific information for the :class:`Task` can be\nprovided via the :obj:`technique_name` and :obj:`task_params` parameters.",
         "properties": {
            "component_role": {
               "title": "Component Role",
               "type": "string"
            },
            "max_duration": {
               "anyOf": [
                  {
                     "type": "number"
                  },
                  {
                     "type": "string"
                  }
               ],
               "title": "Max Duration"
            },
            "sampling_interval": {
               "anyOf": [
                  {
                     "type": "number"
                  },
                  {
                     "type": "string"
                  }
               ],
               "title": "Sampling Interval"
            },
            "polling_interval": {
               "anyOf": [
                  {
                     "type": "number"
                  },
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Polling Interval"
            },
            "technique_name": {
               "title": "Technique Name",
               "type": "string"
            },
            "task_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Task Name"
            },
            "task_params": {
               "anyOf": [
                  {
                     "additionalProperties": true,
                     "type": "object"
                  },
                  {
                     "type": "null"
                  }
               ],
               "title": "Task Params"
            },
            "start_with_task_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Start With Task Name"
            },
            "stop_with_task_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "title": "Stop With Task Name"
            }
         },
         "required": [
            "component_role",
            "max_duration",
            "sampling_interval",
            "technique_name"
         ],
         "title": "Task",
         "type": "object"
      }
   },
   "additionalProperties": false,
   "required": [
      "version",
      "sample",
      "method"
   ]
}

Config:

• extra: str = forbid

Validators:

• extract_methodfile » all fields

• extract_samplefile » all fields

• validate_task_names » method

field version: Literal['2.1'] [Required]

Validated by:

• extract_methodfile

• extract_samplefile

field settings: Settings [Optional]

Additional configuration options for tomato.

Validated by:

• extract_methodfile

• extract_samplefile

field sample: Sample [Required]

Specification of the experimental sample.

Validated by:

• extract_methodfile

• extract_samplefile

field method: Sequence[Task] [Required]

A sequence of the experimental Tasks.

Validated by:

• extract_methodfile

• extract_samplefile

• validate_task_names

validator extract_samplefile  »  all fields

If samplefile is provided in values, parse the file as sample.

validator extract_methodfile  »  all fields

If methodfile is provided in values, parse the file as method.

validator validate_task_names  »  method

pydantic model dgbowl_schemas.tomato.payload.Task

The Task is a driver/device-independent abstraction describing the measurement steps. The driver-specific information for the Task can be provided via the technique_name and task_params parameters.

Show JSON schema

{
   "title": "Task",
   "description": "The :class:`Task` is a driver/device-independent abstraction describing the\nmeasurement steps. The driver-specific information for the :class:`Task` can be\nprovided via the :obj:`technique_name` and :obj:`task_params` parameters.",
   "type": "object",
   "properties": {
      "component_role": {
         "title": "Component Role",
         "type": "string"
      },
      "max_duration": {
         "anyOf": [
            {
               "type": "number"
            },
            {
               "type": "string"
            }
         ],
         "title": "Max Duration"
      },
      "sampling_interval": {
         "anyOf": [
            {
               "type": "number"
            },
            {
               "type": "string"
            }
         ],
         "title": "Sampling Interval"
      },
      "polling_interval": {
         "anyOf": [
            {
               "type": "number"
            },
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Polling Interval"
      },
      "technique_name": {
         "title": "Technique Name",
         "type": "string"
      },
      "task_name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Task Name"
      },
      "task_params": {
         "anyOf": [
            {
               "additionalProperties": true,
               "type": "object"
            },
            {
               "type": "null"
            }
         ],
         "title": "Task Params"
      },
      "start_with_task_name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Start With Task Name"
      },
      "stop_with_task_name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Stop With Task Name"
      }
   },
   "additionalProperties": false,
   "required": [
      "component_role",
      "max_duration",
      "sampling_interval",
      "technique_name"
   ]
}

Config:

• extra: str = forbid

Validators:

• convert_str_to_seconds » max_duration

• convert_str_to_seconds » polling_interval

• convert_str_to_seconds » sampling_interval

• task_names_cannot_be_same » all fields

field component_role: str [Required]

role of the pipeline component on which this Task should run

Validated by:

• task_names_cannot_be_same

field max_duration: float | str [Required]

the maximum duration of this Task, in seconds

Validated by:

• convert_str_to_seconds

• task_names_cannot_be_same

field sampling_interval: float | str [Required]

the interval between measurements, in seconds

Validated by:

• convert_str_to_seconds

• task_names_cannot_be_same

field polling_interval: float | str | None = None

the interval between polling for data by the tomato-job process, in seconds; defaults to the value in driver settings

Validated by:

• convert_str_to_seconds

• task_names_cannot_be_same

field technique_name: str [Required]

the name of the technique; has to match one of the capabilities of the component on which this Task will be executed

Validated by:

• task_names_cannot_be_same

field task_name: str | None = None

the (optional) name of the current Task; can be used for triggering other Task in parallel to this one via start_with_task_name

Validated by:

• task_names_cannot_be_same

field task_params: Dict[str, Any] | None [Optional]

a dict of any additional parameters required to specify the experimental technique; the key-value pairs of this dict will be used as attr-val pairs by the set_attr() method of the component executing this Task

Validated by:

• task_names_cannot_be_same

field start_with_task_name: str | None = None

the task_name of the Task that this Task should be started in parallel with; when set, this Task will wait for execution until a Task with the matching task_name is started

Validated by:

• task_names_cannot_be_same

field stop_with_task_name: str | None = None

the task_name of the Task that, when started, will stop the execution of this Task; when set, this Task will execute normally, but if a a Task with the matching task_name is started, this Task will be stopped

Validated by:

• task_names_cannot_be_same

validator task_names_cannot_be_same  »  all fields

validator convert_str_to_seconds  »  sampling_interval, max_duration, polling_interval