Developing tomato drivers

Since tomato-1.0, all device drivers are developed as separate Python packages with their own documentation and versioning. To ensure compatibility of the Manager between the tomato-driver process and the implementation of the driver, an abstract class ModelInterface is provided. A class inheriting from this abstract class, with the name DriverInterface, has to be available when the selected driver module is imported.

Note

The ModelInterface is versioned. Your driver should target a single version of this ModelInterface by inheriting from only one such abstract class. Any deprecation notices will be provided well in advance directly to driver maintainers. Support for driverinterface_1_0 introduced in tomato-1.0 is guaranteed until at least tomato-3.0.

Bootstrapping a driver process

When the driver process is launched (as a tomato-driver), it’s given information about how to connect to the tomato-daemon process and which device driver to spawn. Once a connection to the tomato-daemon is established, the driver settings are fetched, and the DriverInterface is instantiated passing any settings to the constructor. Then, all components on all devices of this driver type that are known to tomato-daemon are registered using the cmp_register() function.

Note

Each driver creates a separate log file for each port tomato has been executed with. The logfile is stored in the same location as the tomato-daemon logs, i.e. as configured under the logdir option in the settings file. The verbosity of the tomato-driver process is inherited from the tomato-daemon process.

Communication between jobs and drivers

After the driver process is bootstrapped, it enters the main loop, listening for commands to action or pass to the DriverInterface. Therefore, if a job needs to submit a Task, it passes the Task to the tomato-driver process, which actions it on the appropriate component using the task_submit() function. Similarly, if a job decides to poll the driver for data, it does so using the task_data() function.

In general, methods of the DriverInterface that are prefixed with cmp_* deal with managing devices or their components on the driver, methods prefixed with task_* deal with managing Tasks running or submitted to components, and methods without a prefix deal with configuration or status of the driver itself.

Note

The driverinterface_2_0 contains another abstract class ModelDevice. In general, the DriverInterface that you have to implement acts as a pass-through to the (abstract) methods of the ModelDevice; e.g. ModelInterface.cmp_get_attr() is a passthrough function to the appropriate ModelDevice.get_attr().

We expect most of the work in implementing a new driver will actually take place in implementing the ModelDevice class.

Currently, when a Task is submitted to a component, a new Thread is launched on that component that takes care of preparing the component (via ModelDevice.prepare_task()), executing the Task (via ModelDevice.task_runner()), and periodically polling the hardware for data (via the abstract ModelDevice.do_task()). As each component can only run one Task at the same time, subsequently submitted Tasks are stored in an ModelDevice.task_list, which is a Queue used to communicate with the worker Thread. This worker Thread is reinstantiated at the end of every Task.

Note

Direct access to the ModelDevice.data object is not thread-safe. Therefore, a reentrant lock (RLock) object is provided as ModelDevice.datalock. Reading or writing to the ModelDevice.data with the exception of the ModelDevice.get_data() and ModelDevice.do_task() methods should be only carried out when this datalock is acquired, e.g. using a context manager.

Note

The ModelDevice.data object is intended to cache data between ModelDevice.get_data() calls initiated by the job. This object is therefore cleared whenever ModelDevice.get_data() is called; it is the responsibility of the tomato-job process to append or store any new data.

To access the current configuration (i.e. status) of the component, the ModelDevice provides a ModelDevice.status() method. The implementation of what is reported as component status is up to the developers of each driver via the ModelDevice.attrs() function.

To access the latest data of the component, the ModelDevice provides a ModelDevice.get_last_data() method. This will

Best Practices when developing a driver

• The ModelDevice.attrs() defines the variable attributes of the component that should be accessible, using Attr. All entries in attrs() should also be present in ModelDevice.data, as the data likely depends on the settings of these Attrs.

• Each ModelDevice contains a link to its parent ModelInterface in the ModelDevice.driver object.

• Internal functions of the ModelDevice and ModelInterface should be re-used wherever possible. E.g., reading component attributes should always be carried out using ModelDevice.get_attr().

DriverInterface ver. 2.0

class tomato.driverinterface_2_0.ModelInterface(settings=None)

An abstract base class specifying the driver interface.

Individual driver modules should expose a DriverInterface as a top-level import, which inherits from this abstract class. Only the methods of this class are used to interact with drivers and their components.

This class contains one abstract method, DeviceFactory(), that has to be re-implemented by the driver modules.

All methods of this class should return Reply objects (except the DeviceFactory() function). However, for better readability, a decorator function to_reply() is provided, so that the types of the return values can be explicitly defined here.

devmap: dict[tuple, ModelDevice]

Map of registered devices, the tuple keys are component = (address, channel)

constants: dict[str, Any]

A map that should be populated with driver-specific run-time constants.

settings: dict[str, Any]

A settings map to contain driver-specific settings such as dllpath for BioLogic

abstract DeviceFactory(key: tuple[str, str], **kwargs) → ModelDevice

A factory function which is used to pass this instance of the ModelInterface to the new ModelDevice instance.

cmp_register(address: str, channel: str, **kwargs: dict) → tuple[bool, str, set]

Register a new device component in this driver.

Creates a ModelDevice representing a device component, storing it in the self.devmap using the provided address and channel.

Returns the set of capabilities of the registered component as the Reply.data.

cmp_teardown(key: tuple[str, str], **kwargs: dict) → tuple[bool, str, None]

Emergency stop function.

Should set the device component into a documented, safe state. The function is to be only called in case of critical errors, or when the component is being removed, not as part of normal operation (i.e. it is not intended as a clean-up after task completion).

cmp_reset(key: tuple[str, str], **kwargs: dict) → tuple[bool, str, None]

Component reset function.

Should set the device component into a documented, safe state. This function is executed at the end of every job.

cmp_set_attr(attr: str, val: Val, key: tuple[str, str], **kwargs: dict) → tuple[bool, str, Val]

Set value of the Attr of the specified device component.

Pass-through to the ModelDevice.set_attr() function. No type or read-write validation performed here! Returns the validated or coerced value as the Reply.data.

cmp_get_attr(attr: str, key: tuple[str, str], **kwargs: dict) → tuple[bool, str, Val]

Get value of the Attr from the specified device component.

Pass-through to the ModelDevice.get_attr() function. No type coercion is done here. Returns the value as the Reply.data.

cmp_status(key: tuple[str, str], **kwargs: dict) → tuple[bool, str, dict]

Get the status report from the specified device component.

Iterates over all Attrs on the component that have status=True and returns their values in the Reply.data as a dict.

cmp_capabilities(key: tuple[str, str], **kwargs) → tuple[bool, str, set]

Returns the capabilities of the device component.

Pass-through to ModelDevice.capabilities(). Returns the set of capabilities in Reply.data.

cmp_attrs(key: tuple[str, str], **kwargs: dict) → tuple[bool, str, dict]

Query available Attrs on the specified device component.

Pass-through to the ModelDevice.attrs() function. Returns the dict of attributes as the Reply.data.

cmp_constants(key: tuple[str, str], **kwargs: dict) → tuple[bool, str, dict]

Query constants on the specified device component and this driver.

Returns the dict of constants as the Reply.data.

cmp_last_data(key: tuple[str, str], **kwargs: dict) → tuple[bool, str, None | Dataset]

Fetch the last stored data on the component.

Passthrough to ModelDevice.get_last_data(). The data in the form of a xarray.Dataset is returned as the Reply.data.

cmp_measure(key: tuple[str, str], **kwargs: dict) → tuple[bool, str, None]

Do a single measurement on the component according to its current configuration.

Fails if the component already has a running task / measurement.

task_start(key: tuple[str, str], task: Task, **kwargs) → tuple[bool, str, set | Task]

Submit a Task onto the specified device component.

Pushes the supplied Task into the Queue of the component, then starts the worker thread. Checks that the Task is among the capabilities of this component.

task_status(key: tuple[str, str], **kwargs: dict) → tuple[bool, str, dict]

Returns the task readiness status of the specified device component.

The running entry in the data slot of the Reply indicates whether a Task is running. The can_submit entry indicates whether another Task can be queued onto the device component already.

task_stop(key: tuple[str, str], **kwargs) → tuple[bool, str, Dataset | None]

Stops a running task and returns any collected data.

Pass-through to ModelDevice.stop_task() and ModelInterface.task_data().

If there is any cached data, it is returned as a xarray.Dataset in the Reply.data and the cache is cleared.

task_data(key: tuple[str, str], **kwargs) → tuple[bool, str, Dataset | None]

Return cached task data on the device component and clean the cache.

Pass-through for ModelDevice.get_data(), which should return a xarray.Dataset that is fully annotated.

This function gets called by the job thread every device.pollrate, it therefore incurs some IPC cost.

task_validate(key: tuple[str, str], task: Task, **kwargs) → tuple[bool, str, None]

Validate the provided Task for submission on the component identified by key.

status() → Reply

Returns the driver status. Currently that is the names of the components in the devmap.

reset() → Reply

Resets the driver.

Called when the driver process is quitting. Instructs all remaining tasks to stop. Warns when devices linger. Passes through to dev_reset(). This is not a pass-through to dev_teardown().

class tomato.driverinterface_2_0.ModelDevice(driver, key, **kwargs)

An abstract base class specifying a manager for an individual component.

This class should handle determining attributes and capabilities of the component, the reading/writing of those attributes, processing of tasks, and caching and returning of task data.

driver: ModelInterface

The parent DriverInterface instance.

key: tuple[str, str]

The key in self.driver.devmap referring to this object.

task_list: Queue

A Queue used to pass Tasks to the worker Thread.

thread: Thread

The worker Thread.

data: Dataset | None

Container for cached data on this component.

last_data: Dataset | None

Container for last datapoint on this component.

datalock: RLock

Lock object for thread-safe data manipulation.

constants: dict[str, Any]

Constant metadata of this component.

run()

Helper function for starting the self.thread as a task.

measure()

Helper function for starting the self.thread as a measurement.

task_runner()

Target function for the self.thread when handling Tasks.

This function waits for a Task passed using self.task_list, then handles setting all Attrs using the prepare_task() function, and finally handles the main loop of the task, periodically running the do_task() function (using task.sampling_interval) until the maximum task duration (i.e. task.max_duration) is exceeded.

The self.thread is re-primed for future Tasks at the end of this function.

meas_runner()

Target function for the self.thread when performing one shot measurements.

Performs the measurement using self.do_measure(). Resets self.thread for future Tasks.

prepare_task(task: Task, **kwargs: dict)

Given a Task, prepare this component for execution by setting all Attrs as specified in the task.task_params dictionary.

do_task(task: Task, **kwargs: dict)

Periodically called task execution function.

This function is responsible for updating self.data with new data, i.e. performing the measurement. It should also update the value of self.last_data, so that the component status is consistent with the cached data.

abstract do_measure(**kwargs: dict) → None

One shot execution worker function.

This function is performs a measurement using the current configuration of self.attrs, and stores the result in self.last_data.

stop_task(**kwargs: dict)

Stops the currently running task.

abstract set_attr(attr: str, val: Val, **kwargs: dict) → Val

Sets the specified Attr to val.

This function should handle any data type coercion and validation using e.g. Attr.maximum and Attr.minimum.

Returns the coerced value coresponding to val.

abstract get_attr(attr: str, **kwargs: dict) → Val

Reads the value of the specified Attr.

get_data(**kwargs: dict) → Dataset

Returns the cached self.data as a xarray.Dataset before clearing the cache.

get_last_data(**kwargs: dict) → Dataset

Returns the last_data object as a xarray.Dataset.

abstract attrs() → dict[str, Attr]

Returns a dict of all available Attrs.

abstract capabilities() → set

Returns a set of all supported techniques.

status(**kwargs) → dict[str, Val]

Compiles a status report from Attrs marked as status=True.

reset(**kwargs) → None

Resets the component to an initial status.

DriverInterface ver. 1.0

class tomato.driverinterface_1_0.ModelInterface(settings=None)

An abstract base class specifying the driver interface.

Individual driver modules should expose a DriverInterface which inherits from this abstract class. Only the methods of this class should be used to interact with drivers and their devices.

class DeviceManager(driver, key, **kwargs)

An abstract base class specifying a manager for an individual component.

This class should handle determining attributes and capabilities of the component, the reading/writing of those attributes, processing of tasks, and caching and returning of task data.

driver: ModelInterface

The parent DriverInterface instance.

key: tuple

The key in self.driver.devmap referring to this object.

task_list: Queue

A Queue used to pass Tasks to the worker Thread.

thread: Thread

The worker Thread.

data: dict[str, list]

Container for cached data on this component.

datalock: RLock

Lock object for thread-safe data manipulation.

run()

Helper function for starting the self.thread.

task_runner()

Target function for the self.thread.

This function waits for a Task passed using self.task_list, then handles setting all Attrs using the prepare_task() function, and finally handles the main loop of the task, periodically running the do_task() function (using task.sampling_interval) until the maximum task duration (i.e. task.max_duration) is exceeded.

The self.thread is re-primed for future Tasks at the end of this function.

prepare_task(task: Task, **kwargs: dict)

Given a Task, prepare this component for execution by settin all Attrs as specified in the task.task_params dictionary.

abstract do_task(task: Task, **kwargs: dict)

Periodically called task execution function.

This function is responsible for updating self.data with new data, i.e. performing the measurement. It should also update the values of all Attrs, so that the component status is consistent with the cached data.

stop_task(**kwargs: dict)

Stops the currently running task.

abstract set_attr(attr: str, val: Any, **kwargs: dict)

Sets the specified Attr to val.

abstract get_attr(attr: str, **kwargs: dict) → Any

Reads the value of the specified Attr.

get_data(**kwargs: dict) → dict[str, list]

Returns the cached self.data before clearing the cache.

abstract attrs() → dict[str, Attr]

Returns a dict of all available Attrs.

abstract capabilities() → set

Returns a set of all supported techniques.

status(**kwargs) → dict

Compiles a status report from Attrs marked as status=True.

reset(**kwargs) → None

Resets the component to an initial status.

CreateDeviceManager(key, **kwargs)

A factory function which is used to pass this instance of the ModelInterface to the new DeviceManager instance.

devmap: dict[tuple, DeviceManager]

Map of registered devices, the tuple keys are component = (address, channel)

settings: dict[str, Any]

A settings map to contain driver-specific settings such as dllpath for BioLogic

dev_register(address: str, channel: str, **kwargs: dict) → Reply

Register a new device component in this driver.

Creates a DeviceManager representing a device component, storing it in the self.devmap using the provided address and channel.

The returned Reply should contain the capabilities of the registered component in the data slot.

dev_teardown(key: tuple, **kwargs: dict) → Reply

Emergency stop function.

Should set the device component into a documented, safe state. The function is to be only called in case of critical errors, or when the component is being removed, not as part of normal operation (i.e. it is not intended as a clean-up after task completion).

dev_reset(key: tuple, **kwargs: dict) → Reply

Component reset function.

Should set the device component into a documented, safe state. This function is executed at the end of every job.

dev_set_attr(attr: str, val: Any, key: tuple, **kwargs: dict) → Reply

Set value of the Attr of the specified device component.

Pass-through to the DeviceManager.set_attr() function. No type or read-write validation performed here!

dev_get_attr(attr: str, key: tuple, **kwargs: dict) → Reply

Get value of the Attr from the specified device component.

Pass-through to the DeviceManager.get_attr() function. Units are not returned; those can be queried for all Attrs using self.attrs().

dev_status(key: tuple, **kwargs: dict) → Reply

Get the status report from the specified device component.

Iterates over all Attrs on the component that have status=True and returns their values in the Reply.data as a dict.

task_start(key: tuple, task: Task, **kwargs) → Reply

Submit a Task onto the specified device component.

Pushes the supplied Task into the Queue of the component, then starts the worker thread. Checks that the Task is among the capabilities of this component.

task_status(key: tuple, **kwargs: dict) → Reply

Returns the task readiness status of the specified device component.

The running entry in the data slot of the Reply indicates whether a Task is running. The can_submit entry indicates whether another Task can be queued onto the device component already.

task_stop(key: tuple, **kwargs) → Reply

Stops a running task and returns any collected data.

Pass-through to DriverManager.stop_task() and task_data().

task_data(key: tuple, **kwargs) → Reply

Return cached task data on the device component and clean the cache.

Pass-through for DeviceManager.get_data(), with the caveat that the dict[list] which is returned from the component is here converted to a Dataset and annotated using units from attrs().

This function gets called by the job thread every device.pollrate, it therefore incurs some IPC cost.

status() → Reply

Returns the driver status. Currently that is the names of the components in the devmap.

reset() → Reply

Resets the driver.

Called when the driver process is quitting. Instructs all remaining tasks to stop. Warns when devices linger. Passes through to dev_reset(). This is not a pass-through to dev_teardown().

capabilities(key: tuple, **kwargs) → Reply

Returns the capabilities of the device component.

Pass-through to DriverManager.capabilities().

attrs(key: tuple, **kwargs: dict) → Reply

Query available Attrs on the specified device component.

Pass-through to the DeviceManager.attrs() function.