Developing **tomato** --------------------- .. important:: No LLM / No generative AI policy. This means: - Do not generate issues using LLMs. - Do not use LLMs for patches / pull requests. - Do not submit LLM-generated comments on the issue tracker. For further reasoning, please refer to `Johannes Link's excellent blog on this issue `_. **tomato** program flowchart ```````````````````````````` **tomato** currently contains three command line user-facing utilities/executables: - :mod:`~tomato.tomato`, responsible for management of the ``tomato-daemon`` process, - :mod:`~tomato.ketchup`, responsible for job submission and management, and - :mod:`~tomato.passata`, providing an interface to the hardware by communicating to drivers and components. These executables then internally spawn other processes, including the state daemon process ``tomato-daemon``, a ``tomato-job`` process for every running job, and a ``tomato-driver`` process for each driver type, managing all devices of a that type. An incomplete program flowchart is shown below, with the individual components discussed further below. .. mermaid:: flowchart TB subgraph daemon[tomato-daemon] cmd{{port:cmd}} <==> cfg[(config)] cfg --> jm([job manager]) cfg --> pm([pip manager]) cfg --> dm([driver manager]) end jm -.-> job subgraph job[tomato-job] mp([main loop]) --> tp>task process] end jm <-.-> jobdb[(job db
sqlite)] mp -.-> jobdb jm <-.-> pipdb[(pip db
sqlite)] pm <-.-> pipdb mp -.-> pipdb dm <-.-> drvdb[(drv db
sqlite)] drvdb <-.-> dst setf[settings file
devices file] ---> cfg t>tomato] -.-> cmd k>ketchup] -.-> cmd dst -.-> driver dm -..-> driver tp -.-> dcmd subgraph driver[tomato-driver] dcmd{{port:cmd}} ==> dst[(state)] end dst o-.-o h1[hardware] p>passata] -.-> cmd p -.-> dcmd ``tomato`` utility `````````````````` This utility can: - manage the **tomato** installation by creating default configuration files (|setfile|_, |devfile|_) via ``tomato init``, - start the ``tomato-daemon`` process or trigger its configuration reload via ``tomato start`` and ``tomato reload``, - manage the pipeline ``sample`` and ``ready`` state by interacting with the ``pip db`` via ``tomato pipeline``, - retrieve the status of the ``tomato-daemon`` process (i.e. its configuration) via ``tomato status tomato``, - retrieve the status of pipelines, drivers, devices, or components by querying the ``pip db``, ``drv db``, and ``tomato-driver`` processes via ``tomato status pipelines`` etc. ``ketchup`` utility ``````````````````` The utility first retrieves configuration from a running ``tomato-daemon``, in order to find locations of the necessary files. Then, it can: - manage the job submission, status, cancellation by interacting with ``job db``, - manage job snapshotting by traversing the job storage folder. .. tip:: This utility is modelled after common HPC queueing software, including the sub-commands (``submit``, ``cancel``, ``status``) and status codes (e.g. ``q``, ``r``, etc.). ``passata`` utility ``````````````````` The utility first retrieves configuration from a running ``tomato-daemon``, in order to find ports of running ``tomato-drivers``. Then, it can: - register a new component on a driver, - query the status of a component, - query attributes, capabilities, constants on a component, - get and set attribute values on a component, - trigger measurements on a component, - fetch the last measured data from a component - trigger a reset on a component. .. tip:: The `marinara `_ package, which is the web dashboard of the **tomato** ecosystem, uses the :mod:`tomato.passata` API extensively. The CLI interface is currently not as flexible as the API. ``tomato-daemon`` process ````````````````````````` The ``tomato-daemon`` process is stateless, only responsible for reading of the configuration files, validating driver settings, and spawning the *job*, *driver* and *pipeline*--management threads. All configuration and state information that should be persistently stored (i.e. survive restart of these components) is stored in a :mod:`sqlite3` database. ``tomato.daemon.driver.manager`` thread ....................................... This thread ensures that the drivers, as specified in the |devfile|_ and parsed by ``tomato-daemon``, have corresponding ``tomato-driver`` process launched, re-spawned as necessary, and terminated (gracefully or forcefully) once the ``tomato-daemon`` process is about to quit. It also ensures the ``drv db`` is consistent with the running ``tomato-daemon`` processes. The individual driver state, as stored in the ``drv db``, is also managed in this thread, taking care of heartbeating, ``tomato-driver`` spawn retries, or pruning of the ``drv db`` entries if necessary. Finally, once a new ``tomato-driver`` process sets its ``port`` and ``pid``, this thread instructs the ``tomato-driver`` to register its components, if there are none registered so far. ``tomato.daemon.job.manager`` thread .................................... This thread ensures that the ``job db`` and the ``tomato-job`` processes are consistent: - New jobs, submitted via ``ketchup`` and stored in ``job db``, are matched against the existing pipelines considering the capabilities of the components and the sample in the pipeline. The job status is updated to ``qw`` accordingly. - If an appropriate pipeline becomes ready, new ``tomato-job`` processes are launched, and the pipeline is "reserved" for this job by storing its ``jobid`` in the ``pip db``. - Running jobs with inconsistent entries in the ``job db``, (e.g. missing ``tomato-job`` process with a known ``pid``) are cleared appropriately, by resetting their ``pid`` in the ``job db``. - Jobs flagged for cancellation by ``ketchup`` are terminated appropriately and their ``job db`` entry is updated accordingly. .. note:: The manager thread acts on a snapshot of the ``pip db`` state, which is updated once per manager main loop, in order to ensure sequential jobs targetting the same pipeline are submitted in the correct order. ``tomato.daemon.pip.manager`` thread .................................... This thread ensures that the ``pip db`` state is consistent with the running ``tomato-job`` processes. As mentioned above, the ``pip db`` stores a ``jobid`` for every job that should be running. Using the ``job db``, this ``jobid`` allows a lookup of job ``pids`` for all jobs that occupy a pipeline. If a ``tomato-job`` process under this stored ``pid`` no longer exists, or exists and is a zombie, the individual components of that pipeline are reset via their ``tomato-driver`` processes, and the ``jobid`` in ``pip db`` is cleared. The pipeline is then available for further jobs, once marked as ``ready`` via the ``tomato pipeline ready`` command. ``tomato-job`` process `````````````````````` This process manages the execution of a single job. It performs the following actions: - store its own ``pid``, set the ``connected_at`` timestamp, and set the ``status`` to ``r`` in the ``job db``, - determine and create paths for snapshots, results, and temporary job files, storing them in the ``job db``, - if necessary, wait for all required ``tomato-driver`` processes to launch, - plan the job, reorganising tasks by component roles, creating a :obj:`Thread` for each component role, and pushing those plans to these threads, - the individual job threads then execute the :class:`Tasks` within the plan as specified, each thread communicating with a single ``tomato-driver`` process, - the :class:`Tasks` are started and stopped as appropriate by the job thread, - once all :class:`Tasks` in a job thread are completed, a reset of the component is triggered via ``tomato-driver``, - the main thread performs any snapshotting (acting on the temporary job files), message passing of completed and started :class:`Tasks` between threads, and waits for all job threads to join the main thread, - the job data is collated into a NetCDF file and an optional RO-crate, - the ``job db`` is updated to indicate job completion, - the ``pip db`` is updated to free the used pipeline. ``tomato-driver`` process ````````````````````````` This is a process, spawned by the driver manager, that handles all interactions with a single type of devices and their components. These processes are the **only** processes that touch hardware within the **tomato** framework. Please refer to the `tomato driver documentation `_ for more info.