Usage ----- The **tomato** package consists of two key components: - the job scheduler in :mod:`tomato.daemon` executed by ``tomato``, - the queue management app :mod:`tomato.ketchup` executed by ``ketchup``. Starting :mod:`tomato.daemon` `````````````````````````````` The job scheduler ``tomato`` can be started in verbose mode for diagnostic output: .. code:: bash tomato -vv It is used to schedule *jobs* from the *queue* onto *pipelines*, and tracks the overall *state* of the system. A *pipeline* is a way of organising one or many *devices* in a single, addressable unit, see :ref:`devfile` for more details. In general, a single *pipeline* represents a digital twin of an experimental set-up, composed of all *devices* neccessary to carry out a single experimental *payload*. .. note:: Only one instance of the :mod:`tomato.daemon` can be running at a single PC at the same time, excluding any test-suite jobs (``tomato`` executed with the ``-t`` switch). .. note:: For instructions on how to set **tomato** up for a first run, see the :ref:`quickstart`. Using :mod:`~tomato.ketchup` ```````````````````````````` The :mod:`~tomato.ketchup` executable is used to submit *payloads* to the *queue*, to check the status of and to cancel *jobs* in the *queue*, as well as to manage *pipelines* by loading or ejecting *samples* and marking *pipelines* ready for execution. #. **To submit** a *job* using a *payload* contained in a :ref:`payfile` to the *queue*, run: .. code-block:: bash >>> ketchup submit jobid: The *job* will enter the *queue* and wait for a suitable *pipeline* to begin execution. .. note:: For more information about how *jobs* are matched against *pipelines*, see the documentation of the :mod:`~tomato.daemon` module. #. **To check the status** of a *job* with a known ``jobid``, run: .. code-block:: bash >>> ketchup status - jobid: jobname: null status: r submitted: 2022-06-30 11:18:21.538448+00:00 executed: 2022-06-30 11:18:22.983600+00:00 The list of possible *job* statuses is: ======== =========================================================== Status Meaning ======== =========================================================== q Job has entered the queue. qw Job is in the queue, waiting for a pipeline to be ready. r Job is running. rd Job has been marked for cancellation. c Job has completed successfully. ce Job has completed with an error. cd Job has been cancelled. ======== =========================================================== .. note:: The above command can process multiple ``jobids``, returning the information in a ``yaml``-formatted output. #. **To cancel** a submitted *job* with a known ``jobid``, run: .. code-block:: bash >>> ketchup cancel This will mark the `job` for cancellation by setting its status to ``rd``. The :mod:`tomato.daemon` will then proceed with cancelling the `job`. *Jobs* submitted to the *queue* will remain in the *queue* until a *pipeline* meets all of the following criteria: - A *pipeline* which matches all of the ``techniques`` specified in the *payload* by its ``capabilities`` must exist. Once the :mod:`tomato.daemon` finds such a *pipeline*, the status of the *job* will change to ``qw``. - The matching *pipeline* must contain a *sample* with a ``samplename`` that matches the name specified in the *payload*. - The matching *pipeline* must be marked as ``ready``. To manage *samples* in the *pipelines*, use the following :mod:`~tomato.ketchup` commands: #. **To load** the digital twin of a *sample* with a ``samplename`` into a selected *pipeline*, run: .. code-block:: bash >>> ketchup load Trying to load a *sample* in a *pipeline* that already contains a *sample* will fail with a warning. The *pipeline* must be emptied first. #. **To eject** a *sample* from the *pipeline*, run: .. code-block:: bash >>> ketchup eject Attempting to eject a *sample* from a *pipeline* where a *job* is currently running will fail with a warning. The *job* must finish or be cancelled first. #. **To mark** a *pipeline* as **ready**, run: .. code-block:: bash >>> ketchup ready This mechanism is implemented to allow for batch-loading of *samples* into *pipelines*, without having to worry about *jobs* starting prematurely. By default, successfully completed *jobs* (status ``c``) will **not** mark the *pipeline* as ready. This behaviour can be configured in the ``tomato`` section of the *payload*. Jobs that have been cancelled (status ``cd``) or have errored out (status ``ce``) will never mark the *pipeline* as ready, even if configured to do so in the *payload*. This has been implemented to allow the user to investigate the *sample* and/or *pipeline* for any faults. .. note:: Further information about :mod:`~tomato.ketchup` is available in the documentation of the :mod:`~tomato.ketchup` module. Accessing output data ````````````````````` Final job data ************** By default, all data in the *job* folder is processed using ``yadg`` to create a *datagram*, and zipped into a zip archive. This zip archive includes all raw data files, the log file of the **tomato** job, and a copy of the full *payload* in a ``json`` file. The *datagram* contains timestamped, unit-annotated raw data, and includes instrumental uncertainties. Unless specified within the *payload*, the default location where these output files will be placed is the ``cwd()`` where the ``ketchup submit`` command was executed; the default filenames of the returned files are ``results..[zip,json]``. Data snapshotting ***************** While the *job* is running, access to an up-to-date snapshot of the data is provided by :mod:`~tomato.ketchup`: .. code:: bash >>> ketchup snapshot This will create an up-to-date ``snapshot..[zip,json]`` in the ``cwd()``. The files are overwritten on subsequent invocations of ``ketchup snapshot``. An automated, periodic snapshotting can be further configured within the *payload* of the *job*.