Usage

The tomato package consists of two key components:

Starting tomato.daemon

The job scheduler tomato can be started in verbose mode for diagnostic output:

tomato -vv

It is used to schedule jobs from the queue onto pipelines, and tracks the overall state of the system. A pipeline is a way of organising one or many devices in a single, addressable unit, see Devices file for more details. In general, a single pipeline represents a digital twin of an experimental set-up, composed of all devices neccessary to carry out a single experimental payload.

Note

Only one instance of the tomato.daemon can be running at a single PC at the same time, excluding any test-suite jobs (tomato executed with the -t switch).

Note

For instructions on how to set tomato up for a first run, see the Quick start guide.

Using ketchup

The ketchup executable is used to submit payloads to the queue, to check the status of and to cancel jobs in the queue, as well as to manage pipelines by loading or ejecting samples and marking pipelines ready for execution.

  1. To submit a job using a payload contained in a Payload file to the queue, run:

    >>> ketchup submit <payload>
jobid: <jobid>

    The job will enter the queue and wait for a suitable pipeline to begin execution.

    Note

    For more information about how jobs are matched against pipelines, see the documentation of the daemon module.

  2. To check the status of a job with a known jobid, run:

    >>> ketchup status <jobid>
- jobid: <jobid>
  jobname: null
  status: r
  submitted: 2022-06-30 11:18:21.538448+00:00
  executed: 2022-06-30 11:18:22.983600+00:00

    The list of possible job statuses is:

    Status

    Meaning

    q

    Job has entered the queue.

    qw

    Job is in the queue, waiting for a pipeline to be ready.

    r

    Job is running.

    rd

    Job has been marked for cancellation.

    c

    Job has completed successfully.

    ce

    Job has completed with an error.

    cd

    Job has been cancelled.

    Note

    The above command can process multiple jobids, returning the information in a yaml-formatted output.

  3. To cancel a submitted job with a known jobid, run:

    >>> ketchup cancel <jobid>

    This will mark the job for cancellation by setting its status to rd. The tomato.daemon will then proceed with cancelling the job.

Jobs submitted to the queue will remain in the queue until a pipeline meets all of the following criteria:

  • A pipeline which matches all of the techniques specified in the payload by its capabilities must exist. Once the tomato.daemon finds such a pipeline, the status of the job will change to qw.

  • The matching pipeline must contain a sample with a samplename that matches the name specified in the payload.

  • The matching pipeline must be marked as ready.

To manage samples in the pipelines, use the following ketchup commands:

  1. To load the digital twin of a sample with a samplename into a selected pipeline, run:

    >>> ketchup load <samplename> <pipeline>

    Trying to load a sample in a pipeline that already contains a sample will fail with a warning. The pipeline must be emptied first.

  2. To eject a sample from the pipeline, run:

    >>> ketchup eject <pipeline>

    Attempting to eject a sample from a pipeline where a job is currently running will fail with a warning. The job must finish or be cancelled first.

  3. To mark a pipeline as ready, run:

    >>> ketchup ready <pipeline>

    This mechanism is implemented to allow for batch-loading of samples into pipelines, without having to worry about jobs starting prematurely. By default, successfully completed jobs (status c) will not mark the pipeline as ready. This behaviour can be configured in the tomato section of the payload. Jobs that have been cancelled (status cd) or have errored out (status ce) will never mark the pipeline as ready, even if configured to do so in the payload. This has been implemented to allow the user to investigate the sample and/or pipeline for any faults.

Note

Further information about ketchup is available in the documentation of the ketchup module.

Accessing output data

Final job data

By default, all data in the job folder is processed using yadg to create a datagram, and zipped into a zip archive. This zip archive includes all raw data files, the log file of the tomato job, and a copy of the full payload in a json file. The datagram contains timestamped, unit-annotated raw data, and includes instrumental uncertainties.

Unless specified within the payload, the default location where these output files will be placed is the cwd() where the ketchup submit command was executed; the default filenames of the returned files are results.<jobid>.[zip,json].

Data snapshotting

While the job is running, access to an up-to-date snapshot of the data is provided by ketchup:

>>> ketchup snapshot <jobid>

This will create an up-to-date snapshot.<jobid>.[zip,json] in the cwd(). The files are overwritten on subsequent invocations of ketchup snapshot. An automated, periodic snapshotting can be further configured within the payload of the job.