.. _quickstart:

Quick start guide
-----------------

First time set-up
`````````````````

.. note::

    This section assumes that **tomato** has been successfully :ref:`installed<installation>`.

To prepare **tomato** before its first run, a *settings file* containing the basic configuration for the :mod:`tomato.daemon` needs to be placed in **tomato's** *appdir* folder. By default, the *appdir* path is:

- ``$env:localappdata\dgbowl\tomato\<version>`` on Windows,
- ``$HOME/.config/tomato/<version>`` on Linux.

The easiest way to do create this file is using the provided ``tomato init`` command:

.. code-block::
   :linenos:
   :emphasize-lines: 2

    kraus@dorje:/home/kraus/$ tomato init
    Success: wrote default settings into /home/kraus/.config/tomato/1.0a1/settings.toml

Where *appdir* is ``/home/kraus/.config/tomato/1.0a1/``. A custom *appdir* can be specified using the ``--appdir`` argument to **tomato**.


**tomato** program flowchart
````````````````````````````
**tomato** currently contains two command line user-facing utilities/executables:

- :mod:`~tomato.tomato`, responsible for management of the daemon process, and
- :mod:`~tomato.ketchup`, responsible for job submission and management.

These two executables then internally spawn other processes, including the state daemon process ``tomato-daemon``, a ``tomato-job`` process for every running job, and a ``tomato-driver`` process for each driver type, managing all devices of a that type.

.. mermaid::

  flowchart TB
      subgraph daemon[tomato-daemon]
          cmd{{port:cmd}} ==> st[(state)]
          st --> jm([job manager])
          st --> dm([driver manager])
          st <--> io[state io]
      end

      subgraph driver[tomato-driver]
          dcmd{{port:cmd}} ==>  dst[(state)]
      end

      dst o-.-o h1[hardware]

      subgraph job[tomato-job]
          mp([main loop]) --> tp>task process]
      end

      cmd ==> db[(job db\nsqlite)]
      jm -.-> db
      mp -.-> db

      t>tomato] -.-> cmd
      k>ketchup] -.-> cmd

      jm -.-> job
      dst -.-> driver
      dm -..-> driver
      tp -.-> dcmd

.. _concepts:

**tomato** concepts flowchart
`````````````````````````````
The following concepts are used in **tomato**:

- *devices*, which represent separately addressable (and optionally multichannel) instruments in the lab,
- *components*, which represent the individual channels of each *device*, describing the role of this *component* and its associated capabilities,
- *pipelines*, which represent the real world organisation of device *components* into independent experimental set-ups, containing *components* from one or more *device*,
- *jobs*, which are processes that carry out a *payload* (i.e. a set of experimental instructions) on a *pipeline* by instructing its *components* to perform individual *tasks*, and
- *drivers*, which are separate processes, each managing all defined *devices* of a certain driver type as well as their *components*:

.. mermaid::

  flowchart LR

      c1 --- drv1[driver 123]
      c2 --- drv1
      c3 --- drv1

      a100 --- drv2[driver abc]
      b100 --- drv2

      subgraph d1[device 1]
          ip1[address: 192.168.1.1]
          c1[channel 1]
          c2[channel 2]
          c3[channel 3]
      end

      subgraph a[device a]
          com1[address: COM1]
          a100[channel 100]
      end

      subgraph b[device b]
          com2[address: COM2]
          b100[channel 100]
      end

      subgraph pipelines
          pip1
          pip2
          pip3
      end

      subgraph drivers
          drv1
          drv2
      end

      subgraph devices
          d1
          a
          b
      end

      subgraph jobs
          j1
          j2
      end

      j1[complex] --- |two part payload| pip1
      j2[simple] --- |simple payload| pip3

      pip1[pipeline a1] -.-> c1
      pip1[pipeline a1] -.-> a100

      pip2[pipeline b2] -.-> c2
      pip2[pipeline b2] -.-> b100

      pip3[pipeline 3] -.-> c3


.. _setfile:

Settings file
`````````````
The *settings file* contains the basic information required to start the ``tomato-daemon``. The filename is hard-coded to be a ``settings.toml`` file, which has to be located in the *appdir*. The default file generated by ``tomato init`` looks similar to the below example:

.. code-block:: toml
   :linenos:

    datadir = '/home/kraus/.local/share/tomato/1.0a1'
    logdir = '/home/kraus/.cache/tomato/1.0a1/log'

    [jobs]
    storage = '/home/kraus/.local/share/tomato/1.0a1/Jobs'
    dbpath = '/home/kraus/.local/share/tomato/1.0a1/Jobs/dbpath.sqlite'

    [devices]
    config = '/home/kraus/.config/tomato/1.0a1/devices.yml'

    [drivers]
    example_counter.testpar = 1234


In addition to the *appdir*, a second path, *datadir*, is used to specify the location of the data created by **tomato**. The default *datadir* is:

- ``$env:localappdata\dgbowl\tomato\<version>`` on Windows,
- ``$HOME/.local/share/tomato/<version>`` on Linux.

Finally, another path, *logdir*, is used to specify where logs for **tomato** are placed. By default, *logdir* is:

- ``$env:localappdata\dgbowl\tomato\<version>`` on Windows,
- ``$HOME/.cache/tomato/<version>/log`` on Linux.

In the default *settings file* shown above, the following entries are specified:

- ``jobs.storage`` which is the directory where the data of **tomato** *jobs* will be stored,
- ``jobs.dbpath`` which is the location of the ``sqlite3`` database used to track *jobs*, 
- ``devices.config`` which points to a ``yaml``-formatted :ref:`devices file <devfile>`, defining the hardware configuration of the devices managed by **tomato**.

Additional, *driver*-specific settings may be provided in the ``[drivers]`` section, following the example of the ``drivers.example_counter.testpar`` entry. These *driver*-specific settings are passed to each *driver* when its process is launched and the :class:`DriverInterface` is initialised, and can therefore contain paths to various libraries or other files necessary for the *driver* to function.

.. _devfile:

Devices file
````````````
This ``yaml``-formatted file contains information about each *device*, corresponding to an individual piece of hardware managed by **tomato**, as well as information about the organisation of the individually-addressable *components* of those *devices* into *pipelines*.

When the devices file is not present, the default file shipped with **tomato** will be used instead:

**devices** section
*******************
The ``devices`` section of the default *devices file* is shown below:

.. code-block:: yaml
   :linenos:

    devices:
      - name: dev-counter
        driver: "example_counter"
        address: "example-addr"
        channels: ["1"]
        pollrate: 1

Here, we define a single device using the :mod:`~tomato.drivers.example_counter` driver. The definition includes the ``address`` of the device (:class:`str` type) as well as an enumeration of individually-addressable channels the device has (:class:`list[str]`).

For example, the devices shown in the :ref:`concepts flowchart <concepts>` above would be defined as:

.. code-block:: yaml
   :linenos:

    devices:
      - name: device 1
        driver: "driver 123"
        address: "192.168.1.1"
        channels: ["1", "2", "3"]
        pollrate: 1
      - name: device a
        driver: "driver abc"
        address: "COM1"
        channels: ["100"]
        pollrate: 5
      - name: device b
        driver: "driver abc"
        address: "COM2"
        channels: ["100"]
        pollrate: 5


.. note::

    The ``pollrate`` here is an internal setting for **tomato**. It is the frequency with which the *job* interacting with the device *driver* asks the *driver* for data, **not** the frequency with which the *driver* asks the hardware for data! This latter "sampling" frequency can be defined in each individual *task* submitted as a *payload* of a *job*.

**pipelines** section
*********************
The default ``pipelines`` section looks as follows:

.. code-block:: yaml
   :linenos:
   :emphasize-lines: 5

    pipelines:
      - name: pip-counter
        components:
          - role: counter
            device: dev-counter
            channel: "1"

Here, a single *pipeline* called ``pip-counter`` is defined to contain the one available channel of the ``dev-counter`` device (defined on line 5) shown further above. For multi channel devices, it is also possible to define a *pipeline* for each channel automatically, e.g. using the following definition:

.. code-block:: yaml
   :linenos:
   :emphasize-lines: 2,6

    pipelines:
      - name: pip-counter-*
        components:
          - role: counter
            device: dev-counter
            channel: each

Here, a set of pipelines would be created using each of the available channels in ``dev-counter``, replacing the ``*`` in line 2 using the channel. Therefore, one *pipeline*  with name ``pip-counter-1`` would be created. However, if the ``dev-counter`` were to contain more than one channel, a *pipeline* would be generated for each channel.

Finally, a definition for the pipelines shown in the :ref:`concepts flowchart <concepts>` above can be defined as:

.. code-block:: yaml
   :linenos:

    pipelines:
      - name: pipeline a1
        components:
          - role: dev 123
            device: device 1
            channel: "1"
          - role: dev abc
            device: device a
            channel: "100"
      - name: pipeline b2
        components:
          - role: dev 123
            device: device 1
            channel: "2"
          - role: dev abc
            device: device b
            channel: "100"
      - name: pipeline 3
        components:
          - role: dev 123
            device: device 1
            channel: "3"

.. _payfile:

Payload file
````````````
The *payload* file contains all information required to enter a *job* into the queue and allow its assignment onto a *pipeline*. The overall schema of the *payload* is defined in the :mod:`dgbowl_schemas.tomato` module, and is parsed using :func:`dgbowl_schemas.tomato.to_payload`:

.. autopydantic_model:: dgbowl_schemas.tomato.payload.Payload


.. note::

    Of particular importance here is the specification of the individual :class:`Tasks`. Some general :class:`Task` parameters are abstracted by **tomato**, such as the ``max_duration``, ``sampling_interval`` and ``polling_interval``. The other are *driver* or *component* specific, and can be specified using the ``task_params`` :class:`dict`.

.. autopydantic_model:: dgbowl_schemas.tomato.payload.Task

As of ``tomato-2.0``, the ``task_params`` specified in the *payload* are validated by the device *driver*. In particular, the setpoints of any entries within ``task_params`` are checked for compatibility with the type of the matching :class:`~tomato.driverinterface_2_0.Attr`. Similarly, their parsed values are compared against the optional :obj:`Attr.maximum` and :obj:`Attr.minimum`.  

.. warning::
  
  Currently, **tomato** does not check whether the ``sampling_interval`` is realistic. This means your *jobs* may crash when the ``task_params`` are out of bounds when the :class:`Task` is being executed.

  However, such validation is planned for a future release, see https://github.com/dgbowl/tomato/issues/127.