# Cadence and Observing Strategies

The TOM has a couple of unique concepts that may be unfamiliar to some at first, that will be describe here before going
into detail.

The first concept is that of an observing strategy. An observing strategy is something of a template. If an observer is
consistently submitting observations with a lot of similar parameters, it may be useful to save those as a kind of
template, which can just be loaded later. The TOM Toolkit offers an interface that allows facilities to define a
strategy form, that will be saved as an observing strategy. The strategy can then be applied to an observation, with the
remaining parameters filled in or changed. An observing strategy can also be creating from a past observation, with a
button to do so that's available on any ObservationRecord detail page.

The second concept referred to is a cadence strategy. A cadence is as it sounds--a series of observations that are
performed at regular intervals. However, most observatories don't have built-in support for cadences, and, if they do,
they may be limited to a predetermined cadence. The TOM Toolkit, on the other hand, allows for a *reactive* cadence.
Because data is collected programmatically, and observations are submitted programmatically, a user can write their own
cadence to submit observations depending on the success of a prior observation or the data collected from a prior
observation.


## Writing a custom cadence strategy

Many of the TOM modules leverage a plugin architecture that enables you to write your own implementation, and the
cadence strategy plugin is no different. If you're familiar with the other modules, you've already seen examples of this
in the :doc:`Writing an alert broker <../customization/create_broker>`, :doc:`Writing an observation module
<observation_module>`, and :doc:`Customizing data processing <../customization/customizing_data_processing>` tutorials.


Create a cadence strategy file
------------------------------

First, you'll need a file where you'll put your custom cadence strategy. If you have a fresh TOM installation, you'll
have a directory structure that looks something like this:

    ├── data
    ├── db.sqlite3
    ├── manage.py
    ├── mytom
    │   ├── __init__.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── static
    ├── templates
    └── tmp

We'll create a new file called ``mycadence.py`` and place it next to ``settings.py``. To get started, we'll just put a
small skeleton into our new file, so to begin with, it should look like this:

```python
from tom_observations.cadence import CadenceStrategy

class MyCadenceStrategy(CadenceStrategy):
  pass
```

We also need to add the cadence strategy to ``settings.py`` so that our TOM knows that it exists:

```python
TOM_CADENCE_STRATEGIES = [
    'tom_observations.cadence.RetryFailedObservationsStrategy',
    'tom_observations.cadence.ResumeCadenceAfterFailureStrategy',
    'mytom.mycadence.MyCadenceStrategy'
]
```

Add logic to the new cadence strategy
-------------------------------------

You may have noticed that our ``MyCadenceStrategy`` class inherits from ``CadenceStrategy``. The ``CadenceStrategy``
interface only has one method, which is ``run()``. All of the logic for a ``CadenceStrategy`` lives in the ``run()``
method. Rather than demonstrating the implementation of a new cadence strategy, this tutorial is going to walk through
the business logic of a built-in cadence strategy. We're going to review the ``ResumeCadenceAfterFailureStrategy``.

It should also be worth mentioning at this point that the ``CadenceStrategy`` constructor takes an ``observation group``.
The ``observation_group`` is the set of observations that make up the cadence, and is created in the ``ObservationCreateView``
when the first observation of a cadence is submitted.

The ``ResumeCadenceAfterFailureStrategy`` is designed to ensure that, even after an observation fails, the cadence remains
consistent. If, for example, you submit an observation with a cadence of three days, and the observation fails, the cadence
should attempt to get the observation as soon as possible, and then resume observing once every three days.

Let's look at the strategy piece by piece.

```python
last_obs = self.observation_group.observation_records.order_by('-created').first()
facility = get_service_class(last_obs.facility)()
facility.update_observation_status(last_obs.observation_id)
last_obs.refresh_from_db()
```

The first thing this strategy does is get a couple of pieces of information. First, from the observation group that the
cadence consists of, the most recent observation is selected. The facility class for the facility that the cadence is
submitting observations to is also instantiated. With these values, the status of the most recent cadence observation is
updated, and the ``ObservationRecord`` object is refreshed.

```python
start_keyword, end_keyword = facility.get_start_end_keywords()
observation_payload = last_obs.parameters_as_dict
new_observations = []
```

These lines are, again, just more setup. Each facility has its own unique keywords representing the start and the end of
the observation window, so we get those from the facility class. Then, we get the original observation parameters that
were submitted to the facility, and we initialize a list for any new observations that will be submitted when the cadence
is updated.

```python
if not last_obs.terminal:
    return
elif last_obs.failed:
    # Submit next observation to be taken as soon as possible
    window_length = parse(observation_payload[end_keyword]) - parse(observation_payload[start_keyword])
    observation_payload[start_keyword] = datetime.now().isoformat()
    observation_payload[end_keyword] = (parse(observation_payload[start_keyword]) + window_length).isoformat()
else:
    # Advance window normally according to cadence parameters
    observation_payload = self.advance_window(
        observation_payload, start_keyword=start_keyword, end_keyword=end_keyword
    )
```

Here we have some logic for the three cases--either the most recent observation hasn't happened yet, it failed, or it succeeded.
If it hasn't happened, then there's nothing to do--we'll check again later. If if failed, we want to submit it again to be taken
immediately, so we get the original length of the observation window, and set our new observation payload to start immediately,
and end such that the new window length is the same. Finally, if our observation succeeded, we update our new observation
parameters to start 72 hours after the last observation, using a utility method that's part of the
``ResumeCadenceAfterFailureStrategy`` called ``advance_window``.

```python
obs_type = last_obs.parameters_as_dict.get('observation_type')
form = facility.get_form(obs_type)(observation_payload)
form.is_valid()
observation_ids = facility.submit_observation(form.observation_payload())

for observation_id in observation_ids:
    # Create Observation record
    record = ObservationRecord.objects.create(
        target=last_obs.target,
        facility=facility.name,
        parameters=json.dumps(observation_payload),
        observation_id=observation_id
    )
    self.observation_group.observation_records.add(record)
    self.observation_group.save()
    new_observations.append(record)

    for obsr in new_observations:
        facility = get_service_class(obsr.facility)()
        facility.update_observation_status(obsr.observation_id)

    return new_observations
```

The last part of our strategy is when we submit our new observations. Regardless of how we modified the observing window,
we initialize our observation form, validate it, and submit the observation to our facility. The rest of the code is
saving any resulting observations to the database, getting their new status from the facility, and returning them.

Just to review, here is the strategy's ``run()`` in its entirety:

```python
def run(self):
    last_obs = self.observation_group.observation_records.order_by('-created').first()
    facility = get_service_class(last_obs.facility)()
    facility.update_observation_status(last_obs.observation_id)
    last_obs.refresh_from_db()
    start_keyword, end_keyword = facility.get_start_end_keywords()
    observation_payload = last_obs.parameters_as_dict
    new_observations = []
    if not last_obs.terminal:
        return
    elif last_obs.failed:
        # Submit next observation to be taken as soon as possible
        window_length = parse(observation_payload[end_keyword]) - parse(observation_payload[start_keyword])
        observation_payload[start_keyword] = datetime.now().isoformat()
        observation_payload[end_keyword] = (parse(observation_payload[start_keyword]) + window_length).isoformat()
    else:
        # Advance window normally according to cadence parameters
        observation_payload = self.advance_window(
            observation_payload, start_keyword=start_keyword, end_keyword=end_keyword
        )

    obs_type = last_obs.parameters_as_dict.get('observation_type')
    form = facility.get_form(obs_type)(observation_payload)
    form.is_valid()
    observation_ids = facility.submit_observation(form.observation_payload())

    for observation_id in observation_ids:
        # Create Observation record
        record = ObservationRecord.objects.create(
            target=last_obs.target,
            facility=facility.name,
            parameters=json.dumps(observation_payload),
            observation_id=observation_id
        )
        self.observation_group.observation_records.add(record)
        self.observation_group.save()
        new_observations.append(record)

    for obsr in new_observations:
        facility = get_service_class(obsr.facility)()
        facility.update_observation_status(obsr.observation_id)

    return new_observations
```


## Configuring the cadence strategy to run automatically

As you may have noticed, the cadence strategies act on updates to the status of an ``ObservationRecord``. Ideally, we want
the cadence strategies to run as soon as an observation status changes--so, we need to automate that and have it run
periodically.

Fortunately, the TOM Toolkit comes with a built-in management command to update all cadences in the TOM. If you've perused
the TOM Toolkit documentation previously, you may have noticed a section about automation of tasks, and, more specifically,
a subsection about :doc:`Using cron with a management command <../customization/automation>`. You can simply apply the
instructions here, but use the management command ``runcadencestrategies.py`` in place of the example. If you set your cron
to run every few minutes or so, you'll ensure that your cadences are kept up to date!