Automating tasks for your TOM
---

Your TOM may have a need to run a task on a regular schedule without human intervention. With the help of a built-in Django feature and cron, this can be accomplished. Perhaps you want to check for and download data from your scheduled observations every hour, or see if any brokers have published new candidates that meet the criteria of a previous search--all that would be required is a bit of code to call those built-in functions, and a crontab update.

### Create a management command

Django provides the ability to register actions using [management commands](https://docs.djangoproject.com/en/2.2/howto/custom-management-commands/). These actions can then be called from the command line.

#### Starting a new django "app"

Django recommends creating separate "apps" to contain your management commands
(among other things, like custom models and views) so we'll start with creating a
new app called "myapp". You can read more about Django reusable apps
[in the official
documentation](https://docs.djangoproject.com/en/2.2/intro/tutorial01/#creating-the-polls-app).

    ./manage.py startapp myapp

Now your tom should have a new folder in the root directory called "myapp". Next
we need to tell Django to use this new application. In your `settings.py` file
file the `INSTALLED_APPS` settings and add `myapp.apps.MyappConfig` to the array:

```python
INSTALLED_APPS = [
    'django.contrib.admin',
    ...
    'myapp.apps.MyappConfig'
]
```

Now we are read to start writing our new commands.

#### Writing the command

Let's walk through a command to download observation data every hour. The first
thing to be done is to create a `management/commands` directory within your
application to house our script. We'll call it `save_data.py`. The structure should
look like this:

```
mytom/
├── manage.py
└── myapp/
    ├── __init__.py
    ├── models.py
    ├── tests.py
    ├── views.py
    └── management/
        └── commands/
            └── save_data.py
```

A management command simply needs a class called `Command` that inherits from `BaseCommand`, and a `handle` class method that contains the logic for the command.

```python
from django.core.management.base import BaseCommand
from tom_observations.models import ObservationRecord


class Command(BaseCommand):

    help = 'Downloads data for all completed observations'

    def handle(self, *args, **options):
```

Now, we need to add the logic to query the facilities for data. We'll iterate
over each incomplete `ObservationRecord`, and save the data products locally for
that ObservationRecord.

```python
observation_records = ObservationRecord.objects.all()
for record in observation_records:
    if record.terminal:
        record.save_data()

return 'Success!'
```

So our final management command should look like this:

```python
from django.core.management.base import BaseCommand
from tom_observations.models import ObservationRecord


class Command(BaseCommand):

    help = 'Downloads data for all completed observations'

    def handle(self, *args, **options):
        observation_records = ObservationRecord.objects.all()
        for record in observation_records:
            if record.terminal:
                record.save_data()

        return 'Success!'
```

#### Adding parameters

Management commands also provide the ability to accept parameters. Doing this is as simple as implementing `add_arguments` as a class method on your `Command` class. Let's say we want to ensure that our command can be run for a single target:

```python
  def add_arguments(self, parser):
    parser.add_argument('--target_id', help='Download data for a single target')
```

That code will process any additional parameters, and we simply need to handle
them in our,  `handle` class method. We'll attempt to fetch the supplied target
from the database and filter the ObservationRecords accordingly:

```python
  def handle(self, *args, **options):
    if options['target_id']:
      try:
        target = Target.objects.get(pk=options['target_id'])
        observation_records = ObservationRecord.objects.filter(target=target)
      except ObjectDoesNotExist:
        raise Exception('Invalid target id provided')
    else:
        observation_records = ObservationRecord.objects.all()
    ...
```

Finally, we filter our initial set of observation records, so this line:

```python
    observation_records = ObservationRecord.objects.all()
```

will become this:

```python
    observation_records = ObservationRecord.objects.filter(target=target)
```

And our final finished command looks as follows:

```python
from django.core.management.base import BaseCommand
from tom_observations.models import ObservationRecord
from tom_targets.models import Target


class Command(BaseCommand):

    help = 'Downloads data for all completed observations'

    def add_arguments(self, parser):
        parser.add_argument('--target_id', help='Download data for a single target')

    def handle(self, *args, **options):
        if options['target_id']:
            try:
                target = Target.objects.get(pk=options['target_id'])
                observation_records = ObservationRecord.objects.filter(target=target)
            except Target.DoesNotExist:
                raise Exception('Invalid target id provided')
        else:
            observation_records = ObservationRecord.objects.all()
        for record in observation_records:
            if record.terminal:
                record.save_data()

        return 'Success!'
```

### Automating a management command

#### Using cron

On Unix-based systems, [cron](https://linux.die.net/man/8/cron) can be used to automate running of a Django management command. The syntax is very simple, as commands look like this:

`30 2 * 6 3 /path/to/command /path/to/parameters`

In the above case, the first five values, which can either be numbers or asterisks, represent elements of time. From left to right, they are minutes, hours, day of the month, month of the year, and day of the week. Our example would run a command every Wednesday (fourth day of the week, starting from 0) in June (sixth month of the year, starting from 1) at 2:30 AM.

Websites like [crontab.guru](https://crontab.guru/) make it easier to reason about
crontab expressions.

Scheduling can be made more complex as well--values can be comma-separated or presented as a range. Refer to the abundance of cron documentation for more information. An excellent beginner's guide can be found [here](https://www.ostechnix.com/a-beginners-guide-to-cron-jobs/).

Now, how is cron called? Well, cron jobs are run by the system, and it reads the commands that need to be called from a cron table, or crontab. To edit this file, simple call `crontab -e`.

#### Using cron with a management command

To make this more specific to our example, let's say we want to update the observation data every hour. The command we would normally run in our project directory would be the following:

`python manage.py save_data`

However, cron is a system-level operation, so the command needs to be directory-agnostic, and we need to ensure we're using the right Python version. If you have a virtualenv, the command should be the absolute path to the Python interpreter in the virtualenv. If your TOM is in a Docker container, it should be the version of Python running in the container. Otherwise, just ensure that it's at least version 3.6 or higher.

So, the line in our crontab should be as follows:

`0 * * * * /path/to/virtualenv/bin/python /path/to/project/manage.py save_data`

This will run every day on the hour. And that's it! Just exit the crontab and it will automatically restart cron, then your command will run on the next hour.
