# Globus Automate Timer CLI

This is a beta-version CLI for use with the (also in beta) Globus Timer API, to
use primarily for scheduling recurring Globus transfers through Globus Automate.

As the CLI and service are still in beta, please feel free to email support at
globus dot org with feedback or to resolve issues.

## What is this Service/CLI for?

The Globus Timer service can be used to schedule recurring transfer tasks. For
example, let's say we want to have a transfer automatically run every night to
back up data. We submit a job to the Timer Service starting tonight, and with an
interval of 1 day at which it will be re-run. In that request we provide the
Timer service the same input we would give to the [Globus Transfer Action
Provider](https://globus-automate-client.readthedocs.io/en/latest/globus_action_providers.html#globus-transfer-transfer-data);
that part of the request contains the information for what Globus endpoints we
transfer to and from as well as other options relevant to the transfer.

## Installation

This CLI requires Python version 3.5 or higher. See the [Globus CLI
docs](https://docs.globus.org/cli/installation/prereqs/) for help on how to set
up Python.

Once the appropriate version of Python is ready, install with `pip
install globus-timer-cli`. This will create a new command line
utility, `globus-timer` which can be used for all interactions with
the service. Online documentation is always available using the
`--help` option on any command. So, `globus-timer --help` will provide
information about the options on the command while `globus-timer job
--help` will provide help text specific to working with jobs. When in
doubt, add `--help` to a command for guidance on next steps.

## Getting Started

To schedule transfers on your behalf, this CLI requires authentication through
Globus Auth. Initially, the CLI will prompt you to authorize use via the Globus
Auth system, with a link to open in a web browser. The page will request that
you login to your Globus identity and consent to the service looking up your
identity and interacting with the Globus Transfer service on your behalf, and
then give you an authorization code to copy and paste back into the CLI. Some
Globus endpoints require additional authentication for use; when this is
necessary, the CLI will repeat the above process to get consent for using the
Globus Transfer service and the specific endpoint(s) for that job.

Authentication information is thereafter cached in the file
`~/.globus_timer_tokens.json` (so the authentication process is only needed on
the first use of the CLI, or of a given transfer endpoint when required); keep
this file secret.

After first use, you can determine what Globus Auth identity is being used for
interacting with the service by running the command:

```
globus-timer session whoami
```

This will show the identity which will be used for running the Transfers. Be
sure that this identity is authorized to work with the endpoints involved in the
Transfer.

To remove your stored identity information so that you may re-authenticate, for
example to invoke subsequent operations using a different Globus Auth identity,
use the command:

```
globus-timer session logout
```

Note that should you wish to both logout, and revoke the Timer service's
permission to run further operations, you may use the command `globus-timer
session revoke` however this should be an extreme measure as it is much
preferred to properly delete jobs as described below.

## Scheduling Periodic Globus Transfer operations

NOTE: To avoid confusion, please read the entirety of this section before using
the `job transfer` sub-command.

The starting point for working with the Timer service is scheduling transfers to
run periodically. The form of this command is below:

```
globus-timer job transfer \
    --name example-job \
    --label "Timer Transfer Job" \
    --interval 28800 \
    --start '2020-01-01T12:34:56' \
    --source-endpoint ddb59aef-6d04-11e5-ba46-22000b92c6ec \
    --dest-endpoint ddb59af0-6d04-11e5-ba46-22000b92c6ec \
    --item ~/file1.txt ~/new_file1.txt false \
    --item ~/file2.txt ~/new_file2.txt false
```

The parameters to the `job transfer` command are as follows (also documented in
the CLI help: `globus-timer job transfer --help`):

* `name`: A friendly name you can use to identify the job. However,
  note that all operations on the job (see below) are based on the job
  identifier (`job_id`) which is returned in the output of this
  command.

* `label`: A label string which will be used on the Globus Transfer
  tasks which will be created to perform the periodic transfer
  operations. You may use this to help you identify the source of the
  Globus Transfer task when viewing them in the Globus web
  application. If no `label` string is provided, the Globus Transfer
  label will include the `name` of this Timer job.

* `interval`: The time, in seconds, between invocations of the Globus
  Transfer operation. The Transfer will be invoked after this number
  of fixed seconds. Note that this easily allows for intervals such as
  hourly (3600 seconds), daily (86400 seconds) or weekly (604800
  seconds), but it doesn't allow for day of month or provide the
  ability to compensate for other time related changes like daylight
  savings time changes. This is presently a limitation of the Timer
  service. In some cases, the actual time the operation is started in
  Globus Transfer may be a few seconds longer than the specified
  interval, but any delay will not impact scheduling of the next
  operation (i.e. the job will stay on schedule and not "drift" due to
  delays in any particular execution).

* `start`: A timestamp defining when the job should first be
  scheduled. Thus, the first invocation of the Globus Transfer
  operation will occur at the specified time and then again after each
  `interval` number of seconds. The time may contain just a year,
  month and date, or may also contain the exact time of day as shown
  in the example. Unless a timezone is specified, the local timezone
  will be used.

* `source-endpoint` and `dest-endpoint`: Are the ids for Globus
  Transfer endpoints for the source and destination of the transfer
  operation respectively. These values can be retrieved from the
  Globus web application.

* `item`: Specifies the exact locations on the source and destination
  endpoints to perform the transfer operation. `item` should be
  followed by three values: the path on the source endpoint from which
  to start the transfer, the path on the destination endpoint where
  the data should be placed, and a true or false value indicating
  whether the transfer should be performed in a recursive manner. When
  the value is true, if the source path names a folder, the folder and
  all of its contained files and other folders will be transferred. If
  the value is false, the first (source) path should refer to a
  specific file.

Note that, as shown in the example, a single job creation command can
contain multiple specifications of the `item` parameter. This means
that a single job can transfer from multiple paths between a single
source endpoint and a single destination endpoint.

As an alternative to specifying the `item` parameter one or more
times, the item values may be stored in a file in which each line has
the same format as the `item` parameter. The file may be specified
with the `items-file` option which is used *instead of* the `item`
parameter. An example of the file which would perform the same
operations as the example above would look like:

```
# This is my items file
~/file1.txt, ~/new_file1.txt, false
~/file2.txt, ~/new_file2.txt, false
```

If this file was named `transfer_items.csv` it would be specified with the parameter:

```
--items-file transfer_items.csv
```

Note that this is a CSV file. The individual parts of the item are separated
with commas, and lines starting with a `#` are considered to be comments and are
not processed as items for the transfer job.

### Additional Transfer Options

Some optional flags for setting certain Transfer options include:

* `--sync-level`, adjusting how Transfer should behave if files already exist at
  the destination
* `--verify-checksum`, causing Transfer to retry if file checksums don't match
* `--encrypt-data`, causing Transfer to encrypt data using TLS
* `--preserve-timestamp`, causing Transfer to keep file timestamps the same at
  the destination

You may also specify a finite number of times or duration after which a job
should stop:

* `--stop-after-runs` takes a number, telling Timer to run the job only that
  many times.
* `--stop-after-date` takes a date argument, telling Timer to continue running
  the job only until the given date.

When either condition is reached, your job will still exist, and results will
still be visible, but Timer will not run any more transfers. If you specify
both, the job stops running as soon as one stop condition is met. If you update
an existing job to modify either of these, Timer will resume running the job
until it hits your new stop condition.

As always, all of the above options are also documented in `globus-timer job
transfer --help`.

## Monitoring and Controlling Submitted jobs

After submitting the transfer job, the CLI should return some results containing
a `job_id`, which identifies this job in the Timer service. To check the
status of your jobs, use:

```
globus-timer job status <job_id>
```

This command defaults to a summarized version of the job's information, which
does not include the full details for the corresponding task in Globus Transfer. To
check those, use `-v/--verbose`:

```
globus-timer job status --verbose JOB_ID
```

Commands return date-times in ISO format, in UTC time, so the timezone
most likely will not match the local time where you run the
command. No need to worry: the actual start time is still equal to
your submission's start time, etc.

The job description will also contain a value for `status` indicating
the condition of the job. The `status` may contain one of the
following values:

* `new`: The job has been received, but it has not yet been scheduled
  for its first execution (for example the start time has not yet been
  reached).

* `loaded`: The job has reached its execution mode and is executing
  repeatedly on the requested interval. This is the "steady-state" for
  a job when it is executing as expected.

* `updated`: If the job has recently been updated, it will be in this
  state until the job has transitioned to its new regular
  operation. This state should rarely be encountered as it is a
  transient state immediately after a job update.

* `deleted`: Indicates that the job has been deleted. This would not
  normally be seen, but the `job status` command described above as
  well as other commands described below provide the option of
  retrieving jobs marked as deleted using a command line flag
  `--show-deleted`.

A final important note: `Last Result` in the non-verbose output extends only as
far as the Globus Automate system: `SUCCESS` indicates you have successfully submitted
your job to the Timer service, which in turn successfully sent the task to the
Globus Transfer Action Provider. It's possible that the Globus Transfer service will
subsequently encounter some error running your transfer. Check the `--verbose`
output, which includes the actual response from Transfer, to be certain that
Transfer has run your job successfully.

To see all of your jobs which are outstanding, use the command:

```
globus-timer job list
```

This is particularly helpful if you do not have the `job_id` for a
particular job available. It will show all of your jobs in the same
format as the `job status` command described above. Similarly, the
`--verbose/-v` and `--show-deleted` options are supported.

Finally, a job may be removed from operation using the command:

```
globus-timer job delete <job_id>
```

The job will be removed from operation and no further transfers will
be performed. The same information provided via `job status` will be
displayed, showing the final state of the job. As described above, the
`job status` command may still retrieve the job information by
providing the `--show-deleted` flag. This can be helpful to review the
final state of the job, such as the last transfer or error condition,
even if the output for the `job delete` command is lost.
