Upload (EE) — MARV Robotics 21.10.0 documentation

# Upload (EE)

MARV processes and manages continuous recordings performed by large fleets of robots. To ease the handling of a large influx of data MARV implements an upload functionality. It takes care that new data is quickly and securely handed into MARV management. Transmissions are automatically authenticated and encrypted.

In the MARV ecosystem data providers that connect to the MARV server are called leaves. A leaf can be any potential source of data, e.g. a robot, vehicle, CI server, local simulator, or any other technical system.

## Preparations

For secure operation some preparatory steps are necessary.

- Create leaf in MARV via web leaf admin panel or CLI

- Generate authentication token for leaf via web leaf admin panel or CLI

- Download `marv-leaf` binary from web leaf admin panel, copy to robot, and rename

- Register robot as MARV leaf using the authentication token

Example to add a leaf and generate its token via CLI:

    (server)$ marv leaf add C3P0
    (server)$ marv leaf regen-token C3P0
    New token for C3P0: <secret_token>

And use the token to register the leaf:

    (leaf)$ marv-leaf register https://example.com
    Token: <secret_token>
    Registered successfully as leaf 'C3P0'.

The leaf binaries are statically linked and have no other dependencies. They contain system and architecture in their names, please feel free to rename them to just `marv-leaf` when copying them to the leaf system.

Once installed the leaf binary is able to update itself from a running MARV server. This is not necessary with every MARV release, instead `marv-leaf` will inform you if an update is necessary.

## Upload datasets

Once server and leaf have been prepared uploading datasets to MARV is straightforward.

    (leaf)$ marv-leaf upload --collection bags recording.bag

This command will upload the file `recording.bag` to the MARV instance `example.com` and add the dataset to the `bags` collection.

Upload multiple files as one dataset by placing them into one directory and pass this to `marv-leaf upload`. For rosbag2 place additional files into the directory first.

To upload multiple datasets call the command multiple times. See `marv-leaf upload --help` for more information.

### Metadata / userdata

Optionally, pass a json file `marv-leaf upload --meta meta.json ...` containing one top-level `userdata` key with dictionary value. This userdata is available as `dataset.userdata` in nodes and config functions.

Multiple uploads may use the same metadata file, the file name is not important, but the file must not reside in the directory to be uploaded.

### Return code

On success the return value will be zero and the upload process will have made sure that the checksums of all uploaded files are correct.

Rerunning `marv-leaf upload` on the same set of files will not create another dataset. Instead the MARV will recognize that the data already exists on the server and will indicate success by returning zero.

If an upload is interrupted manually e.g. by loss of network connectivity, the return value will be non-zero to indicate failure. Rerunning `marv-leaf upload` with the same parameters will resume the upload from where it was interrupted.

## Data safety

Leaves upload datasets to leavesdir. Make sure to include this directory in backups and consider to delete datasets on leaves only after sufficient time has passed for them to be in backup as well.

## Access control

Uploaded datasets are only visible to admins by default. For a normal user to see a dataset, she needs to receive permissions first through group membership. See Web-based user, group, and leaf management (EE) for more information.