HYPER-SHELL(1)

NAME
hyper-shell - Process shell commands over a distributed, asynchronous queue

SYNOPSIS
hyper-shell [-h] [-v] ...

hyper-shell cluster [-h] FILE [--ssh HOST... | --mpi | --launcher ARGS...] ...

hyper-shell submit [-h] FILE ...

hyper-shell server [-h] FILE ...

hyper-shell client [-h] ...

hyper-shell config [-h] {get | set | which | edit } [--user | --system] ...

hyper-shell task [-h] {submit | info | wait | run | search} ...

DESCRIPTION

hyper-shell is an elegant, cross-platform, high-performance computing utility for processing shell commands over a distributed, asynchronous queue. It is a highly scalable workflow automation tool for many-task scenarios.

Typically, ad hoc usage or batch jobs will use the cluster workflow. This automatically stands up the server and one or more client instances on remote servers and processes the commands from some input FILE until completion.

This can function as a pure in-memory queue; however, one can configure a database in-the-loop to manage task scheduling and persistence. Stand up the server on its own and persistent clients on the nodes in the cluster, and submit tasks independently.

CLUSTER USAGE

hyper-shell cluster [-h]
    [FILE | --restart | --forever] [--no-db | --initdb] [-N NUM] [-t CMD] [-b SIZE] [-w SEC] [--delay-start SEC] [-r NUM [--eager]] [--capture | [-o PATH] [-e PATH]] [-f PATH] [--ssh [HOST... | --ssh-group NAME] [--env] | --mpi | --launcher=ARGS...]

    Start the cluster either locally or with remote clients over ssh or a custom launcher. This mode should be the most common entry-point for general usage. It fully encompasses all of the different agents in the system in a concise workflow.

    The input source for tasks is file-like, either a local path, or from stdin if no argument is given. The command-line tasks are pulled in and either directly published to a distributed queue (see --no-db) or committed to a database first before being scheduled later.

    For large, long running workflows, it might be a good idea to configure a database and run an initial submit job to populate the database, and then run the cluster with --restart and no input FILE. If the cluster is interrupted for whatever reason it can gracefully restart where it left off.

  Arguments

FILE
    Path to input task file (default: <stdin>).

  Modes

--ssh HOST...
    Launch directly with SSH host(s). This can be a single host, a comma-separated list of hosts, or an expandable pattern, e.g., "cluster-a[00-04].xyz".

    See also --ssh-group and --ssh-args.

--mpi
    Same as --launcher=mpirun.

--launcher ARGS...
    Use specific launch interface. This can be any program that handles process management on a distributed system. For example, on a SLURM cluster one might want to use srun. In this case you would specify --launcher=srun; however, the ARGS are not merely the executable but the full listing, e.g., --launcher='srun --mpi=pmi2'.

  Options

-N, --num-tasks NUM
    Number of task executors per client (default: 1).

    For example, -N4 would create four workers, but -N4 --ssh 'cluster-a[00-01].xyz' creates two clients and a total of eight workers.

-t, --template CMD
    Command-line template pattern (default: "{}").

    This is expanded by the client just before execution. With the default "{}" the input command-line will be run verbatim. Specifying a template pattern allows for simple input arguments (e.g., file paths) to be transformed into some common form; such as -t './some_command.py {} >outputs/{/-}.out'.

    See section on templates.

-p, --port NUM
    Port number (default: 50001).

    This is an arbitrary choice and simply must be an available port. The default option chosen here is typically available on most platforms and is not expected by any known major software.

-b, --bundlesize SIZE
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales. This is used by the submit thread to accumulate bundles for either database commits and/or publishing to the queue. If a database is in use, the scheduler thread selects tasks from the database in batches of this size.

    Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also --num-tasks and --bundlewait.

-w, --bundlewait SEC
    Seconds to wait before flushing tasks (default: 5).

    This is used by both the submit thread and forwarded to each client. The client collector thread that accumulates finished task bundles to return to the server will push out a bundle after this period of time regardless of whether it has reached the preferred bundle size.

    See also --bundlesize.

-r, --max-retries NUM
    Auto-retry failed tasks (default: 0).

    If a database is in use, then there is an opportunity to automatically retry failed tasks. A task is considered to have failed if it has a non-zero exit status. Setting this value greater than zero defines the number of attempts for the task. The original is not over-written, a new task is submitted and later scheduled.

    See also --eager.

--eager
    Schedule failed tasks before new tasks. If --max-retries is greater than one, this option defines the appetite for re-submitting failed tasks. By default, failed tasks will only be scheduled when there are no more remaining novel tasks.

--no-db
    Disable database (submit directly to clients).

    By default, a scheduler thread selects tasks from a database that were previously submitted. With --no-db enabled, there is no scheduler and instead the submit thread publishes bundles directly to the queue.

--initdb
    Auto-initialize database.

    If a database is configured for use with the workflow (e.g., PostgreSQL), auto-initialize tables if they don't already exist. This is a short-hand for pre-creating tables with the hyper-shell initdb command. This happens by default with SQLite databases.

    Mutually exclusive to --no-db. See hyper-shell initdb command.

--forever
    Schedule forever.

    Typically, the cluster will process some finite set of submitted tasks. When there are no more tasks left to schedule, the cluster will begin its shutdown procedure. With --forever enabled, the scheduler will continue to wait for new tasks indefinitely.

    Conflicts with --no-db and mutually exclusive to --restart.

--restart
    Start scheduling from last completed task.

    Instead of pulling a new list of tasks from some input FILE, with --restart enabled the cluster will restart scheduling tasks where it left off. Any task in the database that was previously scheduled but not completed will be reverted.

    For very large workflows, an effective strategy is to first use the submit workflow to populate the database, and then to use --restart so that if the cluster is interrupted, it can easily continue where it left off, halting if nothing to be done.

    Conflicts with --no-db and mutually exclusive to --forever.

--ssh-args ARGS...
    Command-line arguments for SSH. For example, --ssh-args '-i ~/.ssh/my_key'.

--ssh-group NAME
    SSH nodelist group in config.

    In your configuration under [ssh.nodelist] can be one or more named lists. These lists should contain host names to associate with the group name.

    See configuration section.

-E, --env
    Send environment variables. Only for --ssh mode, all HYPERSHELL_ prefixed environment variables can be exported to the remote clients.

-d, --delay-start SEC
    Delay time in seconds for launching clients (default: 0).

    At larger scales it can be advantageous to uniformly delay the client launch sequence. Hundreds or thousands of clients connecting to the server all at once is a challenge. Even if the server could handle the load, your task throughput would be unbalanced, coming in waves.

    Use --delay-start with a negative number to impose a uniform random delay up to the magnitude specified (e.g., --delay-start=-600 would delay the client up to ten minutes). This also has the effect of staggering the workload. If your tasks take on the order of 30 minutes and you have 1000 nodes, choose --delay-start=-1800.

-c, --capture
    Capture individual task <stdout> and <stderr>.

    By default, the stdout and stderr streams of all tasks are fused with that of the client thread, and in turn the cluster. If tasks are producing output that needs to be isolated, the tasks need to manage their own output, you can specify a redirect as part of a --template, or use --capture to capture these as .out and .err files.

    These are stored local to the client. Task outputs can be automatically retrieved via SFTP, see task usage.

-o, --output PATH
    File path for task outputs (default: <stdout>).

    If local only (not --ssh, --mpi or --launcher), then the client can redirect all stdout from tasks to some file PATH together.

-e, --errors PATH
    File path for task errors (default: <stderr>).

    If local only (not --ssh, --mpi or --launcher), then the client can redirect all stderr from tasks to some file PATH together.

-f, --failures PATH
    File path to write failed task args (default: <none>).

    The server acts like a sieve, reading task args from stdin and redirecting those original args to stdout if the task had a non-zero exit status. The cluster will run the server for you and if --failures is enabled these task args will be sent to a local file PATH.

SERVER USAGE

hyper-shell server [-h]
    [FILE | --forever | --restart] [-b NUM] [-w SEC] [-r NUM [--eager]] [-H ADDR] [-p PORT] [-k KEY] [--no-db | --initdb] [--print | -f PATH]

    Launch server, schedule directly or asynchronously from database.

    The server includes a scheduler component that pulls tasks from the database and offers them up on a distributed queue to clients. It also has a receiver that collects the results of finished tasks. Optionally, the server can submit tasks (FILE). When submitting tasks, the -w/--bundlewait and -b/bundlesize options are the same as for the submit workflow.

    With --max-retries greater than zero, the scheduler will check for a non-zero exit status for tasks and re-submit them if their previous number of attempts is less.

    Tasks are bundled and clients pull them in these bundles. However, by default the bundle size is one, meaning that at small scales there is greater concurrency.

  Arguments

FILE
    Path to input task file (default: <stdin>).

  Options

-H, --bind ADDR
    Bind address (default: localhost).

    When running locally, the default is recommended. To allow remote clients to connect over the network, bind the server to 0.0.0.0.

-p, --port NUM
    Port number (default: 50001).

    This is an arbitrary choice and simply must be an available port. The default option chosen here is typically available on most platforms and is not expected by any known major software.

-k, --auth KEY
    Cryptographic authorization key to connect with server (default: <not secure>).

    The default KEY used by the server and client is not secure and only a place holder. It is expected that the user choose a secure KEY. The cluster automatically generates a secure one-time KEY.

-b, --bundlesize SIZE
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales. This is used by the submit thread to accumulate bundles for either database commits and/or publishing to the queue. If a database is in use, the scheduler thread selects tasks from the database in batches of this size.

    Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also --num-tasks and --bundlewait.

-w, --bundlewait SEC
    Seconds to wait before flushing tasks (default: 5).

    This is used by both the submit thread and forwarded to each client. The client collector thread that accumulates finished task bundles to return to the server will push out a bundle after this period of time regardless of whether it has reached the preferred bundle size.

    See also --bundlesize.

-r, --max-retries NUM
    Auto-retry failed tasks (default: 0).

    If a database is in use, then there is an opportunity to automatically retry failed tasks. A task is considered to have failed if it has a non-zero exit status. Setting this value greater than zero defines the number of attempts for the task. The original is not over-written, a new task is submitted and later scheduled.

    See also --eager.

--eager
    Schedule failed tasks before new tasks. If --max-retries is greater than one, this option defines the appetite for re-submitting failed tasks. By default, failed tasks will only be scheduled when there are no more remaining novel tasks.

--no-db
    Disable database (submit directly to clients).

    By default, a scheduler thread selects tasks from a database that were previously submitted. With --no-db enabled, there is no scheduler and instead the submit thread publishes bundles directly to the queue.

--initdb
    Auto-initialize database.

    If a database is configured for use with the workflow (e.g., PostgreSQL), auto-initialize tables if they don't already exist. This is a short-hand for pre-creating tables with the hyper-shell initdb command. This happens by default with SQLite databases.

    Mutually exclusive to --no-db. See hyper-shell initdb command.

--forever
    Schedule forever.

    Typically, the cluster will process some finite set of submitted tasks. When there are no more tasks left to schedule, the cluster will begin its shutdown procedure. With --forever enabled, the scheduler will continue to wait for new tasks indefinitely.

    Conflicts with --no-db and mutually exclusive to --restart.

--restart
    Start scheduling from last completed task.

    Instead of pulling a new list of tasks from some input FILE, with --restart enabled the cluster will restart scheduling tasks where it left off. Any task in the database that was previously scheduled but not completed will be reverted.

    For very large workflows, an effective strategy is to first use the submit workflow to populate the database, and then to use --restart so that if the cluster is interrupted, it can easily continue where it left off, halting if nothing to be done.

    Conflicts with --no-db and mutually exclusive to --forever.

--print
    Print failed task args to <stdout>.

    Mutually exclusive to -f/--failures.

-f, --failures PATH
    File path to write failed task args (default: <none>).

    The server acts like a sieve, reading task args from some input source. Tasks with a non-zero exit status can have their original command-line args printed to an output stream. With -f/--failures, specify a local file PATH.

    Mutually exclusive to --print.

CLIENT USAGE

hyper-shell client [-h]
    [-N NUM] [-t TEMPLATE] [-b SIZE] [-w SEC] [-d SEC] [-H ADDR] [-p PORT] [-k KEY] [-c | [-o PATH] [-e PATH]]

    Launch client directly, run tasks in parallel.

    The client connects to the server and pulls bundles of tasks off the shared queue. These tasks are run locally by some number of a parallel task executors.

    The environment for tasks are the same as for the client. Standard output and error for tasks are forwarded to that of the client, unless --capture is used, in which these are directed to individual files for each task.

  Options

-N, --num-tasks NUM
    Number of task executors (default: 1).

-t, --template CMD
    Command-line template pattern (default: "{}").

    This is expanded by the client just before execution. With the default "{}" the input command-line will be run verbatim. Specifying a template pattern allows for simple input arguments (e.g., file paths) to be transformed into some common form; such as -t './some_command.py {} >outputs/{/-}.out'.

    See section on templates.

-b, --bundlesize SIZE
    Size of task bundle (default: 1).

    Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also --num-tasks and --bundlewait.

-w, --bundlewait SEC
    Seconds to wait before flushing tasks (default: 5).

    The client collector thread that accumulates finished task bundles to return to the server will push out a bundle after this period of time regardless of whether it has reached the preferred bundle size.

    See also --bundlesize.

-H, --host ADDR
    Hostname or IP address to connect with server (default: localhost).

-p, --port NUM
    Port number to connect with server (default: 50001).

-k, --auth KEY
    Cryptographic authorization key to connect with server (default: <not secure>).

    The default KEY used by the server and client is not secure and only a place holder. It is expected that the user choose a secure KEY. The cluster automatically generates a secure one-time KEY.

-d, --delay-start SEC
    Delay time in seconds before connecting to server (default: 0).

    At larger scales it can be advantageous to uniformly delay the client launch sequence. Hundreds or thousands of clients connecting to the server all at once is a challenge. Even if the server could handle the load, your task throughput would be unbalanced, coming in waves.

    Use --delay-start with a negative number to impose a uniform random delay up to the magnitude specified (e.g., --delay-start=-600 would delay the client up to ten minutes). This also has the effect of staggering the workload. If your tasks take on the order of 30 minutes and you have 1000 nodes, choose --delay-start=-1800.

-o, --output PATH
    File path for task outputs (default: <stdout>).

-e, --errors PATH
    File path for task errors (default: <stderr>).

-c, --capture
    Capture individual task <stdout> and <stderr>.

    By default, the stdout and stderr streams of all tasks are fused with that of the client thread, and in turn the cluster. If tasks are producing output that needs to be isolated, the tasks need to manage their own output, you can specify a redirect as part of a --template, or use --capture to capture these as .out and .err files.

    These are stored local to the client under <prefix>/lib/task/<uuid>.[out,err]. Task outputs can be automatically retrieved via SFTP, see task usage.

    Mutually exclusive with both --output and --errors.

SUBMIT USAGE

hyper-shell submit [-h]
    [FILE] [-b NUM] [-w SEC] [-t CMD] [--initdb]

    Submit tasks from a file.

    Tasks are accumulated and published in bundles to the database. The -b/--bundlesize and -w/--bundlewait options control the size of these bundles and how long to wait before flushing tasks regardless of how many have accumulated.

    Pre-format tasks at submit-time with template expansion using -t/--template.

  Arguments

FILE
    Path to input task file (default: <stdin>).

  Options

-t, --template CMD
    Command-line template pattern (default: "{}").

    This is expanded at submit-time before sending to the database. With the default "{}" the input command-line will be run verbatim. Specifying a template pattern allows for simple input arguments (e.g., file paths) to be transformed into some common form; such as -t './some_command.py {} >outputs/{/-}.out'.

    See section on templates.

-b, --bundlesize SIZE
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales. Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also --bundlewait.

-w, --bundlewait SEC
    Seconds to wait before flushing tasks (default: 5).

    If this period of time expires since the previous bundle was pushed to the database, The current bundle will be pushed regardless of how many tasks have been accumulated.

    See also --bundlesize.

--initdb
    Auto-initialize database.

    If a database is configured for use with the workflow (e.g., PostgreSQL), auto-initialize tables if they don't already exist. This is a short-hand for pre-creating tables with the hyper-shell initdb command. This happens by default with SQLite databases.

    See hyper-shell initdb command.

INITDB USAGE

hyper-shell initdb [-h] [--truncate [--yes]]

Initialize database.

Create the necessary database tables given your current configuration. This occurs automatically for SQLite. To clear out all metadata for an existing database use --truncate.

  Options

-t, --truncate
    Truncate database (task metadata will be lost).

-y, --yes
    Auto-confirm truncation (default will prompt).

CONFIG GET USAGE

hyper-shell config get [-h]
    [-x] SECTION[...].VAR [--system | --user]

    Get configuration option.

    If --user/--system not specified, the output is the merged configuration from all sources. Use the config which command to see where a specific option originates from.

    Output is pretty-printed using the configured console theme (default: monokai). Colorization is disabled for non-TTY invocations.

  Arguments

SECTION[...].VAR
    Path to variable.

  Options

--system
    Load from system configuration.

--user
    Load from user configuration.

-x, --expand
    Expand variable.

    If the special _env or _eval variant of the option is present in the configuration, it will expand the environment variable or shell command, respectively.

CONFIG SET USAGE

hyper-shell config set [-h]
    SECTION[...].VAR VALUE [--system | --user]

    Set configuration option.

  Arguments

SECTION[...].VAR
    Path to variable.

VALUE
    Value to be set.

  Options

--system
    Apply to system configuration.

--user
    Apply to user configuration (default).

CONFIG EDIT USAGE

hyper-shell config edit [-h]
    [--system | --user]

    Edit configuration with default editor.

    The EDITOR/VISUAL environment variable must be set.

  Options

--system
    Edit system configuration.

--user
    Edit user configuration (default).

CONFIG WHICH USAGE

hyper-shell config which [-h]
    SECTION[...].VAR

    Show origin of configuration option.

  Arguments

SECTION[...].VAR
    Path to variable.

TASK SUBMIT USAGE

hyper-shell task submit [-h] ARGS...

Submit individual task to the database.

A database must be configured. The task will not run until scheduled by the server process. The task UUID will be printed to standard out.

See task info command.

  Arguments

ARGS...
    Command-line arguments.

    The full command-line for some shell task. To use options, preface with leading --.

TASK INFO USAGE

hyper-shell task info [-h]
    ID [--json | --stdout | --stderr | --x FIELD]

    Get metadata and/or task outputs.

    Query for the full metadata on task by ID. Extract a specific field using -x/--extract.

    Alternatively, dump the captured standard output or errors from the finished task (if captured). These files are written on the client host and synced with SFTP if necessary.

  Arguments

ID  Unique task UUID.

  Options

--json
    Format metadata output as JSON.

-x/--extract FIELD
    Print this field only (e.g., -x submit_time).

--stdout
    Print <stdout> of task if captured, fetch from client if necessary.

--stderr
    Print <stderr> of task if captured, fetch from client if necessary.

TASK WAIT USAGE

hyper-shell task wait [-h]
    ID [-n SEC] [--info [--json] | --status]

    Wait for task to complete.

    Poll the database periodically for the completion status of the task. Block until completed.

    Optionally, print task metadata after completion.

  Arguments

ID  Unique task UUID.

  Options

-n, --interval SEC
    Time in seconds to wait between polling (default: 5).

--info
    Print task info after completion.

    See task info command.

--json
    Format metadata output as JSON.

--status
    Print task exit status only.

TASK RUN USAGE

hyper-shell task run [-h] [-n SEC] ARGS...

Submit individual task and wait for completion.

A database must be configured. The task will not run until scheduled by the server process. The <stdout> and <stderr> are written locally as if run locally, requires the client to have --capture enabled.

See task info and task wait commands.

  Arguments

ARGS...
    Command-line arguments.

    The full command-line for some shell task. To use options, preface with leading --.

  Options

-n, --interval SEC
    Time in seconds to wait between polling (default: 5).

TASK SEARCH USAGE

hyper-shell task search [-h]
    [FIELD [FIELD ...]] [--where COND [COND ...]] [--order-by FIELD [--desc]] [-x | --json | --csv] [--count | --limit NUM]

    Search for tasks in database.

    A database must be configured. Fields will be columns in the output. Options relate to SQL concepts. For single column output, use -x/--extract to disable formatting and quotations (useful for piping output).

  Arguments

FIELD
    Select specific named fields to include in output. Default is to include all fields.

  Options

-w, --where COND...
    List of conditional statements to filter results (e.g., -w 'exit_status != 0').

-s, --order-by FIELD
    Order results by field.

-x, --extract
    Disable formatting for single column output.

--json
    Format output as JSON.

--csv
    Format output as CSV.

-l, --limit NUM
    Limit number of returned records.

-c, --count
    Only print number of results that would be returned.

TEMPLATES

Incoming command-line arguments are expanded using a template pattern. Workloads often have a common form and only a small part of the shell command need be different.

Braces, {}, are used in all cases. Empty braces will substitute the full incoming argument line. Use one of the below patterns as a shorthand notation for many common scenarios.

The -t/--template argument is used by the client command to expand templates just prior to execution. The cluster command simply forwards this argument to all clients.

In some situations it may be useful to expand a template with the submit command. These are expanded prior to scheduling as the actual args for the task.

  Filepath Operations

Shell commands often operate on filepaths. In such cases, it may be useful to manipulate these paths. Instead of using a shell interpolation (see below), use one of the available shorthand notations listed here.

{.} Expand to immediate parent directory of given file. E.g., /some/path/to/file.h5 translates to /some/path/to.

{..}
    Expand to second parent directory of given file. E.g., /some/path/to/file.h5 translates to /some/path.

{/} The basename of the given file. E.g., /some/path/to/file.h5 translates to file.h5.

{/-}
    The basename of the given file without its file type extension. E.g., /some/path/to/file.h5 translates to file.

{-} The full path of the given file without the extension. This is useful for targeting adjacent files with a different extension. E.g., /some/path/to/file.h5 translates to /some/path/to/file.

{+} The file type extension for the given file. E.g., /some/path/to/file.h5 translates to .h5.

{++}
    The file type extension for the given file without the leading dot. E.g., /some/path/to/file.h5 translates to h5.

  Argument Slicing

Command-line inputs are understood as individual arguments delimited by whitespace. Slice into the argument vector using the {[]} notation. Arguments follow zero-based indexing. Negative index values are counting backwards from the end.

Select with a singular value.

{[0]}
    The first argument.

{[1]}
    The second argument.

{[-1]}
    The last argument.

    Select a range with a start and stop value, non-inclusive of the stop value. Including a leading or trailing colon implies the default value (inclusive).

{[1:3]}
    The second and third argument.

{[:4]}
    The first four arguments.

{[-2:]}
    The last two arguments.

    Include a third value as a step (or sometimes referred to as stride). Leaving out the start and step value implies starting from the first element (inclusive).

{[::2]}
    Every second argument starting from the first.

{[1::2]}
    Every odd argument.

  Shell Expansion

General purpose shell commands can be expanded with the {% %} notation. The incoming command-line args can be substituted with an @.

{% basename @ %}
    Equivalent to {/}.

{% mktemp -d %}
    Create temporary directory and insert its path.

  Lambda Expressions

Arbitrary Python expressions can be expanded with the {= =} notation. The input argument can be used within the expression with the variable x.

Exposed standard library modules include os, os.path as path, math and datetime as dt.

Incoming arguments are intelligently coerced into the expected type. E.g., 2 will be an integer, 4.67 a float, null and none will be a Python None, and true/false will be the appropriate boolean value.

{= x * math.pi =}
    Multiply the incoming argument (expected to be a float) by Pi.

{= dt.datetime.fromtimestamp(x) =}
    Convert incoming POSIX timestamp to ISO format.

CONFIGURATION

Most of the choices that hyper-shell makes about timing, task bundling, coordination, logging, and such are configurable by the user. This configuration is loaded when hyper-shell starts and is constructed from several sources including an ordered merger of files, environment variables, and command-line options.

In order of precedence (lowest to highest), three files are loaded:
Level | Path
System | /etc/hypershell.toml
User | ~/.hypershell/config.toml
Local | ./.hypershell/config.toml

The TOML format is modern and minimal.

Every configurable option can be set in one of these files. Further, every option can also be set by an environment variable, where the name aligns to the path to that option, delimited by underscores.

For example, set the logging level at the user level with a command:

$ hyper-shell config set logging.level info --user

The file should now look something like this:

# File automatically created on 2022-07-02 11:57:29.332993
# Settings here are merged automatically with defaults and environment variables
[logging]
level = "info"

Alternatively, you can set an environment variable and the runtime configuration would be equivalent:

$ export HYPERSHELL_LOGGING_LEVEL=INFO

Finally, any option defined within a configuration file that ends with _env or _eval is automatically expanded by the given environment variable or shell expression, respectively. This is useful as both a dynamic feature but also as a means to obfuscate sensitive information, such as database connection details.

# File automatically created on 2022-07-02 11:57:29.332993
# Settings here are merged automatically with defaults and environment variables
[logging]
level = "info"
[database]
provider = "postgres"
database = "hypershell"
host = "my.instance.university.edu"
user = "me"
password_eval = "pass hypershell/database/password"  # Decrypt using GNU Pass

  Parameter Reference

[logging]
    Logging configuration. See also logging section.

.level
    One of DEVEL, TRACE, DEBUG, INFO, WARNING, ERROR, or CRITICAL (default: WARNING)

.datefmt
    Date/time format, standard codes apply (default: '%Y-%m-%d %H:%M:%S')

.format
    Log message format. Default set by the default logging.style. See the available attributes defined by the underlying Python logging interface.

.style
    Presets for logging.format which can be difficult to define correctly. Options are default, detailed, and system.

[database]
    Database configuration and connection details. See also database section.

.provider
    Database provider (default: 'sqlite'). Supported alternatives include 'postgres' (or compatible). Support for other providers may be considered in the future.

.file
    Only applicable for SQLite provider. SQLite does not understand any other connection detail.

.database
    Name for database. Not applicable for SQLite.

.schema
    Not applicable for all RDMS providers. For PostgreSQL the default schema is public. Specifying the schema may be useful for having multiple instances within the same database.

.host
    Hostname or address of database server (default: localhost).

.port
    Port number to connect with database server. The default value depends on the provider, e.g., 5432 for PostgreSQL.

.user
    Username for databaser server account. If provided a password must also be provided. Default is the local account.

.password
    Password for database server account. If provided a user must also be provided. Default is the local account.

    See also note on _env and _eval.

.echo
    Special parameter enables verbose logging of all database transactions.

[connection_args]
    Specify additional connection details for the underlying SQL dialect provider, e.g., sqlite3 or psycopg2.

*   Any additional arguments are forwarded to the provider, e.g., encoding = 'utf-8'.

[server]
    Section for server workflow parameters.

.bind
    Bind address (default: localhost).

    When running locally, the default is recommended. To allow remote clients to connect over the network, bind the server to 0.0.0.0.

.port
    Port number (default: 50001).

    This is an arbitrary choice and simply must be an available port. The default option chosen here is typically available on most platforms and is not expected by any known major software.

.auth
    Cryptographic authorization key to connect with server (default: <not secure>).

    The default KEY used by the server and client is not secure and only a place holder. It is expected that the user choose a secure KEY. The cluster automatically generates a secure one-time KEY.

.queuesize
    Maximum number of task bundles on the shared queue (default: 1).

    This blocks the next bundle from being published by the scheduler until a client has taken the current prepared bundle. On smaller scales this is probably best and is only of modest performance impact, limiting the scheduler from getting so far ahead of the currently running tasks.

    On large scale workflows with many clients (e.g., 100) it may be advantageous to allow the scheduler to work ahead in selecting new tasks.

.bundlesize
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales. This is used by the submit thread to accumulate bundles for either database commits and/or publishing to the queue. If a database is in use, the scheduler thread selects tasks from the database in batches of this size.

    Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also -b/--bundlesize command-line option.

.attempts
    Attempts for auto-retry on failed tasks (default: 1).

    If a database is in use, then there is an opportunity to automatically retry failed tasks. A task is considered to have failed if it has a non-zero exit status. The original is not over-written, a new task is submitted and later scheduled.

    Counterpart to the -r/--max-retries command-line option. Setting --max-retries 1 is equivalent to setting .attempts to 2.

    See also .eager.

.eager
    Schedule failed tasks before new tasks (default: false).

    If .attempts is greater than one, this option defines the appetite for re-submitting failed tasks. By default, failed tasks will only be scheduled when there are no more remaining novel tasks.

.wait
    Polling interval in seconds for database queries during scheduling (default: 5). This waiting only occurs when no tasks are returned by the query.

.evict
    Eviction period in seconds for clients (default: 600).

    If a client fails to register a heartbeat after this period of time it is considered defunct and is evicted. When there are no more tasks to schedule the server sends a disconnect request to all registered clients, and waits until a confirmation is returned for each. If a client is defunct, this will hang the shutdown process.

[client]
    Section for client workflow parameters.

.bundlesize
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales.

    Using larger bundles is a good idea for larger distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client. It is also a good idea to coordinate bundle size between the client and server so that the client returns the same sized bundles that it receives.

    See also -b/--bundlesize command-line option.

.bundlewait
    Seconds to wait before flushing task bundle (default: 5).

    If this period of time expires since the previous bundle was returned to the server, the current group of finished tasks will be pushed regardless of bundlesize.

    For larger distributed workflows it is a good idea to make this waiting period sufficiently long so that most bundles are returned whole.

    See also -w/--bundlewait command-line option.

.heartrate
    Interval in seconds between heartbeats sent to server (default 10).

    Even on the largest scales the default interval should be fine.

[submit]
    Section for submit workflow parameters.

.bundlesize
    Size of task bundle (default: 1).

    The default value allows for greater concurrency and responsiveness on small scales. Using larger bundles is a good idea for large distributed workflows; specifically, it is best to coordinate bundle size with the number of executors in use by each client.

    See also -b/--bundlesize command-line option.

.bundlewait
    Seconds to wait before flushing tasks (default: 5).

    If this period of time expires since the previous bundle was pushed to the database, the current bundle will be pushed regardless of how many tasks have been accumulated.

    See also -w/--bundlewait command-line option.

[task]
    Section for task runtime settings.

.cwd
    Explicitly set the working directory for all tasks.

[ssh]
    SSH configuration section.

.args
    SSH connection arguments; e.g., -i ~/.ssh/some.key. It is preferable to configure SSH directly however, in ~/.ssh/config.

[group]
    Setting a list for the .group allows for a global list of available client hosts. Or, set one or more named groups and reference them by name with --ssh-group.

    .<name> = ['host-01', 'host-02', 'host-03']

ENVIRONMENT VARIABLES

A few common environment variables are defined for every task.

TASK_ID
    Universal identifier (UUID) for the current task.

TASK_ARGS
    Original input command-line argument line. Equivalent to {}, see templates section.

TASK_CWD
    Current working directory for the current task.

TASK_OUTPATH
    Absolute file path where standard output is directed (if defined).

TASK_ERRPATH
    Absolute file path where standard error is directed (if defined).

    Further, any environment variable starting with HYPERSHELL_EXPORT_ will be injected into the task environment sans prefix; e.g., HYPERSHELL_EXPORT_FOO would define FOO in the task environment. You can also define such variables in the export section of your configuration file(s); e.g.,

    # File automatically created on 2022-07-02 11:57:29.332993
    # Settings here are merged automatically with defaults and environment variables
    [logging]
    level = "info"
    # Options defined as a list will be joined with a ":" on BSD/Linux or ";" on Windows
    # Environment variables will be in all-caps (e.g., FOO and PATH).
    [export]
    foo = "value"
    path = ["/some/bin", "/some/other/bin"]

EXIT STATUS

0   Success

1   Usage statement printed

2   Bad argument

3   Bad configuration

4   Signal interrupt (keyboard interrupt)

5   Generic runtime error (non-specific)

6   Internal error (unexpected)

SEE ALSO

ssh(1), mpirun(1)

AUTHOR
Geoffrey Lentner <glentner@purdue.edu>.

COPYRIGHT
2019-2022 Geoffrey Lentner