Metadata-Version: 2.1
Name: gravity
Version: 0.13.3
Summary: Command-line utilities to assist in managing Galaxy servers
Home-page: https://github.com/galaxyproject/gravity
Author: The Galaxy Team
Author-email: team@galaxyproject.org
License: MIT
Keywords: gravity galaxy
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Requires-Python: >=3.6
Description-Content-Type: text/x-rst
License-File: LICENSE

========================================
 gravity - Galaxy Server Administration
========================================

A process manager (`supervisor`_) and management tools for `Galaxy`_ servers.

Installing this will give you two executables, ``galaxyctl`` which is used to manage the starting, stopping, and logging
of Galaxy's various processes, and ``galaxy``, which can be used to run a Galaxy server in the foreground.

Installation
============

Python 3.7 or later is required. Gravity can be installed independently of Galaxy, but please read the section on Galaxy
Integration below.

To install::

    $ pip install gravity

To make your life easier, you are encourged to install into a `virtualenv`_. The easiest way to do this is with Python's
built-in `venv`_ module::

    $ python3 -m venv ~/gravity
    $ . ~/gravity/bin/activate

By default, Gravity will store its state, configuration, and log files in ``$XDG_CONFIG_HOME/galaxy-gravity``, where
``$XDG_CONFIG_HOME`` typically defaults to ``~/.config``. You can change this with the ``--state-dir`` option to the
Gravity commands, or by setting ``$GRAVITY_STATE_DIR`` in your environment.

Galaxy 22.01 Integration
========================

Gravity was originally designed to support managing multiple Galaxy (as well as Galaxy Reports and Tool Shed) servers on
a single host, but hides some of this complexity from you if you are working with a single Galaxy server. Additionally,
Galaxy 22.01 has added Gravity as a dependency, and changes have been made to Gravity to support this mode of operation.

- Gravity 0.9.0 has dropped support for running Galaxy with uWSGI in favor of `gunicorn`_ and `FastAPI`_.
- Gravity 0.9.0 cannot be used with Galaxy versions older than 22.01.
- As of Galaxy 22.01, Gravity is automatically installed into Galaxy's virtualenv by Galaxy's setup scripts (as called
  by ``run.sh``)
- Galaxy's setup scripts as of 22.01 also set ``$GRAVITY_STATE_DIR`` to ``<galaxy_root>/database/gravity`` when running
  Galaxy from the source directory. This keeps each Galaxy instance's Gravity configuration separate.

Usage
=====

If running from the root of a Galaxy source tree, you can start and run Galaxy in the foreground with::

    $ galaxy
    Registered galaxy config: /home/nate/work/galaxy/config/galaxy.yml
    Creating or updating service gunicorn
    Creating or updating service celery
    Creating or updating service celery-beat
    celery: added process group
    2022-01-20 14:44:24,619 INFO spawned: 'celery' with pid 291651
    celery-beat: added process group
    2022-01-20 14:44:24,620 INFO spawned: 'celery-beat' with pid 291652
    gunicorn: added process group
    2022-01-20 14:44:24,622 INFO spawned: 'gunicorn' with pid 291653
    celery                           STARTING
    celery-beat                      STARTING
    gunicorn                         STARTING
    ==> /home/nate/work/galaxy/database/gravity/log/gunicorn.log <==
    ...log output follows...

Galaxy will continue to run and output logs to stdout until terminated with ``CTRL+C``.

The ``galaxy`` command is actually a shortcut for three separate steps: 1. register your Galaxy configuration file
(``galaxy.yml``) with Gravity, 2. write out the process manager configurations, and 3. start and run Galaxy in the
foreground using the process manager (`supervisor`_). You can perform these steps separately (and in this example, start
Galaxy as a backgrounded daemon instead of in the foreground)::

    $ galaxyctl register config/galaxy.yml
    Registered galaxy config: /home/nate/work/galaxy/config/galaxy.yml
    $ galaxyctl update
    Creating or updating service gunicorn
    Creating or updating service celery
    Creating or updating service celery-beat
    $ galaxyctl start
    celery                           STARTING
    celery-beat                      STARTING
    gunicorn                         STARTING
    Log files are in /home/nate/work/galaxy/database/gravity/log

When running as a daemon, the ``stop`` subcommand stops your Galaxy server::

    $ galaxyctl stop
    celery-beat: stopped
    gunicorn: stopped
    celery: stopped
    All processes stopped, supervisord will exit
    Shut down

Once a Galaxy configuration file has been registered with Gravity, it doesn't matter where you call ``galaxy`` or
``galaxyctl`` from.

Configuration
=============

The following options in the ``gravity`` section of ``galaxy.yml`` can be used to control Gravity::
unset are shown)::

  # Configuration for Gravity process manager.
  # ``uwsgi:`` section will be ignored if Galaxy is started via Gravity commands (e.g ``./run.sh``, ``galaxy`` or ``galaxyctl``).
  gravity:

    # Specify Galaxy's root directory.
    # Gravity will attempt to find the root directory, but you can set the directory explicitly with this option.
    # galaxy_root:

    # Set to a directory that should contain log files for the processes controlled by Gravity.
    # If not specified defaults to ``<state_dir>/log``.
    # log_dir:

    # Set to Galaxy's virtualenv directory.
    # If not specified, Gravity assumes all processes are on PATH.
    # virtualenv:

    # Select the application server.
    # ``gunicorn`` is the default application server.
    # ``unicornherder`` is a production-oriented manager for (G)unicorn servers that automates zero-downtime Galaxy server restarts,
    # similar to uWSGI Zerg Mode used in the past.
    # Valid options are: gunicorn, unicornherder
    # app_server: gunicorn

    # Override the default instance name.
    # this is hidden from you when running a single instance.
    # instance_name: _default_

    # Configuration for Gunicorn.
    gunicorn:

      # Enable Galaxy gunicorn server.
      # enable: true

      # The socket to bind. A string of the form: ``HOST``, ``HOST:PORT``, ``unix:PATH``, ``fd://FD``. An IP is a valid HOST.
      # bind: localhost:8080

      # Controls the number of Galaxy application processes Gunicorn will spawn.
      # Increased web performance can be attained by increasing this value.
      # If Gunicorn is the only application on the server, a good starting value is the number of CPUs * 2 + 1.
      # 4-12 workers should be able to handle hundreds if not thousands of requests per second.
      # workers: 1

      # Gunicorn workers silent for more than this many seconds are killed and restarted.
      # Value is a positive number or 0. Setting it to 0 has the effect of infinite timeouts by disabling timeouts for all workers entirely.
      # If you disable the ``preload`` option workers need to have finished booting within the timeout.
      # timeout: 300

      # Extra arguments to pass to Gunicorn command line.
      # extra_args:

      # Use Gunicorn's --preload option to fork workers after loading the Galaxy Application.
      # Consumes less memory when multiple processes are configured. Default is ``false`` if using unicornherder, else ``true``.
      # preload:

      # Extra environment variables and their values to set when running the service. A dictionary where keys are the variable
      # names.
      # environment: {}

    # Configuration for Celery Processes.
    celery:

      # Enable Celery distributed task queue.
      # enable: true

      # Enable Celery Beat periodic task runner.
      # enable_beat: true

      # Number of Celery Workers to start.
      # concurrency: 2

      # Log Level to use for Celery Worker.
      # Valid options are: DEBUG, INFO, WARNING, ERROR
      # loglevel: DEBUG

      # Queues to join
      # queues: celery,galaxy.internal,galaxy.external

      # Pool implementation
      # Valid options are: prefork, eventlet, gevent, solo, processes, threads
      # pool: threads

      # Extra arguments to pass to Celery command line.
      # extra_args:

      # Extra environment variables and their values to set when running the service. A dictionary where keys are the variable
      # names.
      # environment: {}

    # Configuration for gx-it-proxy.
    gx_it_proxy:

      # Set to true to start gx-it-proxy
      # enable: false

      # Public-facing IP of the proxy
      # ip: localhost

      # Public-facing port of the proxy
      # port: 4002

      # Routes file to monitor.
      # Should be set to the same path as ``interactivetools_map`` in the ``galaxy:`` section.
      # sessions: database/interactivetools_map.sqlite

      # Include verbose messages in gx-it-proxy
      # verbose: true

      # Forward all requests to IP.
      # This is an advanced option that is only needed when proxying to remote interactive tool container that cannot be reached through the local network.
      # forward_ip:

      # Forward all requests to port.
      # This is an advanced option that is only needed when proxying to remote interactive tool container that cannot be reached through the local network.
      # forward_port:

      # Rewrite location blocks with proxy port.
      # This is an advanced option that is only needed when proxying to remote interactive tool container that cannot be reached through the local network.
      # reverse_proxy: false

      # Extra environment variables and their values to set when running the service. A dictionary where keys are the variable
      # names.
      # environment: {}

    # Configuration for tusd server (https://github.com/tus/tusd).
    # The ``tusd`` binary must be installed manually and made available on PATH (e.g in galaxy's .venv/bin directory).
    tusd:

      # Enable tusd server.
      # If enabled, you also need to set up your proxy as outlined in https://docs.galaxyproject.org/en/latest/admin/nginx.html#receiving-files-via-the-tus-protocol.
      # enable: false

      # Path to tusd binary
      # tusd_path: tusd

      # Host to bind the tusd server to
      # host: localhost

      # Port to bind the tusd server to
      # port: 1080

      # Directory to store uploads in.
      # Must match ``tus_upload_store`` setting in ``galaxy:`` section.
      # upload_dir:

      # Comma-separated string of enabled tusd hooks.
      #
      # Leave at the default value to require authorization at upload creation time.
      # This means Galaxy's web process does not need to be running after creating the initial
      # upload request.
      #
      # Set to empty string to disable all authorization. This means data can be uploaded (but not processed)
      # without the Galaxy web process being available.
      #
      # You can find a list of available hooks at https://github.com/tus/tusd/blob/master/docs/hooks.md#list-of-available-hooks.
      # hooks_enabled_events: pre-create

      # Extra arguments to pass to tusd command line.
      # extra_args:

      # Extra environment variables and their values to set when running the service. A dictionary where keys are the variable
      # names.
      # environment: {}

    # Configure dynamic handlers in this section.
    # See https://docs.galaxyproject.org/en/latest/admin/scaling.html#dynamically-defined-handlers for details.
    # handlers: {}


As a convenience for cases where you may want to have different Gravity configurations but a single Galaxy
configuration (e.g. your Galaxy server is split across multiple hosts), the Gravity configuration can be stored in a
separate file and included into the Galaxy configuration. For example, on a deployment where the web (gunicorn) and job
handler processes run on different hosts, one might have:

In ``galaxy.yml``::

    gravity: !include gravity.yml
    galaxy:
      database_connection: postgresql://...
      ...

In ``gravity.yml`` on the web host::

    log_dir: /var/log/galaxy
    gunicorn:
      bind: localhost:8888
    celery:
      enable: false
      enable_beat: false

In ``gravity.yml`` on the job handler host::

    log_dir: /var/log/galaxy
    gunicorn:
      enable: false
    celery:
      enable: true
      enable_beat: true
    handlers:
      handler:
        processes: 2

Galaxy Job Handlers
-------------------

Gravity has limited support for reading Galaxy's job configuration: it can read statically configured job handlers in
the ``job_conf.xml`` file, but cannot read the newer YAML-format job configuration, or the job configuration inline from
``galaxy.yml``. Improved support for reading Galaxy's job configuration is planned, but for the time being, Gravity will
run standalone Galaxy job handler processes if you:

1. Set ``job_handler_count`` to a number greater than ``0``. **NOTE:** You must also explicitly set the `job handler
   assignment method`_ to ``db-skip-locked`` or ``db-transaction-isolation`` to prevent the web process from also
   handling jobs. This is the preferred method for specifying job handlers.
2. Define static ``<handler id="..."/>`` handlers in the XML-format job configuration file.

Configuration Precedence
------------------------

Gravity's configuration is defined in Galaxy's configuration file to be easy and familiar for Galaxy administrators, but
Gravity maintains its own state in ``$GRAVITY_STATE_DIR/configstate.yaml``.  **If set**, the options in ``galaxy.yml``
will override Gravity's saved state whenever ``galaxyctl update`` is run, but if later **unset**, then the persisted
values in Gravity's saved state are used.

The exception is the values of ``app_server`` and ``job_handler_*``, which will revert to default values if unset in
``galaxy.yml``, because Gravity dynamically adds and removes services based on the Galaxy configuration by design.

Subcommands
===========

Use ``galaxyctl --help`` for help. Subcommands also support ``--help``, e.g. ``galaxy register --help``

register
--------

Register a Galaxy server config (``galaxy.yml``) with Gravity. Does not update or start. Run ``galaxyctl update`` after
registering to apply changes.

list
----

List config files registered with the process manager.

deregister
----------

Deregister a Galaxy server config, Gravity will no longer manage this Galaxy instance. Run ``galaxyctl update`` after
deregistering to apply changes.

start
-----

Start and run Galaxy and associated processes in daemonized (background) mode, or ``-f`` to run in the foreground and
follow log files. The ``galaxy`` command is a shortcut for ``galaxyctl start -f``.

If no config files are registered and you run ``galaxyctl start`` from the root of a Galaxy source tree, it
automatically runs the equivalent of::

    $ galaxyctl register config/galaxy.yml  # or galaxy.yml.sample if galaxy.yml does not exist
    $ galaxyctl update
    $ galaxyctl start

stop
----

Stop daemonized Galaxy server processes. If no processes remain running after this step (which should be the case when
working with a single Galaxy instance), ``supervisord`` will terminate.

restart
-------

Restart Galaxy server processes. This is done in a relatively "brutal" fashion: processes are signaled (by supervisor)
to exit, and then are restarted. See the ``graceful`` subcommand to restart gracefully.

graceful
--------

Restart Galaxy with minimal interruption. If running with `gunicorn`_ this means holding the web socket open while
restarting (connections to Galaxy will block). If running with `unicornherder`_, a new Galaxy application will be
started and the old one shut down only once the new one is accepting connections. A graceful restart with unicornherder
should be transparent to clients.

update
------

Figure out what has changed in configs, which could be:

-  changes to the Gravity configuration options in ``galaxy.yml``
-  adding or removing handlers in ``job_conf.xml``

This may cause service restarts if there are any changes.

Any needed changes to supervisor configs will be performed and then ``supervisorctl update`` will be called.

``update`` is called automatically for the ``start``, ``stop``, ``restart``, and ``graceful`` subcommands.

shutdown
--------

Stop all processes and cause ``supervisord`` to terminate. Similar to ``stop`` but there is no ambiguity as to whether
``supervisord`` remains running.

supervisorctl
-------------

Pass through directly to supervisor. Run ``galaxyctl supervisorctl`` to invoke the supervisorctl shell, or ``galaxyctl
supervisorctl [command]`` to call a supervisorctl command directly. See the `supervisor`_ documentation or ``galaxyctl
supervisorctl help`` for help.

instances
---------

List known (configured) Galaxy instances and services.

show
----

Show stored configuration details for the named config file.

rename
------

If your ``galaxy.yml`` has moved, you can update its path in Gravity's saved state with this command.

configstate.yaml
================

As discussed in the Configuration section, Gravity maintains a state file that also acts as a configuration of sorts.
Administrators deploying Galaxy with a deployment tool (e.g. `Ansible`_) can take advantage of this to deploy a Gravity
state file as part of their Galaxy deployment. See the ``$GRAVITY_STATE_DIR/configstate.yaml`` file after performing a
``register`` and ``update`` command to see what this file looks like, or below for an example. Keep in mind that running
``galaxyctl`` commands after changing the Galaxy configuration can cause changes to the state file because it was not
originally intended to be user-maintainable. See `Issue #6`_ for discussion and development related to this, as we seek
to provide a more consistent experience in working with Gravity's configuration.

Example
-------

A ``configstate.yaml`` file for a Galaxy service might look like::

    config_files:
      /home/nate/work/galaxy/config/galaxy.yml:
        config_type: galaxy
        instance_name: _default_
        attribs:
          app_server: gunicorn
          log_dir: /home/nate/work/galaxy/database/gravity/log
          bind: 'localhost:8080'
          galaxy_root: /home/nate/work/galaxy
        services:
        - config_type: galaxy
          service_type: gunicorn
          service_name: gunicorn
        - config_type: galaxy
          service_type: celery
          service_name: celery
        - config_type: galaxy
          service_type: celery-beat
          service_name: celery-beat

.. _supervisor: http://supervisord.org/
.. _Galaxy: http://galaxyproject.org/
.. _virtualenv: https://virtualenv.pypa.io/
.. _venv: https://docs.python.org/3/library/venv.html
.. _gunicorn: https://gunicorn.org/
.. _FastAPI: https://fastapi.tiangolo.com/
.. _unicornherder: https://github.com/alphagov/unicornherder
.. _job handler assignment method: https://docs.galaxyproject.org/en/master/admin/scaling.html#job-handler-assignment-methods
.. _Ansible: http://www.ansible.com/
.. _Issue #6: https://github.com/galaxyproject/gravity/issues/6


=========
 History
=========

0.13.3
======

- Don't use gunicorn logging options with unicornherder by @natefoo in https://github.com/galaxyproject/gravity/pull/65

0.13.2
======

- Don't override PATH in subprocess call by @jdavcs in https://github.com/galaxyproject/gravity/pull/62
- Only send pre create hook by @mvdbeek in https://github.com/galaxyproject/gravity/pull/64

0.13.1
======

- Set correct default for environment settings by @natefoo in https://github.com/galaxyproject/gravity/pull/58
- Don't catch exceptions in the deregister, register, and rename subcommands by @natefoo in https://github.com/galaxyproject/gravity/pull/59
- ``processes`` in the ``handling`` dict in the job conf dict is a dict, not a list by @natefoo in https://github.com/galaxyproject/gravity/pull/60

0.13.0
======

- Add options to enable/disable gunicorn, celery, and celery-beat services by @natefoo in https://github.com/galaxyproject/gravity/pull/47
- Add ability to include gravity config from a separate file and document by @natefoo in https://github.com/galaxyproject/gravity/pull/48
- Only default to preload = true for gunicorn if not using unicornherder by @natefoo in https://github.com/galaxyproject/gravity/pull/49
- Add option to specify tusd path by @natefoo in https://github.com/galaxyproject/gravity/pull/50
- Support setting per-service environment variables by @natefoo in https://github.com/galaxyproject/gravity/pull/56

0.12.0
======

- Fix typo in ``log_dir`` description by @nsoranzo in https://github.com/galaxyproject/gravity/pull/44
- Shortcut individual services fix by @natefoo in https://github.com/galaxyproject/gravity/pull/45
- Add additional options to celery beat / celery workers by @mvdbeek in https://github.com/galaxyproject/gravity/pull/46

0.11.0
======

- Allow setting supervisor socket path via environment variable by @mvdbeek in https://github.com/galaxyproject/gravity/pull/36
- Automatically switch to non-sample galaxy.yml if it exists by @mvdbeek in https://github.com/galaxyproject/gravity/pull/39
- Add pydantic config schema by @mvdbeek in https://github.com/galaxyproject/gravity/pull/42
- Add --quiet option to galaxy and galaxyctl start by @mvdbeek in https://github.com/galaxyproject/gravity/pull/40
- Add support for yaml job config by @mvdbeek in https://github.com/galaxyproject/gravity/pull/37
- Add --preload support for gunicorn by @mvdbeek in https://github.com/galaxyproject/gravity/pull/41
- Support running tusd by @natefoo in https://github.com/galaxyproject/gravity/pull/23

0.10.0
======

- Fix for the case where a job_conf.xml exists but no handlers are defined by @natefoo in https://github.com/galaxyproject/gravity/pull/24
- Do not raise error if config file section is empty by @nsoranzo in https://github.com/galaxyproject/gravity/pull/25
- Add tests for static handlers and a defined job_conf.xml with no handlers by @natefoo in https://github.com/galaxyproject/gravity/pull/26
- Fix minor typos in readme by @ic4f in https://github.com/galaxyproject/gravity/pull/27
- Move configuration to gravity key of galaxy.yml file by @mvdbeek in https://github.com/galaxyproject/gravity/pull/28
- Fix for resolved galaxy.yml.sample symlink by @mvdbeek in https://github.com/galaxyproject/gravity/pull/31
- Support managing gx-it-proxy via gravity by @mvdbeek in https://github.com/galaxyproject/gravity/pull/32

0.9
===

- Gunicorn/fastAPI support, click support, tests by @mvdbeek in https://github.com/galaxyproject/gravity/pull/14
- Don't test on Python 3.6, which is unsupported by @natefoo in https://github.com/galaxyproject/gravity/pull/17
- Update README. Also some various small bugfixes and fixes for other stuff mentioned in the README by @natefoo in https://github.com/galaxyproject/gravity/pull/18
- Add unicornherder support by @natefoo in https://github.com/galaxyproject/gravity/pull/15
- Expose the log following used by `start -f` as its own subcommand. by @natefoo in https://github.com/galaxyproject/gravity/pull/16
- Better integration with Galaxy's run.sh by @natefoo in https://github.com/galaxyproject/gravity/pull/19
- Use relative paths in supervisord by @natefoo in https://github.com/galaxyproject/gravity/pull/21
- Converted CLI from `argparse`_ to `click`_.
- Stole ideas and code from `planemo`_ in general.
- Improve the AttributeDict so that it can have "hidden" items (anything that
  starts with a ``_``) that won't be serialized. Also, it serializes itself and
  can be created via deserialization from a classmethod. This simplifies using
  it to persist state data in the new GravityState subclass.

.. _argparse: https://docs.python.org/3/library/argparse.html
.. _click: http://click.pocoo.org/
.. _planemo: https://github.com/galaxyproject/planemo

0.8.3
=====

- Merge ``galaxycfg`` and ``galaxyadm`` commands to ``galaxy``.

0.8.2
=====

- Allow for passing names of individual services directly to ``supervisorctl``
  via the ``start``, ``stop``, and ``restart`` methods.
- Fix a bug where uWSGI would not start when using the automatic virtualenv
  install method.

0.8.1
=====

- Version bump because I deleted the 0.8 files from PyPI, and despite the fact
  that it lets you delete them, it doesn't let you upload once they have been
  uploaded once...

0.8
===

- Add auto-register to ``galaxy start`` if it's called from the root (or
  subdirectory) of a Galaxy root directory.
- Make ``galaxycfg remove`` accept instance names as params in addition to
  config file paths.
- Use the same hash generated for an instance name as the hash for a generated
  virtualenv name, so virtualenvs are more easily identified as belonging to a
  config.
- Renamed from ``galaxyadmin`` to ``gravity`` (thanks John Chilton).

0.7
===

- Added the ``galaxyadm`` subcommand ``graceful`` on a suggestion from Nicola
  Soranzo.
- Install uWSGI into the config's virtualenv if requested.
- Removed any dependence on Galaxy and eggs.
- Moved project to its own repository from the Galaxy clone I'd been working
  from.

Older
=====

- Works in progress as part of the Galaxy code.
