``baseplate-serve``
===================

Baseplate comes with a simple Gevent-based server for both Thrift and WSGI
applications called ``baseplate-serve``.

Configuration
-------------

There is one required parameter on the command line, the path to an INI-format
configuration file. There should be two sections in the file: the ``server``
section and the ``app`` section. The section headers look like ``server:main``
or ``app:main`` where the part before the ``:`` is the type of section and the
part after is the "name". Baseplate looks for sections named ``main`` by
default but can be overridden with the ``--server-name`` and ``--app-name``
options.

Shell-like environment variable references in configuration values will be
expanded at server startup. For example, ``foo = $MY_PATH`` or ``foo =
${MY_PATH}`` will result in the application seeing the value from the
``$MY_PATH`` environment variable as the value of the ``foo`` setting as if it
had been written in the config file directly.

.. highlight:: ini

The Server
----------

Here's an example of a ``server`` section::

   [server:main]
   factory = baseplate.server.thrift
   stop_timeout = 30

The ``factory`` tells baseplate what code to use to run the server. Baseplate
comes with two servers built in:

``baseplate.server.thrift``
   A Gevent Thrift server.

``baseplate.server.wsgi``
   A Gevent WSGI server.

Both take two configuration values as well:

``max_concurrency``
   The maximum number of simultaneous clients the server will handle. Note that
   this is how many connections will be accepted, but some of those connections
   may be idle at any given time.

``stop_timeout``
   (Optional) How long, in seconds, to wait for active connections to finish up
   gracefully when shutting down. By default, the server will shut down
   immediately.

The WSGI server takes an additional optional parameter:

``handler``
   A full name of a class which subclasses
   ``gevent.pywsgi.WSGIHandler`` for extra functionality.

There are some additional configuration settings in this section that start
with a ``monitoring`` prefix. For more information on those, see `Process-level
metrics`_.


The Application
---------------

And now the real bread and butter, your ``app`` section::

   [app:main]
   factory = my_app.processor:make_processor
   foo = 3
   bar = 22
   noodles.blah = one, two, three

The ``app`` section also takes a ``factory``.  This should be the name of a
callable in your code which builds and returns your application. The part
before the ``:`` is a Python module. The part after the ``:`` is the name of a
callable object within that module.

The rest of the options in the ``app`` section of the configuration file get
passed as a dictionary to your application callable. You can parse these
options with :py:mod:`baseplate.lib.config`.

The application factory should return an appropriate object for your server:

Thrift
   A ``TProcessor``.

WSGI
   A WSGI callable.

.. _server-logging:

Logging
-------

The baseplate server provides a default configuration for the Python standard
``logging`` system. The root logger will print to ``stdout`` with a format that
includes trace information. The default log level is ``INFO`` or ``DEBUG`` if
the ``--debug`` flag is passed to ``baseplate-serve``.

If more complex logging configuration is necessary, the configuration file will
override the default setup. The :ref:`logging-config-fileformat` is documented
in the standard library.

Automatic reload on source changes
----------------------------------

In development, it's useful for the server to restart itself when you change
code.  You can do this by passing the ``--reload`` flag to ``baseplate-serve``.

This should not be used in production environments.

Einhorn
-------

``baseplate-serve`` can run as a worker in `Stripe's Einhorn socket manager`_.
This allows Einhorn to handle binding the socket, worker management, rolling
restarts, and worker health checks.

Baseplate supports Einhorn's "manual ACK" protocol. Once the application is
loaded and ready to serve, Baseplate notifies the Einhorn master process via
its command socket.

An example command line::

   einhorn -m manual -n 4 --bind localhost:9190 \
      baseplate-serve myapp.ini

.. _Stripe's Einhorn socket manager: https://github.com/stripe/einhorn

Graceful shutdown
-----------------

The flow of graceful shutdown while handling live traffic looks like this:

* The server receives a ``SIGTERM`` from the infrastructure.
* The server sets ``baseplate.server.SERVER_STATE.shutting_down`` to ``True``.
* If the ``drain_time`` setting is set in the server configuration, the server
  will wait the specified amount of time before continuing to the next step.
  This gives your application a chance to use the ``shutting_down`` flag in
  healthcheck responses.
* The server begins graceful shutdown. No new connections will be accepted. The
  server will continue processing the existing in-flight requests until they
  are all done or ``stop_timeout`` time has elapsed.
* The server exits and lets the infrastructure clean up.

During the period between receiving the ``SIGTERM`` and the server exiting, the
application may still be routed new requests. To ensure requests aren't lost
during the graceful shutdown (where they won't be listened for) your
application should set an appropriate ``drain_time`` and use the
``SERVER_STATE.shutting_down`` flag to fail ``READINESS`` healthchecks.

For example:

.. code-block:: py

   def is_healthy(self, context, healthcheck_request):
       if healthcheck_request.probe == IsHealthyProbe.LIVENESS:
           return True
       elif healthcheck_request.probe == IsHealthyProbe.READINESS:
           if SERVER_STATE.shutting_down:
               return False
           return True
       return True


Debug Signal
------------

Applications running under ``baseplate-serve`` will respond to ``SIGUSR1`` by
printing a stack trace to the logger. This can be useful for debugging
deadlocks and other issues.

Note that Einhorn will exit if you send it a ``SIGUSR1``. You can instead open up
``einhornsh`` and instruct the master to send the signal to all workers:

.. code-block:: console

   $ einhornsh
   > signal SIGUSR1
   Successfully sent USR1s to 4 processes: [...]

.. _runtime-metrics:

Prometheus Exporter
-------------------

When enabled, ``baseplate-serve`` will start up a Prometheus exporter on HTTP
port 6060. The exporter is designed for multiprocess use, like if running under
Einhorn.  Each worker process writes its metrics to a file on disk and runs its
own copy of the exporter, all listening on the same port. Any given worker can
serve the metrics for all workers from the same pod. It doesn't matter which
one is hit by the Prometheus collector.

To enable the exporter, install the ``prometheus-client`` package from PyPI:

.. code-block:: console

   $ pip install prometheus-client

If for some reason your service doesn't need Prometheus metrics exported, they can be disabled (they're on by default) with:

   metrics.enabled = false
The ``PROMETHEUS_MULTIPROC_DIR`` environment variable must be set to the path
to an extant writeable directory. This is handled automatically in the official
base Docker images.

As long as the exporter is enabled, you can create Prometheus metrics in your
application using the objects in ``prometheus-client`` and they will get
exported without any extra configuration required.

Note that for non-server jobs, like ``baseplate-script`` scripts, you will need
to export metrics manually. See `exporting to a push gateway`_.

.. _exporting to a push gateway: https://github.com/prometheus/client_python#exporting-to-a-pushgateway

.. versionadded:: 2.3

Process-level metrics
---------------------

If your application has registered a metrics client with
:py:meth:`~baseplate.Baseplate.configure_observers`, ``baseplate-serve``
will automatically send process-level metrics every 10 seconds. Which metrics
are sent depends on your server configuration, for example::

   [server:main]
   factory = baseplate.server.thrift

   monitoring.blocked_hub = 100 milliseconds
   monitoring.concurrency = true

will enable the ``blocked_hub`` reporter (configuring it to trigger at a 100ms
threshold) and the ``concurrency`` reporter (which has no special
configuration).

The following reporters are available:

``monitoring.blocked_hub``
   Enabled if a valid :py:func:`~baseplate.lib.config.Timespan` is set, defaults to disabled.

   This will turn on `Gevent's monitoring thread`_ and report events indicating
   that Gevent detects the main event loop was blocked by a greenlet for longer
   than the given time span. This can indicate excessive CPU usage causing
   event loop starvation or the use of non-patched blocking IO calls. More
   detailed information, including stack traces, is also printed to the logging
   system.

   Each instance of the hub being blocked will be reported as a
   :py:class:`~baseplate.lib.metrics.Timer` measuring the duration of the blockage.

   Note: the performance impact of this reporter is not currently understood.
   Watch your metrics closely if you turn this on.

``monitoring.concurrency``
   Enabled if ``true``, disabled if ``false``. Defaults to enabled.

   This will track the number of in-flight requests being processed
   concurrently by this server process.

   At each report interval, this will update two
   :py:class:`~baseplate.lib.metrics.Gauge` metrics with the current number of
   open connections (``open_connections``) and current number of in-flight
   requests being processed concurrently (``active_requests``).

``monitoring.connection_pool``
   Enabled if ``true``, disabled if ``false``. Defaults to disabled.

   This will track the usage of connection pools for various clients in the
   application. The metrics generated will depend on which clients are used.

``monitoring.gc.stats``
   Enabled if ``true``, disabled if ``false``. Defaults to enabled.

   This will report the Python garbage collector's statistics to the metrics system.

   At each report interval, this will update gauges with the current values
   returned by :py:func:`gc.get_stats()`.

``monitoring.gc.timing``
   Enabled if ``true``, disabled if ``false``. Defaults to disabled.

   This will track the duration of time taken by Python's garbage collector
   doing a collection sweep.

   The duration of each pass of the garbage collector will be reported as a
   timer.

   Note: the performance impact of this reporter is not currently understood.
   Watch your metrics closely if you turn this on.

``monitoring.gc.refcycle``
   Enabled if a path to a writable directory is set, defaults to disabled.

   **This should only be used in debugging, it will certainly have a negative
   performance impact.**

   This will turn off automatic garbage collection and instead run a sweep
   every reporting interval. Any objects found by the collector will be graphed
   using `objgraph`_ to help find reference cycles. The resulting graphs images
   will be written to the directory specified.

   The ``objgraph`` library and ``graphviz`` package must be installed for this
   to work properly.

All metrics generated by ``baseplate-serve`` are prefixed with ``runtime`` and
are tagged with ``hostname`` and ``PID``.

.. versionchanged:: 2.0

   The ``hostname`` and ``PID`` fields of the metric name were moved to tags.

.. _Gevent's monitoring thread: http://www.gevent.org/monitoring.html#the-monitor-thread
.. _objgraph: https://mg.pov.lt/objgraph/
