.. _cluster-config:

======================
Cluster configurations
======================

.. currentmodule:: coiled


When creating Dask clusters, there are a variety of ways in which you can customize
the cluster. For example, you can specify the resources (i.e. CPU and memory) available to the
workers in your cluster, the software environment used throughout the cluster, whether
or not to make GPUs available, etc.

Each of these parameters can be specified when creating a ``coiled.Cluster`` (see the
:ref:`cluster-creation` documentation for more details). Additionally, it's also
useful to be able to save these cluster options and share them with friends and colleagues.
Coiled uses the concept of a **cluster configuration** to specify the parameters that define
a Dask cluster, save them, and then share them with others.

.. note::

    It's important to note that creating a cluster configuration doesn't create a
    cluster or provision any resources. You can think of a cluster configuration as
    a template, or recipe, for a cluster that will be create later.


Creating cluster configurations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Cluster configurations are created using the :meth:`coiled.create_cluster_configuration` function.
You must provide each cluster configuration you create with a name to identify the configuration,
along with a set of optional hardware and software parameters.

For example:

.. code-block:: python

    # Create a software environment named "my-conda-env"
    coiled.create_software_environment(
        name="my-conda-env",
        conda={
            "channels": ["conda-forge", "defaults"],
            "dependencies": ["dask", "xarray=0.15.1", "numba"],
        },
    )

    # Create a cluster configuration named "my-cluster-config"
    coiled.create_cluster_configuration(
        name="my-cluster-config",
        scheduler_cpu=2,
        scheduler_memory="8 GiB",
        worker_cpu=4,
        worker_memory="16 GiB",
        software="my-conda-env",
    )

creates a cluster configuration named "my-cluster-config" where the Dask scheduler
has 2 CPU / 8 GiB of memory, workers each have 4 CPUs / 16 GiB of memory, and
the "my-conda-env" software environment (also created in the above code snippet)
is used for the scheduler and all workers.

.. admonition:: Note
    :class: note

    Software environments used in Coiled clusters must have ``distributed >= 2.23.0``
    installed as `Distributed <https://distributed.dask.org>`_ is required to
    launch Dask scheduler and worker processes.


Using cluster configurations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To create a cluster from a cluster configuration, use the name of configuration for
the ``configuration`` keyword argument to ``coiled.Cluster``. For example:

.. code-block:: python

    # Create a cluster configuration named "my-cluster-config"
    coiled.create_cluster_configuration(
        name="my-cluster-config",
        worker_cpu=4,
        worker_memory="16 GiB",
    )

    # Created a cluster based on the "my-cluster-config" configuration
    # It will use worker_cpu=4 and worker_memory="16 GiB"
    cluster = coiled.Cluster(configuration="my-cluster-config")

Note that you can also use *both* a cluster configuration and pass keyword arguments directly to
``coiled.Cluster``. In this case, keywords which are directly passed will take
precedent over the values stored in the cluster configuration.

For example, in the snippet below ``worker_cpu`` is specified in both a cluster configuration
and passed directly to ``coiled.Cluster``. So the value passed directly to ``coiled.Cluster``
(i.e. ``worker_cpu=4``) will be used:

.. code-block::

    import coiled

    coiled.create_cluster_configuration(
        name="my-config",
        worker_cpu=2,
        worker_memory="16 GiB",
    )

    # Here worker_cpu=4 will be used
    cluster = coiled.Cluster(
        configuration="my-config",
        worker_cpu=4,
    )


Updating cluster configurations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You can update an existing cluster configuration, for example to add a different software
environment, change the number of CPUs or memory, by calling ``coiled.create_cluster_configuration``
with the name of the cluster configuration you want to update and the new specification for the
cluster configuration.

For example, let's update the memory of the cluster configuration that we have previously created.

.. code-block:: python

    coiled.create_cluster_configuration(
        name="my-cluster-config",
        scheduler_cpu=2,
        scheduler_memory="8 GiB",
        worker_cpu=2,
        worker_memory="32 GiB",
        software="my-conda-env",
    )


Listing and deleting cluster configurations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In addition to creating cluster configurations, you can also use the
:meth:`coiled.list_cluster_configurations` function to list all available
cluster configurations:

.. code-block:: python

    coiled.list_cluster_configurations()

There is also a ``account=`` keyword argument which lets you specify the account which you
want to list cluster configurations for.

Similarly, the :meth:`coiled.delete_cluster_configuration` function can be used to delete
individual configurations. For example:

.. code-block:: python

    coiled.delete_cluster_configuration(name="alice/my-cluster-config")

deletes the cluster configuration named "my-cluster-config" in the Coiled account named "alice".


.. _cluster-config-visibility:

Visibility
^^^^^^^^^^

By default, cluster configurations can be viewed and used by any coiled user. If
you want a cluster configuration to be visible only to members of your account,
call :meth:`coiled.create_cluster_configuration` with ``private=True``

.. code-block:: python

   import coiled

   coiled.create_cluster_configuration(
       name="my-config",
       worker_cpu=2,
       worker_memory="16 GiB",
       private=True,
   )
