=====================
Software Environments
=====================

.. currentmodule:: coiled

.. toctree::
   :maxdepth: 1
   :hidden:

   software_environment_management
   software_environment_local
   software_environment_cli

It is important that the packages and versions installed on your local machine are the same as in the cloud computing environment where Dask clusters are created. Otherwise, you could get errors, or even incorrect results.

We recommend using :doc:`package_sync`, which scans your local Python environment and replicates it on the cluster. This may not work for every use case, though, and instead can recreate your local environment in the cloud using :func:`coiled.create_software_environment`.

Creating software environments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can create your Python environment by installing dependencies from conda, pip, or by specifying a pre-built Docker image.

Conda
^^^^^

You can create a conda environment using the ``conda`` keyword argument. You can pass a list of dependencies, for example:

.. code-block:: python

    coiled.create_software_environment(
        name="my-conda-env", conda=["python=3.9", "dask", "coiled", "xarray"]
    )

.. note::

    When creating an environment with ``conda`` it's important to specify the Python version, otherwise, the highest supported version will be used. 

A dictionary of channels and dependencies:

.. code-block:: python

    coiled.create_software_environment(
        name="my-conda-env",
        conda={
            "channels": ["conda-forge", "defaults"],
            "dependencies": ["python=3.9", "dask", "coiled", "xarray"],
        },
    )

or a local environment.yml file:

.. code-block:: python

    coiled.create_software_environment(
        name="my-conda-env",
        conda="environment.yml",
    )

where ``environment.yml`` is a local file that might look something like (see the `conda documentation <https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#exporting-the-environment-yml-file>`_ on how to export your environment.yml file):

.. code-block:: yaml

    # environment.yml
    channels:
      - conda-forge
      - defaults
    dependencies:
      - python==3.9
      - dask==2023.2.0
      - bokeh==2.4.3
      - numba



Pip
^^^

Similarly, you can use the ``pip`` keyword argument to install dependencies using pip. You can pass a list of dependencies, for example:

.. code-block:: python

    coiled.create_software_environment(
        name="my-pip-env",
        pip=["dask[complete]", "coiled", "xarray"],
    )

.. note::

    When creating an environment with ``pip``, your Python version will be detected automatically and used in your cluster.

.. note::

    Pip does not automatically install ``distributed`` along with ``dask``.
    Specify dask with ``dask[complete]`` or ``dask[distributed]`` to ensure
    distributed is installed.

Or you can pass a local requirements.txt file (see the `pip documentation <https://pip.pypa.io/en/stable/user_guide/#requirements-files>`_ for more information on requirements files):

.. code-block:: python

    coiled.create_software_environment(
        name="my-pip-env",
        pip="requirements.txt",
    )

where ``requirements.txt`` might look something like:

.. code-block::

    bokeh==2.4.3
    click==8.1.3
    cloudpickle==2.2.1
    dask==2023.2.0
    distributed==2023.2.0
    fsspec==2023.1.0

Private Repositories
^^^^^^^^^^^^^^^^^^^^

To use pip packages hosted in private repositories you must add a personal
access token to your Coiled profile, which allows Coiled to pip install these
packages on your behalf. To create a GitHub personal access token, follow the
steps in
`GitHub's guide <https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token>`_.
After you've created your access token, add it to your profile page at
https://cloud.coiled.io/profile.

When specifying a pip package from a private repository use the format:

.. code-block:: text

    git+https://GIT_TOKEN@github.com/<github_account>/<github_repo>.git

For example:

.. code-block:: python

    coiled.create_software_environment(
        name="my-pip-env",
        pip=[
            "dask[complete]",
            "git+https://GIT_TOKEN@github.com/coiled/private_package.git",
        ],
    )

.. attention::

    For security reasons, you should **not** use your actual personal access
    token when specifying pip requirements. Instead, use the literal string
    ``GIT_TOKEN`` which acts as a placeholder for your personal access token.
    Your actual access token will be populated when Coiled builds the
    corresponding software environment.

Docker
^^^^^^

You can also build environments based on Docker images using the ``container`` keyword argument, for example:

.. code-block:: python

    coiled.create_software_environment(
        name="my-docker-env",
        container="rapidsai/rapidsai-core:23.02-cuda11.5-runtime-ubuntu20.04-py3.8",
    )

will build a software environment named “my-docker-env” using the latest RAPIDS image. See :doc:`docker` for more details.