Docker images
-------------

In addition to :doc:`/user_guide/package_sync`, you can also create a Coiled Python environment using a Docker image. Coiled supports images on `Docker Hub <https://hub.docker.com/>`_, `Amazon ECR <https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html>`_ or `Google Artifact Registry <https://cloud.google.com/artifact-registry/docs/overview>`_.

.. note::

    Your ability to use private images stored in Docker Hub or a cloud provider-specific registry is limited by which option you chose when initially setting up your Coiled account (see the Container Registry step for :ref:`Google Cloud <setup-gar>` or :ref:`AWS <ecr>`).
    
    For example, if you chose to store your Coiled software environments in ECR, then you will not be able to use private Docker Hub images. If you would like to be able to use both Docker Hub and ECR :doc:`reach out to us </user_guide/support>` for help.

First, you'll create a Coiled Python environment using the ``container=`` keyword argument of of :meth:`coiled.create_software_environment`. For example, to create an environment using the publicly available RAPIDS image on Docker Hub:

.. code-block:: python

    coiled.create_software_environment(
        name="my-docker-env",
        container="rapidsai/rapidsai-core:23.02-cuda11.5-runtime-ubuntu20.04-py3.8",
    )

For Amazon ECR or Google Artifact Registry, you can pass the full registry URL; for Amazon ECR, for example, this would resemble ``789111821368.dkr.ecr.us-east-2.amazonaws.com/prod/coiled``.

Verification
~~~~~~~~~~~~

To test that your container will run successfully on your Coiled cluster, you can run::

    docker run --rm <your_container> python -m distributed.cli.dask_spec \
      --spec '{"cls":"dask.distributed.Scheduler", "opts":{}}'

If successful, this will start the ``dask.distributed`` scheduler (you can use CTRL+C to stop it). For example::

    > docker run --rm daskdev/dask:latest python -m distributed.cli.dask_spec \
        --spec '{"cls":"dask.distributed.Scheduler", "opts":{}}'

    2022-10-06 14:44:43,640 - distributed.scheduler - INFO - State start
    2022-10-06 14:44:43,656 - distributed.scheduler - INFO - Clear task state
    2022-10-06 14:44:43,658 - distributed.scheduler - INFO -   Scheduler at:    tcp://172.17.0.2:41089
    2022-10-06 14:44:43,658 - distributed.scheduler - INFO -   dashboard at:                     :8787

If not, you will see an error like ``/opt/conda/bin/python: Error while finding module specification for 'distributed.cli.dask_spec' (ModuleNotFoundError: No module named 'distributed')``. For example::

    > docker run --rm continuumio/miniconda3:latest python -m distributed.cli.dask_spec \
        --spec '{"cls":"dask.distributed.Scheduler", "opts":{}}'

    Unable to find image 'continuumio/miniconda3:latest' locally
    latest: Pulling from continuumio/miniconda3
    dc1f00a5d701: Already exists
    a7a9c78d89b2: Already exists
    44ac19016d77: Already exists
    Digest: sha256:977263e8d1e476972fddab1c75fe050dd3cd17626390e874448bd92721fd659b
    Status: Downloaded newer image for continuumio/miniconda3:latest
    /opt/conda/bin/python: Error while finding module specification for 'distributed.cli.dask_spec' (ModuleNotFoundError: No module named 'distributed')

If the ``dask.distributed`` scheduler fails to start, it's good to check that ``distributed`` is installed and the environment it is installed in has been activated. If you're having trouble running your Docker container on your Coiled cluster, feel free to :doc:`reach out to us </user_guide/support>` for help.
