.. _datamodel:

.. currentmodule:: ctapipe.io.containers

==================
CTAPipe Data Model
==================

The DL1 files are HDF5 format files, with the following data set hierarchy.
The tables should be written with `pytables <https://www.pytables.org>`_ (not ``h5py``),
ideally with the :py:class:`~ctapipe.io.HDF5TableWriter`, which ensures the unit
and other descriptive metadata are attached to the output.
Containers marked with a ``+`` should be written without their prefix (all others should use column prefixes).

The following describes the contents of data level 1 (DL1) output files
generated by ctapipe (e.g. the ``ctapipe-process`` tool which uses the
:py:class:`~ctapipe.io.DataWriter` component to generate output data).


--------------
DL1 Data Model
--------------

This describes data that change per-event.
The following datasets will be written to the group ``/dl1/event/`` in the  output file:

.. list-table::
    :header-rows: 1

    * - Group/Dataset
      - Description
      - Contents
    * - ``subarray``
      - event-wise data pertaining to a subarray
      - (group)
    * - ``subarray/trigger``
      - subarray trigger information
      - :py:class:`~ctapipe.containers.EventIndexContainer` +, :py:class:`~ctapipe.containers.TriggerContainer`
    * - ``telescope/``
      - Per-telescope Per-event information
      - (group)
    * - ``telescope/parameters/tel_{TEL_ID:03d}``
      - tables of image parameters (one per telescope)
      - :py:class:`~ctapipe.containers.TelEventIndexContainer` +, :py:class:`~ctapipe.containers.ImageParametersContainer`
    * - ``telescope/images/tel_{TEL_ID:03d}``
      - tables of telescope images (one per telescope)
      - :py:class:`~ctapipe.containers.TelEventIndexContainer` +, :py:class:`~ctapipe.containers.DL1CameraContainer`

--------------
DL2 Data Model
--------------

This describes data that change per-event. The following datasets will be
written to the group ``/dl2/event/stereo/<algorithm>/`` and or
``/dl2/event/mono/<algorithm>/``, one for each reconstruction algorithm in the
output file, where ``<algorithm>`` is the identifier of the algorithm (e.g.
"Hillas"):

.. list-table::
    :widths: 25 50 25
    :header-rows: 1

    * - Group/Dataset
      - Description
      - Contents
    * - /geometry
      - shower geometry reconstruction
      - :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.ReconstructedGeometryContainer`
    * - /energy
      - shower energy reconstruction
      - :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.ReconstructedEnergyContainer`
    * - /classification
      - shower classification parameters
      - :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.ParticleClassificationContainer`
        
---------------------
Simulation Data Model
---------------------
    * - ``/simulation/event/subarray/shower``
      - true shower parameters from Monte-Carlo simulation
      - :py:class:`~ctapipe.containers.EventIndexContainer` +, :py:class:`~ctapipe.containers.SimulatedShowerContainer`
    * - ``/simulation/event/telescope/images/tel_{TEL_ID:03d}``
      - simulated camera images
      - :py:class:`~ctapipe.containers.EventIndexContainer` +, :py:class:`~ctapipe.containers.SimulatedCameraContainer`
    * - ``/simulation/event/telescope/parameters/tel_{TEL_ID:03d}``
      - Parameters derived form the simulated camera images
      - :py:class:`~ctapipe.containers.EventIndexContainer` +, :py:class:`~ctapipe.containers.ImageParametersContainer`
    * - ``/simulation/service/shower_distribution``
      - simulated shower distribution histograms
      - :py:class:`~ctapipe.containers.SimulatedShowerDistribution`

------------------------
Configuration Data Model
------------------------

The output file should also contain serializations of the instrument, observation (if
applicable), simulation (if applicable) configuration information, written to the
``/configuration`` group:

.. list-table::
    :widths: 25 50 25
    :header-rows: 1

    * - Group/Dataset
      - Description
      - Contents
    * - ``/instrument``
      - Serialized :py:class:`ctapipe.instrument.SubarrayDescription`, using :py:meth:`ctapipe.instrument.SubarrayDescription.to_hdf`
      - (group)
    * - ``/instrument/subarray/layout``
      - Subarray layout info
      - result of ``SubarrayDescription.to_table()`` output as HDF5 using ``astropy.table`` functionality
    * - ``/instrument/telescope/optics``
      - telescope optics information
      - result of ``SubarrayDescription.to_table(kind='optics')`` output as HDF5 using
        ``astropy.table`` functionality
    * - ``/instrument/telescope/camera/{CAMERA_ID}``
      - camera geometry information
      - result of ``CameraGeometry.to_table()`` output as HDF5 using ``astropy.table``
        functionality
    * - ``/simulation``
      - Monte-Carlo simulation configuration information
      - (group)
    * - ``/simulation/run``
      - Monte-Carlo simulation run information
      - :py:class:`~ctapipe.containers.SimulationConfigContainer`

---------------
Core Provenance
---------------

The root group of the file shall contain all of the "CTA Core Provenance Metadata"
headers as user attributes, with the hierarchy flattened and separated by spaces
(e.g.  ``"CTA ACTIVITY NAME" = "ctapipe-process"``).
