Metadata-Version: 2.4
Name: dmp-af
Version: 0.15.0
Summary: Distibuted dbt runs on Apache Airflow
Project-URL: Homepage, https://github.com/dmp-labs/dmp-af
Project-URL: Documentation, https://dmp-labs.github.io/dmp-af/
Project-URL: Release Notes, https://github.com/dmp-labs/dmp-af/blob/main/CHANGELOG.md
Project-URL: Source Code, https://github.com/dmp-labs/dmp-af
Author-email: Nikita Yurasov <nikitayurasov@ijkos.com>, Evgeny Ermakov <jk.ermakov@ijkos.com>, Leonid Kozhinov <lkozhinov@ijkos.com>
License-Expression: Apache-2.0
License-File: AUTHORS
License-File: LICENSE
License-File: NOTICE
Keywords: airflow,dbt,python
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: <3.13,>=3.10
Requires-Dist: apache-airflow[cncf-kubernetes,fab]<3.2.0,>=2.6
Requires-Dist: attrs>=23.1
Requires-Dist: cachetools<7,>=5.3
Requires-Dist: croniter>=3.0
Requires-Dist: dbt-core<2,>=1.7
Requires-Dist: packaging>=21.0
Requires-Dist: pydantic<3.0.0,>=1.10
Requires-Dist: pytest-env>=1.0.0
Requires-Dist: pytest>=7.4.0
Requires-Dist: typer>=0.9
Requires-Dist: virtualenv>=20.27.0
Provides-Extra: all
Requires-Dist: airflow-mcd<0.4.0,>=0.3.3; extra == 'all'
Requires-Dist: pycarlo>=0.9; extra == 'all'
Requires-Dist: setuptools; extra == 'all'
Requires-Dist: tableauserverclient<0.26.0,>=0.25.0; extra == 'all'
Provides-Extra: mcd
Requires-Dist: airflow-mcd<0.4.0,>=0.3.3; extra == 'mcd'
Requires-Dist: pycarlo>=0.9; extra == 'mcd'
Requires-Dist: setuptools; extra == 'mcd'
Provides-Extra: tableau
Requires-Dist: tableauserverclient<0.26.0,>=0.25.0; extra == 'tableau'
Description-Content-Type: text/markdown

[![PyPI - Version](https://img.shields.io/pypi/v/dmp-af)](https://pypi.org/project/dmp-af/)
[![GitHub Build](https://github.com/dmp-labs/dmp-af/workflows/Tests/badge.svg)](https://github.com/dmp-labs/dmp-af/actions)

[![License](https://img.shields.io/:license-Apache%202-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0.txt)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/dmp-af.svg)](https://pypi.org/project/dmp-af/)
[![PyPI - Downloads](https://img.shields.io/pepy/dt/dmp-af)](https://pypi.org/project/dmp-af/)

[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

# dmp-af: distributed dbt runs on Airflow

## Overview

**dmp-af** runs your dbt models in parallel on Airflow. Each model becomes an independent task while preserving
dependencies across domains.

**Built for scale.** Designed for large dbt projects (1000+ models)
and [data mesh architecture](https://www.datamesh-architecture.com/#what-is-data-mesh). Works with any project size.

![dmp-af](docs/static/airflow_dag_layout.png)
![dbt-af3](docs/static/airflow3_dag_layout.png)

### Why dmp-af?

1. **Domain-driven architecture** - Separate models by domain into different DAGs, run in parallel, perfect for data
   mesh
2. **dbt-first design** - All configuration in dbt model configs, analytics teams stay in dbt, no Airflow knowledge
   required
3. **Flexible scheduling** - Multiple schedules per model (`@hourly`, `@daily`, `@weekly`,
   `@monthly`, [and more](examples/manual_scheduling.md))
4. **Enterprise features** - Multiple dbt targets, configurable test strategies, built-in maintenance, Kubernetes
   support

## Installation

To install `dmp-af` run `pip install dmp-af`.

To contribute we recommend to use `uv` to install package dependencies.
Run `uv sync --all-packages --all-groups --all-extras` to install all dependencies.

## _dmp-af_ by Example

All tutorials and examples are located in the [examples](examples/README.md) folder.

To get basic Airflow DAGs for your dbt project, you need to put the following code into your `dags` folder:

```python
# LABELS: dag, airflow (it's required for airflow dag-processor)
from dmp_af.dags import compile_dmp_af_dags
from dmp_af.conf import Config, DbtDefaultTargetsConfig, DbtProjectConfig

# specify here all settings for your dbt project
config = Config(
    dbt_project=DbtProjectConfig(
        dbt_project_name='my_dbt_project',
        dbt_project_path='/path/to/my_dbt_project',
        dbt_models_path='/path/to/my_dbt_project/models',
        dbt_profiles_path='/path/to/my_dbt_project',
        dbt_target_path='/path/to/my_dbt_project/target',
        dbt_log_path='/path/to/my_dbt_project/logs',
        dbt_schema='my_dbt_schema',
    ),
    dbt_default_targets=DbtDefaultTargetsConfig(default_target='dev'),
    dry_run=False,  # set to True if you want to turn on dry-run mode
)

dags = compile_dmp_af_dags(
    manifest_path='/path/to/my_dbt_project/target/manifest.json',
    config=config,
)
for dag_name, dag in dags.items():
    globals()[dag_name] = dag
```

In _dbt_project.yml_ you need to set up default targets for all nodes in your project
(see [example](examples/dags/dbt_project.yml)):

```yaml
sql_cluster: "dev"
daily_sql_cluster: "dev"
py_cluster: "dev"
bf_cluster: "dev"
```

This will create Airflow DAGs for your dbt project.

Check out the documentation for more details [here](docs/docs.md).

## Key Features

**Auto-generated DAGs**

- Automatically creates Airflow DAGs from your dbt project
- Organizes by domain and schedule
- Handles dependencies across domains

**Idempotent runs**

- Each model is a separate Airflow task
- Date intervals passed to every run
- Reliable backfills and reruns

**Team-friendly**

- Analytics teams stay in dbt
- No Airflow DAG writing required
- Infrastructure handled automatically

## Requirements

`dmp-af` is tested with:

| Airflow version | Python versions | `dbt-core` versions |
|-----------------|-----------------|---------------------|
| 2.6.3           | >=3.10,<3.12    | >=1.7,<=1.10        |
| 2.7.3           | >=3.10,<3.12    | >=1.7,<=1.10        |
| 2.8.4           | >=3.10,<3.12    | >=1.7,<=1.10        |
| 2.9.3           | >=3.10,<3.13    | >=1.7,<=1.10        |
| 2.10.5          | >=3.10,<3.13    | >=1.7,<=1.10        |
| 2.11.0          | >=3.10,<3.13    | >=1.7,<=1.10        |
| 3.0.6           | ≥3.10,<3.13     | ≥1.7,≤1.10          |

## Project Information

- [Docs](https://docs.dmp.af)
- [PyPI](https://pypi.org/project/dmp-af/)
- [Contributing](CONTRIBUTING.md)
- [Original dbt-af Project](https://github.com/Toloka/dbt-af)

## About this fork

This project is a fork of [Toloka AI BV's original repository](https://github.com/Toloka/dbt-af).
It includes substantial modifications by IJKOS & PARTNERS LTD.
This fork is not affiliated with or endorsed by Toloka AI BV.

The original project is licensed under the [Apache License 2.0](./LICENSE).

### Migrating from dbt-af

If you're currently using dbt-af and want to migrate to dmp-af, see our [Migration Guide](MIGRATION.md) for step-by-step
instructions.
