# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['airflow_dbt_python',
 'airflow_dbt_python.hooks',
 'airflow_dbt_python.operators']

package_data = \
{'': ['*']}

install_requires = \
['dbt-core>=1.0,<2.0']

extras_require = \
{'adapters': ['dbt-postgres>=1.0,<2.0',
              'dbt-redshift>=1.0,<2.0',
              'dbt-snowflake>=1.0,<2.0',
              'dbt-bigquery>=1.0,<2.0'],
 'airflow': ['apache-airflow>=1.10.12,<3.0.0'],
 'amazon': ['apache-airflow-providers-amazon>=2,<3'],
 'bigquery': ['dbt-bigquery>=1.0,<2.0'],
 'docs': ['Sphinx==4.2.0',
          'sphinx-rtd-theme==1.0.0',
          'sphinxcontrib-napoleon==0.7'],
 'postgres': ['dbt-postgres>=1.0,<2.0'],
 'redshift': ['dbt-redshift>=1.0,<2.0'],
 'snowflake': ['dbt-snowflake>=1.0,<2.0']}

setup_kwargs = {
    'name': 'airflow-dbt-python',
    'version': '0.12.0',
    'description': 'A dbt operator and hook for Airflow',
    'long_description': '# airflow-dbt-python\n\n[![PyPI version](https://img.shields.io/pypi/v/airflow-dbt-python?style=plastic)](https://pypi.org/project/airflow-dbt-python/)\n[![GitHub build status](https://github.com/tomasfarias/airflow-dbt-python/actions/workflows/test.yaml/badge.svg)](https://github.com/tomasfarias/airflow-dbt-python/actions)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Test coverage](https://codecov.io/gh/tomasfarias/airflow-dbt-python/branch/master/graph/badge.svg?token=HBKZ78F11F)](https://codecov.io/gh/tomasfarias/airflow-dbt-python)\n\nAn [Airflow](https://airflow.apache.org/) operator and hook to interface with the [`dbt-core`](https://pypi.org/project/dbt-core/) Python package.\n\n# Motivation\n\n## Airflow running in a managed environment\n\nAlthough [`dbt`](https://docs.getdbt.com/) is meant to be installed and used as a CLI, we may not have control of the environment where Airflow is running, disallowing us the option of using `dbt` as a CLI.\n\nThis is exactly what happens when using [Amazon\'s Managed Workflows for Apache Airflow](https://aws.amazon.com/managed-workflows-for-apache-airflow/) or MWAA: although a list of Python requirements can be passed, the CLI cannot be found in the worker\'s PATH.\n\nThere is a workaround which involves using Airflow\'s `BashOperator` and running Python from the command line:\n\n``` python\nfrom airflow.operators.bash import BashOperator\n\nBASH_COMMAND = "python -c \'from dbt.main import main; main()\' run"\noperator = BashOperator(\n    task_id="dbt_run",\n    bash_command=BASH_COMMAND,\n)\n```\n\nBut it can get sloppy when appending all potential arguments a `dbt run` command (or other subcommand) can take.\n\nThat\'s where `airflow-dbt-python` comes in: it abstracts the complexity of interfacing with `dbt-core` and exposes one operator for each `dbt` subcommand that can be instantiated with all the corresponding arguments that the `dbt` CLI would take.\n\n## An alternative to `airflow-dbt` that works without the dbt CLI\n\nThe alternative [`airflow-dbt`](https://pypi.org/project/airflow-dbt/) package, by default, would not work if the `dbt` CLI is not in PATH, which means it would not be usable in MWAA. There is a workaround via the `dbt_bin` argument, which can be set to `"python -c \'from dbt.main import main; main()\' run"`, in similar fashion as the `BashOperator` example. Yet this approach is not without its limitations:\n* `airflow-dbt` works by wrapping the `dbt` CLI, which makes our code dependent on the environment in which it runs.\n* `airflow-dbt` does not support the full range of arguments a command can take. For example, `DbtRunOperator` does not have an attribute for `fail_fast`.\n* `airflow-dbt` does not offer access to `dbt` artifacts created during execution. `airflow-dbt-python` does so by pushing any artifacts to [XCom](https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html).\n\n# Additional features\n\nBesides running `dbt` as one would do if doing so manually, `airflow-dbt-python` also supports a few additional features to bring `dbt` closer to being a first-class citizen of Airflow.\n\n## Download dbt projects from S3\n\nThe arguments `profiles_dir` and `project_dir` would normally point to a directory containing a `profiles.yml` file and a dbt project in the local environment respectively. `airflow-dbt-python` extends these arguments to also take an [AWS S3](https://aws.amazon.com/s3/) URL (identified by an `s3://` scheme):\n\n* If an S3 URL is used for `profiles_dir`, then this URL must point to a directory in S3 that contains a `profiles.yml` file. The `profiles.yml` file will be downloaded and made available for the operator to use when running.\n* If an S3 URL is used for `project_dir`, then this URL must point to a directory in S3 containing all the files required for a `dbt` project to run. All of the contents of this directory will be downloaded and made available for the operator. The URL may also point to a zip file containing all the files of a `dbt` project, which will be downloaded, uncompressed, and made available for the operator.\n\nThis feature is intended to work in line with Airflow\'s [description of the task concept](https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html#relationships):\n\n> Tasks don’t pass information to each other by default, and run entirely independently.\n\nIn our world, that means task should be responsible of fetching all the `dbt` related files it needs in order to run independently. This is particularly relevant for an Airflow deployment with a [remote executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html#executor-types), as Airflow does not guarantee which worker will run a particular task.\n\n## Push dbt artifacts to XCom\n\nEach `dbt` execution produces several JSON [artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts/) that may be valuable to obtain metrics, build conditional workflows, for reporting purposes, or other uses. `airflow-dbt-python` can push these artifacts to XCom as requested by exposing a `do_xcom_push_artifacts` argument, which takes a list of artifacts to push. This way, artifacts may be pulled and operated on by downstream tasks. For example:\n\n``` python\nwith DAG(\n    dag_id="example_dbt_artifacts",\n    schedule_interval="0 0 * * *",\n    start_date=days_ago(1),\n    catchup=False,\n    dagrun_timeout=dt.timedelta(minutes=60),\n) as dag:\n    dbt_run = DbtRunOperator(\n        task_id="dbt_run_daily",\n        project_dir="/path/to/my/dbt/project/",\n        profiles_dir="~/.dbt/",\n        select=["+tag:daily"],\n        exclude=["tag:deprecated"],\n        target="production",\n        profile="my-project",\n        full_refresh=True,\n        do_xcom_push_artifacts=["manifest.json", "run_results.json"],\n    )\n\n    process_artifacts = PythonOperator(\n        task_id="process_artifacts",\n        python_callable=process_dbt_artifacts,\n        provide_context=True,\n    )\n    dbt_run >> process_artifacts\n```\n\nSee the full example [here](examples/use_dbt_artifacts_dag.py).\n\n# Usage\n\nCurrently, the following `dbt` commands are supported:\n\n* `clean`\n* `compile`\n* `debug`\n* `deps`\n* `docs generate`\n* `ls`\n* `parse`\n* `run`\n* `run-operation`\n* `seed`\n* `snapshot`\n* `source`\n* `test`\n\n## Examples\n\n``` python\nfrom datetime import timedelta\n\nfrom airflow import DAG\nfrom airflow.utils.dates import days_ago\nfrom airflow_dbt_python.operators.dbt import (\n    DbtRunOperator,\n    DbtSeedOperator,\n    DbtTestoperator,\n)\n\nargs = {\n    \'owner\': \'airflow\',\n}\n\nwith DAG(\n    dag_id=\'example_dbt_operator\',\n    default_args=args,\n    schedule_interval=\'0 0 * * *\',\n    start_date=days_ago(2),\n    dagrun_timeout=timedelta(minutes=60),\n    tags=[\'example\', \'example2\'],\n) as dag:\n    dbt_test = DbtTestOperator(\n        task_id="dbt_test",\n        selector_name=["pre-run-tests"],\n    )\n\n    dbt_seed = DbtSeedOperator(\n        task_id="dbt_seed",\n        select=["/path/to/first.csv", "/path/to/second.csv"],\n        full_refresh=True,\n    )\n\n    dbt_run = DbtRunOperator(\n        task_id="dbt_run",\n        select=["/path/to/models"],\n        full_refresh=True,\n        fail_fast=True,\n    )\n\n    dbt_test >> dbt_seed >> dbt_run\n```\n\nMore examples can be found in the [`examples/`](examples/) directory.\n\n# Requirements\n\nTo line up with `dbt-core`, `airflow-dbt-python` supports Python 3.7, 3.8, and 3.9. We also include Python 3.10 in our testing pipeline, although `dbt-core` does not yet support it.\n\nOn the Airflow side, we unit test with version 1.10.12 and the latest version 2 release.\n\nFinally, `airflow-dbt-python` requires at least `dbt-core` version 1.0.0. Since `dbt-core` follows [semantic versioning](https://semver.org/), we do not impose any restrictions on the minor and patch versions, but do keep in mind that the latest `dbt-core` features incorporated as minor releases may not yet be supported.\n\n# Installing\n\n## From PyPI:\n\n``` shell\npip install airflow-dbt-python\n```\n\nAny `dbt` adapters you require may be installed by specifying extras:\n\n``` shell\npip install airflow-dby-python[snowflake,postgres]\n```\n\n## From this repo:\n\nClone the repo:\n``` shell\ngit clone https://github.com/tomasfarias/airflow-dbt-python.git\ncd airflow-dbt-python\n```\n\nWith poetry:\n``` shell\npoetry install\n```\n\nInstall any extras you need, and only those you need:\n``` shell\npoetry install -E postgres -E redshift\n```\n\n## In MWAA:\n\nAdd `airflow-dbt-python` to your `requirements.txt` file and edit your Airflow environment to use this new `requirements.txt` file.\n\n# Testing\n\nTests are written using `pytest`, can be located in `tests/`, and they can be run locally with `poetry`:\n\n``` shell\npoetry run pytest tests/ -vv\n```\n\n# License\n\nThis project is licensed under the MIT license. See ![LICENSE](LICENSE).\n',
    'author': 'Tomás Farías Santana',
    'author_email': 'tomas@tomasfarias.dev',
    'maintainer': None,
    'maintainer_email': None,
    'url': 'https://github.com/tomasfarias/airflow-dbt-python',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'extras_require': extras_require,
    'python_requires': '>=3.7,<4.0',
}


setup(**setup_kwargs)
