Metadata-Version: 2.1
Name: sqlmesh
Version: 0.123.5
Home-page: https://github.com/TobikoData/sqlmesh
Author: TobikoData Inc.
Author-email: engineering@tobikodata.com
License: Apache License 2.0
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: SQL
Classifier: Programming Language :: Python :: 3 :: Only
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: astor
Requires-Dist: click
Requires-Dist: croniter
Requires-Dist: duckdb!=0.10.3
Requires-Dist: dateparser
Requires-Dist: freezegun
Requires-Dist: hyperscript>=0.1.0
Requires-Dist: ipywidgets
Requires-Dist: jinja2
Requires-Dist: pandas
Requires-Dist: pydantic
Requires-Dist: requests
Requires-Dist: rich[jupyter]
Requires-Dist: ruamel.yaml
Requires-Dist: sqlglot[rs]~=25.21.3
Requires-Dist: importlib-metadata; python_version < "3.12"
Provides-Extra: bigquery
Requires-Dist: google-cloud-bigquery[pandas]; extra == "bigquery"
Requires-Dist: google-cloud-bigquery-storage; extra == "bigquery"
Provides-Extra: cicdtest
Requires-Dist: dbt-databricks; extra == "cicdtest"
Requires-Dist: dbt-redshift; extra == "cicdtest"
Requires-Dist: dbt-sqlserver>=1.7.0; extra == "cicdtest"
Requires-Dist: dbt-trino; extra == "cicdtest"
Provides-Extra: clickhouse
Requires-Dist: clickhouse-connect; extra == "clickhouse"
Provides-Extra: databricks
Requires-Dist: databricks-sql-connector; extra == "databricks"
Provides-Extra: dbt
Requires-Dist: dbt-core<2; extra == "dbt"
Provides-Extra: dev
Requires-Dist: apache-airflow==2.9.1; extra == "dev"
Requires-Dist: agate==1.7.1; extra == "dev"
Requires-Dist: beautifulsoup4; extra == "dev"
Requires-Dist: ruff~=0.4.0; extra == "dev"
Requires-Dist: cryptography~=42.0.4; extra == "dev"
Requires-Dist: dbt-core; extra == "dev"
Requires-Dist: dbt-duckdb>=1.7.1; extra == "dev"
Requires-Dist: dbt-snowflake; extra == "dev"
Requires-Dist: dbt-bigquery; extra == "dev"
Requires-Dist: Faker; extra == "dev"
Requires-Dist: google-auth; extra == "dev"
Requires-Dist: google-cloud-bigquery; extra == "dev"
Requires-Dist: google-cloud-bigquery-storage; extra == "dev"
Requires-Dist: mypy~=1.10.0; extra == "dev"
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: pandas-stubs; extra == "dev"
Requires-Dist: psycopg2-binary; extra == "dev"
Requires-Dist: pydantic<2.6.0; extra == "dev"
Requires-Dist: PyGithub; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-asyncio<0.23.0; extra == "dev"
Requires-Dist: pytest-mock; extra == "dev"
Requires-Dist: pytest-xdist; extra == "dev"
Requires-Dist: pyspark~=3.5.0; extra == "dev"
Requires-Dist: pytz; extra == "dev"
Requires-Dist: snowflake-connector-python[pandas,secure-local-storage]>=3.0.2; extra == "dev"
Requires-Dist: sqlalchemy-stubs; extra == "dev"
Requires-Dist: tenacity==8.1.0; extra == "dev"
Requires-Dist: types-croniter; extra == "dev"
Requires-Dist: types-dateparser; extra == "dev"
Requires-Dist: types-python-dateutil; extra == "dev"
Requires-Dist: types-pytz; extra == "dev"
Requires-Dist: types-requests==2.28.8; extra == "dev"
Requires-Dist: typing-extensions; extra == "dev"
Requires-Dist: custom-materializations; extra == "dev"
Requires-Dist: pytest-retry; python_version >= "3.9" and extra == "dev"
Provides-Extra: gcppostgres
Requires-Dist: cloud-sql-python-connector[pg8000]; extra == "gcppostgres"
Provides-Extra: github
Requires-Dist: PyGithub; extra == "github"
Provides-Extra: llm
Requires-Dist: langchain; extra == "llm"
Requires-Dist: openai; extra == "llm"
Provides-Extra: mssql
Requires-Dist: pymssql; extra == "mssql"
Provides-Extra: mwaa
Requires-Dist: boto3; extra == "mwaa"
Provides-Extra: mysql
Requires-Dist: mysql-connector-python; extra == "mysql"
Provides-Extra: postgres
Requires-Dist: psycopg2; extra == "postgres"
Provides-Extra: redshift
Requires-Dist: redshift-connector; extra == "redshift"
Provides-Extra: slack
Requires-Dist: slack-sdk; extra == "slack"
Provides-Extra: snowflake
Requires-Dist: cryptography~=42.0.4; extra == "snowflake"
Requires-Dist: snowflake-connector-python[pandas,secure-local-storage]; extra == "snowflake"
Requires-Dist: snowflake-snowpark-python; python_version < "3.12" and extra == "snowflake"
Provides-Extra: trino
Requires-Dist: trino; extra == "trino"
Provides-Extra: web
Requires-Dist: fastapi==0.110.2; extra == "web"
Requires-Dist: watchfiles>=0.19.0; extra == "web"
Requires-Dist: uvicorn[standard]==0.22.0; extra == "web"
Requires-Dist: sse-starlette>=0.2.2; extra == "web"
Requires-Dist: pyarrow; extra == "web"

<p align="center">
  <img src="https://github.com/TobikoData/sqlmesh/blob/main/docs/readme/sqlmesh.png?raw=true" alt="SQLMesh logo" width="50%" height="50%">
</p>

SQLMesh is a next-generation data transformation and modeling framework that is backwards compatible with dbt. It aims to be easy to use, correct, and efficient.

SQLMesh enables data teams to efficiently run and deploy data transformations written in SQL or Python.

It is more than just a [dbt alternative](https://tobikodata.com/reduce_costs_with_cron_and_partitions.html).

<p align="center">
  <img src="https://github.com/TobikoData/sqlmesh/blob/main/docs/readme/architecture_diagram.png?raw=true" alt="Architecture Diagram" width="100%" height="100%">
</p>

## Core Features
<img src="https://github.com/TobikoData/sqlmesh-public-assets/blob/main/sqlmesh_plan_mode.gif?raw=true" alt="SQLMesh Plan Mode">

> Get instant SQL impact analysis of your changes, whether in the CLI or in [SQLMesh Plan Mode](https://sqlmesh.readthedocs.io/en/stable/guides/ui/?h=modes#working-with-an-ide)

  <details>
  <summary><b>Virtual Data Environments</b></summary>

  * See a full diagram of how [Virtual Data Environments](https://whimsical.com/virtual-data-environments-MCT8ngSxFHict4wiL48ymz) work
  * [Watch this video to learn more](https://www.youtube.com/watch?v=weJH3eM0rzc)

  </details>

  * Plan / Apply workflow like [Terraform](https://www.terraform.io/) to understand potential impact of changes
  * Automatic [column level lineage](https://sqlmesh.readthedocs.io/en/stable/guides/ui/?h=column+lineage#lineage-module) and data contracts
  * Easy to use [CI/CD bot](https://sqlmesh.readthedocs.io/en/stable/integrations/github/)

<details>
<summary><b>Efficiency and Testing</b></summary>

Running this command will generate a unit test file in the `tests/` folder: `test_stg_payments.yaml`

Runs a live query to generate the expected output of the model

```bash
sqlmesh create_test tcloud_demo.stg_payments --query tcloud_demo.seed_raw_payments "select * from tcloud_demo.seed_raw_payments limit 5"

# run the unit test
sqlmesh test
```

```sql
MODEL (
  name tcloud_demo.stg_payments,
  cron '@daily',
  grain payment_id,
  audits (UNIQUE_VALUES(columns = (
      payment_id
  )), NOT_NULL(columns = (
      payment_id
  )))
);

SELECT
    id AS payment_id,
    order_id,
    payment_method,
    amount / 100 AS amount, /* `amount` is currently stored in cents, so we convert it to dollars */
    'new_column' AS new_column, /* non-breaking change example  */
FROM tcloud_demo.seed_raw_payments
```

```yaml
test_stg_payments:
model: tcloud_demo.stg_payments
inputs:
    tcloud_demo.seed_raw_payments:
    - id: 66
    order_id: 58
    payment_method: coupon
    amount: 1800
    - id: 27
    order_id: 24
    payment_method: coupon
    amount: 2600
    - id: 30
    order_id: 25
    payment_method: coupon
    amount: 1600
    - id: 109
    order_id: 95
    payment_method: coupon
    amount: 2400
    - id: 3
    order_id: 3
    payment_method: coupon
    amount: 100
outputs:
    query:
    - payment_id: 66
    order_id: 58
    payment_method: coupon
    amount: 18.0
    new_column: new_column
    - payment_id: 27
    order_id: 24
    payment_method: coupon
    amount: 26.0
    new_column: new_column
    - payment_id: 30
    order_id: 25
    payment_method: coupon
    amount: 16.0
    new_column: new_column
    - payment_id: 109
    order_id: 95
    payment_method: coupon
    amount: 24.0
    new_column: new_column
    - payment_id: 3
    order_id: 3
    payment_method: coupon
    amount: 1.0
    new_column: new_column
```
</details>

* Never builds a table [more than once](https://tobikodata.com/simplicity-or-efficiency-how-dbt-makes-you-choose.html)
* Partition-based [incremental models](https://tobikodata.com/correctly-loading-incremental-data-at-scale.html)
* [Unit tests](https://tobikodata.com/we-need-even-greater-expectations.html) and audits

<details>
<summary><b>Take SQL Anywhere</b></summary>
Write SQL in any dialect and SQLMesh will transpile it to your target SQL dialect on the fly before sending it to the warehouse.
<img src="https://github.com/TobikoData/sqlmesh/blob/main/docs/readme/transpile_example.png?raw=true" alt="Transpile Example">
</details>

* Compile time error checking and can transpile [10+ different SQL dialects](https://sqlmesh.readthedocs.io/en/stable/integrations/overview/#execution-engines)
* Definitions using [simply SQL](https://sqlmesh.readthedocs.io/en/stable/concepts/models/sql_models/#sql-based-definition) (no need for redundant and confusing Jinja + YAML)
* [Self documenting queries](https://tobikodata.com/metadata-everywhere.html) using native SQL Comments

For more information, check out the [website](https://sqlmesh.com) and [documentation](https://sqlmesh.readthedocs.io/en/stable/).

## Getting Started
Install SQLMesh through [pypi](https://pypi.org/project/sqlmesh/) by running:

```bash
mkdir sqlmesh-example
cd sqlmesh-example
python -m venv .env
source .env/bin/activate
pip install sqlmesh
sqlmesh init duckdb # get started right away with a local duckdb instance
```

Follow the [quickstart guide](https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/#1-create-the-sqlmesh-project) to learn how to use SQLMesh. You already have a head start!

## Join Our Community
We want to ship better data with you. Connect with us in the following ways:

* Join the [Tobiko Slack Community](https://tobikodata.com/slack) to ask questions, or just to say hi!
* File an issue on our [GitHub](https://github.com/TobikoData/sqlmesh/issues/new)
* Send us an email at [hello@tobikodata.com](mailto:hello@tobikodata.com) with your questions or feedback
* Read our [blog](https://tobikodata.com/blog)

## Contribution
Contributions in the form of issues or pull requests are greatly appreciated. [Read more](https://sqlmesh.readthedocs.io/en/stable/development/) on how to contribute to SQLMesh open source.
