Metadata-Version: 2.4
Name: shift_left
Version: 0.1.50
Summary: The shift-left CLI to manage your shift-left project
Project-URL: Homepage, https://jbcodeforce.github.io/shift_left_utils/
Project-URL: Repository, https://github.com/jbcodeforce/shift_left_utils
Project-URL: Documentation, https://jbcodeforce.github.io/shift_left_utils/
Project-URL: Issues, https://github.com/jbcodeforce/shift_left_utils/issues
Author-email: jbcodeforce <jerome.boyer@gmail.com>
License: Apache-2.0
Keywords: cli,confluent,data-pipeline,flink,kafka,migration,sql,streaming
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Code Generators
Requires-Python: <3.13,>=3.12
Requires-Dist: click<8.2.0
Requires-Dist: confluent-kafka>=2.10.0
Requires-Dist: jinja2==3.1.6
Requires-Dist: matplotlib>=3.10.1
Requires-Dist: mcp>=1.16.0
Requires-Dist: networkx>=3.4.2
Requires-Dist: openai>=1.97.1
Requires-Dist: pydantic-yaml>=1.4.0
Requires-Dist: pydantic>=2.6.10
Requires-Dist: pytz>=2025.2
Requires-Dist: pyvis>=0.3.2
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: requests>=2.32.3
Requires-Dist: rich>=13.9.4
Requires-Dist: sqlglot>=27.19.0
Requires-Dist: toml>=0.10.2
Requires-Dist: typer>=0.15.2
Provides-Extra: dev
Requires-Dist: pytest>=8.3.5; extra == 'dev'
Provides-Extra: rag
Requires-Dist: chromadb>=0.4.0; extra == 'rag'
Requires-Dist: sentence-transformers>=2.2.0; extra == 'rag'
Description-Content-Type: text/markdown

# Shift_left

CLI to manage Flink projects and migrate batch SQL pipelines to real-time Apache Flink on Confluent Cloud. Supports KSQL, Spark SQL, and dbt as sources.

**Documentation:** [https://jbcodeforce.github.io/shift_left_utils/](https://jbcodeforce.github.io/shift_left_utils/)

## Install

```bash
pip install shift_left
```

## Major features


- **Project management**: Initialize Flink projects (Kimball or data-product layout), validate config, list Kafka topics and Flink compute pools, and report cross-product or single-child tables.
- **Pipeline deployment**: Build table inventory and pipeline metadata, deploy DDL/DML with execution planning. Optional parallel deploy, dml-only, and blue-green via modified-files tracking.
- **Table lifecycle**: Create table scaffolding, run AI migration per table (with optional recursion), and manage unit tests (init, run, delete) against Confluent Cloud.
- **SQL migration**: Migrate KSQL, Spark SQL, and dbt to Flink SQL using LLM-based agents (OpenAI-compatible, including local Ollama). Handles table detection, DDL creation, and translation with validation.
- **Unit test harness**: Develop synthetic test data from SQL DML semantic, with test suite and test case metadata to help Data Engineers to unit test their SQL on Confluent Cloud.
- **Blue-green and operations**: List modified files by branch for safe deployments, integrate with your CI/CD and Confluent Cloud APIs.

## Quick start

```bash
# Initialize a Flink project
shift_left project init my-flink-project /path/to/project --project-type data_product

# Validate Confluent Cloud config
shift_left project validate-config

# Migrate a table from Spark SQL to Flink SQL (one time work)
shift_left table migrate my_table /path/to/script.sql /path/to/staging --source-type spark --recursive

# Build pipeline execution plan
shift_left table build-inventory
shift_left pipeline build-all-metadata
shift_left pipeline build-execution-plan  /path/to/inventor --table-name my_table --may-start-de

# Deploy pipelines from inventory
shift_left pipeline deploy /path/to/inventory --table-name my_table
```

## Documentation

Full guides, tutorials, CLI reference, and MCP server setup:

**[https://jbcodeforce.github.io/shift_left_utils/](https://jbcodeforce.github.io/shift_left_utils/)**

- User guide: environment setup, recipes, pipeline management, blue-green deployment
- Tutorials: setup, project management, migration with AI
- CLI reference and MCP (Model Context Protocol) integration for Cursor and other tools

## Requirements

- Python 3.12
- Confluent Cloud (Kafka and Flink). For AI migration: OpenAI-compatible API or local Ollama.

## License

Apache-2.0
