Metadata-Version: 2.1
Name: airbyte-cdk
Version: 0.79.2
Summary: A framework for writing Airbyte Connectors.
Home-page: https://github.com/airbytehq/airbyte
License: MIT
Keywords: airbyte,connector-development-kit,cdk
Author: Airbyte
Author-email: contact@airbyte.io
Requires-Python: >=3.9,<4.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.8
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Provides-Extra: file-based
Provides-Extra: sphinx-docs
Provides-Extra: vector-db-based
Requires-Dist: Deprecated (>=1.2,<1.3)
Requires-Dist: Jinja2 (>=3.1.2,<3.2.0)
Requires-Dist: PyYAML (>=6.0.1,<7.0.0)
Requires-Dist: Sphinx (>=4.2,<4.3) ; extra == "sphinx-docs"
Requires-Dist: airbyte-protocol-models (==0.5.1)
Requires-Dist: avro (>=1.11.2,<1.12.0) ; extra == "file-based"
Requires-Dist: backoff
Requires-Dist: cachetools
Requires-Dist: cohere (==4.21) ; extra == "vector-db-based"
Requires-Dist: dpath (>=2.0.1,<2.1.0)
Requires-Dist: fastavro (>=1.8.0,<1.9.0) ; extra == "file-based"
Requires-Dist: genson (==1.2.2)
Requires-Dist: isodate (>=0.6.1,<0.7.0)
Requires-Dist: jsonref (>=0.2,<0.3)
Requires-Dist: jsonschema (>=3.2.0,<3.3.0)
Requires-Dist: langchain (==0.0.271) ; extra == "vector-db-based"
Requires-Dist: markdown ; extra == "file-based"
Requires-Dist: openai[embeddings] (==0.27.9) ; extra == "vector-db-based"
Requires-Dist: pdf2image (==1.16.3) ; extra == "file-based"
Requires-Dist: pdfminer.six (==20221105) ; extra == "file-based"
Requires-Dist: pendulum (<3.0.0)
Requires-Dist: pyarrow (>=15.0.0,<15.1.0) ; extra == "file-based"
Requires-Dist: pydantic (>=1.10.8,<2.0.0)
Requires-Dist: pyrate-limiter (>=3.1.0,<3.2.0)
Requires-Dist: pytesseract (==0.3.10) ; extra == "file-based"
Requires-Dist: python-dateutil
Requires-Dist: requests
Requires-Dist: requests_cache
Requires-Dist: sphinx-rtd-theme (>=1.0,<1.1) ; extra == "sphinx-docs"
Requires-Dist: tiktoken (==0.4.0) ; extra == "vector-db-based"
Requires-Dist: unstructured.pytesseract (>=0.3.12) ; extra == "file-based"
Requires-Dist: unstructured[docx,pptx] (==0.10.27) ; extra == "file-based"
Requires-Dist: wcmatch (==8.4)
Project-URL: Documentation, https://docs.airbyte.io/
Project-URL: Repository, https://github.com/airbytehq/airbyte
Description-Content-Type: text/markdown

# Connector Development Kit \(Python\)

The Airbyte Python CDK is a framework for rapidly developing production-grade Airbyte connectors.The CDK currently offers helpers specific for creating Airbyte source connectors for:

- HTTP APIs \(REST APIs, GraphQL, etc..\)
- Generic Python sources \(anything not covered by the above\)

The CDK provides an improved developer experience by providing basic implementation structure and abstracting away low-level glue boilerplate.

This document is a general introduction to the CDK. Readers should have basic familiarity with the [Airbyte Specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/) before proceeding.

# Setup

## Prerequisites

#### Poetry

Before you can start working on this project, you will need to have Poetry installed on your system. Please follow the instructions below to install Poetry:

1. Open your terminal or command prompt.
2. Install Poetry using the recommended installation method:

```bash
curl -sSL https://install.python-poetry.org | POETRY_VERSION=1.5.1 python3 -
```

Alternatively, you can use `pip` to install Poetry:

```bash
pip install --user poetry
```

3. After the installation is complete, close and reopen your terminal to ensure the newly installed `poetry` command is available in your system's PATH.

For more detailed instructions and alternative installation methods, please refer to the official Poetry documentation: https://python-poetry.org/docs/#installation

### Concepts & Documentation

See the [concepts docs](docs/concepts/) for a tour through what the API offers.

### Example Connectors

**HTTP Connectors**:

- [Stripe](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/source.py)
- [Slack](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-slack/source_slack/source.py)

**Simple Python connectors using the bare-bones `Source` abstraction**:

- [Google Sheets](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-sheets/google_sheets_source/google_sheets_source.py)
- [Mailchimp](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mailchimp/source_mailchimp/source.py)

## Contributing

### First time setup

Install the project dependencies and development tools:

```bash
poetry install --all-extras
```

Installing all extras is required to run the full suite of unit tests.

#### Iteration

- Iterate on the CDK code locally
- Run tests via `poetry run poe unit-test-with-cov`, or `python -m pytest -s unit_tests` if you want to pass pytest options.
- Run `poetry run poe check-local` to lint all code, type-check modified code, and run unit tests with coverage in one command.

To see all available scripts, run `poetry run poe`.

##### Autogenerated files

If the iteration you are working on includes changes to the models or the connector generator, you might want to regenerate them. In order to do that, you can run:

```bash
poetry run poe build
```

This will generate the code generator docker image and the component manifest files based on the schemas and templates.

#### Testing

All tests are located in the `unit_tests` directory. Run `poetry run poe unit-test-with-cov` to run them. This also presents a test coverage report. For faster iteration with no coverage report and more options, `python -m pytest -s unit_tests` is a good place to start.

#### Building and testing a connector with your local CDK

When developing a new feature in the CDK, you may find it helpful to run a connector that uses that new feature. You can test this in one of two ways:

- Running a connector locally
- Building and running a source via Docker

##### Installing your local CDK into a local Python connector

In order to get a local Python connector running your local CDK, do the following.

First, make sure you have your connector's virtual environment active:

```bash
# from the `airbyte/airbyte-integrations/connectors/<connector-directory>` directory
source .venv/bin/activate

# if you haven't installed dependencies for your connector already
pip install -e .
```

Then, navigate to the CDK and install it in editable mode:

```bash
cd ../../../airbyte-cdk/python
pip install -e .
```

You should see that `pip` has uninstalled the version of `airbyte-cdk` defined by your connector's `setup.py` and installed your local CDK. Any changes you make will be immediately reflected in your editor, so long as your editor's interpreter is set to your connector's virtual environment.

##### Building a Python connector in Docker with your local CDK installed

_Pre-requisite: Install the [`airbyte-ci` CLI](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)_

You can build your connector image with the local CDK using

```bash
# from the airbytehq/airbyte base directory
airbyte-ci connectors --use-local-cdk --name=<CONNECTOR> build
```

Note that the local CDK is injected at build time, so if you make changes, you will have to run the build command again to see them reflected.

##### Running Connector Acceptance Tests for a single connector in Docker with your local CDK installed

_Pre-requisite: Install the [`airbyte-ci` CLI](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)_

To run acceptance tests for a single connectors using the local CDK, from the connector directory, run

```bash
airbyte-ci connectors --use-local-cdk --name=<CONNECTOR> test
```

#### When you don't have access to the API

There may be a time when you do not have access to the API (either because you don't have the credentials, network access, etc...) You will probably still want to do end-to-end testing at least once. In order to do so, you can emulate the server you would be reaching using a server stubbing tool.

For example, using [mockserver](https://www.mock-server.com/), you can set up an expectation file like this:

```json
{
  "httpRequest": {
    "method": "GET",
    "path": "/data"
  },
  "httpResponse": {
    "body": "{\"data\": [{\"record_key\": 1}, {\"record_key\": 2}]}"
  }
}
```

Assuming this file has been created at `secrets/mock_server_config/expectations.json`, running the following command will allow to match any requests on path `/data` to return the response defined in the expectation file:

```bash
docker run -d --rm -v $(pwd)/secrets/mock_server_config:/config -p 8113:8113 --env MOCKSERVER_LOG_LEVEL=TRACE --env MOCKSERVER_SERVER_PORT=8113 --env MOCKSERVER_WATCH_INITIALIZATION_JSON=true --env MOCKSERVER_PERSISTED_EXPECTATIONS_PATH=/config/expectations.json --env MOCKSERVER_INITIALIZATION_JSON_PATH=/config/expectations.json mockserver/mockserver:5.15.0
```

HTTP requests to `localhost:8113/data` should now return the body defined in the expectations file. To test this, the implementer either has to change the code which defines the base URL for Python source or update the `url_base` from low-code. With the Connector Builder running in docker, you will have to use domain `host.docker.internal` instead of `localhost` as the requests are executed within docker.

#### Publishing a new version to PyPi

1. Open a PR
2. Once it is approved and **merged**, an Airbyte member must run the `Publish CDK Manually` workflow from master using `release-type=major|manor|patch` and setting the changelog message.

