Metadata-Version: 2.4
Name: pytest-openapi
Version: 0.1.5
Summary: `pytest --openapi` - an opinionated, lightweight black-box contract tester against a live API using its OpenAPI specification as the source of truth
Author-email: Sinan Ozel <coding@sinan.slmail.me>
License: MIT License
        
        Copyright (c) 2025 Sinan Ozel
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/sinan-ozel/pytest-openapi
Project-URL: Issues, https://github.com/sinan-ozel/pytest-openapi/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pytest>=7.0.0
Requires-Dist: requests>=2.31.0
Requires-Dist: exrex>=0.11.0
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-depends>=1.0.1; extra == "test"
Requires-Dist: pytest-mock>=3.14.0; extra == "test"
Requires-Dist: httpx>=0.28.1; extra == "test"
Provides-Extra: dev
Requires-Dist: isort>=5.12.0; extra == "dev"
Requires-Dist: ruff>=0.12.11; extra == "dev"
Requires-Dist: black>=24.0.0; extra == "dev"
Requires-Dist: docformatter>=1.7.5; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.0.0; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
Requires-Dist: mike>=2.0.0; extra == "docs"
Provides-Extra: publish
Requires-Dist: packaging>=25.0; extra == "publish"
Dynamic: license-file

![Tests & Lint](https://github.com/sinan-ozel/pytest-openapi/actions/workflows/ci.yaml/badge.svg?branch=main)
![PyPI](https://img.shields.io/pypi/v/pytest-openapi.svg)
![Downloads](https://static.pepy.tech/badge/pytest-openapi)
![Monthly Downloads](https://static.pepy.tech/badge/pytest-openapi/month)
[![Documentation](https://img.shields.io/badge/docs-mkdocs-blue)](https://sinan-ozel.github.io/pytest-openapi/)
![License](https://img.shields.io/github/license/sinan-ozel/pytest-openapi.svg)
![Python](https://img.shields.io/badge/Python-3.11%2B-blue)
![Made with Love](https://img.shields.io/badge/Made%20with-❤️-red)

# 🧪 OpenAPI Contract Tester

An opinionated, lightweight **black-box contract tester** against a **live API** using its OpenAPI specification as the source of truth.

This tool validates OpenAPI quality, generates test cases from schemas, and verifies that real HTTP responses match the contract.
This "certifies" that the documentation is complete with descriptions, example, and schema, and that the endpoint behaves as the documentation suggests.

## Guiding Principles:
1. A service needs to document clearly. (This means schemas, descriptions, and examples)
2. When the examples and schemas are used, it should respond as expected from the documentation.

📚 **[Read the full documentation](https://sinan-ozel.github.io/pytest-openapi/)**

## ✨ What it does

### ▶️ Quick Example

![Swagger POST endpoint /email](swagger-screenshot-1.png)

```bash
pytest --openapi=http://localhost:8000
```

```
Test #10 ✅
POST /email
Requested:
  {
    "body": "Lorem ipsum dolor sit amet",
    "from": "Lorem ipsum dolor sit amet",
    "subject": "Lorem ipsum dolor sit amet",
    "to": "Test!@#$%^&*()_+-=[]{}|;:<>?,./`~"
  }

Expected 201
  {
    "body": "Hi Bob, how are you?",
    "from": "alice@example.com",
    "id": 1,
    "subject": "Hello",
    "to": "bob@example.com"
  }

Actual 201
  {
    "body": "Lorem ipsum dolor sit amet",
    "from": "Lorem ipsum dolor sit amet",
    "id": 10,
    "subject": "Lorem ipsum dolor sit amet",
    "to": "Test!@#$%^&*()_+-=[]{}|;:<>?,./`~"
  }

```
Generates multiple QA tests.

✔️ Validates OpenAPI request/response definitions
✔️ Enforces schema field descriptions
✔️ Generates test cases from schemas, checks response codes and types in the response
✔️ Tests the exanples
✔️ Tests **GET / POST / PUT / DELETE** endpoints
✔️ Compares live responses against examples
✔️ Produces a readable test report


# ▶️ Detailed Example

## Install
```bash
pip install pytest-openapi
```

## Run

Say that you have a service running at port `8000` on `localhost`. Then, run:

```bash
pytest --openapi=http://localhost:8000
```

### Options

- `--openapi=BASE_URL`: Run contract tests against the API at the specified base URL
- `--openapi-no-strict-example-checking`: Use lenient validation for example-based tests
- `--openapi-markdown-output=FILENAME`: Write test results in Markdown format to the specified file
- `--openapi-no-stdout`: Suppress all output to stdout
- `--openapi-ignore=REGEXP`: Completely ignore endpoints whose path matches the given regular expression. Useful to skip known-broken or auth-protected paths.

Examples:

```bash
pytest --openapi=http://localhost:8000 --openapi-ignore=mcp
pytest --openapi=http://localhost:8000 --openapi-ignore=(auth|mcp)
pytest --openapi=http://localhost:8000 --openapi-ignore=(v[0-9]+/auth|mcp)
```

#### Strict vs Lenient Example Checking

By default, pytest-openapi performs **strict matching** on example-based tests:
- When your OpenAPI spec includes explicit request/response examples, the actual response must match the example values exactly
- This ensures examples accurately reflect real API behavior

However, sometimes examples contain placeholder values (like `[1, 2, 3]`) that don't match actual responses (like `[]`). Use `--openapi-no-strict-example-checking` for lenient validation:

```bash
pytest --openapi=http://localhost:8000 --openapi-no-strict-example-checking
```

**Lenient mode** validates:
- Structure and types match (all expected keys present, correct types)
- But ignores exact values and array lengths

**Note**: Schema-generated tests always use schema validation (not affected by this flag).

#### Markdown Output Format

You can generate test reports in Markdown format and save them to a file:

```bash
pytest --openapi=http://localhost:8000 --openapi-markdown-output=report.md
```

This creates a `report.md` file with:
- Summary statistics (total, passed, failed tests)
- Formatted code blocks for JSON data
- Clear sections for expected vs actual responses
- Error details in formatted blocks

The markdown report is written independently of stdout output.

**Example output**: See [example_report.md](example_report.md) for a sample markdown report.

#### Suppress Stdout

For CI/CD pipelines where you want completely silent operation or only care about the exit code:

```bash
pytest --openapi=http://localhost:8000 --openapi-no-stdout
```

This will:
- Suppress all output to stdout
- Still return appropriate exit codes (0 for success, 1 for failure)
- Can be combined with `--openapi-markdown-output` to only generate a file

You can combine flags for silent operation with file output:
```bash
pytest --openapi=http://localhost:8000 --openapi-markdown-output=report.md --openapi-no-stdout
```

## Server
See here an example server - `email-server`: [tests/test_servers/email_server/server.py](tests/test_servers/email_server/server.py)

## Resulting Tests

[tests/test_servers/email_server/email_test_output.txt](tests/test_servers/email_server/email_test_output.txt)

# Future Plans / TODO

This is a work in progress.
- [ ] A check that the example matches the schema
- [ ] Ask that 400 responses be in the documentation.
- [ ] A check for regexp and email formats.

## In Consideration
- [ ] Use LLM-as-a-judge to assess the error messages and check their spelling.

# Contributing
Contributions are welcome!

The only requirement is 🐳 Docker.

Test are containerized, run them using the VS Code task `test`. If you don't want to use VS Code, the command is `docker compose -f ./tests/docker-compose.yaml --project-directory ./tests up --build --abort-on-container-exit --exit-code-from test`. Run this before making a PR, please.

There is also a development environment for VS Code, if you need it. On this environment, you can run the task `run-mock-server` to run one of the [mock servers](tests/test_servers) and see the output.

You can add your own mock server, and then add integration tests. Just follow the same pattern as every test to make a call - `subprocess.run('pytest', '--openapi=http://your-server:8000`.

Please reformat and lint before making a PR. The VS Task is `lint`, and if you don't want to use VS Code, the command is: `docker compose -f ./lint/docker-compose.yaml --project-directory ./lint up --build --abort-on-container-exit --exit-code-from linter`. Run this before making a PR, please.

If you add a functionality, please add to the the documentation.

Please submit a pull request or open an issue for any bugs or feature requests.

The moment your PR is merged, you get a dev release. You can then set up the version number to use your changes.

# License
MIT License. See [LICENSE](LICENSE) file for the specific wording.

