Metadata-Version: 2.4
Name: ptm-client
Version: 0.12.0
Summary: Lightweight PTM API client for integration with external Python services
Author: 15Five Engineering
License: Proprietary
Classifier: Development Status :: 4 - Beta
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Testing
Classifier: Intended Audience :: Developers
Classifier: Typing :: Typed
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: requests<3.0,>=2.32
Provides-Extra: dev
Requires-Dist: pytest<9.0,>=8.2; extra == "dev"
Requires-Dist: responses<1.0,>=0.25; extra == "dev"
Requires-Dist: ruff<1.0,>=0.6; extra == "dev"

# ptm-client

Lightweight Python client for the Prompt Test Manager (PTM) API. Single runtime dependency: `requests`.

## Install

```bash
pip install ptm-client
```

## Quick start

```python
from ptm_client import PTMClient

client = PTMClient(base_url="https://ptm.example.com", token="your-api-token")

# List prompts, filtered by tag
prompts = client.list_prompts(tag="my_team")

# Fetch a prompt and its test cases
detail = client.get_prompt("my_team.summarizer")
tests = client.get_prompt_tests("my_team.summarizer")

# Run an eval against the library prompt
run = client.run_eval(
    prompt_ids=["my_team.summarizer"],
    provider_ids=["openai_gpt41_mini"],
)

# Or run a one-off manual eval
run = client.run_manual_eval({
    "prompt_text": "Summarize: {{text}}",
    "tests": [{"description": "smoke", "vars": {"text": "Hello, world."}}],
    "provider_profiles": ["openai_gpt41_mini"],
})

# Block until complete, then fetch a report
result = client.wait_for_run(run["run_key"], timeout=120)
html = client.run_report(run["run_key"])
json_report = client.run_report(run["run_key"], format="json")
```

## Constructor

```python
PTMClient(base_url, token, timeout=30, *, verify_ssl=None)
```

`token` is a PTM personal access token or service account token. `timeout` is the HTTP request timeout in seconds.

`verify_ssl` controls TLS certificate verification. When `None` (the default), the value is read from the `PTM_SSL_VERIFY` environment variable, which defaults to `true`. Set `PTM_SSL_VERIFY=false` (or pass `verify_ssl=False`) to allow self-signed or otherwise invalid certificates - intended for local dev against a homelab PTM instance, never for production. The matching `urllib3.InsecureRequestWarning` is silenced when verification is disabled.

## Public methods

### Prompts

- `list_prompts(tag=None, team=None, service=None, source=None, search=None, group=None)`
- `get_prompt(prompt_id)`
- `get_prompt_tests(prompt_id)`
- `list_prompt_versions(prompt_id)` *(v0.3.0)*
- `get_prompt_version(prompt_id, version_number)` *(v0.3.0)*

### Providers

- `list_providers()`

### Evaluations

- `run_eval(prompt_ids, provider_ids, **kwargs)`
- `run_manual_eval(payload)`
- `run_prompt_eval(prompt_id, provider_ids, *, inject_vars=None, extra_tests=None, visibility_scope="org_visible", label=None)`

### Runs

- `list_runs(limit=50, terminal_only=False, mine_only=False)` *(v0.3.0)*
- `get_run(run_key)`
- `wait_for_run(run_key, timeout=300, poll_interval=5)`
- `run_report(run_key, format="html")` - `html` / `json` / `markdown` / `csv`

### Optimization *(v0.3.0)*

- `submit_optimization(prompt_id, provider_profiles=None, judge_profile=None, max_cycles=10, target_score=90.0, min_improvement=2.0, max_cost_usd=20.0, comparison_strategy=None, visibility_scope=None, *, stability_samples=None, validation_samples=None, flakiness_threshold=None, min_consistent_improvement=None, variance_aware_mutator=None, variance_signal=None, enforce_target_score=None)` *(variance-aware fields added in v0.6.0; omit to inherit server admin defaults)*
- `optimize_prompt(...)` - deprecated alias for `submit_optimization`; emits `DeprecationWarning`; removed in v1.0.0
- `get_optimization_status(prompt_id)`
- `get_optimization_history(prompt_id)`
- `get_optimization_detail(optimization_id)` *(v0.3.0)*
- `cancel_optimization(optimization_id)`
- `wait_for_optimization(prompt_id, *, timeout=600, poll_interval=10)`

## Test-case shapes

PTM evaluates with three optional scoring layers. Use any combination.

### Promptfoo assertions (deterministic)

Go in the `assert` array inside each test case:

```python
{
    "description": "mention the topic with enough length",
    "vars": {"transcript": "..."},
    "assert": [
        {"type": "icontains", "value": "API migration"},
        {"type": "javascript", "value": "output.length >= 100"},
    ],
}
```

### DeepEval metrics (semantic, judge-LLM)

Go in `additional_metrics` at the payload root:

```python
{
    "additional_metrics": [
        {"name": "relevance", "criteria": "Output addresses the input topic.", "threshold": 0.7},
    ],
    "judge_profile": "openai_gpt41_mini",
}
```

### KPI configs (custom weighted expressions)

Go in `additional_kpis` at the payload root:

```python
{
    "additional_kpis": [
        {"name": "cost_ok", "description": "Under $0.05", "expression": "1 if cost < 0.05 else 0", "weight": 1.0},
    ],
}
```

## Inline examples

### `run_manual_eval` - full control

```python
run = client.run_manual_eval({
    "label": "my_custom_eval",
    "prompt_text": "Summarize: {{text}}",
    "tests": [{"description": "short text", "vars": {"text": "The quick brown fox."}}],
    "provider_profiles": ["openai_gpt41_mini"],
    "cost_threshold": 1.0,
    "latency_threshold_ms": 30000,
})
```

### `run_prompt_eval` - fetch from PTM + inject live data

```python
run = client.run_prompt_eval(
    prompt_id="my_team.summarizer",
    provider_ids=["openai_gpt41_mini"],
    inject_vars={"transcript": real_transcript, "meeting_title": "Weekly 1:1"},
)
result = client.wait_for_run(run["run_key"], timeout=120)
```

## Error handling

```python
from ptm_client import PTMClient, PTMError, PTMTimeoutError

try:
    result = client.wait_for_run(run_key, timeout=60)
except PTMTimeoutError:
    print("Run did not complete in time")
except PTMError as e:
    print(f"PTM API error ({e.status_code}): {e}")
```

`PTMError` wraps all HTTP errors, `ConnectionError`, and `requests.Timeout`. Check `e.status_code` (0 for connection/timeout failures).

## Compatibility

- Python 3.12+
- PTM backend compatible (some v0.3.0 methods require a recent backend release for full functionality; older backends work for all other methods)
