Metadata-Version: 2.4
Name: veris-cli
Version: 2.15.0
Summary: CLI to connect local agents to the Veris backend
Project-URL: Homepage, https://github.com/veris-ai/veris-cli
Project-URL: Bug Tracker, https://github.com/veris-ai/veris-cli/issues
Author: Veris
Requires-Python: >=3.11
Requires-Dist: click>=8.1.7
Requires-Dist: httpx>=0.27.0
Requires-Dist: pathspec>=0.12.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: questionary>=2.0.0
Requires-Dist: rich>=13.7.0
Provides-Extra: dev
Requires-Dist: pre-commit>=4.3.0; extra == 'dev'
Provides-Extra: test
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'test'
Requires-Dist: pytest-cov>=4.1.0; extra == 'test'
Requires-Dist: pytest>=8.2; extra == 'test'
Description-Content-Type: text/markdown

# Veris CLI

[![PyPI version](https://badge.fury.io/py/veris-cli.svg)](https://badge.fury.io/py/veris-cli)
[![Tests](https://github.com/veris-ai/veris-cli/actions/workflows/test.yml/badge.svg)](https://github.com/veris-ai/veris-cli/actions/workflows/test.yml)
[![Python](https://img.shields.io/pypi/pyversions/veris-cli.svg)](https://pypi.org/project/veris-cli/)

**Connect your existing agent to Veris simulations.**

The Veris CLI lets you package your agent as a Docker image and run it against simulation scenarios.

## Installation

```bash
uv tool install veris-cli
```

Or from source:
```bash
uv tool install git+https://github.com/veris-ai/veris-cli.git
```

## Quick Start

### 1. Login

```bash
# Browser-based Google login (recommended)
veris login

# Or with an API key directly (for CI/scripts)
veris login YOUR_API_KEY
```

This saves your credentials to `~/.veris/config.yaml`.

### 2. Initialize Your Project

```bash
veris init
```

This creates a `.veris/` folder with template files:
- **`Dockerfile.sandbox`** — Docker image template for your agent
- **`veris.yaml`** — Simulation configuration (services, persona, agent settings)
- **`.dockerignore`** — Build context exclusions

### 3. Create an Environment

```bash
veris env create --name "my-agent"
```

This registers your environment on the backend and saves the environment ID to `.veris/config.yaml`.

### 4. Configure Your Agent

Edit `.veris/Dockerfile.sandbox` and `.veris/veris.yaml` to match your agent. Set secrets:

```bash
veris env vars set OPENAI_API_KEY=sk-your-key --secret
```

### 5. Build and Push

```bash
veris env push
```

### 6. Generate Scenarios

```bash
veris scenarios create --num 10
veris scenarios status <SET_ID> --watch
```

### 7. Run Simulations

```bash
veris simulations create --scenario-set-id <SET_ID>
veris simulations status <SIM_RUN_ID> --watch
```

### 8. Evaluate Results

```bash
veris evaluations create --sim-run-id <SIM_RUN_ID>
veris evaluations status <SIM_RUN_ID> <EVAL_RUN_ID> --watch
```

### 9. Generate Report

```bash
veris reports create <SIM_RUN_ID>
veris reports status <REPORT_ID> --watch
veris reports get <REPORT_ID> -o results.html
```

### Full Pipeline (One Command)

```bash
# Interactive — prompts for each step
veris run

# CI — all flags, markdown summary to stdout
veris run --scenario-set-id <SET_ID> --grader-id <GRADER_ID> --report
```

## Command Reference

### Top-Level Commands

```bash
veris login [API_KEY]              # Authenticate (browser or API key)
  --backend-url URL                # Custom backend URL
  --console-url URL                # Custom console URL
  --org ORG_ID                     # Scope to organization

veris init                         # Scaffold .veris/ directory (local only)

veris run                          # Full pipeline: simulations → evaluations → reports
  --scenario-set-id ID             # Scenario set (prompts if omitted)
  --grader-id ID                   # Grader (prompts if omitted)
  --env-id ID                      # Environment (uses config if omitted)
  --concurrency N                  # Parallel jobs (default: 10)
  --image-tag TAG                  # Image tag (default: latest)
  --simulation-timeout N           # Timeout per sim in seconds
  --report                         # Generate HTML report after evaluation
```

### Environment (`veris env`)

```bash
veris env create --name NAME       # Register environment on backend
veris env push [--tag TAG]         # Build and push image (Cloud Build)
veris env list                     # List environments
veris env delete ENV_ID            # Delete environment
```

### Environment Variables (`veris env vars`)

```bash
veris env vars set K=V [K=V ...]   # Set variables
  --secret                         # Mark as secret
  --env-id ID                      # Override environment
veris env vars list                # List variables
veris env vars rm KEY              # Remove a variable
```

### Scenarios (`veris scenarios`)

```bash
veris scenarios create             # Generate scenario set + grader
  --num N                          # Number of scenarios (default: 5)
  --env-id ID                      # Environment
  --image-tag TAG                  # Image tag
veris scenarios status SET_ID      # Check generation progress
  --watch                          # Poll until done
veris scenarios list               # List scenario sets (with grader column)
veris scenarios get SET_ID         # Open in console browser
veris scenarios delete SET_ID      # Delete scenario set
```

### Simulations (`veris simulations`)

```bash
veris simulations create           # Create simulation run (interactive)
  --scenario-set-id ID             # Scenario set
  --env-id ID                      # Environment
  --concurrency N                  # Parallel jobs
  --simulation-timeout N           # Timeout per sim
  --image-tag TAG                  # Image tag
veris simulations status SIM_RUN_ID  # Run progress + sim list
  --watch                          # Poll until done
  --log                            # Append event stream
veris simulations list             # List simulation runs
  --status STATUS                  # Filter by status
  --env-id ID                      # Filter by environment
veris simulations cancel SIM_RUN_ID  # Cancel a run
```

### Evaluations (`veris evaluations`)

```bash
veris evaluations create           # Trigger grading (interactive)
  --sim-run-id ID                  # Simulation run
  --grader-id ID                   # Grader (pre-selected by scenario set)
veris evaluations status SIM_RUN_ID EVAL_RUN_ID  # Eval progress
  --watch                          # Poll until done
veris evaluations list [SIM_RUN_ID]  # List eval runs
                                   # Without args: all evals for current env
                                   # With arg: evals for specific run
veris evaluations get SIM_RUN_ID EVAL_RUN_ID  # Open in console browser
```

### Reports (`veris reports`)

```bash
veris reports create [SIM_RUN_ID]  # Trigger report generation
  --eval-run-id ID                 # Specific eval run
veris reports status REPORT_ID     # Report progress
  --watch                          # Poll until done
veris reports list                 # List reports
veris reports get REPORT_ID        # Download report HTML
  -o PATH                          # Output path
```

### Profiles (`veris profile`)

```bash
veris profile list                 # List all profiles
veris profile get                  # Show active profile
veris profile use [NAME]           # Set active profile + org (interactive)
  --org ORG_ID                     # Set organization
veris profile delete NAME          # Remove a profile

# Use a profile for any command
veris --profile staging env list
```

## CI/CD Integration

```yaml
name: Veris Simulation
on:
  pull_request:
    branches: [main]

jobs:
  simulate:
    runs-on: ubuntu-latest
    environment: veris-sim-ci
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install veris-cli
      - name: Build & push
        env:
          VERIS_API_KEY: ${{ secrets.VERIS_API_KEY }}
        run: |
          veris login "$VERIS_API_KEY"
          veris env push --tag ${{ github.sha }}
      - name: Run pipeline
        run: veris run --image-tag ${{ github.sha }} --report > summary.md
      - uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: summary.md
```

## How It Works

```
veris init                    → Scaffold .veris/ directory
veris env create              → Register environment on backend
veris env push                → Build and push Docker image
veris scenarios create        → Generate scenarios + grader
veris simulations create      → Run agent against scenarios
veris evaluations create      → Grade simulation results
veris reports create          → Generate failure analysis report
```

Or use `veris run` to chain all steps interactively.

## Configuration Files

### `~/.veris/config.yaml`

Global config with named profiles:
```yaml
active_profile: default
profiles:
  default:
    api_key: vrs_abc123
    backend_url: https://sandbox.api.veris.ai
    console_url: https://console.veris.ai
  staging:
    api_key: vrs_staging
    backend_url: https://sandbox.api.veris.ai
    organization_id: org_abc123
```

### `.veris/config.yaml`

Project config (auto-generated):
```yaml
profiles:
  default:
    environment_id: env_abc123
    environment_name: my-agent
```

## Profiles

The CLI supports named profiles for managing multiple environments. Each profile stores its own API key, backend URL, console URL, organization, and environment ID.

```bash
# Login to a named profile
veris --profile acme login <api-key> --org org_abc123

# Switch active profile
veris profile use acme

# Override for a single command
veris --profile production env list
```

Profile resolution: `--profile` flag → `active_profile` in config → `"default"`.

Existing flat configs (without `profiles` key) auto-migrate on first write.

## Development

```bash
git clone https://github.com/veris-ai/veris-cli.git
cd veris-cli
uv sync
uv run pytest
uv tool install --force -e .
```

## Troubleshooting

- **"No API key found"** — Run `veris login`
- **Cloud Build fails** — Check build logs in the output. Ensure Dockerfile.sandbox is valid.
- **Image push fails** — Credentials are fetched automatically. Just retry.

## Support

- GitHub Issues: https://github.com/veris-ai/veris-cli/issues
- Email: developers@veris.ai

## License

MIT
