Metadata-Version: 2.4
Name: veris-cli
Version: 2.13.0rc1
Summary: CLI to connect local agents to the Veris backend
Project-URL: Homepage, https://github.com/veris-ai/veris-cli
Project-URL: Bug Tracker, https://github.com/veris-ai/veris-cli/issues
Author: Veris
Requires-Python: >=3.11
Requires-Dist: click>=8.1.7
Requires-Dist: httpx>=0.27.0
Requires-Dist: pathspec>=0.12.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: questionary>=2.0.0
Requires-Dist: rich>=13.7.0
Provides-Extra: dev
Requires-Dist: pre-commit>=4.3.0; extra == 'dev'
Provides-Extra: test
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'test'
Requires-Dist: pytest-cov>=4.1.0; extra == 'test'
Requires-Dist: pytest>=8.2; extra == 'test'
Description-Content-Type: text/markdown

# Veris CLI

[![PyPI version](https://badge.fury.io/py/veris-cli.svg)](https://badge.fury.io/py/veris-cli)
[![Tests](https://github.com/veris-ai/veris-cli/actions/workflows/test.yml/badge.svg)](https://github.com/veris-ai/veris-cli/actions/workflows/test.yml)
[![Python](https://img.shields.io/pypi/pyversions/veris-cli.svg)](https://pypi.org/project/veris-cli/)

**Connect your existing agent to Veris simulations.**

The Veris CLI lets you package your agent as a Docker image and run it against simulation scenarios.

## Installation

```bash
uv tool install veris-cli
```

Or from source:
```bash
uv tool install git+https://github.com/veris-ai/veris-cli.git
```

## Quick Start

> **🚀 To run locally** Skip to [Local Development & Testing](#local-development--testing) to run scenarios on your machine without cloud deployment.

### 1. Login

```bash
# Browser-based Google login (recommended)
veris login

# Or with an API key directly (for CI/scripts)
veris login YOUR_API_KEY
```

This saves your credentials to `~/.veris/config.yaml`.

> **Note:** Login is only required for cloud runs. For local testing with `veris run local`, you only need Docker and your agent's required environment variables in a `.env` file.

### 2. Initialize Your Project

In your agent's project directory:

```bash
veris init
```

This will:
1. Create a `.veris/` folder with configuration files
2. Prompt you for an environment name (e.g., "my-customer-support-agent")
3. Create the environment in Veris
4. Save the environment ID to `.veris/config.yaml`

Files created:
- **`Dockerfile.sandbox`** - Docker image template for your agent
- **`veris.yaml`** - Simulation configuration (services, persona, agent settings)
- **`config.yaml`** - Environment ID (auto-generated, don't edit)

### 3. Configure Your Agent

Edit the generated files to match your agent:

**`.veris/Dockerfile.sandbox`** - Update paths to your agent code:
```dockerfile
# Copy your agent's dependencies
COPY pyproject.toml uv.lock /agent/
COPY your_agent_module /agent/your_agent_module

# Install dependencies
WORKDIR /agent
RUN uv sync --frozen --no-dev
```

**`.veris/veris.yaml`** - Configure your agent's entry point, port, and environment variables:
```yaml
agent:
  code_path: /agent
  entry_point: your_agent_module.main:app  # Update this!
  port: 8000  # Update if your agent uses a different port
  environment:
    DATABASE_URL: postgresql://postgres:postgres@localhost:5432/SIMULATION_ID
```

For secrets (API keys), use `veris env set` for cloud runs or a `.env` file at project root for local runs:
```bash
# Cloud runs: stored securely, injected at runtime
veris env set OPENAI_API_KEY=sk-your-key --secret

# Local runs: create .env at project root
echo 'OPENAI_API_KEY=sk-your-key' > .env
```

### 4. Build and Push Your Agent Image

```bash
# Build and push in one command
veris env push

# Or build only (for testing)
veris env build
```

This will:
1. Use the environment created during `veris init`
2. Generate push credentials for the `latest` tag
3. **Automatically build your Docker image** (handles buildx on Mac)
4. **Automatically push to the registry**

**Note:** On macOS, this uses `docker buildx` for multi-platform builds targeting `linux/amd64` (GKE platform).

### 5. Generate Scenarios (Optional)

You can write scenarios by hand (see [Local Development](#local-development--testing)) or generate them automatically:

```bash
# Generate 5 scenarios + graders using Claude Code
veris scenarios generate --num 5
```

This launches a K8s job that explores your agent's source code and produces test scenarios and graders. Poll status with:

```bash
veris scenarios list
```

### 6. List Available Scenarios

```bash
veris scenarios list
```

### 7. Create and Run a Simulation

```bash
# Interactive mode (prompts for scenario and environment)
veris run create

# Or specify directly
veris run create --scenario-set-id scenset_abc123 --env-id env_xyz789
```

### 8. Monitor Your Run

```bash
# Check status
veris run status run_abc123

# Watch status (updates every 3 seconds)
veris run status run_abc123 --watch

# View logs
veris run logs run_abc123

# Follow logs (like tail -f)
veris run logs run_abc123 --follow
```

### 9. Evaluate Results (Optional)

Once a run completes and graders are available:

```bash
# List available graders
veris eval list

# Trigger evaluation (interactive prompts for run and grader)
veris evaluation-runs create

# Check evaluation status
veris evaluation-runs list --run-id run_abc123
veris evaluation-runs status evalrun_abc123 --run-id run_abc123
```

### 10. Cancel a Run (if needed)

```bash
veris run cancel run_abc123
```

## Complete Command Reference

### Authentication

```bash
# Browser-based Google login (recommended)
veris login

# Login with API key (for CI/scripts)
veris login <api-key>

# Login and scope to an organization
veris login --org <org-id>

# Specify a custom backend URL (for developers)
veris login [--backend-url https://sandbox.api.veris.ai]

# Login to a specific profile
veris --profile staging login <api-key>
```

### Project Setup

```bash
# Initialize .veris/ directory and create environment
veris init [--name "my-agent"]

# If name not provided, you'll be prompted interactively
```

### Profiles

```bash
# List all profiles
veris profile list

# Show active profile's settings
veris profile show

# Set the active profile
veris profile use <name>

# Set organization scope for the active profile
veris profile set-org <org-id>

# Clear organization scope (revert to personal)
veris profile set-org

# Delete a profile
veris profile delete <name>

# Use a specific profile for any command
veris --profile <name> <command>
```

### Environment Management

```bash
# Build Docker image only
veris env build [--tag latest] [--no-cache]

# Build and push Docker image (locally or via Cloud Build)
veris env push [--tag latest] [--no-cache] [--remote]

# List all environments
veris env list [--status ready]

# Set environment variables (for cloud runs)
veris env set KEY=VALUE [...] [--secret] [--env-id <id>]

# List environment variables
veris env vars [--env-id <id>]

# Remove an environment variable
veris env rm KEY [--env-id <id>]
```

### Scenarios

```bash
# List scenario sets
veris scenarios list [--env-id <id>]

# Generate scenarios + graders via K8s job
veris scenarios generate [--env-id <id>] [--num 5] [--image-tag latest]

# View scenario set details
veris scenarios get <set-id>
```

### Eval (Graders)

```bash
# List graders for an environment
veris eval list [--env-id <id>]
```

### Evaluation Runs

```bash
# Trigger grading on a completed run
veris evaluation-runs create [--run-id <id>] [--grader-id <id>]

# List evaluation runs for a run
veris evaluation-runs list --run-id <id>

# Get evaluation run status and results
veris evaluation-runs status <eval-run-id> --run-id <id> [--watch]
```

### CI

```bash
# Run simulation + evaluation pipeline (config-driven)
veris ci run [--scenario-set-id <id>] [--env-id <id>] [--concurrency <n>] [--image-tag <tag>] [--simulation-timeout <seconds>]
```

### Runs

```bash
# List runs
veris run list [--status <status>] [--env-id <id>]

# Create run (interactive or with flags)
veris run create [--scenario-set-id <id>] [--env-id <id>] [--concurrency 10] [--simulation-timeout <seconds>]

# Get run status
veris run status <run-id> [--watch]

# Get run logs
veris run logs <run-id> [--follow]

# Cancel run
veris run cancel <run-id>

# Run scenarios locally in Docker (no cloud deployment needed)
veris run local [scenario...] [--skip-build] [--image <name>] [--platform <platform>] [--scenarios-dir <path>] [--concurrency <n>]
```

### Simulations

```bash
# List simulations for a run
veris simulations get <run-id>

# Get a specific simulation result
veris simulations get <run-id> <sim-id>
```

### Reports

```bash
# Create a failure analysis report
veris reports create [--env-id <id>] [--eval-run-id <id>]

# List reports
veris reports list [--env-id <id>]

# Check report status (with optional polling)
veris reports status <report-id> [--watch]

# Download report HTML to file
veris reports get <report-id> [-o <output-path>]
```

## Local Development & Testing

You can run scenarios locally without deploying to Veris cloud infrastructure. This is useful for:
- Testing your agent during development
- Debugging scenarios offline
- Running simulations without network dependency

### Prerequisites

1. Docker installed and running
2. `.veris/Dockerfile.sandbox` and `.veris/veris.yaml` configured (run `veris init` first)
3. Any required API keys or environment variables set in `.env` file or environment (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.)

### Quick Start

```bash
# Create .env file with your API keys
echo 'OPENAI_API_KEY=sk-your-key' > .env

# Run all scenarios in ./scenarios/ directory
veris run local

# Run specific scenarios
veris run local checkout payment

# Skip Docker build (faster, use existing image)
veris run local --skip-build
```

### Command Options

```bash
veris run local [SCENARIO...] [OPTIONS]

Arguments:
  SCENARIO            One or more scenario IDs (default: all scenarios in scenarios-dir)

Options:
  --skip-build           Skip building the Docker image
  --image TEXT           Docker image name (default: veris-sandbox)
  --platform TEXT        Docker platform (default: linux/amd64)
  --scenarios-dir PATH   Path to scenarios folder (default: ./scenarios)
  --concurrency INT      Max parallel containers (default: unbounded)
```

### Examples

```bash
# Run all scenarios in ./scenarios/
veris run local

# Run two specific scenarios in parallel
veris run local checkout payment

# Use custom scenarios directory
veris run local --scenarios-dir ./tests/scenarios

# Skip build and use custom image name
veris run local --skip-build --image my-agent-sandbox

# Limit to 2 parallel containers
veris run local --concurrency 2

# Use different platform (e.g., for Intel Macs)
veris run local --platform linux/amd64
```

### How Scenario Resolution Works

If you don't specify scenario arguments, the CLI scans your `--scenarios-dir` (default: `./scenarios/`):

- **Files**: `checkout.yaml` → scenario ID `checkout`
- **Directories**: `checkout/` → scenario ID `checkout`
- **Fallback**: If no scenarios found, uses default `customer_browse_and_purchase`

The scenario ID is passed to the container via `SCENARIO_ID` environment variable.

### Output

After running, you'll see:
1. **Summary table**: scenario → sim_id → exit code → logs path
2. **Complete logs**: All `.log` files from each simulation run

Logs are saved to `.veris/logs/<sim_id>/` for each run.

### Environment Variables

The command loads environment variables from two sources (in order):
1. **`agent.environment`** in `.veris/veris.yaml` — non-secret config shared with cloud runs
2. **`.env`** at project root — local-only secrets (takes precedence)

Each container also receives a `SCENARIO_ID` environment variable.

### Docker Details

Each scenario runs in an isolated container with:
- Mounted `.veris/veris.yaml` at `/config/veris.yaml` (read-only)
- Mounted scenarios folder at `/scenarios` (read-only)
- Mounted logs directory at `/sessions` (for output)
- Environment variables from `.env` plus `SCENARIO_ID`

## CI/CD Integration

Run simulations automatically on every pull request and post results as a PR comment.

### Setup

1. Run `veris ci run` interactively once to select a scenario set — this saves the config to `.veris/config.yaml`:

```bash
veris ci run
```

2. Commit `.veris/config.yaml` to your repo (make sure it's not gitignored).

3. Add a `VERIS_API_KEY` secret to your repo (Settings → Secrets → Actions).

### GitHub Actions Workflow

```yaml
name: Veris Simulation
on:
  pull_request:
    branches: [main]

jobs:
  simulate:
    runs-on: ubuntu-latest
    # Restrict to maintainers via GitHub environment protection rules
    environment: veris-sim-ci
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install veris-cli
        run: pip install veris-cli

      - name: Run simulation & evaluation
        env:
          VERIS_API_KEY: ${{ secrets.VERIS_API_KEY }}
        run: |
          veris login "$VERIS_API_KEY"
          veris ci run --image-tag ${{ github.sha }} > veris-summary.md

      - name: Comment on PR
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: veris-summary.md
```

The `environment: veris-sim-ci` line uses [GitHub environment protection rules](https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment) — secrets are only exposed to approved maintainers, even on public repos.

### CLI Usage

```bash
# Everything from config — zero flags needed
veris ci run

# Override image tag (common in CI for PR builds)
veris ci run --image-tag $(git rev-parse --short HEAD)

# Override everything
veris ci run --scenario-set-id X --env-id Y --concurrency 5
```

Progress is printed to stderr, clean markdown to stdout. The command exits non-zero if the run or evaluation fails.

## How It Works

```
┌─────────────────────────────────────────────────────────────┐
│  1. One-Time Setup                                          │
│     veris init → creates environment + config files         │
│                → saves env_id to .veris/config.yaml         │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  2. Push Updates                                            │
│     Edit code → veris env push → docker build & push        │
│     (can push multiple versions using --tag v1, v2, etc.)   │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  3. Generate Scenarios (optional)                           │
│     veris scenarios generate → Claude Code explores agent   │
│                              → produces scenarios + graders │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  4. Run Simulations                                         │
│     veris run create → Veris spawns your agent in K8s       │
│                     → Runs scenarios against it             │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  5. Evaluate Results (optional)                             │
│     veris evaluation-runs create → grades simulation traces │
│     veris evaluation-runs status → view grading results     │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  6. Generate Reports (optional)                             │
│     veris reports generate → failure analysis report        │
│     veris reports get → download HTML report                │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  7. Monitor & Analyze                                       │
│     veris run status → check progress                       │
│     veris run list → list all runs                          │
│     veris simulations get → view per-simulation results     │
└─────────────────────────────────────────────────────────────┘
```

## Profiles

The CLI supports named profiles for managing multiple Veris environments (e.g., dev, staging, production) — similar to AWS CLI profiles. Each profile stores its own API key, backend URL, organization, and environment ID.

### Setting Up Profiles

```bash
# Login to the default profile (same as before)
veris login <api-key>

# Login to a named profile with an organization
veris --profile acme login <api-key> --org org_abc123

# Initialize a project under a specific profile
veris --profile acme init --name my-agent
```

### Organizations

Profiles can be scoped to an organization. When set, all created assets (environments, runs) belong to that org, and list commands only show that org's assets.

```bash
# Set org during login
veris login --org org_abc123

# Or set/change org on an existing profile
veris profile set-org org_abc123

# Clear org (use personal scope)
veris profile set-org

# View current profile settings including org
veris profile show
```

You can find your organization ID in the Veris Console under the Admin page.

### Switching Profiles

```bash
# Set the active profile (persists across commands)
veris profile use acme

# Override the active profile for a single command
veris --profile production env list

# See which profiles are configured
veris profile list
```

### Profile Resolution

Profiles are resolved in priority order:

1. `--profile <name>` CLI flag (per-command override)
2. `active_profile` field in `~/.veris/config.yaml` (set via `veris profile use`)
3. `"default"` (implicit fallback)

### Backwards Compatibility

Existing flat config files (without `profiles` key) continue to work. On first write, the flat format is automatically migrated to the profiled format — no manual migration needed.

## Configuration Files

### `~/.veris/config.yaml`
Created by `veris login`. Supports named profiles:
```yaml
active_profile: default
profiles:
  default:
    api_key: vrs_abc123xyz
    backend_url: https://sandbox.api.veris.ai
  acme:
    api_key: vrs_acme_key
    backend_url: https://sandbox.api.veris.ai
    organization_id: org_abc123
```

### `.veris/config.yaml`
Project configuration (auto-generated by `veris init`). Each profile can have its own environment:
```yaml
profiles:
  default:
    environment_id: env_abc123
    environment_name: my-agent
  staging:
    environment_id: env_def456
    environment_name: my-agent-staging
```

### `.veris/Dockerfile.sandbox`
Template for building your agent's Docker image. **Important:** Build context is project root, so `COPY` paths are relative to your project root, not `.veris/`.

### `.veris/veris.yaml`
Simulation configuration including:
- Services your agent uses (with DNS aliases)
- Persona modality (http/ws/email)
- Agent entry point and port

## Development

```bash
# Clone repo
git clone https://github.com/veris-ai/veris-cli.git
cd veris-cli

# Install dependencies
uv sync

# Run tests
uv run pytest

# Install locally for testing
uv tool install --force .
```

## Troubleshooting

### "No API key found"
Run `veris login` to authenticate via Google, or `veris login <your-api-key>` with an API key.

### Docker build fails
- Make sure Docker is running
- On macOS, Docker Desktop must be installed (required for `docker buildx`)
- Try `veris env build --no-cache` to force a clean build

### Image push fails
Check that Docker is running and try again. Credentials are fetched automatically — you don't need to run `docker login` manually.

## Support

- GitHub Issues: https://github.com/veris-ai/veris-cli/issues
- Email: developers@veris.ai

## License

MIT
