Metadata-Version: 2.4
Name: veris-cli
Version: 2.5.0
Summary: CLI to connect local agents to the Veris backend
Project-URL: Homepage, https://github.com/veris-ai/veris-cli
Project-URL: Bug Tracker, https://github.com/veris-ai/veris-cli/issues
Author: Veris
Requires-Python: >=3.11
Requires-Dist: click>=8.1.7
Requires-Dist: httpx>=0.27.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: questionary>=2.0.0
Requires-Dist: rich>=13.7.0
Provides-Extra: dev
Requires-Dist: pre-commit>=4.3.0; extra == 'dev'
Provides-Extra: test
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'test'
Requires-Dist: pytest-cov>=4.1.0; extra == 'test'
Requires-Dist: pytest>=8.2; extra == 'test'
Description-Content-Type: text/markdown

# Veris CLI

[![PyPI version](https://badge.fury.io/py/veris-cli.svg)](https://badge.fury.io/py/veris-cli)
[![Tests](https://github.com/veris-ai/veris-cli/actions/workflows/test.yml/badge.svg)](https://github.com/veris-ai/veris-cli/actions/workflows/test.yml)
[![Python](https://img.shields.io/pypi/pyversions/veris-cli.svg)](https://pypi.org/project/veris-cli/)

**Connect your existing agent to Veris simulations.**

The Veris CLI lets you package your agent as a Docker image and run it against simulation scenarios.

## Installation

```bash
uv tool install veris-cli
```

Or from source:
```bash
uv tool install git+https://github.com/veris-ai/veris-cli.git
```

## Quick Start

> **🚀 To run locally** Skip to [Local Development & Testing](#local-development--testing) to run scenarios on your machine without cloud deployment.

### 1. Login

```bash
# Browser-based Google login (recommended)
veris login

# Or with an API key directly (for CI/scripts)
veris login YOUR_API_KEY
```

This saves your credentials to `~/.veris/config.yaml`.

> **Note:** Login is only required for cloud runs. For local testing with `veris run local`, you only need Docker and your agent's required environment variables in a `.env` file.

### 3. Initialize Your Project

In your agent's project directory:

```bash
veris init
```

This will:
1. Create a `.veris/` folder with configuration files
2. Prompt you for an environment name (e.g., "my-customer-support-agent")
3. Create the environment in Veris
4. Save the environment ID to `.veris/config.yaml`

Files created:
- **`Dockerfile.sandbox`** - Docker image template for your agent
- **`veris.yaml`** - Simulation configuration (services, persona, agent settings)
- **`.env.simulation`** - Environment variables for your agent
- **`config.yaml`** - Environment ID (auto-generated, don't edit)

### 4. Configure Your Agent

Edit the generated files to match your agent:

**`.veris/Dockerfile.sandbox`** - Update paths to your agent code:
```dockerfile
# Copy your agent's dependencies
COPY pyproject.toml uv.lock /agent/
COPY your_agent_module /agent/your_agent_module

# Install dependencies
WORKDIR /agent
RUN uv sync --frozen --no-dev
```

**`.veris/veris.yaml`** - Configure your agent's entry point and port:
```yaml
agent:
  code_path: /agent
  entry_point: your_agent_module.main:app  # Update this!
  port: 8000  # Update if your agent uses a different port
```

**`.veris/.env.simulation`** - Add your agent's environment variables:
```bash
OPENAI_API_KEY=sk-your-key
DATABASE_URL=postgresql://...
```

### 5. Build and Push Your Agent Image

```bash
# Build and push in one command
veris env push

# Or build only (for testing)
veris env build
```

This will:
1. Use the environment created during `veris init`
2. Generate push credentials for the `latest` tag
3. **Automatically build your Docker image** (handles buildx on Mac)
4. **Automatically push to the registry**

**Note:** On macOS, this uses `docker buildx` for multi-platform builds targeting `linux/amd64` (GKE platform).

### 6. Generate Scenarios (Optional)

You can write scenarios by hand (see [Local Development](#local-development--testing)) or generate them automatically:

```bash
# Generate 5 scenarios + graders using Claude Code
veris scenarios generate --num 5
```

This launches a K8s job that explores your agent's source code and produces test scenarios and graders. Poll status with:

```bash
veris scenarios list
```

### 7. List Available Scenarios

```bash
veris scenarios list
```

### 8. Create and Run a Simulation

```bash
# Interactive mode (prompts for scenario and environment)
veris run create

# Or specify directly
veris run create --scenario-set-id scenset_abc123 --env-id env_xyz789
```

### 9. Monitor Your Run

```bash
# Check status
veris run status run_abc123

# Watch status (updates every 3 seconds)
veris run status run_abc123 --watch

# View logs
veris run logs run_abc123

# Follow logs (like tail -f)
veris run logs run_abc123 --follow
```

### 10. Evaluate Results (Optional)

Once a run completes and graders are available:

```bash
# List available graders
veris eval list

# Trigger evaluation (interactive prompts for run and grader)
veris evaluation-runs create

# Check evaluation status
veris evaluation-runs list --run-id run_abc123
veris evaluation-runs status evalrun_abc123 --run-id run_abc123
```

### 11. Cancel a Run (if needed)

```bash
veris run cancel run_abc123
```

## Complete Command Reference

### Authentication

```bash
# Browser-based Google login (recommended)
veris login

# Login with API key (for CI/scripts)
veris login <api-key>

# Specify a custom backend URL (for developers)
veris login [--backend-url https://sandbox.api.veris.ai]
```

### Project Setup

```bash
# Initialize .veris/ directory and create environment
veris init [--name "my-agent"]

# If name not provided, you'll be prompted interactively
```

### Environment Management

```bash
# Build Docker image only
veris env build [--tag latest] [--no-cache]

# Build and push Docker image
veris env push [--tag latest] [--no-cache]

# List all environments
veris env list [--status ready]
```

### Scenarios

```bash
# List scenario sets
veris scenarios list [--env-id <id>]

# Generate scenarios + graders via K8s job
veris scenarios generate [--env-id <id>] [--num 5] [--image-tag latest]
```

### Eval (Graders)

```bash
# List graders for an environment
veris eval list [--env-id <id>]
```

### Evaluation Runs

```bash
# Trigger grading on a completed run
veris evaluation-runs create [--run-id <id>] [--grader-id <id>]

# List evaluation runs for a run
veris evaluation-runs list --run-id <id>

# Get evaluation run status and results
veris evaluation-runs status <eval-run-id> --run-id <id> [--watch]
```

### Runs

```bash
# Create run (interactive or with flags)
veris run create [--scenario-set-id <id>] [--env-id <id>] [--concurrency 10]

# Get run status
veris run status <run-id> [--watch]

# Get run logs
veris run logs <run-id> [--follow]

# Cancel run
veris run cancel <run-id>

# Run scenarios locally in Docker (no cloud deployment needed)
veris run local [scenario...] [--skip-build] [--image <name>] [--platform <platform>] [--scenarios-dir <path>] [--concurrency <n>]
```

## Local Development & Testing

You can run scenarios locally without deploying to Veris cloud infrastructure. This is useful for:
- Testing your agent during development
- Debugging scenarios offline
- Running simulations without network dependency

### Prerequisites

1. Docker installed and running
2. `.veris/Dockerfile.sandbox` and `.veris/veris.yaml` configured (run `veris init` first)
3. Any required API keys or environment variables set in `.env` file or environment (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.)

### Quick Start

```bash
# Create .env file with your API keys
echo 'OPENAI_API_KEY=sk-your-key' > .env

# Run all scenarios in ./scenarios/ directory
veris run local

# Run specific scenarios
veris run local checkout payment

# Skip Docker build (faster, use existing image)
veris run local --skip-build
```

### Command Options

```bash
veris run local [SCENARIO...] [OPTIONS]

Arguments:
  SCENARIO            One or more scenario IDs (default: all scenarios in scenarios-dir)

Options:
  --skip-build           Skip building the Docker image
  --image TEXT           Docker image name (default: veris-sandbox)
  --platform TEXT        Docker platform (default: linux/arm64)
  --scenarios-dir PATH   Path to scenarios folder (default: ./scenarios)
  --concurrency INT      Max parallel containers (default: unbounded)
```

### Examples

```bash
# Run all scenarios in ./scenarios/
veris run local

# Run two specific scenarios in parallel
veris run local checkout payment

# Use custom scenarios directory
veris run local --scenarios-dir ./tests/scenarios

# Skip build and use custom image name
veris run local --skip-build --image my-agent-sandbox

# Limit to 2 parallel containers
veris run local --concurrency 2

# Use different platform (e.g., for Intel Macs)
veris run local --platform linux/amd64
```

### How Scenario Resolution Works

If you don't specify scenario arguments, the CLI scans your `--scenarios-dir` (default: `./scenarios/`):

- **Files**: `checkout.yaml` → scenario ID `checkout`
- **Directories**: `checkout/` → scenario ID `checkout`
- **Fallback**: If no scenarios found, uses default `customer_browse_and_purchase`

The scenario ID is passed to the container via `SCENARIO_ID` environment variable.

### Output

After running, you'll see:
1. **Summary table**: scenario → sim_id → exit code → logs path
2. **Complete logs**: All `.log` files from each simulation run

Logs are saved to `.veris/logs/<sim_id>/` for each run.

### Environment Variables

The command automatically:
- Loads `.env` from project root if it exists
- Passes all environment variables from `.env` to containers
- Adds `SCENARIO_ID` environment variable for each scenario

### Docker Details

Each scenario runs in an isolated container with:
- Mounted `.veris/veris.yaml` at `/config/veris.yaml` (read-only)
- Mounted scenarios folder at `/scenarios` (read-only)
- Mounted logs directory at `/sessions` (for output)
- Environment variables from `.env` plus `SCENARIO_ID`

## How It Works

```
┌─────────────────────────────────────────────────────────────┐
│  1. One-Time Setup                                          │
│     veris init → creates environment + config files         │
│                → saves env_id to .veris/config.yaml         │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  2. Push Updates                                            │
│     Edit code → veris env push → docker build & push        │
│     (can push multiple versions using --tag v1, v2, etc.)   │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  3. Generate Scenarios (optional)                           │
│     veris scenarios generate → Claude Code explores agent   │
│                              → produces scenarios + graders │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  4. Run Simulations                                         │
│     veris run create → Veris spawns your agent in K8s       │
│                     → Runs scenarios against it             │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  5. Evaluate Results (optional)                             │
│     veris evaluation-runs create → grades simulation traces │
│     veris evaluation-runs status → view grading results     │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  6. Monitor & Analyze                                       │
│     veris run status → check progress                       │
│     veris run logs → view events                            │
└─────────────────────────────────────────────────────────────┘
```

## Configuration Files

### `~/.veris/config.yaml`
Created by `veris login`:
```yaml
api_key: vrs_abc123xyz
backend_url: https://sandbox.api.veris.ai
```

### `.veris/config.yaml`
Project configuration (auto-generated by `veris init`):
```yaml
environment_id: env_abc123
environment_name: my-agent
```

### `.veris/Dockerfile.sandbox`
Template for building your agent's Docker image. **Important:** Build context is project root, so `COPY` paths are relative to your project root, not `.veris/`.

### `.veris/veris.yaml`
Simulation configuration including:
- Services your agent uses (with DNS aliases)
- Persona modality (http/ws/email)
- Agent entry point and port

### `.veris/.env.simulation`
Environment variables loaded into your agent container at runtime.

## Development

```bash
# Clone repo
git clone https://github.com/veris-ai/veris-cli.git
cd veris-cli

# Install dependencies
uv sync

# Run tests
uv run pytest

# Install locally for testing
uv tool install --force .
```

## Troubleshooting

### "No API key found"
Run `veris login` to authenticate via Google, or `veris login <your-api-key>` with an API key.

### Docker build fails
- Make sure Docker is running
- On macOS, Docker Desktop must be installed (required for `docker buildx`)
- Try `veris env build --no-cache` to force a clean build

### Image push fails
Check that Docker is running and try again. Credentials are fetched automatically — you don't need to run `docker login` manually.

## Support

- GitHub Issues: https://github.com/veris-ai/veris-cli/issues
- Email: developers@veris.ai

## License

MIT
