Metadata-Version: 2.4
Name: skillscan-trace
Version: 1.0.0
Summary: Behavioral execution engine for MCP-based AI agent skills
Project-URL: Repository, https://github.com/kurtpayne/skillscan-trace
Project-URL: Issues, https://github.com/kurtpayne/skillscan-trace/issues
Project-URL: skillscan-security, https://github.com/kurtpayne/skillscan-security
License: Apache-2.0
License-File: LICENSE
Keywords: agents,ai,behavioral-analysis,mcp,security
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Security
Requires-Python: >=3.11
Requires-Dist: bashlex>=0.18
Requires-Dist: click>=8.0
Requires-Dist: jsonschema>=4.0
Requires-Dist: mcp>=1.0
Requires-Dist: openai>=1.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: python-frontmatter>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: requests>=2.0
Requires-Dist: rich>=13.0
Provides-Extra: dev
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Requires-Dist: types-requests>=2.0; extra == 'dev'
Provides-Extra: serve
Requires-Dist: boto3>=1.34; extra == 'serve'
Requires-Dist: fastapi>=0.110; extra == 'serve'
Requires-Dist: pydantic>=2.0; extra == 'serve'
Requires-Dist: uvicorn[standard]>=0.29; extra == 'serve'
Description-Content-Type: text/markdown

# skillscan-trace

[![CI](https://github.com/kurtpayne/skillscan-trace/actions/workflows/ci.yml/badge.svg)](https://github.com/kurtpayne/skillscan-trace/actions/workflows/ci.yml)
[![CodeQL](https://github.com/kurtpayne/skillscan-trace/actions/workflows/codeql.yml/badge.svg)](https://github.com/kurtpayne/skillscan-trace/actions/workflows/codeql.yml)
[![PyPI](https://img.shields.io/pypi/v/skillscan-trace.svg)](https://pypi.org/project/skillscan-trace/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](pyproject.toml)

**Behavioral execution engine for MCP-based AI agent skills.**

skillscan-trace runs a skill against a real language model inside an instrumented, isolated environment and records everything the model does: every file it reads, every network request it makes, every environment variable it accesses, every binary it probes. The output is a structured, machine-readable trace report.

It is the dynamic analysis counterpart to [skillscan](https://github.com/kurtpayne/skillscan-security), which performs static analysis. Together they form two legs of the skillscan family:

```
skillscan family
├── skillscan        — static analysis (pattern matching, ML classifier)
├── skillscan-trace  — behavioral execution engine  ← this repo
└── skillscan-lint   — schema and format validation (planned)
```

---

## What it does

A skill is a Markdown file that becomes a system prompt for an AI agent. Most skills are benign. Some are malicious — they instruct the agent to exfiltrate credentials, probe the filesystem, call attacker-controlled servers, or hijack the agent's behavior through prompt injection embedded in tool output.

Static analysis catches the obvious cases. Behavioral analysis catches the rest: conditional payloads that only activate in certain environments, obfuscated instructions that decode at runtime, and prompt injection delivered through external content the skill fetches.

skillscan-trace works by:

1. Loading the skill's `SKILL.md` as the system prompt for a local language model
2. Sending a realistic user prompt that exercises the skill's stated functionality
3. Driving the model's tool-use loop through an **instrumented MCP server** that intercepts every tool call
4. Checking each call against a canary taxonomy (credential files, wallet paths, ENV vars, binary probing, network destinations)
5. Emitting a structured trace report (JSON + SARIF) with every observed behavior and any findings

The model runs locally via [Ollama](https://ollama.com/) — no API key required, no cloud dependency. API providers (OpenAI, OpenRouter) are supported for users who prefer them or want access to more capable models.

---

## Status

**v0.2.0 — verdict banner + full CI matrix.** All phases implemented and 199/199 tests pass across Python 3.11, 3.12, and 3.13. The tool is installable and usable today.

---

## Quick start

```bash
# Install from source
git clone https://github.com/kurtpayne/skillscan-trace
cd skillscan-trace
pip install -e .

# Run a trace with OpenAI
export OPENAI_API_KEY=sk-...
skillscan-trace run ./path/to/skill/

# Run with OpenRouter (200+ models via one key)
export OPENROUTER_API_KEY=sk-or-...
skillscan-trace run ./skill/ --provider openrouter --model mistralai/mistral-7b-instruct

# Run with a local Ollama model (no API key required)
# Model must support tool calling — llama3.1:8b and llama3.2:3b are verified
skillscan-trace run ./skill/ --provider ollama --model llama3.1:8b

# Dry run — validate config and skill loading without calling the LLM
skillscan-trace run ./skill/ --dry-run

# Run with explicit base URL (Azure, Mistral, etc.)
skillscan-trace run ./skill/ --base-url https://api.mistral.ai/v1 --api-key $MISTRAL_KEY

# Output formats
skillscan-trace run ./skill/ --format sarif   # SARIF 2.1.0 for CI
skillscan-trace run ./skill/ --format json    # native trace format
skillscan-trace run ./skill/ --format text    # human-readable summary

# Verify connectivity
skillscan-trace check
skillscan-trace check --provider openrouter
skillscan-trace check --provider ollama
```

---

## Skill format support

skillscan-trace handles all skill formats found in the wild:

| Format | Description | Example |
|---|---|---|
| Single `SKILL.md` | Standard Claude Code / MCP skill | `./my-skill/SKILL.md` |
| Single `.md` file | Flat file, no directory | `./my-skill.md` |
| Directory with `SKILL.md` | Standard with supporting files | `./my-skill/` |
| Frontmatter + body | YAML frontmatter + Markdown body | `name:`, `allowed-tools:` |
| Plain Markdown | No frontmatter | Any `.md` file |
| Multi-file skill | Directory with multiple `.md` files | Loaded in alphabetical order |

---

## Docker

A single image supports both `run` (single trace) and `serve` (HTTP API) modes. The same image powers the hosted service at [trace.skillscan.sh](https://trace.skillscan.sh) and enterprise self-hosting.

Pull the image:

```bash
docker pull kurtpayne/skillscan-trace:latest
```

### Run mode (single trace)

Mount your skill and pass an API key via env var. The container exits after the trace completes.

```bash
docker run --rm \
  -v "$(pwd)":/data \
  -e OPENAI_API_KEY=sk-... \
  kurtpayne/skillscan-trace run /data/SKILL.md
```

Use OpenRouter instead:

```bash
docker run --rm \
  -v "$(pwd)":/data \
  -e OPENROUTER_API_KEY=sk-or-... \
  kurtpayne/skillscan-trace run /data/SKILL.md --provider openrouter
```

Fully local with Ollama on the host:

```bash
docker run --rm --network host \
  -v "$(pwd)":/data \
  kurtpayne/skillscan-trace run /data/SKILL.md --provider ollama
```

### Serve mode (HTTP API)

Starts the FastAPI server on port 8080. Callers provide their own API key per request (BYOK) — keys are never logged or stored.

```bash
docker run -d -p 8080:8080 --name sst \
  kurtpayne/skillscan-trace serve

curl http://localhost:8080/v1/health
```

For persistent caching across restarts, mount volumes for `/trace-cache` and `/trace-output`. A full `docker-compose.yml` is provided in the repository root.

---

## Output: Trace Report

Every trace produces a JSON trace report and optionally a SARIF report.

```json
{
  "schema_version": "1.0.0",
  "trace_id": "trc_20260320_abc123",
  "skill": {
    "path": "./my-skill/SKILL.md",
    "name": "git-helper",
    "sha256": "a1b2c3..."
  },
  "model": {
    "provider": "ollama",
    "model": "llama3.1:8b",
    "version": "..."
  },
  "prompt": "Help me commit my changes",
  "duration_ms": 4823,
  "tool_calls": [...],
  "findings": [...],
  "summary": {
    "total_tool_calls": 7,
    "finding_count": 1,
    "severity_max": "HIGH",
    "clean": false
  }
}
```

---

## Relationship to skillscan-security

skillscan-trace is a sibling project to [skillscan-security](https://github.com/kurtpayne/skillscan-security). The two projects share:

- **Finding schema**: The same finding IDs (`EXF-001`, `MAL-001`, `IOC-001`, etc.) and severity levels
- **Canary taxonomy**: The same list of high-value target paths and ENV var names
- **Domain allowlist**: `trace/domains/verified.yml` from skillscan-security is the source of truth; skillscan-trace consumes it
- **Corpus feedback loop**: Traces that produce findings can be reviewed and added to the skillscan-security corpus as `sandbox_verified/` examples, improving the ML classifier's recall on behavioral patterns that static analysis misses

---

## Privacy & Security

**Your API keys are never stored, logged, or transmitted by SkillScan** — they are passed directly to the LLM provider you chose over HTTPS and held in memory only for the duration of the run.

- **No telemetry, no analytics, no phone-home** — skillscan-trace makes zero network requests beyond the LLM API calls you explicitly authorize
- **Local-first** — the canary server runs in-process on your machine; nothing leaves your network except LLM API calls
- **Hosted service (trace.skillscan.sh)** — follows the BYOK model; your API key is forwarded to the LLM provider and never stored or cached; only the trace report is cached
- **No user identity** — no accounts, no login, no tracking
- **Ollama** — fully local, zero external network requests, no API key required

See [`PRIVACY.md`](./PRIVACY.md) for the full data flow diagram and detailed explanation.

---

## Contributing

See [`CONTRIBUTING.md`](./CONTRIBUTING.md).

---

## License

Apache-2.0
