Metadata-Version: 2.4
Name: penbot
Version: 2.2.0
Summary: AI Chatbot Penetration Testing Framework
Author: terminal48
License: MIT
Project-URL: Homepage, https://gitlab.com/yan-ban/penbot
Project-URL: Documentation, https://gitlab.com/yan-ban/penbot/-/tree/main/docs
Project-URL: Repository, https://gitlab.com/yan-ban/penbot
Project-URL: Issues, https://gitlab.com/yan-ban/penbot/-/issues
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: langgraph>=0.2.0
Requires-Dist: langgraph-checkpoint-sqlite>=2.0.0
Requires-Dist: langchain>=0.2.0
Requires-Dist: langchain-anthropic>=0.1.0
Requires-Dist: pydantic>=2.7.0
Requires-Dist: pydantic-settings>=2.2.0
Requires-Dist: click>=8.1.7
Requires-Dist: rich>=13.7.1
Requires-Dist: aiohttp>=3.9.4
Requires-Dist: httpx>=0.27.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: python-dateutil>=2.9.0
Requires-Dist: jsonpath-ng>=1.6.0
Requires-Dist: jinja2>=3.1.3
Requires-Dist: prometheus-client>=0.20.0
Requires-Dist: python-dotenv>=1.0.1
Requires-Dist: structlog>=24.1.0
Requires-Dist: mcp>=1.0.0
Requires-Dist: langchain-openai>=0.1.0
Provides-Extra: full
Requires-Dist: fastapi>=0.110.0; extra == "full"
Requires-Dist: uvicorn[standard]>=0.29.0; extra == "full"
Requires-Dist: slowapi>=0.1.9; extra == "full"
Requires-Dist: PyJWT>=2.8.0; extra == "full"
Requires-Dist: playwright>=1.43.0; extra == "full"
Requires-Dist: weasyprint>=62.0; extra == "full"
Requires-Dist: reportlab>=4.1.0; extra == "full"
Requires-Dist: python-docx>=1.1.0; extra == "full"
Requires-Dist: pypdf>=4.0.0; extra == "full"
Requires-Dist: Pillow>=10.0.0; extra == "full"
Requires-Dist: prometheus-fastapi-instrumentator>=7.0.0; extra == "full"
Requires-Dist: tavily-python>=0.5.0; extra == "full"
Provides-Extra: recon
Requires-Dist: tavily-python>=0.5.0; extra == "recon"
Provides-Extra: think
Provides-Extra: ml
Requires-Dist: sentence-transformers>=2.2.0; extra == "ml"
Requires-Dist: faiss-cpu>=1.7.0; extra == "ml"
Requires-Dist: numpy>=1.24.0; extra == "ml"
Provides-Extra: ml-viz
Requires-Dist: sentence-transformers>=2.2.0; extra == "ml-viz"
Requires-Dist: faiss-cpu>=1.7.0; extra == "ml-viz"
Requires-Dist: numpy>=1.24.0; extra == "ml-viz"
Requires-Dist: scikit-learn>=1.3.0; extra == "ml-viz"
Requires-Dist: matplotlib>=3.8.0; extra == "ml-viz"
Provides-Extra: dev
Requires-Dist: pytest>=8.1.1; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23.6; extra == "dev"
Requires-Dist: pytest-cov>=5.0.0; extra == "dev"
Requires-Dist: black>=24.3.0; extra == "dev"
Requires-Dist: ruff>=0.3.5; extra == "dev"
Dynamic: license-file

<div align="center">

```
██████╗ ███████╗███╗   ██╗██████╗  ██████╗ ████████╗
██╔══██╗██╔════╝████╗  ██║██╔══██╗██╔═══██╗╚══██╔══╝
██████╔╝█████╗  ██╔██╗ ██║██████╔╝██║   ██║   ██║   
██╔═══╝ ██╔══╝  ██║╚██╗██║██╔══██╗██║   ██║   ██║   
██║     ███████╗██║ ╚████║██████╔╝╚██████╔╝   ██║   
╚═╝     ╚══════╝╚═╝  ╚═══╝╚═════╝  ╚═════╝    ╚═╝   
```

<img src="docs/evidence/penbot_logo.png" alt="PenBot Logo" width="180"/>

### AI Chatbot Penetration Testing Framework

**Multi-Agent Security Testing for AI Systems**

[![PyPI version](https://img.shields.io/pypi/v/penbot.svg)](https://pypi.org/project/penbot/)
[![Pipeline Status](https://gitlab.com/yan-ban/penbot/badges/main/pipeline.svg)](https://gitlab.com/yan-ban/penbot/-/pipelines)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![OWASP LLM Top 10](https://img.shields.io/badge/OWASP-LLM%20Top%2010-orange.svg)](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
[![OWASP ASI](https://img.shields.io/badge/OWASP-ASI%202026-red.svg)](https://genai.owasp.org/initiatives/agentic-security/)
[![Contributions Welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg)](CONTRIBUTING.md)

</div>

> Multi-agent adversarial testing framework for AI chatbots and agentic systems. Combines domain-aware campaign planning, OWASP LLM Top 10 + ASI 2026 coverage, and **first-class Model Context Protocol (MCP) attack surface testing** — built for teams red-teaming production AI.

---

## Why PenBot?

The AI red-team tooling landscape has matured — `PyRIT`, `garak`, Meta's
`PurpleLlama`, and several commercial scanners all ship competent pattern
libraries. PenBot's position is deliberately narrow:

| Capability | Pattern scanners (garak, PurpleLlama) | Campaign frameworks (PyRIT) | **PenBot** |
|---|---|---|---|
| Static attack corpus | ✅ | ✅ | ✅ (1,120+ patterns) |
| Multi-turn orchestration | ⚠️ limited | ✅ | ✅ |
| Domain-aware campaign planning | ❌ | ⚠️ manual | ✅ automatic |
| Guardrail fingerprinting & targeted bypass | ❌ | ❌ | ✅ |
| Finding chaining (compound exploits) | ❌ | ❌ | ✅ |
| **Model Context Protocol (MCP) attack surface** | ❌ | ❌ | ✅ **20 patterns, 8 vectors** |
| OWASP ASI 2026 (ASI-03 Tool Misuse) | ❌ | ❌ | ✅ |
| Agentic-system action-safety probes | ⚠️ | ⚠️ | ✅ |
| Regression baselines (CI-friendly) | ⚠️ | ⚠️ | ✅ |
| Purple-team / defense simulation mode | ❌ | ❌ | ✅ |

### Where PenBot is distinctive

1. **MCP protocol-surface testing** — the only open framework (as of
   April 2026) with a dedicated agent for tool-description poisoning,
   resource URI traversal, `list_changed` bait-and-switch, cross-server
   pivots, and sampling API abuse. Matches the OWASP ASI-03 Tool Misuse
   category directly.
2. **Domain-aware coordinator** — agents don't just run in sequence;
   the coordinator picks agents per campaign phase, applies refusal-rate
   boosts, and composes hybrid attacks when agent pairs have proven
   compound vectors.
3. **Two-pass generation** — attacks are drafted, critiqued, and refined
   by a separate reasoning LLM before hitting the target, measurably
   improving first-pass success rates against guarded targets.
4. **Finding chaining** — the detector layer cross-references findings
   across rounds, flagging compound exploits (e.g., information
   disclosure → privilege escalation → exfiltration) as a single
   higher-severity chain rather than three isolated issues.

### Precision & recall caveat

PenBot reports coverage at the *category* level ("full coverage of LLM01")
— it does **not** claim every instance will be caught in every target.
We publish per-category precision/recall against the internal benchmark
in each test report, and `penbot benchmark` scores detection against
intentionally vulnerable mock chatbots so you can calibrate thresholds
for your environment before trusting a finding count.

### Production data point

First production test against a live AI chatbot:

| Metric | Result |
|--------|--------|
| Vulnerabilities found | 15 |
| Test duration | 63 minutes (60 rounds) |
| Success rate | 25% |
| Domain identification | Round 1 |
| Key finding | **CRITICAL** stored XSS in admin panel via payload logging (fixed immediately) |

---

## Quick Start

### Option 1: Install from PyPI (Recommended)

```bash
# Core install — CLI + REST API testing
pip install penbot

# Full install — adds dashboard, Playwright browser automation, PDF/DOCX reports, OpenAI support
pip install penbot[full]

# ML install — adds embedding-based attack memory (sentence-transformers, FAISS)
pip install penbot[ml]
```

### Option 2: Install from Source

```bash
git clone https://gitlab.com/yan-ban/penbot.git
cd penbot
pip install -e .        # Core
pip install -e ".[full]" # Full (optional)
```

### Option 3: Docker

```bash
docker pull registry.gitlab.com/yan-ban/penbot:latest
docker run -it -e ANTHROPIC_API_KEY=sk-ant-... registry.gitlab.com/yan-ban/penbot penbot --help
```

### Run PenBot

```bash
# 1. First-run setup (creates .env, configures API keys, installs browsers)
penbot onboard

# 2. Configure target (interactive wizard)
penbot wizard

# 3. Run test
penbot test --config configs/clients/your-target.yaml

# Verify your environment anytime
penbot doctor
```

**Quick smoke test:**
```bash
penbot test --config configs/example.yaml --quick
```

**Start dashboard:**
```bash
penbot dashboard
# Home / overview:       http://localhost:8000/dashboard
# Live Mission Control:  http://localhost:8000/dashboard/live
# Session replay:        http://localhost:8000/dashboard/session?id=<session_id>
# OWASP compliance:      http://localhost:8000/dashboard/owasp
```

---

## Features

### Security Testing
- **14 specialized agents** — Jailbreak, encoding, social engineering, RAG, tool exploitation, **MCP exploit**, exfiltration, indirect injection, action safety, compliance, and more
- **1,398+ attack patterns** — Curated across 27 pattern libraries (including 20 MCP protocol-attack patterns) and continuously evolved
- **22 vulnerability detectors** — Two-layer detection (pattern + LLM) including SSRF, guardrail fingerprinting, and finding chaining
- **OWASP LLM Top 10 (2025) + ASI (2026) coverage** — 9/10 LLM categories + dedicated ASI-03 Tool Misuse coverage
- **Model Context Protocol (MCP) testing** — Tool-description poisoning, resource URI traversal, list_changed bait-and-switch, cross-server pivots, sampling API abuse
- **Automatic tool & API discovery** — Runtime probing detects tools, functions, and APIs exposed by the target
- **Persistence verification** — Post-test replay confirms findings are reproducible, not transient
- **Endpoint reconnaissance** — Two-phase systematic API surface mapping with framework detection

### Intelligence
- **Think-MCP reasoning** — Draft→refine critique cycle, consensus validation, post-response learning
- **Domain awareness** — LLM-powered domain adaptation in subagent pipeline
- **Attack graphs** — UCB1 planning + live vis.js dashboard graph
- **Strategic guidance** — Think-MCP generates per-round strategy that flows to agents
- **Structured session summaries** — JSON summaries replace lossy text for agent context
- **Cross-agent learning** — Patterns persist across sessions
- **Agent learning loop** — Agents track success/failure per round, restore state on restart
- **Phase intelligence** — Multi-turn attack phases (recon→probe→exploit→persist) with agent boost/penalize
- **Evolutionary generation** — Novel attacks via genetic algorithms

### ML-Enhanced Attack Memory (v2.1)
- **Semantic retrieval** — `sentence-transformers` + FAISS replaces filter+recency with cosine-similarity nearest-neighbour search
- **Automatic migration** — Existing JSONL attack history is indexed on first use, zero manual steps
- **Evolutionary boost** — `EvolutionaryAgent` selects parent attacks by semantic relevance to the current campaign, not just recency
- **Evaluation framework** — MRR, Precision@k, Recall@k comparison between old and new retrieval
- **Embedding visualisation** — Jupyter notebook with t-SNE projection, cluster analysis, similarity heatmaps
- **Graceful degradation** — Falls back to original `AttackMemoryStore` when ML deps are absent

### Monitoring
- **Web dashboard** — Home overview, live Mission Control, session replay, OWASP report
- **Real-time streaming** — WebSocket push of attacks, findings, and graph updates
- **Attack chain replay** — Step-by-step post-test analysis
- **Interactive graph** — Visualize attack paths
- **Detailed reports** — HTML with OWASP mapping
- **Benchmark suite** — Score PenBot against intentionally vulnerable mock chatbots

### Flexibility
- **REST API** or **browser automation** (Playwright)
- **YAML configuration** — Easy target setup
- **Docker deployment** — Production-ready
- **Checkpointing** — Resume long-running tests
- **JWT auth + API keys** — Multi-tenant API access for teams
- **Continuous testing** — `penbot watch` re-runs on config/code changes, CI templates included

---

## Screenshots

### Mission Control Dashboard

Real-time attack monitoring with interactive graph visualization, campaign metrics, and confirmed findings.

<p align="center">
  <img src="docs/evidence/dashboard_with_findings.png" alt="PenBot Dashboard with Findings" width="100%"/>
</p>

### CLI Orchestration

Multi-agent coordination with dual-model architecture (Claude Sonnet 4.5 for analysis, Claude 3.7 Sonnet for attack generation).

<p align="center">
  <img src="docs/evidence/cli_initialization.png" alt="CLI Initialization" width="80%"/>
</p>

### Agent Voting & Consensus

Transparent decision-making: agents vote on attack strategies with scored reasoning.

<p align="center">
  <img src="docs/evidence/agent_voting.png" alt="Agent Voting Mechanism" width="80%"/>
</p>

### Subagent Refinement Pipeline

Attacks refined through psychological enhancement and stealth layers before execution.

<p align="center">
  <img src="docs/evidence/subagent_refinement.png" alt="Subagent Refinement" width="80%"/>
</p>

---

## CLI Commands

```bash
penbot onboard   # First-run setup (env, API keys, browsers)
penbot doctor    # Check environment health
penbot wizard    # Configure new target
penbot test      # Run security test
penbot dashboard # Start Mission Control
penbot sessions  # Manage past sessions
penbot agents    # Browse 14 agents
penbot patterns  # Search attack library
penbot report    # Generate report
penbot benchmark # Score detection against mock chatbots
penbot watch     # Continuous testing (re-run on config change)
```

See [CLI Reference](docs/CLI_REFERENCE.md) for full documentation.

---

## Documentation

| Document | Description |
|----------|-------------|
| [**Developer Guide**](docs/DEVELOPER_GUIDE.md) | How PenBot works under the hood |
| [**Architecture**](docs/ARCHITECTURE.md) | System design & diagrams |
| [**Methodology**](docs/METHODOLOGY.md) | Attack strategies |
| [**Configuration**](docs/CONFIGURATION.md) | YAML & environment setup |
| [**CLI Reference**](docs/CLI_REFERENCE.md) | Command-line usage |
| [**API Reference**](docs/API_REFERENCE.md) | REST & WebSocket |
| [**Agents**](docs/AGENTS.md) | Agent system details |
| [**Detection**](docs/DETECTION.md) | Vulnerability detectors |
| [**Advanced**](docs/ADVANCED.md) | RAG, tools, evolutionary |
| [**OWASP Coverage**](docs/OWASP_COVERAGE.md) | Compliance mapping |
| [**Test Example**](docs/TEST_EXAMPLE.md) | Real test walkthrough |

---

## Responsible Use

### ⚠️ Authorized Testing Only

This tool is for **authorized security testing only**.

**Permitted:**
- Testing your own AI chatbots
- Security research with written permission
- Red team exercises (with contract)
- Pre-deployment validation

**Prohibited:**
- Testing without authorization
- Attacking production systems maliciously
- Extracting proprietary data
- Bypassing security for unauthorized access

**Built-in safeguards:**
- Authorization verification
- Blocklist for public AI services
- Rate limiting
- Comprehensive audit logging

---

## Technology

- **LangGraph** — Multi-agent workflow orchestration
- **Claude Sonnet 4.5** — Attack generation
- **FastAPI** — API + WebSocket server (requires `penbot[full]`)
- **Playwright** — Browser automation (requires `penbot[full]`)
- **SQLite** — Session persistence

### Install Extras

| Extra | Command | What it adds |
|-------|---------|-------------|
| Core | `pip install penbot` | CLI, REST API testing, 13 security agents, 26 attack pattern libraries |
| Full | `pip install penbot[full]` | Dashboard, Playwright, PDF/DOCX reports, OpenAI provider, Tavily recon |
| Recon | `pip install penbot[recon]` | Tavily web search for target reconnaissance |
| Think | `pip install penbot[think]` | MCP-based enhanced reasoning |
| ML | `pip install penbot[ml]` | Embedding-based attack memory (sentence-transformers, FAISS) |
| ML-Viz | `pip install penbot[ml-viz]` | ML + scikit-learn & matplotlib for notebooks |

---

## Project Status

| Aspect | Status |
|--------|--------|
| Development | Under Active Development |
| Tests | 1,330+ passing ✅ |
| Skipped | 11 (optional PDF/DOCX deps) |
| Docker | Multi-stage build |

---

## License

MIT License — See [LICENSE](LICENSE)

---

## References

### Academic Papers

- Kumar, V., Liao, Z., Jones, J., & Sun, H. (2024). *"AmpleGCG-Plus: A Strong Generative Model of Adversarial Suffixes to Jailbreak LLMs with Higher Success Rates in Fewer Attempts."* [arXiv:2410.22143](https://arxiv.org/abs/2410.22143)

- Zhang, J., et al. (2025). *"Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity."* [arXiv:2510.01171](https://arxiv.org/abs/2510.01171)

### Acknowledgments

- [Elder Plinius / L1B3RT4S](https://github.com/elder-plinius) — Jailbreak pattern research
- [Manus AI](https://manus.im) — Context engineering principles
- [LangChain](https://github.com/langchain-ai/langgraph) — LangGraph framework
- [Anthropic](https://anthropic.com)
- [OWASP](https://owasp.org) — LLM Top 10 framework

---

<div align="center">

**Built for a more secure AI future**

[📚 Docs](docs/) · [🏗️ Architecture](docs/ARCHITECTURE.md) · [📝 Example](docs/TEST_EXAMPLE.md)

</div>
