Metadata-Version: 2.4
Name: securitybench
Version: 0.1.9
Summary: Security Bench - Security testing for LLM pipelines
Author: Security Bench Team
License: Elastic-2.0
License-File: LICENSE
Requires-Python: >=3.9
Requires-Dist: click>=8.0
Requires-Dist: httpx>=0.24
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# Security Bench

**Website:** [securitybench.ai](https://securitybench.ai)

Security testing framework for AI/LLM pipelines - Test your AI systems for prompt injection, jailbreaks, data leakage, and other security vulnerabilities.

## Overview

Security Bench is a comprehensive security testing tool designed for real-world AI deployments. Unlike benchmarks that only test base models, Security Bench tests your **entire pipeline** - including RAG systems, tool-calling agents, and multi-component architectures.

### Key Features

- **Two Testing Modes** - LLM endpoint scanning AND local code/config auditing
- **Pipeline-First Testing** - Test deployed systems, not just models
- **Lynis-Style Auditing** - Rich terminal output with grades, findings, and remediation
- **Privacy-Preserving** - 100% local execution, no data leaves your environment
- **Comprehensive Coverage** - 500+ LLM tests, 266 local checks across 32 categories
- **Simple to Advanced** - Zero-config for quick scans, detailed config for complex systems

## Quick Start

```bash
# Install
pip install securitybench

# Scan an LLM endpoint for vulnerabilities
sb scan https://api.example.com/chat

# Balanced scan (recommended for benchmarking)
sb scan https://api.example.com/chat --balanced

# Audit local project for security issues
sb audit

# Code-only analysis
sb code ./src
```

### LLM Endpoint Testing

```bash
# Quick security check
sb scan https://api.example.com/chat --header "Authorization: Bearer sk-..." --limit 20

# Balanced scan across all 31 attack categories
sb scan https://api.example.com/chat --balanced

# Full scan with configuration file
sb scan --config securitybench.yaml
```

### Local Security Auditing (Lynis-style)

```bash
# Full audit (code + config + infrastructure)
sb audit

# Code analysis only (prompt injection patterns, secrets, etc.)
sb code

# Configuration checks only (exposed keys, insecure settings)
sb config

# Infrastructure checks only (Docker, K8s, permissions)
sb infra

# Get remediation guidance for a specific finding
sb fix CODE-001
```

## Test Modes

Security Bench provides flexible test modes to balance speed vs. coverage:

| Mode | Tests | Use Case |
|------|-------|----------|
| `--limit 20` | 20 random | Quick smoke test |
| `--limit 50` | 50 random | Default, general testing |
| `--balanced` | 155 (5 × 31 categories) | **Recommended for benchmarking** |
| `--balanced --per-category 10` | 310 | Thorough coverage |
| `--categories SPE,PIN` | Filtered | Focus on specific attacks |

### Balanced Mode (Recommended)

The `--balanced` flag ensures even sampling across all 31 attack categories:

```bash
sb scan https://api.example.com/chat --balanced --delay 2
```

- **Default**: 5 tests per category = 155 tests total
- **Adjustable**: `--per-category 10` for 310 tests
- **Stable**: Always includes all categories, even as the test database grows
- **Comparable**: Results are comparable across models on the leaderboard

### Rate Limiting

Use `--delay` to avoid rate limits when testing external APIs:

```bash
sb scan https://openrouter.ai/api/v1 -m gpt-4 --balanced --delay 2
```

This adds a 2-second pause between API calls (155 tests × 2s ≈ 5 minutes).

## Test Categories (32 Total)

Security Bench provides comprehensive coverage across **32 attack categories**, expandable as new techniques are discovered.

### Core Categories (10 with tests, 22 planned)

**Injection & Manipulation:**
- **SPE** - System Prompt Extraction (14,138 tests)
- **PIN** - Prompt Injection Direct (64 tests)
- **IND** - Indirect Injection (114 tests)
- **JBR** - Jailbreak (123 tests)
- **OBF** - Obfuscation (84 tests)
- **MTM** - Multi-Turn Manipulation (19 tests)
- **GHJ** - Goal Hijacking
- **CTX** - Context Manipulation

**Information Leakage:**
- **ILK** - Information Leakage (3,298 tests)
- **SEC** - Secret Extraction (115 tests)
- **EXF** - Data Exfiltration
- **MEX** - Model Extraction

**Code & Execution:**
- **CEX** - Code Execution (331 tests)
- **OPS** - Output Manipulation

**Advanced Threats:**
- **AGY** - Excessive Agency
- **RAG** - RAG/Vector Poisoning
- **VEC** - Vector/Embedding Attacks
- **MEM** - Memory Poisoning
- **IAT** - Inter-Agent Trust
- **MCP** - Model Context Protocol
- **COT** - Chain-of-Thought Manipulation
- **IMG** - Multi-modal Injection

**Plus 10 more:** Social Engineering, Compliance, Hallucination Exploitation, Resource Exhaustion, and others.

See [SB_TAXONOMY_V2.1_FLAT.md](SB_TAXONOMY_V2.1_FLAT.md) for complete taxonomy.

## Editions

### Community (Open Source)
- Core testing engine
- ~330 baseline security tests
- CLI and MCP server
- Apache 2.0 License

### PRO (Commercial)
- +750 advanced tests
- Industry-specific attack scenarios
- Continuous updates with latest techniques
- Priority support

## Documentation

- [Installation Guide](docs/installation.md)
- [Configuration](docs/configuration.md)
- [MCP Server Setup](docs/mcp-server.md)
- [Test Categories](docs/categories.md)

## Evaluation

Security Bench uses an LLM-as-Judge approach for evaluating test results. The judge model analyzes responses to determine if attacks succeeded (e.g., system prompt leaked, jailbreak bypassed safety filters). For specific attack types like TensorTrust, pattern matching is used instead.

## Project Status

**Status:** Beta
**Version:** 0.1.0

**Working:**
- ✅ LLM endpoint scanning (`sb scan`)
- ✅ Local security auditing (`sb audit`, `sb code`, `sb config`, `sb infra`)
- ✅ Rich terminal output with grades and findings
- ✅ JSON output for CI/CD integration

**Coming Soon:**
- MCP server for AI assistant integration
- HTML/PDF report generation

## Contributing

We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## Acknowledgments

See [ACKNOWLEDGMENTS.md](ACKNOWLEDGMENTS.md) for research inspiration and credits.

## License

Elastic License 2.0 (ELv2) - Free to use, but you may not offer it as a hosted service. See [LICENSE](LICENSE) for details.

---

**Made with ❤️ for the AI security community**
