Metadata-Version: 2.4
Name: docpull
Version: 1.3.0
Summary: Pull documentation from the web and convert to clean markdown
Author-email: Zachary Roth <support@raintree.technology>
Maintainer-email: Raintree Technology <support@raintree.technology>
License-Expression: MIT
Project-URL: Homepage, https://github.com/raintree-technology/docpull
Project-URL: Documentation, https://github.com/raintree-technology/docpull#readme
Project-URL: Repository, https://github.com/raintree-technology/docpull
Project-URL: Source Code, https://github.com/raintree-technology/docpull
Project-URL: Bug Tracker, https://github.com/raintree-technology/docpull/issues
Project-URL: Changelog, https://github.com/raintree-technology/docpull/blob/main/CHANGELOG.md
Keywords: python,markdown,documentation,web-scraping,developer-tools,claude,ai-training-data
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Education
Classifier: Environment :: Console
Classifier: Topic :: Documentation
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Documentation
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Topic :: Utilities
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31.0
Requires-Dist: beautifulsoup4>=4.12.0
Requires-Dist: html2text>=2020.1.16
Requires-Dist: defusedxml>=0.7.1
Requires-Dist: extruct>=0.15.0
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: rich>=13.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: gitpython>=3.1.40
Provides-Extra: js
Requires-Dist: playwright>=1.40.0; extra == "js"
Provides-Extra: all
Requires-Dist: playwright>=1.40.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: bandit>=1.7.0; extra == "dev"
Requires-Dist: pip-audit>=2.0.0; extra == "dev"
Requires-Dist: pre-commit>=3.0.0; extra == "dev"
Requires-Dist: types-requests>=2.31.0; extra == "dev"
Requires-Dist: types-beautifulsoup4>=4.12.0; extra == "dev"
Requires-Dist: types-defusedxml>=0.7.0; extra == "dev"
Requires-Dist: types-pyyaml>=6.0.0; extra == "dev"
Dynamic: license-file

# docpull

**Pull documentation from any website and converts it into clean, AI-ready Markdown.**
Fast, type-safe, secure, and optimized for building knowledge bases or training datasets.

**NEW in v1.3.0**: Rich structured metadata extraction (Open Graph, JSON-LD) for enhanced AI/RAG integration.

**v1.2.0**: 15 major features including language filtering, deduplication, auto-indexing, multi-source configuration, and more. Real-world testing shows **58% size reduction** with automatic optimization.

[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![PyPI version](https://badge.fury.io/py/docpull.svg)](https://badge.fury.io/py/docpull)
[![License: MIT](https://img.shields.io/github/license/raintree-technology/docpull)](https://github.com/raintree-technology/docpull/blob/main/LICENSE)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Type checked: mypy](https://img.shields.io/badge/type%20checked-mypy-blue.svg)](http://mypy-lang.org/)
[![Security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg)](https://github.com/PyCQA/bandit)

## Why docpull?

Unlike tools like wget or httrack, docpull extracts only the main content, removing ads, navbars, and clutter. Output is clean Markdown with optional YAML frontmatter—ideal for RAG systems, offline docs, or ML pipelines.

## Key Features

### Core Features (v1.0+)
- Works on any documentation site
- Smart extraction of main content
- Async + parallel fetching (up to 10× faster)
- Optional JavaScript rendering via Playwright
- Sitemap + link crawling
- Rate limiting, timeouts, content-type checks
- Saves docs in structured Markdown with YAML metadata
- Built-in Stripe profile as reference implementation (custom profiles easily added)

### NEW in v1.3.0: Rich Metadata Extraction
- **Structured Metadata**: Extract Open Graph, JSON-LD, and microdata during fetch
- **Enhanced Frontmatter**: Adds author, description, keywords, images, publish dates, and more
- **AI/RAG Ready**: Richer context for embeddings and retrieval systems
- **Opt-in Feature**: Enabled with `--rich-metadata` flag

### v1.2.0: Advanced Optimization
- **Language Filtering**: Auto-detect and filter by language (skip 352+ translation files)
- **Deduplication**: Remove duplicates with SHA-256 hashing (save 10+ MB on duplicate content)
- **Auto-Index Generation**: Create navigable INDEX.md with tree/TOC/categories/stats
- **Size Limits**: Control file and total download size (skip/truncate oversized files)
- **Multi-Source Configuration**: Configure multiple docs in one YAML file
- **Selective Crawling**: Include/exclude URL patterns for targeted fetching
- **Content Filtering**: Remove verbose sections (Examples, Changelog, etc.)
- **Format Conversion**: Output to Markdown, TOON (compact), JSON, or SQLite
- **Smart Naming**: 4 naming strategies (full, short, flat, hierarchical)
- **Metadata Extraction**: Extract titles, URLs, stats to metadata.json
- **Update Detection**: Only download changed files (checksums, ETags)
- **Incremental Mode**: Resume interrupted downloads with checkpointing
- **Hooks & Plugins**: Python plugin system for custom processing
- **Git Integration**: Auto-commit changes with customizable messages
- **Archive Mode**: Create tar.gz/zip archives for distribution

**Real-world impact**: Testing with 1,914 files (31 MB) → **13 MB (58% reduction)** with all optimizations enabled.

## Quick Start

```bash
pip install docpull
docpull --doctor         # verify installation

# Basic usage
docpull https://aptos.dev
docpull stripe           # use a built-in profile

# NEW: Simple optimization (v1.2.0)
docpull https://code.claude.com/docs --language en --create-index

# NEW: Rich metadata extraction (v1.3.0)
docpull https://docs.anthropic.com --rich-metadata --create-index

# NEW: Advanced optimization (v1.2.0)
docpull https://aptos.dev \
  --deduplicate \
  --keep-variant mainnet \
  --max-file-size 200kb \
  --create-index

# NEW: Multi-source configuration (v1.2.0)
docpull --sources-file examples/multi-source-optimized.yaml
```

### JavaScript-heavy sites

```bash
pip install docpull[js]
python -m playwright install chromium
docpull https://site.com --js
```

## Python API

```python
from docpull import GenericAsyncFetcher

fetcher = GenericAsyncFetcher(
    url_or_profile="https://aptos.dev",
    output_dir="./docs",
    max_pages=100,
    max_concurrent=20,
)
fetcher.fetch()
```

## Common Options

### Core Options
- `--doctor` – verify installation and dependencies
- `--max-pages N` – limit crawl size
- `--max-depth N` – restrict link depth
- `--max-concurrent N` – control parallel fetches
- `--js` – enable Playwright rendering
- `--output-dir DIR` – output directory
- `--rate-limit X` – seconds between requests
- `--no-skip-existing` – re-download existing files
- `--dry-run` – test without downloading

### NEW in v1.2.0: Optimization Options
- `--language LANG` – filter by language (e.g., `en`)
- `--exclude-languages LANG [LANG ...]` – exclude languages
- `--deduplicate` – remove duplicate files
- `--keep-variant PATTERN` – keep files matching pattern when deduplicating
- `--max-file-size SIZE` – max file size (e.g., `200kb`, `1mb`)
- `--max-total-size SIZE` – max total download size
- `--include-paths PATTERN [PATTERN ...]` – only crawl matching URLs
- `--exclude-paths PATTERN [PATTERN ...]` – skip matching URLs
- `--exclude-sections NAME [NAME ...]` – remove sections by header name
- `--format {markdown,toon,json,sqlite}` – output format
- `--naming-strategy {full,short,flat,hierarchical}` – file naming strategy
- `--create-index` – generate INDEX.md with navigation
- `--extract-metadata` – extract metadata to metadata.json
- `--rich-metadata` – extract rich structured metadata (Open Graph, JSON-LD) during fetch
- `--update-only-changed` – only download changed files
- `--incremental` – enable incremental mode with resume
- `--git-commit` – auto-commit changes
- `--git-message MSG` – commit message template
- `--archive` – create compressed archive
- `--archive-format {tar.gz,tar.bz2,tar.xz,zip}` – archive format
- `--sources-file PATH` – multi-source configuration file

See `docpull --help` for complete list of options.

## Performance

Async fetching drastically reduces runtime:

| Pages | Sync | Async | Speedup |
|-------|------|-------|---------|
| 50 | ~50s | ~6s | 8× faster |

Higher concurrency yields even better results.

## Output Format

Each downloaded page becomes a Markdown file:

```markdown
---
url: https://stripe.com/docs/payments
fetched: 2025-11-13
---
# Payment Intents
...
```

With `--rich-metadata`, the frontmatter includes Open Graph, JSON-LD, and other structured metadata:

```markdown
---
url: https://stripe.com/docs/payments
fetched: 2025-11-13
title: Accept a payment
description: Learn how to accept payments with the Payment Intents API
author: Stripe
keywords: [payments, api, stripe, checkout]
image: https://stripe.com/img/docs-preview.png
type: article
site_name: Stripe Documentation
---
# Payment Intents
...
```

Directory layout mirrors the target site's structure.

## Configuration File

### Simple Configuration (v1.0+)

```yaml
output_dir: ./docs
rate_limit: 0.5
sources:
  - stripe  # Built-in profile
  - https://docs.example.com  # Or any URL
```

Run with:
```bash
docpull --config config.yaml
```

### NEW: Multi-Source Configuration (v1.2.0)

```yaml
sources:
  anthropic:
    url: https://docs.anthropic.com
    language: en
    max_file_size: 200kb
    create_index: true
    rich_metadata: true  # Extract Open Graph, JSON-LD metadata

  claude-code:
    url: https://code.claude.com/docs
    language: en  # Skips 352 translation files!
    create_index: true

  aptos:
    url: https://aptos.dev
    deduplicate: true
    keep_variant: mainnet  # Skips 304 duplicates!
    max_file_size: 200kb
    include_paths:
      - "build/guides/*"

output_dir: ./docs
rate_limit: 0.5
git_commit: true
git_message: "Update docs - {date}"
extract_metadata: true
archive: true
```

Run with:
```bash
docpull --sources-file config.yaml
```

See `examples/` directory for more configuration examples.

## Custom Profiles

docpull includes a Stripe profile as reference. Create custom profiles for other sites:

```python
from docpull.profiles.base import SiteProfile

MY_PROFILE = SiteProfile(
    name="mysite",
    domains={"docs.mysite.com"},
    include_patterns=["/docs/", "/api/"],
    sitemap_url="https://docs.mysite.com/sitemap.xml",
    rate_limit=0.5,
)
```

**Want to contribute profiles?** Submit a PR with your custom profile! Popular ones may be added to the core or a community profiles repository.

## Security

- HTTPS-only
- Blocks private network IPs
- 50MB page size limit
- Timeout controls
- Validates content-type
- Playwright sandboxing

## Troubleshooting

- **Installation issues**: Run `docpull --doctor` to diagnose problems
- **Missing dependencies**: See [TROUBLESHOOTING.md](TROUBLESHOOTING.md) for common fixes
- **Site requires JS**: install Playwright + `--js`
- **Slow or rate limited**: lower concurrency or raise `--rate-limit`
- **Large sites**: set `--max-pages`

For detailed troubleshooting, see [TROUBLESHOOTING.md](TROUBLESHOOTING.md).

## v1.2.0 Feature Examples

### Language Filtering
Automatically detect and filter documentation by language:
```bash
# English only (auto-detects /en/, _en_, docs_en_, etc.)
docpull https://code.claude.com/docs --language en --create-index
```
**Impact**: Claude Code docs downloaded in 9 languages = 352 unnecessary files for English-only users.

### Deduplication
Remove duplicate files based on content hash:
```bash
# Keep mainnet version, skip testnet/devnet duplicates
docpull https://aptos.dev --deduplicate --keep-variant mainnet --create-index
```
**Impact**: Aptos Move reference docs across 3 environments = 304 duplicate files (~10 MB saved).

### Format Conversion
Convert to different formats for various use cases:
```bash
# TOON format (40-60% size reduction, optimized for LLMs)
docpull https://docs.anthropic.com --format toon --language en

# SQLite database with full-text search
docpull https://docs.anthropic.com --format sqlite --language en

# Structured JSON
docpull https://docs.anthropic.com --format json --language en
```

### Incremental Updates
Only download changed files:
```bash
docpull https://docs.anthropic.com \
  --incremental \
  --update-only-changed \
  --git-commit \
  --git-message "Update docs - {date}"
```
**Use case**: Regular documentation updates with minimal bandwidth usage.

### Complete Optimization Pipeline
Combine all optimizations:
```bash
docpull --sources-file examples/multi-source-optimized.yaml
```
See `examples/` directory for comprehensive configuration examples.

**Real-world results**: Testing with 4 documentation sources (Anthropic, Claude Code, Aptos, Shelby):
- **Before**: 1,914 files, 31 MB, no navigation
- **After**: 1,250 files, 13 MB (58% reduction), full indexes generated
- **One command** instead of 4+ separate commands with manual optimization

## What's New in v1.3.0

This release adds rich structured metadata extraction for better AI/RAG integration.

**New Feature**:
- **Rich Metadata Extraction**: Extract Open Graph, JSON-LD, microdata, and other structured metadata during fetch
  - Adds author, description, keywords, images, publish dates, and more to frontmatter
  - Enhances AI/RAG systems with richer context
  - Enabled with `--rich-metadata` flag or `rich_metadata: true` in config
  - Powered by the extruct library

**Example enhanced frontmatter**:
```yaml
---
url: https://docs.example.com/guide
fetched: 2025-11-20
title: Getting Started Guide
description: Learn the basics of our platform
author: John Doe
keywords: [tutorial, guide, api]
image: https://docs.example.com/og-image.png
type: article
published_time: 2024-01-15T10:00:00Z
---
```

**Backward Compatible**: All existing workflows continue to work unchanged. Rich metadata is opt-in.

## What's New in v1.2.0

This release adds 15 major features across 4 phases. See [CHANGELOG.md](CHANGELOG.md) for complete release notes.

**Highlights**:
- Multi-source YAML configuration
- Language filtering with auto-detection
- SHA-256 based deduplication
- Auto-index generation (tree, TOC, categories, stats)
- 4 output formats (Markdown, TOON, JSON, SQLite)
- Incremental mode with resume capability
- Git integration and archive creation
- Python plugin/hook system

**Backward Compatible**: All v1.0+ workflows continue to work unchanged.

## Links

- [PyPI](https://pypi.org/project/docpull/)
- [GitHub](https://github.com/raintree-technology/docpull)
- [Issues](https://github.com/raintree-technology/docpull/issues)
- [Changelog](https://github.com/raintree-technology/docpull/blob/main/CHANGELOG.md)
- [Examples](https://github.com/raintree-technology/docpull/tree/main/examples)

## License

MIT License - see [LICENSE](LICENSE) file for details
