Metadata-Version: 2.4
Name: graphsense-lib
Version: 2.9.12
Summary: Graphsense backend lib and automation cli
Author-email: Iknaio Cryptoasset Analytics GmbH <contact@iknaio.com>
Project-URL: Homepage, https://graphsense.github.io/
Project-URL: Source, https://github.com/graphsense/graphsense-lib
Project-URL: Changelog, https://github.com/graphsense/graphsense-lib/blob/master/CHANGELOG.md
Project-URL: Tracker, https://github.com/graphsense/graphsense-lib/issues
Project-URL: Download, https://github.com/graphsense
Keywords: graphsense
Platform: any
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Intended Audience :: Developers
Classifier: Topic :: Utilities
Requires-Python: <3.14,>=3.10
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
License-File: LICENSE
Requires-Dist: setuptools<80.9,>=80.0.0
Requires-Dist: filelock>=3.8.0
Requires-Dist: click>=8.0.3
Requires-Dist: pandas>=2.3.3
Requires-Dist: methodtools>=0.4
Requires-Dist: simplejson>=3.17.6
Requires-Dist: goodconf[yaml]>=3.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydantic-settings<2.13.0,>=2.0.0
Requires-Dist: requests>=2.32.5
Requires-Dist: parsy<3.0,>=2.0
Requires-Dist: rich>=12.6.0
Requires-Dist: cashaddress>=1.0.6
Requires-Dist: base58>=2.1
Requires-Dist: bitarray>=2.9
Requires-Dist: eth-event>=1.2.3
Requires-Dist: eth-hash>=0.3.0
Requires-Dist: cassandra-driver>=3.27.0
Requires-Dist: async-lru>=2.0.4
Requires-Dist: tenacity>=8.2.3
Requires-Dist: coincurve>=21.0.0
Requires-Dist: bech32>=1.2.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: ecdsa>=0.13
Requires-Dist: eth-account>=0.13.7
Provides-Extra: ingest
Requires-Dist: graphsense-ethereum-etl>=2.4.6; extra == "ingest"
Requires-Dist: chainside-btcpy>=0.6; extra == "ingest"
Requires-Dist: graphsense-bitcoin-etl>=1.5.5; extra == "ingest"
Requires-Dist: lz4>=4.3.2; extra == "ingest"
Requires-Dist: pyarrow>=23.0.0; extra == "ingest"
Requires-Dist: deltalake>=1.4.2; extra == "ingest"
Requires-Dist: duckdb>=1.0.0; extra == "ingest"
Requires-Dist: grpcio>=1.49.0; extra == "ingest"
Requires-Dist: redis>=5.0.0; extra == "ingest"
Provides-Extra: conversions
Requires-Dist: networkx>=3.2.1; extra == "conversions"
Requires-Dist: matplotlib; extra == "conversions"
Provides-Extra: tagstore
Requires-Dist: sqlmodel>=0.0.22; extra == "tagstore"
Requires-Dist: asyncpg>=0.30.0; extra == "tagstore"
Requires-Dist: psycopg2-binary>=2.9.3; extra == "tagstore"
Requires-Dist: colorama>=0.4.6; extra == "tagstore"
Requires-Dist: fastapi[all]>=0.115.0; extra == "tagstore"
Requires-Dist: httpx<1.0.0; extra == "tagstore"
Requires-Dist: gitpython>=3.1; extra == "tagstore"
Requires-Dist: giturlparse>=0.10; extra == "tagstore"
Requires-Dist: lz4>=4.3.2; extra == "tagstore"
Requires-Dist: python-dotenv>=1.0.1; extra == "tagstore"
Requires-Dist: pyyaml-include~=1.3; extra == "tagstore"
Requires-Dist: pyyaml>=6.0; extra == "tagstore"
Requires-Dist: tabulate>=0.9.0; extra == "tagstore"
Requires-Dist: anytree>=2.9.0; extra == "tagstore"
Provides-Extra: tagpacks
Requires-Dist: graphsense-lib[tagstore]; extra == "tagpacks"
Requires-Dist: rapidyaml>=0.10.0; extra == "tagpacks"
Requires-Dist: rapidfuzz>=3.13.0; extra == "tagpacks"
Provides-Extra: web
Requires-Dist: graphsense-lib[conversions,tagstore]; extra == "web"
Requires-Dist: asgi-lifespan>=2.1.0; extra == "web"
Requires-Dist: fastapi>=0.115.0; extra == "web"
Requires-Dist: python-dateutil>=2.9.0; extra == "web"
Requires-Dist: redis>=5.0.0; extra == "web"
Requires-Dist: uvicorn[standard]>=0.32.0; extra == "web"
Requires-Dist: gunicorn>=23.0.0; extra == "web"
Provides-Extra: all
Requires-Dist: graphsense-lib[ingest]; extra == "all"
Requires-Dist: graphsense-lib[conversions]; extra == "all"
Requires-Dist: graphsense-lib[tagpacks]; extra == "all"
Requires-Dist: graphsense-lib[web]; extra == "all"
Dynamic: license-file

# GraphSense Library

[![Test and Build Status](https://github.com/graphsense/graphsense-lib/actions/workflows/run_tests.yaml/badge.svg)](https://github.com/graphsense/graphsense-lib/actions) [![PyPI version](https://badge.fury.io/py/graphsense-lib.svg)](https://badge.fury.io/py/graphsense-lib) [![Python](https://img.shields.io/pypi/pyversions/graphsense-lib)](https://pypi.org/project/graphsense-lib/) [![Downloads](https://static.pepy.tech/badge/graphsense-lib)](https://pepy.tech/project/graphsense-lib)

A comprehensive Python library for the GraphSense crypto-analytics platform. It provides database access, data ingestion, maintenance tools, and analysis capabilities for cryptocurrency transactions and networks.

> **Note:** This library uses optional dependencies. Use `graphsense-lib[all]` to install all features.

## Quick Start

### Installation

```bash
# Install with all features
uv add graphsense-lib[all]

# Install from source
git clone https://github.com/graphsense/graphsense-lib.git
cd graphsense-lib
make install
```

### Serving the REST API locally

The web API requires two backend connections: a **Cassandra** cluster (blockchain data) and a **TagStore** (PostgreSQL). You can configure them via environment variables or a YAML config file.

#### Option A: Environment variables only

```bash
GS_CASSANDRA_ASYNC_NODES='["<cassandra-host>"]' \
GRAPHSENSE_TAGSTORE_READ_URL='postgresql+asyncpg://<user>:<password>@<host>:<port>/tagstore' \
GS_CASSANDRA_ASYNC_CURRENCIES='{"btc":{"raw": "btc_raw", "transformed": "btc_transformed"},"eth":{}}' \
uv run --extra web uvicorn graphsenselib.web.app:create_app --factory --host localhost --port 9000 --reload
```

#### Option B: YAML config file

Point `CONFIG_FILE` to a REST-specific config (see `instance/config.yaml` for a full example):

```bash
CONFIG_FILE=./instance/config.yaml make serve-web
```

Or without Make:

```bash
CONFIG_FILE=./instance/config.yaml \
uv run --extra web uvicorn graphsenselib.web.app:create_app --factory --host localhost --port 9000 --reload
```

#### Option C: `.graphsense.yaml` with a `web` key

If you already have a `.graphsense.yaml` (or `~/.graphsense.yaml`) for the CLI, you can add a `web` key containing the REST config. The app will pick it up automatically without setting `CONFIG_FILE`:

```yaml
# .graphsense.yaml
environments:
  # ... your existing CLI config ...

web:
  database:
    nodes: ["<cassandra-host>"]
    currencies:
      btc:
      eth:
  gs-tagstore:
    url: "postgresql+asyncpg://<user>:<password>@<host>:<port>/tagstore"
```

```bash
make serve-web
```

**Config resolution order:** explicit `config_file` param > `CONFIG_FILE` env var > `./instance/config.yaml` > `.graphsense.yaml` `web` key > env vars only.

#### Optional REST settings (env vars)

| Variable | Default | Description |
|---|---|---|
| `GSREST_DISABLE_AUTH` | `false` | Disable API key authentication |
| `GSREST_ENSURE_TAGSTORE_SCHEMA_ON_STARTUP` | `false` | Auto-initialize TagStore tables/views at startup when missing |
| `GSREST_ALLOWED_ORIGINS` | `*` | CORS allowed origins |
| `GSREST_LOGGING_LEVEL` | — | Logging level (DEBUG, INFO, …) |
| `GS_CASSANDRA_ASYNC_PORT` | `9042` | Cassandra port |
| `GS_CASSANDRA_ASYNC_USERNAME` | — | Cassandra username |
| `GS_CASSANDRA_ASYNC_PASSWORD` | — | Cassandra password |

When enabling `GSREST_ENSURE_TAGSTORE_SCHEMA_ON_STARTUP=true`, keep in mind:

- The DB user must have DDL privileges (create tables/views/indexes/extensions/procedures).
- Startup may be slower because schema checks and potential initialization run before the app serves traffic.
- In multi-replica deployments, initialize schema once (migration/init job) to avoid startup races.

If TagStore is not configured (`gs-tagstore` missing) or the TagStore URL is unreachable, the REST app now falls back to a mock TagStore so endpoints still work. In this mode, tag-specific responses (labels, actors, taxonomies, tag counts) are empty.

### Basic Usage

#### Database Access with Configuration File

```python
from graphsenselib.db import DbFactory

# Using GraphSense config file (default: ~/.graphsense.yaml)
with DbFactory().from_config("development", "btc") as db:
    highest_block = db.transformed.get_highest_block()
    print(f"Highest BTC block: {highest_block}")

    # Get block details
    block = db.transformed.get_block(100000)
    print(f"Block 100000: {block.block_hash}")
```

#### Direct Database Connection

```python
from graphsenselib.db import DbFactory

# Direct connection without config file
with DbFactory().from_name(
    raw_keyspace_name="eth_raw",
    transformed_keyspace_name="eth_transformed",
    schema_type="account",
    cassandra_nodes=["localhost"],
    currency="eth"
) as db:
    print(f"Highest block: {db.transformed.get_highest_block()}")
```

#### Async Database Services

The async services are used internally by the REST API and can also be used standalone. `AddressesService` depends on several other services:

```python
from graphsenselib.db.asynchronous.services import (
    BlocksService, AddressesService, TagsService,
    EntitiesService, RatesService,
)

# Services are initialized with their dependencies
blocks_service = BlocksService(db, rates_service, config, logger)
addresses_service = AddressesService(
    db, tags_service, entities_service, blocks_service, rates_service, logger
)

address_info = await addresses_service.get_address("btc", "1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa")
txs = await addresses_service.list_address_txs("btc", "1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa")
```

## Command Line Interface

GraphSense-lib exposes a comprehensive CLI tool: `graphsense-cli`

### Basic Commands

```bash
# Show help and available commands
graphsense-cli --help

# Check version
graphsense-cli version

# Show current configuration
graphsense-cli config show

# Generate config template
graphsense-cli config template > ~/.graphsense.yaml

# Show config file path
graphsense-cli config path
```

## Modules

### Database Management

Query and manage the GraphSense database state.

```bash
# Show database management options
graphsense-cli db --help

# Check database state/summary
graphsense-cli db state -e development

# Get block information
graphsense-cli db block info -e development -c btc --height 100000

# Query logs (for Ethereum-based chains)
graphsense-cli db logs -e development -c eth --from-block 1000000 --to-block 1000100
```

### Schema Operations

Create and validate database schemas.

```bash
# Show schema options
graphsense-cli schema --help

# Create database schema for a currency
graphsense-cli schema create -e dev -c btc

# Validate existing schema
graphsense-cli schema validate -e dev -c btc

# Show expected schema for currency
graphsense-cli schema show-by-currency btc

# Show schema by type (utxo/account)
graphsense-cli schema show-by-schema-type utxo
```

### Data Ingestion

Ingest raw cryptocurrency data from nodes.

```bash
# Show ingestion options
graphsense-cli ingest --help

# Ingest blocks from cryptocurrency node
graphsense-cli ingest from-node \
    -e dev \
    -c btc \
    --start-block 0 \
    --end-block 1000 \
    --create-schema

# Ingest with custom batch size
graphsense-cli ingest from-node \
    -e dev \
    -c eth \
    --start-block 1000000 \
    --end-block 1001000 \
    --batch-size 100
```

### Delta Updates

Update transformed keyspace from raw keyspace.

```bash
# Show delta update options
graphsense-cli delta-update --help

# Check update status
graphsense-cli delta-update status -e dev -c btc

# Perform delta update
graphsense-cli delta-update update -e dev -c btc

# Validate delta update consistency
graphsense-cli delta-update validate -e dev -c btc

# Patch exchange rates for specific blocks
graphsense-cli delta-update patch-exchange-rates \
    -e dev \
    -c btc \
    --start-block 100000 \
    --end-block 200000
```

### Exchange Rates

Fetch and ingest exchange rates from various sources.

```bash
# Show exchange rate options
graphsense-cli exchange-rates --help

# Fetch from CoinDesk
graphsense-cli exchange-rates coindesk -e dev -c btc

# Fetch from CoinMarketCap (requires API key in config)
graphsense-cli exchange-rates coinmarketcap -e dev -c btc
```

### Monitoring

Monitor GraphSense infrastructure health and state.

```bash
# Show monitoring options
graphsense-cli monitoring --help

# Get database summary
graphsense-cli monitoring get-summary -e dev

# Get summary for specific currency
graphsense-cli monitoring get-summary -e dev -c btc

# Send notifications to configured handlers
graphsense-cli monitoring notify \
    --topic "database-update" \
    --message "BTC ingestion completed"
```

### Event Watching (Alpha)

Watch for cryptocurrency events and generate notifications.

```bash
# Show watch options
graphsense-cli watch --help

# Watch for money flows on specific addresses
graphsense-cli watch money-flows \
    -e dev \
    -c btc \
    --address 1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa \
    --threshold 1000000  # satoshis
```

### File Conversion Tools

Convert between different file formats.

```bash
# Show conversion options
graphsense-cli convert --help
```

## Configuration

GraphSense-lib uses a YAML configuration file that defines database connections and environment settings. Default locations: `./.graphsense.yaml`, `~/.graphsense.yaml`.

### Generate Configuration Template

```bash
graphsense-cli config template > ~/.graphsense.yaml
```

### Example Configuration Structure

```yaml
# Optional: default environment to use
default_environment: dev

environments:
  dev:
    # Cassandra cluster configuration
    cassandra_nodes: ["localhost"]
    port: 9042
    # Optional authentication
    # username: "cassandra"
    # password: "cassandra"

    # Currency/keyspace configurations
    keyspaces:
      btc:
        raw_keyspace_name: "btc_raw"
        transformed_keyspace_name: "btc_transformed"
        schema_type: "utxo"

        # Node connection for ingestion
        ingest_config:
          node_reference: "http://localhost:8332"
          # Optional authentication for node
          # username: "rpcuser"
          # password: "rpcpassword"

        # Keyspace setup for schema creation
        keyspace_setup_config:
          raw:
            replication_config: "{'class': 'SimpleStrategy', 'replication_factor': 1}"
          transformed:
            replication_config: "{'class': 'SimpleStrategy', 'replication_factor': 1}"

      eth:
        raw_keyspace_name: "eth_raw"
        transformed_keyspace_name: "eth_transformed"
        schema_type: "account"

        ingest_config:
          node_reference: "http://localhost:8545"

        keyspace_setup_config:
          raw:
            replication_config: "{'class': 'SimpleStrategy', 'replication_factor': 1}"
          transformed:
            replication_config: "{'class': 'SimpleStrategy', 'replication_factor': 1}"

  prod:
    cassandra_nodes: ["cassandra1.prod", "cassandra2.prod", "cassandra3.prod"]
    username: "gs_user"
    password: "secure_password"

    keyspaces:
      btc:
        raw_keyspace_name: "btc_raw"
        transformed_keyspace_name: "btc_transformed"
        schema_type: "utxo"

        ingest_config:
          node_reference: "http://bitcoin-node.internal:8332"

        keyspace_setup_config:
          raw:
            replication_config: "{'class': 'NetworkTopologyStrategy', 'datacenter1': 3}"
          transformed:
            replication_config: "{'class': 'NetworkTopologyStrategy', 'datacenter1': 3}"

# Optional: Slack notification configuration
slack_topics:
  database-update:
    hooks: ["https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"]

  payment_flow_notifications:
    hooks: ["https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"]

# Optional: API keys for external services
coingecko_api_key: ""
coinmarketcap_api_key: "YOUR_CMC_API_KEY"

# Optional: cache directory for temporary files
cache_directory: "~/.graphsense/cache"
```

## Advanced Features

### Tagpack Management

GraphSense-lib includes comprehensive tagpack management tools (formerly standalone tagpack-tool). For detailed documentation, see [Tagpack README](tagpack/docs/README.md).

```bash
# Validate tagpacks
graphsense-cli tagpack-tool tagpack validate /path/to/tagpack

# Insert tagpack into tagstore
graphsense-cli tagpack-tool insert \
    --url "postgresql://user:pass@localhost/tagstore" \
    /path/to/tagpack

# Show quality measures
graphsense-cli tagpack-tool quality show-measures \
    --url "postgresql://user:pass@localhost/tagstore"
```

### Tagstore Operations

```bash
# Initialize tagstore database
graphsense-cli tagstore init

# Initialize with custom database URL
graphsense-cli tagstore init --db-url "postgresql://user:pass@localhost/tagstore"

# Get DDL SQL for manual setup
graphsense-cli tagstore get-create-sql
```

### Cross-chain Analysis

```python
# Using an initialized AddressesService (see above for setup)
related = await addresses_service.get_cross_chain_pubkey_related_addresses(
    "1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa"
)

for addr in related:
    print(f"Network: {addr.network}, Address: {addr.address}")
```

### Function Call Parsing

```python
from graphsenselib.utils.function_call_parser import parse_function_call

# Parse Ethereum function calls
function_signatures = {
    "0xa9059cbb": [{
        "name": "transfer",
        "inputs": [
            {"name": "to", "type": "address"},
            {"name": "value", "type": "uint256"}
        ]
    }]
}

parsed = parse_function_call(tx_input_bytes, function_signatures)
if parsed:
    print(f"Function: {parsed['name']}")
    print(f"Parameters: {parsed['parameters']}")
```

## Development

**Important:** Requires Python >=3.10, <3.13.

### Setup Development Environment

```bash
# Initialize development environment (installs deps + pre-commit hooks)
make dev

# Or install dev dependencies only
make install-dev
```

### Code Quality and Testing

Before committing, please format, lint, and test your code:

```bash
# Format code
make format

# Lint code
make lint

# Run fast tests
make test

# Or run all steps at once
make pre-commit
```

For comprehensive testing:

```bash
# Run complete test suite (including slow tests)
make test
```

#### Podman Notes

If you run the test suite with Podman, make sure your shell points at the Podman socket:

```bash
export DOCKER_HOST="unix://${XDG_RUNTIME_DIR}/podman/podman.sock"
```

The test fixtures automatically disable Ryuk when `DOCKER_HOST` contains `podman.sock` and rely on explicit fixture cleanup instead.

### Release Process

This repository uses two source-of-truth versions in the root `Makefile`:

- **Library version**: `RELEASESEM` (released with `vX.Y.Z`, `vX.Y.Z-rc.N`, or `vX.Y.Z-dev.N` tags)
- **OpenAPI/API version**: `WEBAPISEM` (written to `src/graphsenselib/web/version.py`)

The Python client package version is derived from the API version and should match it.

Library package versioning is dynamic via `setuptools_scm` (`pyproject.toml`):

- Git tag `v2.9.8` -> package version `2.9.8`
- Git tag `v2.9.8-rc.1` -> package version `2.9.8rc1`
- Git tag `v2.9.8-dev.1` -> package version `2.9.8.dev1`
- Commits after a tag append local metadata, for example `2.9.8.dev1+g<sha>.d<date>`

Use the root Makefile helpers:

```bash
# Show all current versions
make show-versions

# Update and validate OpenAPI contract version
make update-api-version WEBAPISEM=v2.10.0
make check-api-version WEBAPISEM=v2.10.0

# Sync client version from API version and validate
make sync-client-version WEBAPISEM=v2.10.0
make check-client-version WEBAPISEM=v2.10.0

# Generate Python client (package version = OpenAPI info.version)
make generate-python-client

# Create both release tags from Makefile versions
make tag-version
```

Tagging behavior:

- Library release tag: `vX.Y.Z`, `vX.Y.Z-rc.N`, or `vX.Y.Z-dev.N` (from `RELEASESEM`)
- Client release tag: `webapi-vA.B.C` (from `WEBAPISEM`)

Recommended library versioning routine:

1. For development prereleases, set `RELEASESEM` to `vX.Y.Z-dev.N` (for example `v2.10.0-dev.1`)
2. For release candidates, set `RELEASESEM` to `vX.Y.Z-rc.N`
3. For stable releases, set `RELEASESEM` to `vX.Y.Z`
4. Create tags with `make tag-version`
5. Push tags with `git push origin --tags`

CI trigger background:

- Stable library tags (`vX.Y.Z`) trigger:
  - GitHub Release creation
  - Python library package build/publish (`graphsense-lib`)
  - Docker image build/publish
- Client tags (`webapi-vA.B.C`) trigger Python client package build/publish (`clients/python`)
- Other library tags (`vX.Y.Z-rc.N`, `vX.Y.Z-dev.N`) do not trigger GitHub Release or Python package publish; they only trigger Docker image build/publish

1. Update CHANGELOG.md with new features and fixes
2. Update relevant versions (library/API/client) based on what changed
3. Sync API/client versions if needed (`make update-api-version` + `make sync-client-version`)
4. Create and push tags:

```bash
make tag-version
git push origin --tags
```

## Troubleshooting

### OpenSSL Errors

Some components use OpenSSL hash functions that aren't available by default in OpenSSL 3.0+ (e.g., ripemd160). This can cause test suite failures. To fix this, enable legacy providers in your OpenSSL configuration. See the "fix openssl legacy mode" step in `.github/workflows/run_tests.yaml` for an example.

### Common Issues

1. **Connection Refused**: Verify Cassandra is running and accessible
2. **Schema Validation Errors**: Ensure database schema matches expected version
3. **Import Errors**: Install with `[all]` option for complete feature set
4. **Python Version**: Requires Python >=3.10, <3.13

### Getting Help

- Check [GitHub Issues](https://github.com/graphsense/graphsense-lib/issues)
- Review [GraphSense Documentation](https://graphsense.github.io/)
- Use `--help` with any CLI command for detailed usage information
- For tagpack-specific issues, see [Tagpack Documentation](tagpack/docs/README.md)

## License

See LICENSE file for licensing details.

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run `make pre-commit` to ensure code quality
5. Submit a pull request

---

**GraphSense** - Open Source Crypto Analytics Platform
Website: https://graphsense.github.io/
