Metadata-Version: 2.4
Name: pattern_lens
Version: 0.1.0
Project-URL: Homepage, https://miv.name/pattern-lens
Project-URL: Documentation, https://miv.name/pattern-lens
Project-URL: Repository, https://github.com/mivanit/pattern-lens
Project-URL: Issues, https://github.com/mivanit/pattern-lens/issues
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: beartype>=0.14.1
Requires-Dist: ipykernel>=6.29.5
Requires-Dist: ipywidgets>=8.1.5
Requires-Dist: jaxtyping>=0.2.33
Requires-Dist: matplotlib>=3.8.0
Requires-Dist: muutils>=0.6.19
Requires-Dist: numpy<2.0.0,>=1.26.1
Requires-Dist: pandas>=2.2.2
Requires-Dist: pillow>=11.0.0
Requires-Dist: scipy>=1.14.1
Requires-Dist: torch>=2.5.1
Requires-Dist: tqdm>=4.66.5
Requires-Dist: transformer-lens>=2.10.0
Requires-Dist: typeguard>=4.4.1
Requires-Dist: zanj>=0.3.1
Description-Content-Type: text/markdown

# pattern-lens

visualization of LLM attention patterns and things computed about them

`pattern-lens` makes it easy to:

- Generate visualizations of attention patterns, or figures computed from attention patterns, from models supported by [TransformerLens](https://github.com/TransformerLensOrg/TransformerLens)
- Compare generated figures across models, layers, and heads in an [interactive web interface](https://miv.name/pattern-lens/demo/)

# Installation

```bash
pip install pattern-lens
```


# Usage

The pipeline is as follows:

- Generate attention patterns using `pattern_lens.activations.acitvations_main()`, saving them in `npz` files
- Generate visualizations using `pattern_lens.figures.figures_main()` -- read the `npz` files, pass each attention pattern to each visualization function, and save the resulting figures
- Serve the web interface using `pattern_lens.server` -- web interface reads metadata in json/jsonl files, then lets the user select figures to show


## Basic CLI

Generate attention patterns and default visualizations:

```bash
# generate activations
python -m pattern_lens.activations --model gpt2 --prompts data/pile_1k.jsonl --save-path attn_data
# create visualizations
python -m pattern_lens.figures --model gpt2 --save-path attn_data
```

serve the web UI:

```bash
python -m pattern_lens.server --path attn_data
```


## Web UI

View a demo of the web UI at [miv.name/pattern-lens/demo](https://miv.name/pattern-lens/demo/).

## Custom Figures

Add custom visualization functions by decorating them with `@register_attn_figure_func`. You should still generate the activations first:
```
python -m pattern_lens.activations --model gpt2 --prompts data/pile_1k.jsonl --save-path attn_data
```

and then write+run a script/notebook that looks something like this:

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.linalg import svd

# these functions simplify writing a function which saves a figure
from pattern_lens.figure_util import matplotlib_figure_saver, save_matrix_wrapper
# decorator to register your function, such that it will be run by `figures_main`
from pattern_lens.attn_figure_funcs import register_attn_figure_func
# runs the actual figure generation pipeline
from pattern_lens.figures import figures_main

# define your own functions
# this one uses `matplotlib_figure_saver` -- define a function that takes matrix and `plt.Axes`, modify the axes
@register_attn_figure_func
@matplotlib_figure_saver(fmt="svgz")
def svd_spectra(attn_matrix: np.ndarray, ax: plt.Axes) -> None:
    # Perform SVD
    U, s, Vh = svd(attn_matrix)

    # Plot singular values
    ax.plot(s, "o-")
    ax.set_yscale("log")
    ax.set_xlabel("Singular Value Index")
    ax.set_ylabel("Singular Value")
    ax.set_title("Singular Value Spectrum of Attention Matrix")


# run the figures pipelne
# run the pipeline
figures_main(
	model_name="pythia-14m",
	save_path=Path("docs/demo/"),
	n_samples=5,
	force=False,
)
```

see `demo.ipynb` for a full example