Metadata-Version: 2.4
Name: diffusionLM
Version: 0.1.6
Summary: A diffusion-based language model implementation
Home-page: https://github.com/codewithdark-git/PIP-DifffusionLM
Author: Dark Coder
Author-email: codewithdark90@gmail.com
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Natural Language :: English
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: torch
Requires-Dist: transformers
Requires-Dist: datasets
Requires-Dist: numpy
Requires-Dist: tqdm
Requires-Dist: wandb
Requires-Dist: hydra-core
Requires-Dist: omegaconf
Requires-Dist: pytorch-lightning
Requires-Dist: accelerate
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# DiffusionLM: Large Language Models with Diffusion

[![PyPI version](https://badge.fury.io/py/diffusionLM.svg)](https://badge.fury.io/py/diffusionLM)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

DiffusionLM is a novel approach to language modeling that combines transformer architectures with diffusion processes for high-quality text generation. This package provides a flexible and efficient implementation of diffusion-based language models.

## Features

- **Advanced Architecture**
  - Transformer-based backbone with diffusion capabilities
  - Configurable model sizes (small, medium, large)
  - Time step conditioning
  - Attention mechanisms optimized for text

- **Multiple Generation Strategies**
  - Auto-regressive generation
  - Parallel generation
  - Confidence-based masking
  - Semi-autoregressive generation
  - Top-p (nucleus) sampling
  - Beam search

- **Training Features**
  - Distributed training support
  - Mixed precision training
  - Gradient checkpointing
  - Early stopping
  - Model checkpointing
  - Learning rate scheduling

- **Utilities**
  - Real-time token generation streaming
  - Model saving and loading
  - HuggingFace Hub integration
  - Comprehensive logging
  - Error handling

## Installation

```bash
pip install diffusionLM
```

For development installation:

```bash
git clone https://github.com/codewithdark-git/PIP-DifffusionLM.git
cd PIP-DifffusionLM
pip install -e .
```

## Quick Start

```python
from diffusionLM.utils import prepare_dataset
from diffusionLM.model import DiffusionConfig, DiffusionLLM
from transformers import AutoTokenizer

# Load tokenizer and prepare dataset
tokenizer = AutoTokenizer.from_pretrained("gpt2")
train_dataset, val_dataset, _ = prepare_dataset(
    dataset_name="wikitext/wikitext-103-v1",
    tokenizer_name="gpt2"
)

# Initialize model
config = DiffusionConfig(
        vocab_size=len(tokenizer),
        max_position_embeddings=256,
        num_timesteps=50,
        pad_token_id=tokenizer.pad_token_id,
        mask_token_id=tokenizer.mask_token_id,
        # **config_kwargs
    )

model = DiffusionLLM(config)


```

## Training

### Basic Training

```python
from diffusionLM import trainer

train_model = trainer(
        model=model,
        train_dataset=train_dataset,
        val_dataset=val_dataset,
        batch_size=batch_size,
        num_epochs=num_epochs,
        learning_rate=learning_rate,
        num_timesteps=num_timesteps,
        save_path=save_dir,
        device=device,
    )
```

### Model Registry

```python
from diffusionLM import registerANDpush

registerANDpush(
    model=trained_model,
    tokenizer=tokenizer,
    model_type="diffusionLM",
    repo_id="your-username/model-name"
)
```

## Error Handling

The package includes comprehensive error handling:

```python
from diffusionLM import DiffusionLMError, handle_errors

@handle_errors()
def your_function():
    # Your code here
    pass
```

## Contributing

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

## Requirements

- Python ≥ 3.8
- PyTorch ≥ 1.9.0
- Transformers ≥ 4.21.0
- For full requirements, see `requirements.txt`

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Citation

```bibtex
@article{diffusionllm2025,
  title={DiffusionLM: Large Language Models with Diffusion},
  author={Dark Coder},
  journal={GitHub Repository},
  year={2025},
  publisher={GitHub},
  url={https://github.com/codewithdark-git/PIP-DifffusionLM}
}
```

## Contact

- GitHub: [@codewithdark-git](https://github.com/codewithdark-git)
- Email: codewithdark90@gmail.com
# PIP-DifffusionLM
