Metadata-Version: 2.4
Name: pyarallel
Version: 0.1.0
Summary: A powerful parallel execution library for Python
Project-URL: Homepage, https://github.com/oneryalcin/pyarallel
Project-URL: Repository, https://github.com/oneryalcin/pyarallel.git
Project-URL: Documentation, https://github.com/oneryalcin/pyarallel
Author-email: Mehmet Oner Yalcin <oneryalcin@gmail.com>
License: MIT License
        
        Copyright (c) 2025 Pyarallel Contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE.md
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.12
Description-Content-Type: text/markdown

# Pyarallel

A powerful,feature-rich parallel execution library for Python that makes concurrent programming easy and efficient.

## Features

- **Simple Decorator-Based API**: Just add `@parallel` to your functions
- **Flexible Parallelism**: Choose between threads (I/O-bound) and processes (CPU-bound)
- **Smart Rate Limiting**: Control execution rates with per-second, per-minute, or per-hour limits
- **Batch Processing**: Handle large datasets efficiently with automatic batching
- **Performance Optimized**: 
  - Automatic worker pool reuse
  - Optional worker prewarming for latency-critical applications
  - Smart defaults based on your system
- **Production Ready**:
  - Thread-safe implementation
  - Memory-efficient with automatic cleanup
  - Comprehensive error handling

## Installation

```bash
pip install pyarallel
```

## Quick Start

```python
from pyarallel import parallel

# Basic parallel processing
@parallel(max_workers=4)
def fetch_url(url: str) -> dict:
    return requests.get(url).json()

# Process multiple URLs in parallel
urls = ["http://api1.com", "http://api2.com"]
results = fetch_url(urls)

# Rate-limited CPU-intensive task
@parallel(
    max_workers=4,
    executor_type="process",
    rate_limit=(100, "minute")  # 100 ops/minute
)
def process_image(image: bytes) -> bytes:
    return heavy_processing(image)

# Memory-efficient batch processing
@parallel(max_workers=4, batch_size=10)
def analyze_text(text: str) -> dict:
    return text_analysis(text)
```

## Advanced Usage

### Rate Limiting

Control execution rates using various formats:

```python
# Operations per second
@parallel(rate_limit=2.0)
def func1(): ...

# Operations per minute
@parallel(rate_limit=(100, "minute"))
def func2(): ...

# Custom rate limit object
from pyarallel import RateLimit
rate = RateLimit(1000, "hour")
@parallel(rate_limit=rate)
def func3(): ...
```

### CPU-Bound Tasks

Use process-based parallelism for CPU-intensive operations:

```python
@parallel(
    max_workers=4,
    executor_type="process",  # Use processes instead of threads
    batch_size=10            # Process in batches of 10
)
def cpu_intensive(data: bytes) -> bytes:
    return heavy_computation(data)
```

### Latency-Critical Applications

Prewarm workers to minimize cold start latency:

```python
@parallel(
    max_workers=4,
    prewarm=True  # Start workers immediately
)
def latency_critical(item): ...
```

### Memory-Efficient Processing

Handle large datasets with batch processing:

```python
@parallel(
    max_workers=4,
    batch_size=100  # Process items in batches of 100
)
def process_large_dataset(item): ...

# Process millions of items without memory issues
items = range(1_000_000)
results = process_large_dataset(items)
```

## Best Practices

1. **Choose the Right Executor**:
   - Use `executor_type="thread"` (default) for I/O-bound tasks (network, disk)
   - Use `executor_type="process"` for CPU-bound tasks (computation)

2. **Optimize Worker Count**:
   - For I/O-bound: `max_workers = cpu_count * 5` (default)
   - For CPU-bound: `max_workers = cpu_count` (default)

3. **Control Resource Usage**:
   - Use `batch_size` for large datasets
   - Use `rate_limit` to prevent overwhelming resources
   - Only use `prewarm=True` when cold start latency is critical

4. **Handle Errors Properly**:
   ```python
   @parallel()
   def my_func(item):
       try:
           return process(item)
       except Exception as e:
           return {"error": str(e), "item": item}
   ```

## Roadmap

### Observability & Debugging
- **Advanced Telemetry System**
  - Task execution metrics (duration, wait times, queue times)
  - Worker utilization tracking
  - Error frequency analysis
  - SQLite persistence for historical data
  - Interactive visualizations with Plotly
  - Performance bottleneck identification

- **Rich Logging System**
  - Configurable log levels per component
  - Structured logging for machine parsing
  - Contextual information for debugging
  - Log rotation and management
  - Integration with popular logging frameworks

### Advanced Features
- **Callback System**
  - Pre/post execution hooks
  - Error handling callbacks
  - Progress tracking
  - Custom metrics collection
  - State management hooks

- **Smart Scheduling**
  - Priority queues for tasks
  - Deadline-aware scheduling
  - Resource-aware task distribution
  - Adaptive batch sizing
  - Dynamic worker scaling

- **Fault Tolerance**
  - Automatic retries with backoff
  - Circuit breaker pattern
  - Fallback strategies
  - Dead letter queues
  - Task timeout handling

- **Resource Management**
  - Memory usage monitoring
  - CPU utilization tracking
  - Network bandwidth control
  - Disk I/O rate limiting
  - Resource quotas per task

### Developer Experience
- **CLI Tools**
  - Task monitoring dashboard
  - Performance profiling
  - Configuration management
  - Log analysis utilities
  - Telemetry visualization


### Enterprise Features
- **Integration**
  - Distributed tracing (OpenTelemetry)
  - Metrics export (Prometheus)
  - Log aggregation (ELK Stack)

Want to contribute? Check out our [CONTRIBUTING.md](CONTRIBUTING.md) guide!

## API Reference

### @parallel Decorator

```python
@parallel(
    max_workers: int = None,          # Maximum workers (default: based on CPU)
    batch_size: int = None,           # Items per batch (default: all at once)
    rate_limit: Union[                # Rate limiting configuration
        float,                        # - Operations per second
        Tuple[float, str],           # - (count, interval)
        RateLimit                     # - RateLimit object
    ] = None,
    executor_type: str = "thread",    # "thread" or "process"
    prewarm: bool = False            # Prewarm workers
)
```

### RateLimit Class

```python
class RateLimit:
    def __init__(self, count: float, interval: str = "second"):
        """
        Args:
            count: Operations allowed per interval
            interval: "second", "minute", or "hour"
        """
```

## Contributing

Contributions are welcome! Please check out our [Contributing Guide](CONTRIBUTING.md).

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.