Metadata-Version: 2.4
Name: opensensor-enviroplus
Version: 0.4.6
Summary: Modern CLI-based environmental sensor collector using Polars, Arrow, and Delta Lake for Enviro+
Author-email: Youssef Harby <yharby@walkthru.earth>
License: MIT
Keywords: sensors,enviroplus,raspberry-pi,polars,arrow,iot,opensensor-space,walkthru-earth
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: croniter>=6.0.0
Requires-Dist: obstore>=0.8.2
Requires-Dist: polars>=1.35.2
Requires-Dist: pyarrow>=22.0.0
Requires-Dist: pydantic>=2.12.4
Requires-Dist: pydantic-settings>=2.12.0
Requires-Dist: python-dotenv>=1.2.1
Requires-Dist: typer>=0.20.0
Requires-Dist: uuid6>=2025.0.1
Requires-Dist: pimoroni-bme280>=1.0.0; sys_platform == "linux"
Requires-Dist: pms5003>=1.0.1; sys_platform == "linux"
Requires-Dist: ltr559>=1.0.0; sys_platform == "linux"
Requires-Dist: ads1015>=1.0.0; sys_platform == "linux"
Requires-Dist: gpiod>=2.1.3; sys_platform == "linux"
Requires-Dist: gpiodevice>=0.0.3; sys_platform == "linux"

# OpenSensor Enviroplus

Modern, CLI-based environmental sensor data collector using Polars, Apache Arrow, and Hive-partitioned Parquet for Raspberry Pi Enviro+.

Part of the [OpenSensor.Space](https://opensensor.space) network for open environmental data.

## Features

- **UUID v7 Station IDs**: Time-ordered UUIDs for better database performance
- **Modern Stack**: Polars streaming, Apache Arrow, Hive-partitioned Parquet
- **Memory Efficient**: Optimized for Raspberry Pi with limited RAM
- **CLI-First**: Simple Python commands replace bash scripts
- **Smart Logging**: Rich console output for easy debugging
- **Cloud Sync**: Multi-provider sync using obstore (S3, R2, GCS, Azure, MinIO, Wasabi, Backblaze, Hetzner)
- **Prefix-based IAM**: S3 bucket access control per station
- **Type Safe**: Pydantic settings with validation
- **Production Ready**: Graceful error handling, automatic retries
- **Browser-queryable**: DuckDB-wasm compatible Parquet output
- **Temperature & Humidity Compensation**: CPU heat correction using Pimoroni's dewpoint formula (applied in collector and CLI test)
- **System Health Monitoring**: Optional CPU, memory, disk, WiFi signal, NTP sync tracking

## Quick Start

### One-Line Install (Recommended)

```bash
curl -LsSf https://raw.githubusercontent.com/walkthru-earth/opensensor-enviroplus/main/install.sh | sudo bash
```

This installs everything: system dependencies, UV, opensensor-enviroplus, and configures permissions. After reboot:

```bash
cd ~/opensensor
opensensor setup                  # Configure station
opensensor test                   # Verify sensors
sudo opensensor service setup     # Install as service
```

### Manual Installation

<details>
<summary>Click to expand manual installation steps</summary>

#### Prerequisites

```bash
# Update system packages
sudo apt-get update

# Install git and required system libraries
sudo apt-get install -y git python3-dev python3-cffi libportaudio2

# Enable I2C and SPI interfaces (required for sensors)
sudo raspi-config nonint do_i2c 0
sudo raspi-config nonint do_spi 0

# Install UV package manager (fast Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
```

> **Note:** The I2C interface is required for BME280, LTR559, and gas sensors. SPI is needed for the LCD display. A reboot may be required after enabling these interfaces.

#### Install from PyPI

```bash
uv tool install opensensor-enviroplus
```

#### Or install from source

```bash
git clone https://github.com/walkthru-earth/opensensor-enviroplus.git
cd opensensor-enviroplus
uv sync
```

#### Fix Permissions (reboot required)

```bash
sudo $(which opensensor) fix-permissions
sudo reboot
```

#### Configure and Start

```bash
opensensor setup                  # Configure station ID
opensensor test                   # Verify sensors work
sudo opensensor service setup     # Install & start as service
```

</details>

## CLI Commands

| Command | Description |
|---------|-------------|
| `opensensor setup` | Interactive configuration wizard |
| `opensensor test` | Test sensors with live readings table |
| `opensensor info` | Show config, sensors, data stats, service status |
| `opensensor start` | Run collector in foreground (for debugging) |
| `opensensor sync` | Manual sync to cloud storage |
| `opensensor fix-permissions` | Fix serial port permissions (sudo required) |

### Service Commands

| Command | Description |
|---------|-------------|
| `opensensor service setup` | Install, enable, and start service |
| `opensensor service status` | Show service status |
| `opensensor service logs -f` | Follow live logs |
| `opensensor service stop` | Stop the service |
| `opensensor service restart` | Restart the service |
| `opensensor service remove` | Completely remove service |

## Configuration

Configuration via `.env` file (auto-generated by `opensensor setup`):

```bash
# Station identification (UUID v7 - auto-generated)
OPENSENSOR_STATION_ID=019ab383-d789-74e2-a460-bb92b1c13681

# Data collection
OPENSENSOR_READ_INTERVAL=5              # Seconds between sensor reads
OPENSENSOR_BATCH_DURATION=900           # 15-minute batches

# Temperature/humidity compensation (for Raspberry Pi CPU heat)
OPENSENSOR_TEMP_COMPENSATION_ENABLED=true
OPENSENSOR_TEMP_COMPENSATION_FACTOR=2.25  # Pimoroni's official factor
OPENSENSOR_PMS5003_DEVICE=/dev/serial0  # Serial port for PMS5003

# Health monitoring (CPU, memory, disk, WiFi, NTP sync)
OPENSENSOR_HEALTH_ENABLED=true

# Output settings
OPENSENSOR_OUTPUT_DIR=output
OPENSENSOR_COMPRESSION=snappy           # Fast compression (snappy, zstd, gzip)

# Cloud sync (optional)
OPENSENSOR_SYNC_ENABLED=true
OPENSENSOR_SYNC_INTERVAL_MINUTES=15

# Storage provider (s3, r2, gcs, azure, minio, wasabi, backblaze, hetzner)
OPENSENSOR_STORAGE_PROVIDER=s3
OPENSENSOR_STORAGE_BUCKET=my-sensor-bucket
OPENSENSOR_STORAGE_PREFIX=sensors/station-019ab383  # For IAM scoping
OPENSENSOR_STORAGE_REGION=us-west-2

# Provider credentials (see .env.example for all providers)
OPENSENSOR_AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
OPENSENSOR_AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCY

# Logging
OPENSENSOR_LOG_LEVEL=INFO
OPENSENSOR_LOG_DIR=logs
```

See `.env.example` for a complete template with all provider configurations and IAM policy examples.

### Supported Cloud Storage Providers

| Provider | Type | Notes |
|----------|------|-------|
| `s3` | AWS S3 | Native support |
| `r2` | Cloudflare R2 | S3-compatible, no egress fees |
| `gcs` | Google Cloud Storage | Native support |
| `azure` | Azure Blob Storage | Native support |
| `minio` | MinIO | S3-compatible, self-hosted |
| `wasabi` | Wasabi | S3-compatible, affordable |
| `backblaze` | Backblaze B2 | S3-compatible |
| `hetzner` | Hetzner Object Storage | S3-compatible |

## Architecture

### Data Flow

```
Sensors (5s) -> Polars Collector -> Hive-Partitioned Parquet (15min) -> S3/MinIO (obstore)
                    ↓
              Health Metrics (~1min) -> Separate Parquet (output-health/)
```

### Output Format (Hive-Partitioned Parquet)

```
output/                                           # Sensor data
  station=019ab383-d789-74e2-a460-bb92b1c13681/
    year=2025/
      month=11/
        day=24/
          data_1430.parquet  # Batch written at 14:30
          data_1445.parquet  # Batch written at 14:45

output-health/                                    # System health (optional)
  station=019ab383-d789-74e2-a460-bb92b1c13681/
    year=2025/
      month=11/
        day=24/
          health_1430.parquet  # ~15 health records per batch
```

**Benefits:**
- Browser-queryable with DuckDB-wasm
- Partition pruning for fast time-range queries
- Simple, universal format (no proprietary transaction logs)
- Perfect for append-only time-series data

See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed diagrams and scalability analysis.

## Differences from Original

| Feature | Old (enviroplus-python) | New (opensensor-enviroplus) |
|---------|-------------------------|----------------------------|
| Station IDs | Manual/random | UUID v7 (time-ordered) |
| Data library | pandas + DuckDB | Polars + Apache Arrow |
| Storage | Partitioned Parquet | Hive-partitioned Parquet |
| Configuration | bash scripts + env vars | Pydantic Settings + .env |
| Setup | install.sh | `opensensor setup` CLI |
| Cloud sync | rclone (process spawn) | obstore (8 providers, Rust) |
| IAM policies | N/A | Prefix-based scoping |
| Logging | print statements | Rich + structured logging |
| Memory usage | Higher (pandas) | 50% lower (Polars streaming) |
| CLI | None | Typer with 6 simple commands |
| Read interval | 1 second | 5 seconds (configurable) |
| Batch duration | Variable | 15 minutes (900s) |
| Humidity correction | None | Dewpoint-based compensation |
| Health monitoring | None | CPU, memory, WiFi, NTP sync |

## Development

```bash
# Install with dev dependencies
uv sync --group dev

# Format code
uv run ruff format .

# Lint code
uv run ruff check .

# Run with UV (no venv activation needed)
uv run opensensor --help
```

## Tech Stack

- **Python 3.10+** - Modern Python with type hints
- **UV** - Fast Rust-based package manager (10-100x faster than pip)
- **Polars 1.35+** - High-performance DataFrames with streaming
- **PyArrow 22+** - Columnar memory format (zero-copy operations)
- **uuid6** - RFC 9562 UUID v7 implementation
- **obstore** - Rust-powered object storage (S3/GCS/Azure)
- **Pydantic Settings** - Type-safe configuration
- **Typer + Rich** - Beautiful CLI with auto-completion
- **Ruff** - Extremely fast Python linter and formatter

## License

MIT License - see [LICENSE](LICENSE) file for details

## Credits

Built by the [WalkThru Earth](https://walkthru.earth) team for the [OpenSensor.Space](https://opensensor.space) network.

**Dependencies:**
- [pimoroni-bme280](https://github.com/pimoroni/bme280-python) - Temperature, pressure, humidity sensor
- [pms5003](https://github.com/pimoroni/pms5003-python) - Particulate matter sensor
- [ltr559](https://github.com/pimoroni/ltr559-python) - Light and proximity sensor
- [ads1015](https://github.com/pimoroni/ads1015-python) - ADC for MICS6814 gas sensor
- [Polars](https://pola.rs/) - Lightning-fast DataFrames
- [obstore](https://developmentseed.org/obstore/) - Object storage abstraction
- [Typer](https://typer.tiangolo.com/) - CLI framework
- [Rich](https://rich.readthedocs.io/) - Terminal formatting
- [Pydantic](https://docs.pydantic.dev/) - Data validation

## Contributing

Contributions welcome! Please:
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Run tests and linting (`uv run ruff check .`)
4. Commit your changes (`git commit -m 'feat: add amazing feature'`)
5. Push to the branch (`git push origin feature/amazing-feature`)
6. Open a Pull Request

## Support

- **Issues**: [GitHub Issues](https://github.com/walkthru-earth/opensensor-enviroplus/issues)
- **Discussions**: [GitHub Discussions](https://github.com/walkthru-earth/opensensor-enviroplus/discussions)
- **Documentation**: [ARCHITECTURE.md](ARCHITECTURE.md)
