Metadata-Version: 2.4
Name: trustcv
Version: 1.0.0
Summary: Trustworthy Cross-Validation: Framework-agnostic CV with data leakage detection
Home-page: https://github.com/ki-smile/trustcv
Author: SMAILE Team, Karolinska Institutet
Author-email: "SMAILE Team, Karolinska Institutet" <contact@smile.ki.se>
License-Expression: MIT
Project-URL: Homepage, https://github.com/ki-smile/trustcv
Project-URL: Documentation, https://ki-smile.github.io/trustcv
Project-URL: Repository, https://github.com/ki-smile/trustcv
Project-URL: Bug Tracker, https://github.com/ki-smile/trustcv/issues
Keywords: cross-validation,medical,machine-learning,healthcare,validation
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Healthcare Industry
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: AUTHORS.md
Requires-Dist: numpy>=1.20.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: scipy>=1.7.0
Requires-Dist: matplotlib>=3.4.0
Requires-Dist: seaborn>=0.11.0
Requires-Dist: plotly>=5.0.0
Requires-Dist: joblib>=1.0.0
Requires-Dist: tqdm>=4.60.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=3.0.0; extra == "dev"
Requires-Dist: flake8>=4.0.0; extra == "dev"
Requires-Dist: mypy>=0.950; extra == "dev"
Requires-Dist: black>=22.0.0; extra == "dev"
Requires-Dist: isort>=5.10.0; extra == "dev"
Requires-Dist: bandit>=1.7.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=4.5.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.0.0; extra == "docs"
Requires-Dist: sphinx-autodoc-typehints>=1.12.0; extra == "docs"
Requires-Dist: myst-parser>=0.18.0; extra == "docs"
Requires-Dist: nbsphinx>=0.9.0; extra == "docs"
Provides-Extra: viz
Requires-Dist: bokeh>=2.4.0; extra == "viz"
Provides-Extra: pytorch
Requires-Dist: torch; extra == "pytorch"
Requires-Dist: torchvision; extra == "pytorch"
Provides-Extra: tensorflow
Requires-Dist: tensorflow>=2.16; extra == "tensorflow"
Provides-Extra: monai
Requires-Dist: monai>=1.0.0; extra == "monai"
Requires-Dist: nibabel>=3.2.0; extra == "monai"
Provides-Extra: all
Requires-Dist: torch; extra == "all"
Requires-Dist: torchvision; extra == "all"
Requires-Dist: tensorflow>=2.16; extra == "all"
Requires-Dist: monai>=1.0.0; extra == "all"
Requires-Dist: nibabel>=3.2.0; extra == "all"
Requires-Dist: pytest>=7.0.0; extra == "all"
Requires-Dist: flake8>=4.0.0; extra == "all"
Requires-Dist: mypy>=0.950; extra == "all"
Requires-Dist: black>=22.0.0; extra == "all"
Requires-Dist: sphinx>=4.5.0; extra == "all"
Requires-Dist: nbsphinx>=0.9.0; extra == "all"
Requires-Dist: pytest-cov>=3.0.0; extra == "all"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# trustcv — Trustworthy Cross-Validation Toolkit

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![Documentation](https://img.shields.io/badge/docs-available-brightgreen.svg)](https://github.com/ki-smile/trustcv)



**TrustCV** is a framework-agnostic toolkit for **reliable cross-validation** in safety-critical and regulated settings.
It builds on familiar scikit-learn idioms, but adds:

- Carefully designed cross-validation splitters (starting with IID in v0.1).
- Automatic **data leakage** and **class balance** checks.
- **Clinical/industrial metrics** with confidence intervals.
- Simple, **regulatory-ready reporting**.

> **Status:** v1.0.0 – Production release with 29 CV methods across IID, Grouped, Temporal, and Spatial categories. Full data leakage detection and regulatory compliance features.

---

## Why TrustCV?

Standard cross-validation is easy to misuse:

- Train/test splits can accidentally **leak information** (e.g., shared patients, timestamps, engineered features).
- Imbalanced datasets can give **overly optimistic** metrics if not stratified or monitored.
- For clinical and industrial applications, we often need **meaningful metrics** and reproducible reports, not just accuracy.

TrustCV addresses these issues by:

- Providing **well-tested IID splitters** with clear semantics.
- Running **leakage and balance checks** alongside your CV.
- Exposing **clinical metrics** and simple **reporting utilities** for audits and regulatory files.

---

## What's in v1.0.0

This release includes **29 cross-validation methods** across four categories:

- **IID splitters** (9 methods):
  `HoldOut`, `KFold`, `StratifiedKFold`, `RepeatedKFold`, `LeaveOneOut`, `LeavePOut`, `BootstrapValidation`, `MonteCarloCV`, `NestedCV`

- **Grouped splitters** (6 methods):
  `GroupKFold`, `StratifiedGroupKFold`, `LeaveOneGroupOut`, `RepeatedGroupKFold`, `NestedGroupedCV`, `HierarchicalGroupKFold`

- **Temporal splitters** (8 methods):
  `TimeSeriesSplit`, `BlockedTimeSeriesSplit`, `RollingWindowCV`, `ExpandingWindowCV`, `PurgedKFold`, `CombinatorialPurgedKFold`, `PurgedGroupTimeSeriesSplit`, `NestedTemporalCV`

- **Spatial splitters** (4 methods):
  `SpatialBlockCV`, `BufferedSpatialCV`, `SpatiotemporalBlockCV`, `EnvironmentalHealthCV`

- **Framework-agnostic runner**:
  `UniversalCVRunner` + `CVResults` for consistent, reusable CV loops across scikit-learn, PyTorch, TensorFlow, MONAI, and JAX.

- **High-level validator**:
  `TrustCVValidator` with automatic method selection and leakage detection.

- **Data integrity checks**:
  `DataLeakageChecker` (6 leakage types), `BalanceChecker`, and `LeakageReport`.

- **Clinical/medical metrics**:
  `ClinicalMetrics` with confidence intervals (sensitivity, specificity, PPV/NPV, etc.).

- **Regulatory reporting**:
  `RegulatoryReport` for FDA/CE MDR compliance documentation.

---


## Quick Start

### Installation

```bash
# Install from source (recommended for latest features)
git clone https://github.com/ki-smile/trustcv.git
cd trustcv
pip install -e .

# Or install from PyPI (when released)
pip install trustcv
```


## Quickstart – IID CV with TrustCV

Here is a minimal example using `StratifiedKFold` and `UniversalCVRunner`:


```python
from trustcv import TrustCVValidator
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), RandomForestClassifier(random_state=42))

# Validates with leakage checks and computes clinical CIs
validator = TrustCVValidator(
    method="StratifiedKfold",  # IID, stratified
    n_splits=5,
    random_state=42,
    check_leakage=True,
    check_balance=True,
)
results = validator.validate(model=model, X=X, y=y)

print(results.summary()) 

# Output:
#=== Trustworthy Cross-Validation Results ===

#Performance Metrics (mean +/- std) (method: bootstrap):
#  accuracy: 0.956 +/- 0.014 [95% CI (bootstrap): 0.946-0.967]
#  roc_auc: 0.989 +/- 0.009 [95% CI (bootstrap): 0.981-0.995]
#  sensitivity: 0.966 +/- 0.030 [95% CI (bootstrap): 0.939-0.986]
#  specificity: 0.939 +/- 0.056 [95% CI (bootstrap): 0.893-0.981]
#  precision: 0.965 +/- 0.031 [95% CI (bootstrap): 0.937-0.989]
#  recall: 0.966 +/- 0.030 [95% CI (bootstrap): 0.939-0.986]
#  f1: 0.965 +/- 0.011 [95% CI (bootstrap): 0.957-0.973]


#Data Integrity Checks:
#  Leakage Check: PASSED
#  Class Balance: PASSED
```

For a higher-level workflow with leakage and balance checks, see the
 [Quickstart: IID CV with TrustCV](https://github.com/ki-smile/trustcv/blob/main/docs/Quickstart%3A%20IID%20Cross-Validation%20with%20TrustCV.md) and  [IID Splitters](https://github.com/ki-smile/trustcv/blob/main/docs/iid_splitters.md) tutorial.

 ### Run the example notebooks

Prefer to learn by running code? From the repo root, open the notebooks in `notebooks/`:

- `notebooks/01_IID_Methods_Showcase.ipynb` – quick tour of the IID splitters and metrics.
- `notebooks/02_Advanced_Workflow_UniversalRunner.ipynb` – end-to-end UniversalCVRunner workflow.
- `notebooks/03_TrustCVValidator_Showcase.ipynb` – TrustCVValidator examples with leakage/balance checks.
- `notebooks/04_TrustCVValidator_IID_Comparison.ipynb` – side-by-side IID method comparison.
- `notebooks/05_CrossValidation_Comparison.ipynb` – comprehensive CV methods comparison.
- Reports generated by the notebooks are saved in `notebooks/reports/` (HTML/PDF).



------

## How TrustCV relates to scikit-learn

**Similarities:**

- Uses familiar scikit-learn idioms: estimators with `fit`/`predict`, splitter objects with `split(X, y)`.
- Works seamlessly with scikit-learn models, pipelines, and metrics.
- IID splitters follow scikit-learn semantics (e.g., `KFold`, `StratifiedKFold`).

**Added value:**

- **Leakage and balance checks**: `DataLeakageChecker` and `BalanceChecker` run alongside your CV.
- **Clinical metrics**: `ClinicalMetrics` computes sensitivity, specificity, PPV/NPV, ROC/PR metrics, and CIs.
- **Structured results**: `CVResults` and `ValidationResult` standardize fold-level outputs.
- **Reporting**: `UniversalRegulatoryReport` turns your evaluation into a reproducible HTML/JSON report.

------

## Roadmap

TrustCV v1.0.0 includes all 29 CV methods. Future versions will focus on:

- **v1.1 – Enhanced reporting**
   Extended regulatory report templates and audit trail features.
- **v1.2 – Additional frameworks**
   Expanded support for JAX/Flax, XGBoost, LightGBM, and CatBoost.
- **v1.3 – AutoML integration**
   Integration with Optuna for hyperparameter tuning within nested CV.

------

## Contributors

See [AUTHORS.md](https://github.com/ki-smile/trustcv/blob/main/AUTHORS.md) for a full list of contributors and acknowledgments.

### Lead Contributors
- **[Farhad Abtahi](https://github.com/farhad-abtahi)**
- **[Abdelamir Karbalaie](https://github.com/abdkar)**





### Contributing
We welcome contributions!
- Code contributions
- Medical use case examples
- Documentation improvements
- Bug reports and feature requests

Please see:

- [`CONTRIBUTING.md`](https://github.com/ki-smile/trustcv/blob/main/CONTRIBUTING.md)
- [`CODE_OF_CONDUCT.md`](https://github.com/ki-smile/trustcv/blob/main/CODE_OF_CONDUCT.md)

------



## 3. Quickstart: IID CV with TrustCV – outline

**See file:** `docs/quickstart_iid.md`  
(and a matching notebook: `notebooks/Quickstart_IID_TrustCV.ipynb`)


## Repository Structure

```
trustcv/       # Python package (splitters, validators, metrics, core)
docs/          # Documentation & guides
notebooks/     # Jupyter tutorials
examples/      # Real-world examples
tests/         # Unit & integration tests
website/       # Static site and visualizations
```

## Development

```bash
pip install -e .[dev]
pytest tests/
cd docs && make html
```

## Citation

If you use trustcv in your research, please cite:

```bibtex
@software{trustcv2025,
  title = {trustcv: Trustworthy Cross-Validation Toolkit},
  author = {Abtahi, Farhad and Karbalaie, Abdolamir},
  year = {2025},
  url = {https://github.com/ki-smile/trustcv}
}
```

## License

MIT License — see [LICENSE](LICENSE).

## Contact & Support

- GitHub Issues: https://github.com/ki-smile/trustcv/issues

---

## ⚠️ Disclaimer

This toolkit is for research and educational purposes. Always validate results with domain experts before clinical deployment.

---

Advancing Medical AI Through Rigorous Validation
