Metadata-Version: 2.1
Name: pfio
Version: 2.9.0
Summary: PFN IO library
Home-page: http://github.com/pfnet/pfio
Author: Tianqi Xu, Kota Uenishi
Author-email: tianqi@preferred.jp, kota@preferred.jp
Keywords: filesystem hdfs chainer development
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Filesystems
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3
Requires-Dist: deprecation
Requires-Dist: urllib3
Provides-Extra: bench
Requires-Dist: numpy >=1.19.5 ; extra == 'bench'
Requires-Dist: torch >=1.9.0 ; extra == 'bench'
Requires-Dist: Pillow <=8.2.0 ; extra == 'bench'
Provides-Extra: doc
Requires-Dist: sphinx ; extra == 'doc'
Requires-Dist: sphinx-rtd-theme ; extra == 'doc'
Provides-Extra: hdfs
Requires-Dist: pyarrow >=6.0.0 ; extra == 'hdfs'
Provides-Extra: test
Requires-Dist: pytest ; extra == 'test'
Requires-Dist: flake8 ; extra == 'test'
Requires-Dist: autopep8 ; extra == 'test'
Requires-Dist: parameterized ; extra == 'test'
Requires-Dist: isort ; extra == 'test'
Requires-Dist: moto[server] >=5.0.0 ; extra == 'test'
Requires-Dist: numpy ; extra == 'test'
Requires-Dist: mypy ; extra == 'test'
Provides-Extra: trace
Requires-Dist: pytorch-pfn-extras ; extra == 'trace'

## PFIO

![Doc Status Badge](https://readthedocs.org/projects/pfio/badge/?version=master&style=flat)

PFIO is an IO abstraction library developed by PFN, optimized for deep
learning training with batteries included. It supports

- Filesystem API abstraction with unified error semantics,
- Explicit user-land caching system,
- IO performance tracing and metrics stats, and
- Fileset container utilities to save metadata.


## Dependency

- HDFS client and libhdfs for HDFS access
- CPython >= 3.9

## Installation and Document build

Installation

```shell
$ git clone https://github.com/pfnet/pfio.git
$ cd pfio
$ pip install .
```

Documentation
```sh
$ cd pfio/docs
$ make html
$ open build/html/index.html
```

## How to use

Please refer to the [official document](https://pfio.readthedocs.io) for more information about the usage.

## Release

Check [the official document](https://packaging.python.org/tutorials/packaging-projects/) for latest release procedure.

Run tests locally:

```sh
$ pip install tox
$ tox
```

Bump version numbers in `pfio/version.py` .

Push and open a pull request to invoke CI. Once CI passed and the pull request merged,
tag a release:

```sh
$ git tag -s X.Y.Z
$ git push --tags
```

Build:

```sh
$ rm -rf dist
$ pip3 install --user build
$ python3 -m build
```

Release to PyPI:

```sh
$ python3 -m pip install --user --upgrade twine
$ python3 -m twine upload --repository testpypi dist/*
```
