Metadata-Version: 2.1
Name: tokenwise
Version: 0.1.0
Summary: Tokenizer evaluation library
License: Apache 2.0
Author: Kacper Łukawski
Author-email: kacper.lukawski@embassy.ai
Requires-Python: >=3.10,<4.0
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Dist: sentence-transformers (>=2.5.1,<3.0.0)
Description-Content-Type: text/markdown

# tokeval

A toolkit for evaluating the quality of the text tokenizers.

