Metadata-Version: 2.4
Name: contextflow
Version: 1.6.0
Summary: Large Language Models Tools
Author-email: Vladimir Petrukhin <man4j@ya.ru>
Project-URL: Homepage, https://github.com/Equiron-AI/contextflow
Project-URL: Repository, https://github.com/Equiron-AI/contextflow
Project-URL: Issues, https://github.com/Equiron-AI/contextflow/issues
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: transformers
Requires-Dist: jinja2
Requires-Dist: sseclient-py
Requires-Dist: pytz

## Environment
To create a Python virtual environment, use the command:
```console
conda env create -f environment.yml
```

## Installation
```console
pip install contextflow
```

## Supported Models
The following LLM models are supported:
- Gemma Family
- Qwen Family
- YandexGPT Family

## LLM backends
The following LLM backends are supported:
- Llama.cpp Server API

## Run Llama.CPP Server backend
```console
llama.cpp/build/bin/llama-server -m model_q5_k_m.gguf -ngl 99 -fa -np 2 -c 8192 --host 0.0.0.0 --port 8000
```

## Install CUDA toolkit for Llama.cpp compilation
Please note that the toolkit version must match the driver version. The driver version can be found using the nvidia-smi command.
Аor example, to install toolkit for CUDA 12.4 you need to run the following commands:
```console
CUDA_TOOLKIT_VERSION=12-4
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt -y install cuda-toolkit-${CUDA_TOOLKIT_VERSION}
echo -e '
export CUDA_HOME=/usr/local/cuda
export PATH=${CUDA_HOME}/bin:${PATH}
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
' >> ~/.bashrc
```

## Set GPU Max Temp
```console
nvidia-smi -pm 1
sudo nvidia-smi -gtt 80
nvidia-smi -q | grep Target
```
