Metadata-Version: 2.1
Name: llmlite
Version: 0.0.14
Summary: A library helps to chat with all kinds of LLMs consistently.
License: MIT
Author: InftyAI
Requires-Python: >=3.10,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: accelerate (==0.22.0)
Requires-Dist: openai (>=0.28.0,<0.29.0)
Requires-Dist: pydantic (<2)
Requires-Dist: sentencepiece (==0.1.99)
Requires-Dist: tokenizers (==0.14)
Requires-Dist: torch (>=2.1.1,<3.0.0)
Requires-Dist: transformers (==4.34.0)
Requires-Dist: vllm (==0.2.3)
Description-Content-Type: text/markdown

# llmlite

A library helps to communicate with all kinds of LLMs consistently.

| Model | State | System Prompt | Note |
| ---- | ---- | ---- | ---- |
| ChatGPT | Done ✅ | Yes | |
| Llama-2 | Done ✅ | Yes | |
| CodeLlama | Done ✅ | Yes | |
| ChatGLM2 | Done ✅ | No | |
| ChatGLM3 | WIP ⏳ | Yes | |
| Baichuan2 | Done ✅ | Yes | |
| Claude-2 | RoadMap 📋 | | [issue#7](https://github.com/InftyAI/ChatLLM/issues/7)
| Falcon | RoadMap 📋 | | [issue#8](https://github.com/InftyAI/ChatLLM/issues/8)
| StableLM | RoadMap 📋 | | [issue#11](https://github.com/InftyAI/ChatLLM/issues/11) |
| Baichuan2 | RoadMap 📋 | | [issue#34](https://github.com/InftyAI/llmlite/issues/34)
| ... | ... | ... | ... |

llmlite also supports different inference backends as below:

| backend | State | Note |
| ---- | ---- | ---- |
| [huggingface](https://github.com/huggingface) | Done ✅ | Support by huggingface pipeline |
| [vLLM](https://github.com/vllm-project/vllm) | Done ✅ | |
| ... | ... | ... |

## How to install

```cmd
pip install llmlite==0.0.9
```

## How to use

### Chat

```python
from llmlite.apis import ChatLLM, ChatMessage

chat = ChatLLM(
    model_name_or_path="meta-llama/Llama-2-7b-chat-hf", # required
    task="text-generation",
    backend="vllm",
    )

result = chat.completion(
  messages=[
    ChatMessage(role="system", content="You're a honest assistant."),
    ChatMessage(role="user", content="There's a llama in my garden, what should I do?"),
  ]
)

# Output: Oh my goodness, a llama in your garden?! 😱 That's quite a surprise! 😅 As an honest assistant, I must inform you that llamas are not typically known for their gardening skills, so it's possible that the llama in your garden may have wandered there accidentally or is seeking shelter. 🐮 ...

```

`llmlite` also supports other parameters like `temperature`, `max_length`, `do_sample`, `top_k`, `top_p` to help control the length, randomness and diversity of the generated text.

See **[examples](./examples/)** for reference.

### Prompting

You can use `llmlite` to help you generate full prompts, for instance:

```python
from llmlite.apis import ChatMessage, LlamaChat

messages = [
    ChatMessage(role="system", content="You're a honest assistant."),
    ChatMessage(role="user", content="There's a llama in my garden, what should I do?"),
]

LlamaChat.prompt(messages)

# Output:
# <s>[INST] <<SYS>>
# You're a honest assistant.
# <</SYS>>

# There's a llama in my garden, what should I do? [/INST]
```

### Logging

Set the env variable `LOG_LEVEL` for log configuration, default to `INFO`, others like DEBUG, INFO, WARNING etc..

## Roadmap

- Adapter support
- Quantization
- Streaming

## Contributions

🚀 All kinds of contributions are welcomed ! Please follow [Contributing](/CONTRIBUTING.md).

## Contributors

🎉 Thanks to all these contributors.

<a href="https://github.com/InftyAI/ChatLLM/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=InftyAI/ChatLLM" />
</a>

