Metadata-Version: 2.4
Name: lapi
Version: 1.0.1
Summary: A lightweight CLI tool and OpenAI-compatible server for querying multiple Large Language Model (LLM) providers
Home-page: https://github.com/ServiceStack/llms
Author: ServiceStack
Author-email: ServiceStack <team@servicestack.net>
Maintainer-email: ServiceStack <team@servicestack.net>
License-Expression: BSD-3-Clause
Project-URL: Homepage, https://github.com/ServiceStack/llms
Project-URL: Documentation, https://github.com/ServiceStack/llms#readme
Project-URL: Repository, https://github.com/ServiceStack/llms
Project-URL: Bug Reports, https://github.com/ServiceStack/llms/issues
Keywords: llm,ai,openai,anthropic,google,gemini,groq,mistral,ollama,cli,server,chat,completion
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Systems Administration
Classifier: Topic :: Utilities
Classifier: Environment :: Console
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aiohttp
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# ServiceStack LLMs

A lightweight CLI tool and OpenAI-compatible server for querying multiple Large Language Model (LLM) providers.

Mix and match local models with models from different API providers to best fit for your needs. Requests are automatically routed to available providers that supports the requested model (in defined order). Define
free/cheapest/local providers first to save on costs, any failures are automatically retried on the next available provider.

## Features

- **Lightweight**: Single [llms.py](llms.py) Python file with single `aiohttp` dependency
- **Multi-Provider Support**: OpenAI, Anthropic, Google (Gemini), Groq, Mistral, Ollama, OpenRouter, and more
- **OpenAI-Compatible API**: Works with any client that supports OpenAI's chat completion API
- **Configuration Management**: Easy provider enable/disable and configuration management
- **CLI Interface**: Simple command-line interface for quick interactions
- **Server Mode**: Run an OpenAI-compatible HTTP server at `http://localhost:{PORT}/v1/chat/completions`
- **Image Support**: Process images through vision-capable models
- **Auto-Discovery**: Automatically discover available Ollama models
- **Unified Models**: Define custom model names that map to different provider-specific names

## Installation

### Option 1: Install from PyPI (Recommended)

```bash
pip install lapi
```

### Option 2: Download directly

1. Download `llms.py`

```bash
curl -O https://raw.githubusercontent.com/ServiceStack/llms/main/llms.py
chmod +x llms.py
mv llms.py ~/.local/bin/llms
```

2. Install single dependency:

```bash
pip install aiohttp
```

## Quick Start

### 1. Initialize Configuration

Create a default configuration file:

```bash
llms --init
```

This saves the latest [llms.json](llms.json) configuration to `~/.llms/llms.json`.

Modify `~/.llms/llms.json` to enable providers, add required API keys, additional models or any custom
OpenAI-compatible providers.

### 2. Set API Keys

Set environment variables for the providers you want to use:

```bash
export OPENROUTER_API_KEY="..."
export GROQ_API_KEY="..."
export GOOGLE_API_KEY="..."
export ANTHROPIC_API_KEY="..."
# ... etc
```

### 3. Enable Providers

Enable the providers you want to use:

```bash
# Enable providers with free models and free tiers
llms --enable openrouter_free google_free groq

# Enable paid providers
llms --enable openrouter anthropic google openai mistral
```

### 4. Start Chatting

```bash
llms "What is the capital of France?"
```

## Configuration

The configuration file (`llms.json`) defines available providers, models, and default settings. Key sections:

### Defaults
- `headers`: Common HTTP headers for all requests
- `text`: Default chat completion request template for text prompts

### Providers

Each provider configuration includes:
- `enabled`: Whether the provider is active
- `type`: Provider class (OpenAiProvider, GoogleProvider, etc.)
- `api_key`: API key (supports environment variables with `$VAR_NAME`)
- `base_url`: API endpoint URL
- `models`: Model name mappings (local name → provider name)

## Command Line Usage

### Basic Chat

```bash
# Simple question
llms "Explain quantum computing"

# With specific model
llms -m gemini-2.5-pro "Write a Python function to sort a list"

# With system prompt
llms -s "You are a helpful coding assistant" "How do I reverse a string in Python?"

# With image (vision models)
llms --image image.jpg "What's in this image?"
llms --image https://example.com/photo.png "Describe this photo"

# Display full JSON Response
llms "Explain quantum computing" --raw
```

### Using a Chat Template

By default llms uses the `defaults/text` chat completion request defined in [llms.json](llms.json). 

You can instead use a custom chat completion request with `--chat`, e.g:

```bash
# Load chat completion request from JSON file
llms --chat request.json

# Override user message
llms --chat request.json "New user message"

# Override model
llms -m kimi-k2 --chat request.json
```

Example `request.json`:

```json
{
  "model": "kimi-k2",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user",   "content": ""}
  ],
  "temperature": 0.7,
  "max_tokens": 150
}
```

### Image Requests

Send images to vision-capable models using the `--image` option:

```bash
# Use defaults/image Chat Template (Describe the key features of the input image)
llms --image ./screenshot.png

# Local image file
llms --image ./screenshot.png "What's in this image?"

# Remote image URL
llms --image https://example.org/photo.jpg "Describe this photo"

# Data URI
llms --image "data:image/png;base64,$(base64 -w 0 image.png)" "Describe this image"

# With a specific vision model
llms -m gemini-2.5-flash --image chart.png "Analyze this chart"

# Combined with system prompt
llms -s "You are a data analyst" --image graph.png "What trends do you see?"

# With custom chat template
llms --chat image-request.json --image photo.jpg
```

Example of `image-request.json`:

```json
{
    "model": "qwen2.5vl",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": ""
                    }
                },
                {
                    "type": "text",
                    "text": "Caption this image"
                }
            ]
        }
    ]
}
```

**Supported image formats**: PNG, WEBP, JPG, JPEG, GIF, BMP, TIFF, ICO

**Image sources**:
- **Local files**: Absolute paths (`/path/to/image.jpg`) or relative paths (`./image.png`, `../image.jpg`)
- **Remote URLs**: HTTP/HTTPS URLs are automatically downloaded
- **Data URIs**: Base64-encoded images (`data:image/png;base64,...`)

Images are automatically processed and converted to base64 data URIs before being sent to the model.

### Vision-Capable Models

Popular models that support image analysis:
- **OpenAI**: GPT-4o, GPT-4o-mini, GPT-4.1
- **Anthropic**: Claude Sonnet 4.0, Claude Opus 4.1
- **Google**: Gemini 2.5 Pro, Gemini Flash
- **Ollama**: qwen2.5vl, llava

Images are automatically downloaded and converted to base64 data URIs.

### Configuration Management

```bash
# List enabled providers and models
llms --list
llms ls

# List specific providers
llms ls ollama
llms ls google anthropic

# Enable providers
llms --enable openrouter
llms --enable anthropic google_free groq

# Disable providers
llms --disable ollama
llms --disable openai anthropic

# Set default model
llms --default grok-4

# Update to latest version
llms --update
```

### Advanced Options

```bash
# Use custom config file
llms --config /path/to/config.json "Hello"

# Get raw JSON response
llms --raw "What is 2+2?"

# Enable verbose logging
llms --verbose "Tell me a joke"

# Custom log prefix
llms --verbose --logprefix "[DEBUG] " "Hello world"

# Set default model (updates config file)
llms --default grok-4

# Update llms.py to latest version
llms --update
```

### Default Model Configuration

The `--default MODEL` option allows you to set the default model used for all chat completions. This updates the `defaults.text.model` field in your configuration file:

```bash
# Set default model to gpt-oss
llms --default gpt-oss:120b

# Set default model to Claude Sonnet
llms --default claude-sonnet-4-0

# The model must be available in your enabled providers
llms --default gemini-2.5-pro
```

When you set a default model:
- The configuration file (`~/.llms/llms.json`) is automatically updated
- The specified model becomes the default for all future chat requests
- The model must exist in your currently enabled providers
- You can still override the default using `-m MODEL` for individual requests

### Updating llms.py

The `--update` option downloads and installs the latest version of `llms.py` from the GitHub repository:

```bash
# Update to latest version
llms --update
```

This command:
- Downloads the latest `llms.py` from `https://raw.githubusercontent.com/ServiceStack/llms/refs/heads/main/llms.py`
- Overwrites your current `llms.py` file with the latest version
- Preserves your existing configuration file (`llms.json`)
- Requires an internet connection to download the update

### Beautiful rendered Markdown

Pipe Markdown output to [glow](https://github.com/charmbracelet/glow) to beautifully render it in the terminal:

```bash
llms "Explain quantum computing" | glow
```

## Server Mode

Run as an OpenAI-compatible HTTP server:

```bash
# Start server on port 8000
llms --serve 8000
```

The server exposes a single endpoint:
- `POST /v1/chat/completions` - OpenAI-compatible chat completions

Example client usage:

```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k2",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }'
```

## Supported Providers

### OpenAI
- **Type**: `OpenAiProvider`
- **Models**: GPT-4o, GPT-4o-mini, GPT-4.1, O4-mini, etc.
- **Features**: Text, images, function calling

```bash
export OPENAI_API_KEY="your-key"
llms --enable openai
```

### Anthropic (Claude)
- **Type**: `OpenAiProvider` 
- **Models**: Claude Opus 4.1, Sonnet 4.0, Haiku 3.5, etc.
- **Features**: Text, images, large context windows

```bash
export ANTHROPIC_API_KEY="your-key"
llms --enable anthropic
```

### Google Gemini
- **Type**: `GoogleProvider`
- **Models**: Gemini 2.5 Pro, Flash, Flash-Lite
- **Features**: Text, images, safety settings

```bash
export GOOGLE_API_KEY="your-key"
llms --enable google_free
```

### Groq
- **Type**: `OpenAiProvider`
- **Models**: Llama 3.3, Gemma 2, Kimi K2, etc.
- **Features**: Fast inference, competitive pricing

```bash
export GROQ_API_KEY="your-key" 
llms --enable groq
```

### Ollama (Local)
- **Type**: `OllamaProvider`
- **Models**: Auto-discovered from local Ollama installation
- **Features**: Local inference, privacy, no API costs

```bash
# Ollama must be running locally
llms --enable ollama
```

### OpenRouter
- **Type**: `OpenAiProvider`
- **Models**: 100+ models from various providers
- **Features**: Access to latest models, free tier available

```bash
export OPENROUTER_API_KEY="your-key"
llms --enable openrouter
```

### Mistral
- **Type**: `OpenAiProvider`
- **Models**: Mistral Large, Codestral, Pixtral, etc.
- **Features**: Code generation, multilingual

```bash
export MISTRAL_API_KEY="your-key"
llms --enable mistral
```

## Model Routing

The tool automatically routes requests to the first available provider that supports the requested model. If a provider fails, it tries the next available provider with that model.

Example: If both OpenAI and OpenRouter support `kimi-k2`, the request will first try OpenRouter (free), then fall back to Groq than OpenRouter (Paid) if requests fails.

## Environment Variables

| Variable | Description | Example |
|----------|-------------|---------|
| `LLMS_CONFIG_PATH`        | Custom config file path | `/path/to/llms.json` |
| `OPENAI_API_KEY`          | OpenAI API key | `sk-...` |
| `ANTHROPIC_API_KEY`       | Anthropic API key | `sk-ant-...` |
| `GOOGLE_API_KEY`          | Google API key | `AIza...` |
| `GROQ_API_KEY`            | Groq API key | `gsk_...` |
| `MISTRAL_API_KEY`         | Mistral API key | `...` |
| `OPENROUTER_API_KEY`      | OpenRouter API key | `sk-or-...` |
| `OPENROUTER_FREE_API_KEY` | OpenRouter free tier key | `sk-or-...` |
| `CODESTRAL_API_KEY`       | Codestral API key | `...` |

## Configuration Examples

### Minimal Configuration

```json
{
  "defaults": {
    "headers": {"Content-Type": "application/json"},
    "text": {
      "model": "kimi-k2",
      "messages": [{"role": "user", "content": ""}]
    }
  },
  "providers": {
    "openai": {
      "enabled": true,
      "type": "OpenAiProvider",
      "base_url": "https://api.groq.com/openai",
      "api_key": "$GROQ_API_KEY",
      "models": {
        "llama3.3:70b": "llama-3.3-70b-versatile",
        "llama4:109b": "meta-llama/llama-4-scout-17b-16e-instruct",
        "llama4:400b": "meta-llama/llama-4-maverick-17b-128e-instruct",
        "kimi-k2": "moonshotai/kimi-k2-instruct-0905",
        "gpt-oss:120b": "openai/gpt-oss-120b",
        "gpt-oss:20b": "openai/gpt-oss-20b",
        "qwen3:32b": "qwen/qwen3-32b"
      }
    }
  }
}
```

### Multi-Provider Setup

```json
{
"providers": {
    "openrouter": {
      "enabled": true, 
        "type": "OpenAiProvider",
        "base_url": "https://openrouter.ai/api",
        "api_key": "$OPENROUTER_API_KEY",
        "models": {
            "grok-4": "x-ai/grok-4",
            "glm-4.5-air": "z-ai/glm-4.5-air",
            "kimi-k2": "moonshotai/kimi-k2",
            "deepseek-v3.1:671b": "deepseek/deepseek-chat",
            "llama4:400b": "meta-llama/llama-4-maverick"
        }
    },
    "anthropic": {
      "enabled": true, 
      "type": "OpenAiProvider",
      "base_url": "https://api.anthropic.com",
      "api_key": "$ANTHROPIC_API_KEY",
      "models": {
        "claude-sonnet-4-0": "claude-sonnet-4-0"
      }
    },
    "ollama": {
      "enabled": true,
      "type": "OllamaProvider", 
      "base_url": "http://localhost:11434",
      "models": {},
      "all_models": true
    }
  }
}
```

## Usage

Run `llms` without arguments to see the help screen:

    usage: llms.py [-h] [--config FILE] [-m MODEL] [--logprefix PREFIX] [--verbose] [--raw] [--chat REQUEST]
                [-s PROMPT] [--image IMAGE] [--list] [--serve PORT] [--init] [--enable PROVIDER]
                [--disable PROVIDER] [--default MODEL] [--update]

    llms v1

    options:
    -h, --help            show this help message and exit
    --config FILE         Path to config file
    -m MODEL, --model MODEL
                            Model to use
    --logprefix PREFIX    Prefix used in log messages
    --verbose             Verbose output
    --raw                 Return raw AI JSON response
    --chat REQUEST        OpenAI Chat Completion Request to send
    -s PROMPT, --system PROMPT
                            System prompt to use for chat completion
    --image IMAGE         Image prompt to use in chat completion
    --list                Show list of enabled providers and their models (alias ls provider?)
    --serve PORT          Port to start an OpenAI Chat compatible server on
    --init                Create a default llms.json
    --enable PROVIDER     Enable a provider
    --disable PROVIDER    Disable a provider
    --default MODEL       Configure the default model to use
    --update              Update to latest version

## Troubleshooting

### Common Issues

**Config file not found**
```bash
# Initialize default config
llms --init

# Or specify custom path
llms --config ./my-config.json
```

**No providers enabled**

```bash
# Check status
llms --list

# Enable providers
llms --enable google anthropic
```

**API key issues**
```bash
# Check environment variables
echo $ANTHROPIC_API_KEY

# Enable verbose logging
llms --verbose "test"
```

**Model not found**

```bash
# List available models
llms --list

# Check provider configuration
llms ls openrouter
```

### Debug Mode

Enable verbose logging to see detailed request/response information:

```bash
llms --verbose --logprefix "[DEBUG] " "Hello"
```

This shows:
- Enabled providers
- Model routing decisions
- HTTP request details
- Error messages with stack traces

## Development

### Project Structure

- `llms.py` - Main script with CLI and server functionality
- `llms.json` - Default configuration file
- `requirements.txt` - Python dependencies

### Provider Classes

- `OpenAiProvider` - Generic OpenAI-compatible provider
- `OllamaProvider` - Ollama-specific provider with model auto-discovery
- `GoogleProvider` - Google Gemini with native API format
- `GoogleOpenAiProvider` - Google Gemini via OpenAI-compatible endpoint

### Adding New Providers

1. Create a provider class inheriting from `OpenAiProvider`
2. Implement provider-specific authentication and formatting
3. Add provider configuration to `llms.json`
4. Update initialization logic in `init_llms()`

## Contributing

Contributions are welcome! Please submit a PR to add support for any missing OpenAI-compatible providers.
