diff --git a/docs/docs/how-to/backends-and-configuration.md b/docs/docs/how-to/backends-and-configuration.md index 150754ba0..adceda481 100644 --- a/docs/docs/how-to/backends-and-configuration.md +++ b/docs/docs/how-to/backends-and-configuration.md @@ -30,11 +30,11 @@ The following table shows all available backends, their class names, import path | Backend | Class | Import | Required extras | `start_session()` | | ------- | ----- | ------ | --------------- | ----------------- | -| [Ollama](../integrations/ollama) | `OllamaModelBackend` | `mellea.backends.ollama` | base | `backend_name="ollama"` | -| [OpenAI](../integrations/openai) | `OpenAIBackend` | `mellea.backends.openai` | base | `backend_name="openai"` | -| [LiteLLM](../integrations/bedrock) | `LiteLLMBackend` | `mellea.backends.litellm` | `mellea[litellm]` | `backend_name="litellm"` | -| [Hugging Face](../integrations/huggingface) | `LocalHFBackend` | `mellea.backends.huggingface` | `mellea[hf]` | `backend_name="hf"` | -| [WatsonX](../integrations/watsonx) | `WatsonxAIBackend` | `mellea.backends.watsonx` | `mellea[watsonx]` | `backend_name="watsonx"` (deprecated) | +| [Ollama](../integrations/ollama.md) | `OllamaModelBackend` | `mellea.backends.ollama` | base | `backend_name="ollama"` | +| [OpenAI](../integrations/openai.md) | `OpenAIBackend` | `mellea.backends.openai` | base | `backend_name="openai"` | +| [LiteLLM](../integrations/litellm.md) | `LiteLLMBackend` | `mellea.backends.litellm` | `mellea[litellm]` | `backend_name="litellm"` | +| [Hugging Face](../integrations/huggingface.md) | `LocalHFBackend` | `mellea.backends.huggingface` | `mellea[hf]` | `backend_name="hf"` | +| [WatsonX](../integrations/watsonx.md) | `WatsonxAIBackend` | `mellea.backends.watsonx` | `mellea[watsonx]` | `backend_name="watsonx"` (deprecated) | > **Note:** Vertex AI uses the LiteLLM backend with appropriate model IDs. See the [Vertex AI integration](../integrations/vertex-ai) for details. For detailed setup instructions, click the backend name in the table above. @@ -112,7 +112,8 @@ m = MelleaSession( > See the [LiteLLM docs](https://docs.litellm.ai/) for your provider's setup. LiteLLM provides unified access to 100+ providers — Anthropic, AWS Bedrock, Azure, -and more: +IBM WatsonX, and more. Set the provider's credentials in the environment and pass a +`/` model string: ```python # Requires: mellea[litellm] @@ -128,6 +129,10 @@ print(str(result)) # Output will vary — LLM responses depend on model and temperature. ``` +See the [LiteLLM integration](../integrations/litellm.md) for direct-mode vs proxy usage, +self-hosted proxy setup, per-provider configuration examples, an environment variable +reference, and the WatsonX migration path. + ## Hugging Face backend > **Backend note:** Requires `pip install "mellea[hf]"`. Models are downloaded from diff --git a/docs/docs/integrations/litellm.md b/docs/docs/integrations/litellm.md new file mode 100644 index 000000000..215628774 --- /dev/null +++ b/docs/docs/integrations/litellm.md @@ -0,0 +1,321 @@ +--- +title: "LiteLLM" +description: "Reach 100+ LLM providers through Mellea's LiteLLMBackend: direct calls, a self-hosted proxy, and the recommended migration path off the deprecated WatsonX backend." +sidebar_label: "LiteLLM" +# diataxis: how-to +--- + +The [`LiteLLMBackend`](../reference/glossary.md#litellm--litellmbackend) gives Mellea a +single code path to 100+ model providers (Anthropic, AWS Bedrock, Azure OpenAI, IBM +WatsonX, Google Vertex AI, and more) by delegating provider auth and request +translation to [LiteLLM](https://docs.litellm.ai/). You switch providers by changing +the `model_id` prefix; your Mellea code stays the same. + +**Prerequisites:** `pip install 'mellea[litellm]'` and provider credentials set as +environment variables (each provider is listed below). + +## When to use LiteLLM + +- **Multi-provider access.** One backend reaches most hosted providers, so you can + compare models or fail over between vendors without rewriting session code. +- **WatsonX migration.** The native `WatsonxAIBackend` is deprecated since v0.4; + LiteLLM (or the OpenAI backend) is the recommended replacement. See + [Migrating from the WatsonX backend](#migrating-from-the-watsonx-backend). +- **Central governance.** Paired with a self-hosted proxy, LiteLLM centralizes API + keys, routing, rate limits, and cost tracking across teams. + +For providers that already have a dedicated Mellea page, prefer it for provider-specific +detail: [AWS Bedrock](./bedrock.md) and [Vertex AI](./vertex-ai.md) both use this +backend under the hood. + +## Direct mode vs the LiteLLM Proxy + +There are two ways to use LiteLLM. **Direct mode** calls the provider straight from +your process. The **LiteLLM Proxy** is a standalone server you (or your platform team) +run; Mellea talks to the proxy, and the proxy talks to the providers. + +| | Direct mode | LiteLLM Proxy | +| --- | --- | --- | +| Setup | `pip install 'mellea[litellm]'` only | Run a separate proxy server | +| Credentials | Provider keys live in each app's environment | Keys live on the proxy; apps hold only a proxy key | +| Routing / fallback / rate limits | Per-app | Centralized on the proxy | +| Cost & usage tracking | Per-app | Centralized on the proxy | +| Best for | Single apps, local dev, quick provider comparison | Shared infrastructure, many apps, governed key management | + +## Direct mode + +The quickest path is [`start_session()`](../reference/glossary.md#melleasession) with +`backend_name="litellm"`. The `model_id` is a LiteLLM model string of the form +`/`: + +```python +# Requires: mellea[litellm] +# Returns: str +import mellea + +m = mellea.start_session( + backend_name="litellm", + model_id="anthropic/claude-sonnet-4-20250514", +) +result = m.chat("Give me three facts about the Amazon rainforest.") +print(str(result)) +# Output will vary — LLM responses depend on model and temperature. +``` + +> **Note:** For cloud providers, leave `base_url` unset (the default). LiteLLM infers +> the correct endpoint from the `model_id` prefix. Only set `base_url` when you target +> a proxy or a local server (see [Self-hosted LiteLLM Proxy](#self-hosted-litellm-proxy)). + +For full control, construct the [`Backend`](../reference/glossary.md#backend) directly and +pass it to [`MelleaSession`](../reference/glossary.md#melleasession): + +```python +# Requires: mellea[litellm] +# Returns: MelleaSession +from mellea import MelleaSession +from mellea.backends.litellm import LiteLLMBackend + +m = MelleaSession( + LiteLLMBackend(model_id="anthropic/claude-sonnet-4-20250514"), +) +``` + +## Provider configuration examples + +Set the provider's credentials in the environment, then pass the matching `model_id` +prefix. The [environment variable reference](#environment-variable-reference) below +lists every provider in one table. + +### Anthropic + +```bash +export ANTHROPIC_API_KEY=your-api-key-here +``` + +```python +# Requires: mellea[litellm] +# Returns: MelleaSession +from mellea import MelleaSession +from mellea.backends.litellm import LiteLLMBackend + +m = MelleaSession( + LiteLLMBackend(model_id="anthropic/claude-sonnet-4-20250514"), +) +``` + +### Azure OpenAI + +The `model_id` is `azure/`, your Azure deployment name rather +than a base model name: + +```bash +export AZURE_API_KEY=your-api-key-here +export AZURE_API_BASE=https://your-resource.openai.azure.com +export AZURE_API_VERSION=2024-02-15-preview +``` + +```python +# Requires: mellea[litellm] +# Returns: MelleaSession +from mellea import MelleaSession +from mellea.backends.litellm import LiteLLMBackend + +m = MelleaSession( + LiteLLMBackend(model_id="azure/my-gpt-4o-deployment"), +) +``` + +### IBM WatsonX + +LiteLLM reaches WatsonX with the `watsonx/` prefix. Note the API-key variable is +`WATSONX_APIKEY` (no underscore before `KEY`), which differs from the native backend's +`WATSONX_API_KEY`: + +```bash +export WATSONX_URL=https://us-south.ml.cloud.ibm.com +export WATSONX_APIKEY=your-api-key-here +export WATSONX_PROJECT_ID=your-project-id +``` + +```python +# Requires: mellea[litellm] +# Returns: MelleaSession +from mellea import MelleaSession +from mellea.backends.litellm import LiteLLMBackend + +m = MelleaSession( + LiteLLMBackend(model_id="watsonx/ibm/granite-3-3-8b-instruct"), +) +``` + +### AWS Bedrock and Google Vertex AI + +Both use `LiteLLMBackend` and have dedicated pages with credential setup and model-string +tables: + +- **Bedrock**: `bedrock/converse/`. See [AWS Bedrock](./bedrock.md). +- **Vertex AI**: `vertex_ai/`. See [Vertex AI](./vertex-ai.md). + +## Self-hosted LiteLLM Proxy + +The [LiteLLM Proxy](https://docs.litellm.ai/docs/simple_proxy) is a server that holds +your provider keys centrally and exposes them behind a single proxy key. Define your +models in a `config.yaml`: + +```yaml +model_list: + - model_name: my-model + litellm_params: + model: anthropic/claude-sonnet-4-20250514 + api_key: os.environ/ANTHROPIC_API_KEY +``` + +Start the proxy (defaults to port 4000): + +```bash +pip install 'litellm[proxy]' +litellm --config config.yaml +``` + +Point Mellea at the proxy with the `litellm_proxy/` prefix and the proxy URL as +`base_url` (forwarded to LiteLLM as `api_base`). Authenticate with the proxy's key via +`LITELLM_PROXY_API_KEY`: + +```bash +export LITELLM_PROXY_API_KEY=sk-... +``` + +```python +# Requires: mellea[litellm] +# Returns: MelleaSession +from mellea import MelleaSession +from mellea.backends.litellm import LiteLLMBackend + +m = MelleaSession( + LiteLLMBackend( + model_id="litellm_proxy/my-model", + base_url="http://localhost:4000", + ), +) +``` + +`my-model` is the `model_name` you defined in the proxy's `config.yaml`; the app never +sees which provider or model backs it. To pass the proxy key explicitly instead of using +the environment variable, add it to `model_options`: + +```python +# Requires: mellea[litellm] +# Returns: MelleaSession +from mellea import MelleaSession +from mellea.backends.litellm import LiteLLMBackend + +m = MelleaSession( + LiteLLMBackend( + model_id="litellm_proxy/my-model", + base_url="http://localhost:4000", + model_options={"api_key": "sk-..."}, + ), +) +``` + +## Migrating from the WatsonX backend + +The native `WatsonxAIBackend` is deprecated since v0.4. To move an existing WatsonX +session onto LiteLLM, swap the backend and prefix the model with `watsonx/`. + +Before (deprecated): + +```python +# Requires: mellea[watsonx] +# Returns: MelleaSession +from mellea import MelleaSession +from mellea.backends.watsonx import WatsonxAIBackend + +m = MelleaSession( + WatsonxAIBackend(model_id="ibm/granite-3-3-8b-instruct"), +) +``` + +After (LiteLLM): + +```python +# Requires: mellea[litellm] +# Returns: MelleaSession +from mellea import MelleaSession +from mellea.backends.litellm import LiteLLMBackend + +m = MelleaSession( + LiteLLMBackend(model_id="watsonx/ibm/granite-3-3-8b-instruct"), +) +``` + +> **Warning:** LiteLLM reads the WatsonX API key from `WATSONX_APIKEY`, whereas the +> native backend read `WATSONX_API_KEY`. Rename the variable when you migrate, or the +> new backend will not find your credentials. `WATSONX_URL` and `WATSONX_PROJECT_ID` +> are unchanged. + +## Environment variable reference + +Set these before creating the session. LiteLLM reads them automatically based on the +`model_id` prefix. + +| Provider | `model_id` prefix | Environment variables | +| --- | --- | --- | +| Anthropic | `anthropic/` | `ANTHROPIC_API_KEY` | +| Azure OpenAI | `azure/` | `AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION` | +| AWS Bedrock | `bedrock/converse/` | `AWS_BEARER_TOKEN_BEDROCK` (or standard AWS credentials) | +| IBM WatsonX | `watsonx/` | `WATSONX_URL`, `WATSONX_APIKEY`, `WATSONX_PROJECT_ID` | +| Google Vertex AI | `vertex_ai/` | `VERTEXAI_PROJECT`, `VERTEXAI_LOCATION` (see [Vertex AI](./vertex-ai.md)) | +| LiteLLM Proxy | `litellm_proxy/` | `LITELLM_PROXY_API_KEY`, `LITELLM_PROXY_API_BASE` (or pass `base_url`) | + +See the [LiteLLM providers documentation](https://docs.litellm.ai/docs/providers) for +the full list and any provider-specific variables. + +## Model options + +Pass generation parameters with [`ModelOption`](../reference/glossary.md#modeloption), the +same as any other backend. Options set at construction apply to all calls; options +passed to `instruct()` or `chat()` apply to that call only and take precedence: + +```python +# Requires: mellea[litellm] +# Returns: MelleaSession +from mellea import MelleaSession +from mellea.backends import ModelOption +from mellea.backends.litellm import LiteLLMBackend + +m = MelleaSession( + LiteLLMBackend( + model_id="anthropic/claude-sonnet-4-20250514", + model_options={ModelOption.TEMPERATURE: 0.2, ModelOption.MAX_NEW_TOKENS: 512}, + ), +) +``` + +See [Configure Model Options](../how-to/configure-model-options.md) for the full list of +`ModelOption` keys. + +## Troubleshooting + +**`ImportError: The LiteLLM backend requires extra dependencies`:** + +```bash +pip install 'mellea[litellm]' +``` + +**"litellm allows for unknown / non-openai input params" or "litellm may drop the +following openai keys" warnings:** LiteLLM supports different parameters per provider. +Mellea logs which model options are unrecognized or may be dropped for the current +model, and passes the request through anyway. These warnings are informational: the +unsupported OpenAI parameters are dropped automatically (`drop_params=True`), and there +are occasional false positives. Remove the flagged options if a call misbehaves. + +**"There is a known bug with litellm. This generation call may fail" warning:** This +appears for WatsonX-over-LiteLLM when Mellea detects generation calls running across +multiple asyncio event loops. Run only synchronous Mellea functions, or run your async +Mellea code from a single `asyncio.run()` call. + +--- + +**See also:** [Backends and Configuration](../how-to/backends-and-configuration.md) | +[AWS Bedrock](./bedrock.md) | [Vertex AI](./vertex-ai.md) | [IBM WatsonX](./watsonx.md) diff --git a/docs/sidebars.ts b/docs/sidebars.ts index 0c6db0ba1..fd02bfa1d 100644 --- a/docs/sidebars.ts +++ b/docs/sidebars.ts @@ -79,6 +79,7 @@ const sidebars: SidebarsConfig = { 'integrations/ollama', 'integrations/huggingface', 'integrations/openai', + 'integrations/litellm', 'integrations/vertex-ai', 'integrations/bedrock', 'integrations/watsonx',