Skip to content

proxy.md

Latest commit

 

History

History
83 lines (59 loc) · 3.01 KB

proxy.md

File metadata and controls

83 lines (59 loc) · 3.01 KB
Raw
title Proxy API (OpenAI-Compatible)
description /v1 endpoints exposing your configured models to OpenAI SDKs and tools
sidebar_position 12

Proxy API

The proxy exposes your model registry as a standard OpenAI-compatible API. Point any OpenAI SDK at the server base URL and it "just works":

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:3000/v1",
  apiKey: "aut_...",
});

See LLM API Proxy for setup, authentication, and model resolution. All /v1 endpoints are CORS-enabled (*) and support Authorization: Bearer or x-api-key.

POST /v1/chat/completions

OpenAI-compatible chat completions, streaming or not. The model field is special — see Model Resolution.

Request Body:

{
  "model": "auto",
  "messages": [{ "role": "user", "content": "Explain SQLite WAL mode" }],
  "stream": true,
  "max_tokens": 512,
  "temperature": 0.7
}

Supported fields (forwarded to the upstream provider):

messages (required), temperature, top_p, n, stop, max_tokens, max_completion_tokens, presence_penalty, frequency_penalty, logit_bias, user, tools, tool_choice, response_format, seed, logprobs, top_logprobs, parallel_tool_calls, metadata, reasoning_effort, service_tier, store

stream and model are handled by the proxy itself: stream_options.include_usage is force-set, and the upstream model is always the resolved registry modelId while the response echoes your requested string.

Errors use the OpenAI shape:

{ "error": { "message": "...", "type": "authentication_error", "code": null } }
Status Meaning
400 Malformed JSON, missing/empty messages, empty model
401 Missing or invalid API key
403 Key not allowed to use any configured models
404 model didn't resolve (type: "model_not_found", lists available ids)
500 No models configured, or all fallback models failed

Source: app/v1/chat/completions/route.ts, lib/proxy/completions.ts

Streaming

With "stream": true the response is text/event-stream. Chunks are relayed verbatim from the first model in the chain that responds, framed as data: {chunk}\n\n with a final data: [DONE]\n\n. If a model in the chain errors at connect time, the proxy records an error usage row and transparently tries the next model. Mid-stream errors are emitted as a data: {error} chunk.

GET /v1/models

List the models a key is allowed to use.

Response:

{
  "object": "list",
  "data": [
    { "id": "gpt-4o-mini", "object": "model", "created": 1735689600, "owned_by": "openai" }
  ]
}

id is the registry modelId; owned_by is the provider preset (defaults to "autark").

Source: app/v1/models/route.ts

OPTIONS /v1/*

Preflight CORS — returns 204 with Access-Control-Allow-Origin: *, allowed methods GET, POST, OPTIONS, and allowed headers Authorization, Content-Type, x-api-key.