Run on local Ollama models by default

.env.example

@@ -3,15 +3,23 @@
 ANTHROPIC_API_KEY=
 OPENAI_API_KEY=
 
+# Default: beam:cloud through a local Ollama daemon (run `ollama signin`
+# once for cloud models). Any tagged Ollama name works (e.g. qwen3:8b).
 # chat-latest follows ChatGPT Instant. gpt-* uses OpenAI; claude-* uses Anthropic.
 # Model access/availability must be verified with your provider.
-# Blank or unsupported model prefixes fail at startup.
-# Keys are optional for BYOK-only hosting; visitors enter both keys in the app.
-LLM_MODEL=chat-latest
+# Blank or unsupported model names fail at startup.
+# With Ollama models no keys are needed; visitors may still add their own.
+LLM_MODEL=beam:cloud
+# OLLAMA_BASE_URL=http://localhost:11434/v1
+# OLLAMA_API_KEY=
 
 # Jev selects the renderer and visualization before each new user turn.
+# Default: clef-flash in local Ollama (`ollama pull clef-flash`), no key needed.
+# For hosted Jev set JEV_URL=https://api.typesafe.ai/v1/systemone,
+# JEV_MODEL=jev-latest and TYPESAFE_API_KEY.
+JEV_MODEL=clef-flash:latest
+# JEV_URL=http://localhost:11434/v1/systemone
 TYPESAFE_API_KEY=
-JEV_MODEL=jev-latest
 
 # Frontend settings: place these in apps/app/.env.local.
 # LANGGRAPH_DEPLOYMENT_URL defaults to http://localhost:8123.

CLAUDE.md

@@ -29,7 +29,7 @@ Use the existing validated `Websandbox.connection.remote.sendPrompt({ text })` a
 
 ## Configuration and checks
 
-The local default model is `chat-latest` (ChatGPT Instant). `claude-*` requires `ANTHROPIC_API_KEY`; `chat-latest` and `gpt-*` require `OPENAI_API_KEY`. Jev routing requires `TYPESAFE_API_KEY`. Do not invent native model APIs or silently change pinned model IDs. See `render.yaml` for its separate explicit deployment model setting.
+The local default answer model is `beam:cloud` through Ollama's OpenAI-compatible API (`OLLAMA_BASE_URL`, default `http://localhost:11434/v1`); any tagged Ollama name works without a key. `claude-*` requires `ANTHROPIC_API_KEY`; `chat-latest` and `gpt-*` require `OPENAI_API_KEY`. Jev routing defaults to `clef-flash:latest` at Ollama's `/v1/systemone` (`JEV_URL`, `JEV_MODEL`); only hosted TypeSafe requires `TYPESAFE_API_KEY`. Visitor BYOK keys always use OpenAI `chat-latest` and hosted Jev, and the chat asks for them only when `GET /credentials/status` reports the server's own models unconfigured. Do not invent native model APIs or silently change pinned model IDs. See `render.yaml` for its separate explicit deployment model setting, which stays on OpenAI and hosted Jev.
 
 ```bash
 pnpm install

Makefile

@@ -8,8 +8,8 @@ install: ## Install all dependencies (Node + Python)
 
 setup: install ## Full setup: install deps and create .env from template
 	@if [ ! -f apps/agent/.env ]; then \
-		printf '%s\n' 'OPENAI_API_KEY=' 'TYPESAFE_API_KEY=' 'LLM_MODEL=chat-latest' 'JEV_MODEL=jev-latest' > apps/agent/.env; \
-		echo "Created apps/agent/.env — set OPENAI_API_KEY and TYPESAFE_API_KEY"; \
+		printf '%s\n' 'LLM_MODEL=beam:cloud' 'JEV_MODEL=clef-flash:latest' 'OPENAI_API_KEY=' 'TYPESAFE_API_KEY=' > apps/agent/.env; \
+		echo "Created apps/agent/.env for Ollama — run 'ollama signin' and 'ollama pull clef-flash'"; \
 	else \
 		echo "apps/agent/.env already exists, skipping"; \
 	fi

README.md

@@ -140,28 +140,27 @@ These starters run the agent; they are not prerecorded responses. Jev chooses th
 
 ## Get started
 
-Prerequisites: Node.js 22+, pnpm 9+, Python 3.12+, and [uv](https://docs.astral.sh/uv/).
+Prerequisites: Node.js 22+, pnpm 9+, Python 3.12+, [uv](https://docs.astral.sh/uv/), and [Ollama](https://ollama.com) signed in with `ollama signin` (needed for `:cloud` models).
 
 ```bash
 git clone https://github.com/CopilotKit/OpenIntelligentUI.git
 cd OpenIntelligentUI
 make setup
-# Optional: set shared OPENAI_API_KEY and TYPESAFE_API_KEY in apps/agent/.env
-# Or use API keys in the chat header after starting the app
+ollama pull clef-flash:latest   # local visualization router
 make dev
 ```
 
 Open the [app](http://localhost:3000). The [agent health endpoint](http://localhost:8123/health) confirms the agent service is running. See [Getting started](docs/getting-started.md) for configuration and verification.
 
-The default model is `chat-latest`, OpenAI’s documented alias for the latest ChatGPT Instant model, requiring `OPENAI_API_KEY`. Jev (`jev-latest`, requiring `TYPESAFE_API_KEY`) selects the renderer and visualization for each user turn. Basic tables use A2UI; charts, diagrams, calculators, and maps use Open Generative UI. Provider failures are surfaced instead of silently substituting another router or model. See [Visualization routing](docs/visualization-routing.md).
+The default answer model is `beam:cloud`, served through your local Ollama daemon's OpenAI-compatible API. The visualization router defaults to `clef-flash:latest`, which answers Jev's System One API in the same daemon. Neither needs an API key. To use hosted providers instead, see [Getting started](docs/getting-started.md#configure-the-agent). The router selects the renderer and visualization for each user turn. Basic tables use A2UI; charts, diagrams, calculators, and maps use Open Generative UI. Provider failures are surfaced instead of silently substituting another router or model. See [Visualization routing](docs/visualization-routing.md).
 
 ## Use your own API keys
 
-The chat asks for both keys before sending your first message or example prompt and keeps your draft while you set them up. Open **API keys** in the chat header, enter an OpenAI key and a Jev key, then choose **Test connection** and **Save keys**. The connection test makes a small request to each provider. Your keys use `chat-latest` for answers and `jev-latest` for visualization routing, with usage billed to your provider accounts.
+With the default local models the chat never asks for keys. If the server's own models are not configured, the chat asks for both keys before sending your first message or example prompt and keeps your draft while you set them up. Open **API keys** in the chat header, enter an OpenAI key and a Jev key, then choose **Test connection** and **Save keys**. The connection test makes a small request to each provider. Your keys use `chat-latest` for answers and `jev-latest` for visualization routing, with usage billed to your provider accounts.
 
 Keys stay in browser memory until refresh or **Clear keys**; the app does not save them in browser storage. Requests send them through the application server to their respective providers. They are kept out of chat state and checkpoints, and hosted LangSmith tracing is disabled for requests using your keys. Only use this feature on a deployment whose operator you trust. Saving or clearing keys starts a new chat; **New chat** keeps your keys for the current session.
 
-The agent can run without shared provider keys. Visitors must then supply both keys before chatting. Configured server credentials remain available to direct API clients that omit visitor keys; the chat interface requires visitors to add their own keys.
+The chat checks `GET /credentials/status` on the agent. It asks visitors for keys only when the server's own answer model or router is unconfigured, for example a hosted `LLM_MODEL` without its key. Visitor keys always use OpenAI `chat-latest` and hosted Jev, never the server's Ollama models.
 
 ## How it works
 

apps/agent/main.py

@@ -20,6 +20,7 @@
     CredentialsMiddleware,
     CredentialScopedAgent,
     current_credentials,
+    server_credentials_available,
     validate_credentials,
 )
 from src.visualization_router import JevVisualizationMiddleware
@@ -61,6 +62,12 @@ async def validate_provider_credentials():
     return await validate_credentials(credentials)
 
 
+@app.get("/credentials/status")
+def provider_credentials_status():
+    # Lets the chat skip the API-key prompt when the server's models are configured.
+    return {"required": not server_credentials_available()}
+
+
 @app.get("/health")
 def health():
     return {"status": "ok"}

apps/agent/src/credentials.py

@@ -46,9 +46,12 @@ def parse_credentials(headers):
 
 
 def server_credentials_available():
-    model = os.environ.get('LLM_MODEL', 'chat-latest').strip()
-    key = 'ANTHROPIC_API_KEY' if model.startswith('claude-') else 'OPENAI_API_KEY'
-    return bool(os.environ.get(key, '').strip() and os.environ.get('TYPESAFE_API_KEY', '').strip())
+    """Whether the server's own models can answer without visitor keys."""
+    from src.model import DEFAULT_MODEL, provider_key_name
+    from src.visualization_router import router_key_required
+    model_key = provider_key_name(os.environ.get('LLM_MODEL', DEFAULT_MODEL).strip())
+    router_key = 'TYPESAFE_API_KEY' if router_key_required() else None
+    return all(os.environ.get(key, '').strip() for key in (model_key, router_key) if key)
 
 
 class CredentialsMiddleware:

apps/agent/src/model.py

@@ -1,5 +1,6 @@
 """Chat model factory for the agent."""
 
+import json
 import os
 from contextlib import contextmanager
 
@@ -10,7 +11,11 @@
 from langchain_anthropic import ChatAnthropic
 from langchain_core.language_models.chat_models import BaseChatModel
 
-DEFAULT_MODEL = "chat-latest"
+# The server default runs through a local Ollama daemon (cloud models need
+# `ollama signin`). Visitor BYOK keys always use OpenAI's chat-latest instead.
+DEFAULT_MODEL = "beam:cloud"
+BYOK_MODEL = "chat-latest"
+DEFAULT_OLLAMA_URL = "http://localhost:11434/v1"
 
 # langchain-anthropic defaults max_tokens to 4096, which truncates
 # generateSandboxedUi tool args mid-stream (jsFunctions/jsExpressions never
@@ -19,26 +24,53 @@
 MAX_TOKENS = 64000
 
 
+def provider_key_name(model_name: str) -> str | None:
+    """Name the API key a model needs; None means an Ollama model or an invalid name."""
+    if is_ollama_model(model_name):
+        # Checked first: Ollama tags such as gpt-oss:20b-cloud are not OpenAI models.
+        return None
+    if model_name == "chat-latest" or model_name.startswith("gpt-"):
+        return "OPENAI_API_KEY"
+    if model_name.startswith("claude-"):
+        return "ANTHROPIC_API_KEY"
+    return None
+
+
+def is_ollama_model(model_name: str) -> bool:
+    # Ollama names carry an explicit tag (beam:cloud); this keeps typos
+    # like "gemini-example" failing at startup instead of at the first request.
+    name, _, tag = model_name.partition(":")
+    return bool(name and tag) and not any(c.isspace() for c in model_name)
+
+
 def build_model(*, allow_unconfigured: bool = False) -> BaseChatModel:
     model_name = os.environ.get("LLM_MODEL", DEFAULT_MODEL).strip()
-    is_openai = model_name == "chat-latest" or model_name.startswith("gpt-")
-    if not is_openai and not model_name.startswith("claude-"):
+    key_name = provider_key_name(model_name)
+    if key_name is None and not is_ollama_model(model_name):
         raise ValueError(
-            "LLM_MODEL must name a supported provider: claude-* (Anthropic) "
-            "or gpt-*/chat-latest (OpenAI). Unset LLM_MODEL to use the default."
+            "LLM_MODEL must name a supported provider: an Ollama model with a tag "
+            "(e.g. beam:cloud), claude-* (Anthropic) or gpt-*/chat-latest "
+            "(OpenAI). Unset LLM_MODEL to use the default."
+        )
+    from langchain_openai import ChatOpenAI
+
+    if key_name is None:
+        # Ollama serves an OpenAI-compatible API; a local daemon ignores the key.
+        return ChatOpenAI(
+            model=model_name,
+            base_url=os.environ.get("OLLAMA_BASE_URL", "").strip() or DEFAULT_OLLAMA_URL,
+            api_key=os.environ.get("OLLAMA_API_KEY", "").strip() or "ollama",
+            max_tokens=MAX_TOKENS,
         )
-    key_name = "OPENAI_API_KEY" if is_openai else "ANTHROPIC_API_KEY"
     if not os.environ.get(key_name, "").strip():
         if allow_unconfigured:
             return UnconfiguredModel()
         raise ValueError(
             f"{key_name} is required for the selected LLM_MODEL provider. "
             "Set it in the agent environment and restart the agent."
         )
-    if is_openai:
+    if key_name == "OPENAI_API_KEY":
         # chat-latest follows ChatGPT Instant; explicit gpt-* pins also work.
-        from langchain_openai import ChatOpenAI
-
         return ChatOpenAI(model=model_name)
     return ChatAnthropic(
         model=model_name,
@@ -91,7 +123,7 @@ def request_model(fallback):
         # Clear that merge before any request so host Authorization cannot win.
         client._custom_headers = {}
     model = ChatOpenAI(
-        model=DEFAULT_MODEL,
+        model=BYOK_MODEL,
         api_key=credentials.openai,
         base_url=options["base_url"],
         openai_proxy="",
@@ -163,17 +195,84 @@ async def _agenerate(self, messages, stop=None, run_manager=None, **kwargs):
     def _stream(self, messages, stop=None, run_manager=None, **kwargs):
         with provider_error_boundary():
             model, options = self._target(kwargs)
-            for chunk in model.stream(messages, stop=stop, **options):
-                yield ChatGenerationChunk(message=as_chunk(chunk))
+            for chunk in model.stream(messages, stop=stop, config=INNER_STREAM, **options):
+                for part in split_tool_call_chunk(as_chunk(chunk)):
+                    yield ChatGenerationChunk(message=part)
 
     async def _astream(self, messages, stop=None, run_manager=None, **kwargs):
         with provider_error_boundary():
             model, options = self._target(kwargs)
-            async for chunk in model.astream(messages, stop=stop, **options):
-                yield ChatGenerationChunk(message=as_chunk(chunk))
+            async for chunk in model.astream(messages, stop=stop, config=INNER_STREAM, **options):
+                for part in split_tool_call_chunk(as_chunk(chunk)):
+                    yield ChatGenerationChunk(message=part)
+
+
+# The wrapped model's own stream events would reach AG-UI alongside this
+# wrapper's, emitting every tool call twice; only the wrapper's are relayed.
+INNER_STREAM = {"metadata": {"emit-messages": False, "emit-tool-calls": False}}
 
 
 def as_chunk(message):
     if isinstance(message, AIMessageChunk):
         return message
     return AIMessageChunk(**message.model_dump(exclude={"type"}))
+
+
+def split_tool_call_chunk(chunk):
+    """Send a tool call's name before its arguments, as OpenAI streams them.
+
+    Ollama delivers each tool call whole in one chunk, but AG-UI starts the
+    call on its named chunk and drops any arguments that chunk carries.
+    """
+    calls = chunk.tool_call_chunks
+    if len(calls) != 1 or not calls[0].get("name") or not calls[0].get("args"):
+        return [chunk]
+    call = calls[0]
+    head = AIMessageChunk(
+        content=chunk.content,
+        id=chunk.id,
+        additional_kwargs=chunk.additional_kwargs,
+        response_metadata=chunk.response_metadata,
+        usage_metadata=chunk.usage_metadata,
+        tool_call_chunks=[{**call, "args": ""}],
+    )
+    args = AIMessageChunk(
+        content="",
+        id=chunk.id,
+        tool_call_chunks=[
+            {**call, "name": None, "id": None, "args": widget_args(call["name"], call["args"])}
+        ],
+    )
+    return [head, args]
+
+
+# CopilotKit's OpenGenerativeUIMiddleware creates the widget when it reads
+# initialHeight and drops any css/html patches that arrive before it.
+WIDGET_TOOL = "generateSandboxedUi"
+DEFAULT_WIDGET_HEIGHT = 200
+WIDGET_ARG_ORDER = ("initialHeight", "placeholderMessages", "css", "html", "jsFunctions", "jsExpressions")
+
+
+def widget_args(name, args):
+    """Put a whole widget call's arguments in the order the middleware needs.
+
+    Ollama re-serializes arguments with its own key order, and smaller models
+    sometimes send jsExpressions as one string instead of a list.
+    """
+    if name != WIDGET_TOOL:
+        return args
+    try:
+        parsed = json.loads(args)
+    except ValueError:
+        return args
+    if not isinstance(parsed, dict):
+        return args
+    if isinstance(parsed.get("jsExpressions"), str):
+        parsed["jsExpressions"] = [parsed["jsExpressions"]]
+    if not isinstance(parsed.get("initialHeight"), (int, float)):
+        # Without it the middleware never creates the widget; the renderer
+        # uses the same default and autosizes from there.
+        parsed["initialHeight"] = DEFAULT_WIDGET_HEIGHT
+    ordered = {key: parsed[key] for key in WIDGET_ARG_ORDER if key in parsed}
+    ordered.update((key, value) for key, value in parsed.items() if key not in ordered)
+    return json.dumps(ordered)

apps/agent/src/visualization_router.py

@@ -10,8 +10,16 @@
 from langchain.agents.middleware import AgentMiddleware
 from langchain_core.messages import SystemMessage
 from pydantic import BaseModel, Field, ValidationError
+from urllib.parse import urlparse
 
+# Hosted Jev. Visitor BYOK keys are TypeSafe keys, so they always route here.
 JEV_URL = "https://api.typesafe.ai/v1/systemone"
+# Server default: clef-flash answers the same System One API in local Ollama.
+DEFAULT_ROUTER_URL = "http://localhost:11434/v1/systemone"
+DEFAULT_ROUTER_MODEL = "clef-flash:latest"
+HOSTED_TIMEOUT = 15
+# A local router loads its model on the first request, which takes 10-12s cold.
+LOCAL_TIMEOUT = 60
 VISUALIZATIONS = {
     "text": "Direct facts, prose, writing, code, conversation, or a clarification; no visual needed.",
     "table": "Basic table: exact values, records, simple side-by-side comparison; no custom interaction.",
@@ -113,19 +121,47 @@ def routing_input(state):
     return turn, context
 
 
+def server_router_url():
+    return os.environ.get("JEV_URL", "").strip() or DEFAULT_ROUTER_URL
+
+
+def router_key_required():
+    """Only hosted TypeSafe needs a key; a self-hosted System One server does not."""
+    host = urlparse(server_router_url()).hostname or ""
+    return host == "typesafe.ai" or host.endswith(".typesafe.ai")
+
+
+def request_url():
+    from src.credentials import current_credentials
+
+    return JEV_URL if current_credentials.get() else server_router_url()
+
+
+def request_timeout():
+    from src.credentials import current_credentials
+
+    if current_credentials.get() or router_key_required():
+        return HOSTED_TIMEOUT
+    return LOCAL_TIMEOUT
+
+
 def request_options(context):
     from src.credentials import current_credentials
 
     credentials = current_credentials.get()
     key = credentials.jev if credentials else os.environ.get("TYPESAFE_API_KEY", "").strip()
-    if not key:
+    if not key and router_key_required():
         raise ValueError(
             "TYPESAFE_API_KEY is required for Jev visualization routing. Configure it and restart the agent."
         )
+    if credentials:
+        model = "jev-latest"
+    else:
+        model = os.environ.get("JEV_MODEL", "").strip() or DEFAULT_ROUTER_MODEL
     return {
-        "headers": {"Authorization": f"Bearer {key}"},
+        "headers": {"Authorization": f"Bearer {key}"} if key else {},
         "json": {
-            "model": "jev-latest" if credentials else os.environ.get("JEV_MODEL", "jev-latest"),
+            "model": model,
             "state": {"conversation": context},
             "questions": {
                 "visualization": {
@@ -158,8 +194,8 @@ def provider_trust_env():
 
 def route(context):
     try:
-        with httpx.Client(timeout=15, trust_env=provider_trust_env()) as client:
-            return parse_response(client.post(JEV_URL, **request_options(context)))
+        with httpx.Client(timeout=request_timeout(), trust_env=provider_trust_env()) as client:
+            return parse_response(client.post(request_url(), **request_options(context)))
     except httpx.HTTPError:
         raise ValueError(
             "Jev visualization routing is unavailable or timed out. Please retry."
@@ -168,9 +204,9 @@ def route(context):
 
 async def aroute(context):
     try:
-        async with httpx.AsyncClient(timeout=15, trust_env=provider_trust_env()) as client:
+        async with httpx.AsyncClient(timeout=request_timeout(), trust_env=provider_trust_env()) as client:
             return parse_response(
-                await client.post(JEV_URL, **request_options(context))
+                await client.post(request_url(), **request_options(context))
             )
     except httpx.HTTPError:
         raise ValueError(