Skip to content

feat(llm): per-node local/edge VLM endpoints + JSONC URL fix + provider-surface tests - #132

Open
RanjithRagavan wants to merge 6 commits into
google:mainfrom
RanjithRagavan:upstream-on-device-endpoint-config
Open

RanjithRagavan wants to merge 6 commits into
google:mainfrom
RanjithRagavan:upstream-on-device-endpoint-config

Conversation

@RanjithRagavan

Copy link
Copy Markdown

Summary

Unblocks the On-Device Lightweight VLMs roadmap item by making local/edge OpenAI-compatible endpoints (Ollama, llama.cpp-server, vLLM, LM Studio) configurable per agent node — today only the global OPENAI_BASE_URL env var works, forcing every node onto one endpoint.

Closes #131

Relation to #128: complementary, not overlapping. #128 adds admin-UI/env credential plumbing (CUSTOM_LLM_API_KEY, Settings.get_api_key) and validate_provider branches. This PR adds the per-node config→endpoint passthrough in _resolve_endpoint(), repairs the local-ollama preset, fixes a latent JSONC parser bug, and adds provider-surface tests. Happy to rebase whichever lands second.

Changes

Endpoint plumbing

  • artemis/config/llm.py: LLM gains optional api_base, api_key, max_tokens, timeout_seconds, is_multimodal (types match ModelEndpoint); validate_provider() no longer requires cloud keys for ollama / vllm / custom
  • artemis/services/llm.py: _resolve_endpoint() forwards all five fields into ModelEndpoint, for primary and fallback nodes

Preset + parser fixes

  • local-ollama preset (both config copies): provider: "ollama", model: "qwen2.5vl:7b", api_base: "http://localhost:11434/v1" — boots with zero cloud keys
  • artemis/utils/file.py: strip_json_comments() rewritten as a string-aware scanner — the old regex stripped // inside quoted strings, corrupting any config containing a URL value (blocking bug for any config-level endpoint fix)
  • .env.example documents local-endpoint variables

Tests

  • New tests/unit/test_llm_router.py: 22 tests — provider alias parsing, mocked ChatOpenAI construction for OLLAMA/VLLM/CUSTOM, _resolve_endpoint passthrough/defaults/fallback, local-provider validation, JSONC string safety, preset regression

Docs

  • docs/design/on-device-vlm.md: design doc for the on-device VLM roadmap item (deployment shapes, endpoint schema with hybrid local-flash + cloud-pro example, perception-pipeline next steps, privacy model, latency measurement plan); roadmap bullets in both READMEs link to it

Drive-by CI fix

  • playground/backend_manager: auto-fixed 11 pre-existing ruff 0.16.3 lint errors (datetime.UTC alias style) that fail the Lint Python step under the ruff version CI resolves today — verified failing on a clean checkout of main

Test plan

  • pytest tests/unit/test_llm_router.py tests/unit/test_llm_reliability.py tests/unit/test_llm_structured.py tests/unit/test_llm_gateway.py44 passed
  • Full pytest tests/unit — failure set identical before/after this branch (pre-existing, environment-related e.g. missing GOOGLE_API_KEY)
  • ruff check + ruff format --check clean on the whole tree
  • LLM(provider="ollama", model="qwen2.5vl:7b", api_base="http://localhost:11434/v1") validates with no cloud keys set

Example: hybrid on-device setup now possible

"default": { "provider": "google", "model": "gemini-2.5-pro" },
"nodes": {
  "explorer": {
    "provider": "ollama",
    "model": "qwen2.5vl:7b",
    "api_base": "http://localhost:11434/v1",
    "is_multimodal": true
  }
}

Add api_base, api_key, max_tokens, timeout_seconds and is_multimodal to
the LLM config schema (artemis/config/llm.py) and forward them in
_resolve_endpoint() (artemis/services/llm.py) for primary and fallback
nodes, so local OpenAI-compatible endpoints (Ollama/vLLM/custom) can be
configured per agent. LLM.validate_provider no longer requires cloud API
keys for ollama/vllm/custom providers.

Refs #1, refs #2
The local-ollama preset used provider 'openai' with no api_base, which
failed credential validation without OPENAI_API_KEY. Switch it to
provider 'ollama' with api_base http://localhost:11434/v1 and a local VLM
default (qwen2.5vl:7b), in both config/artemis.jsonc and the bundled
artemis/resources/config/artemis.jsonc.

That requires strip_json_comments (artemis/utils/file.py) to ignore '//'
and '/* */' inside string literals; the previous regex truncated URL
values such as 'http://localhost:11434/v1' and broke config loading.

Also document local-endpoint env fallbacks in .env.example.

Refs #2
@google-cla

google-cla Bot commented Sep 18, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Cover ModelProvider.from_string alias parsing (incl. ollama/vllm/custom),
ModelFactory.create_model construction for local endpoints with
langchain_openai.ChatOpenAI mocked (no network), _resolve_endpoint
passthrough of api_base/api_key/max_tokens/timeout_seconds/is_multimodal
for primary and fallback nodes, local-provider credential validation,
string-safe JSONC comment stripping, and the local-ollama presets.

Refs #6
Add docs/design/on-device-vlm.md covering deployment shapes (host-side
Ollama/llama.cpp/vLLM, LAN edge box, future on-SoC), the new endpoint
config schema with a hybrid local-flash + cloud-pro JSONC example,
perception pipeline next steps (#3/#4/#5), the privacy model, and a
latency measurement plan. Link it from the roadmap bullets in README.md
and README_CN.md.

Refs #7
Pre-existing on main: 11 lint errors (datetime.UTC alias style) fail the
Lint Python step under the ruff version CI resolves today. Applied
ruff check --fix + format so the pipeline goes green again.
@RanjithRagavan
RanjithRagavan force-pushed the upstream-on-device-endpoint-config branch from f72781b to 1c9be8b Compare September 19, 2026 02:06
@RanjithRagavan

Copy link
Copy Markdown
Author

@google-cla I signed it!

1 similar comment
@RanjithRagavan

Copy link
Copy Markdown
Author

@google-cla I signed it!

@RanjithRagavan
RanjithRagavan force-pushed the upstream-on-device-endpoint-config branch from 1c9be8b to e052219 Compare September 19, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-node local/edge LLM endpoints not configurable; local-ollama preset broken; JSONC parser corrupts URL strings

1 participant