Skip to content

Rebuild omop-llm on any-llm, with oa-configurator 1.0 compatibility - #2

Draft
nicoloesch wants to merge 15 commits into
mainfrom
feat/anyllm
Draft

Rebuild omop-llm on any-llm, with oa-configurator 1.0 compatibility#2
nicoloesch wants to merge 15 commits into
mainfrom
feat/anyllm

Conversation

@nicoloesch

Copy link
Copy Markdown
Collaborator

Depends on: oa-configurator. The oa-configurator compatibility slice of this PR bumps pyproject.toml's upper bound (<1.0.0 today) once 1.0 is published.

Summary

This PR is predominantly the any-llm rewrite. The means rebuilding omop-llm as the shared chat/embedding contract for the whole OMOP stack, replacing five repos' worth of independently hand-rolled provider backends (Ollama-specific clients, OpenAI-compat wrappers, Anthropic wrappers, a bespoke ModelBackend ABC in groundcrew, native /api/chat in cava-nlp-shard) with one interface.

oa-configurator 1.0 is required to configure LLMs and store them in the configuration. This is essentially functionality for this module.

any-llm rewrite

  • ModelBackend (backend.py): one sync+async class wrapping Mozilla's any-llm SDK
    • complete/async_complete, embed_texts/async_embed_texts, extract/async_extract (native structured output, no instructor dependency required), dimensions/async_dimensions, is_available/async_is_available.
  • Six supported providers:
    • ollama, llamacpp, vllm (local/self-hosted) and openai, anthropic, gemini (cloud).
    • Can be natively extended if others are required. any-llm supports roughly 50 with various different capabilities.
  • ModelCapabilities (capabilities.py):
    • per-model capability flags
      • streaming, embeddings, extended_thinking read from any-llm's own metadata;
      • tool_use/structured_output declared by omop_llm itself, since any-llm has no equivalent
  • omop_llm.embeddings:
    • EmbeddingRole (DOCUMENT/QUERY) + prefix application for asymmetric embedding models (nomic-embed-text, E5, BGE, ...)
      • absorbed the prefix logic from omop-emb
    • KNOWN_EMBEDDING_PREFIXES flags an unrecognized configured prefix to prevent model prefix injection
  • providers.canonical_model_name() normalizes a model name across providers, and rejects Ollama's mutable :latest tag
  • structured.py:
    • an explicitly-scoped, optional instructor-based fallback for structured extraction, kept out of backend.py so importing omop_llm never requires the optional instructor dependency.
    • Checked directly against instructor's own provider-builder source rather than assumed:
      • ollama is unsafe to route through it (instructor's Ollama builder uses the OpenAI-compat shim, not native /api/chat)
      • llamacpp/vllm have no dedicated builder there;
      • anthropic/gemini aren't offered here either. Not wired in as a silent alternative for every provider.
  • Typed error hierarchy:
    • OmopLlmError, UnsupportedProviderError, UnsupportedCapabilityError, NoParsedOutputError.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking Incompatible API change. MAJOR: x+1.y.z

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant