Migrate embedding generation to omop-llm, plus oa-configurator 1.0

[nicoloesch]

Depends on: omop-llm, must merge and publish first (this PR is omop-llm's first real consumer, not just an oa-configurator upgrade). Also depends on oa-configurator's new vector_stores domain (see that PR's "Vector-store configuration is new" section) landing first, since OmopEmbConfig migrates onto it below. Bumps pyproject.toml's oa-configurator pin (main + dev extra) and activates the currently-commented-out omop-llm dependency line, both marked # TODO in the diff. Also drops the openai dependency entirely: it was only ever used inside the now-deleted EmbeddingClient.

Summary

This PR is predominantly the omop-llm rewrite: replacing omop-emb's own hand-rolled embedding-model client with omop_llm.ModelBackend, the same shared contract the rest of the stack is converging on. The oa-configurator 1.0 RefTo migration is a smaller piece riding along at the end, since it's what makes the new embedding_model_name field resolvable in the first place.

omop-llm rewrite (the bulk of this PR)

• Deleted omop-emb's entire embedding-client layer, about 590 lines of model-calling logic this package no longer owns:
  • embeddings/embedding_client.py (EmbeddingClient, its own EmbeddingRole enum, batching, dimension auto-discovery, cosine-similarity helpers)
  • embeddings/embedding_providers.py (EmbeddingProvider/OllamaProvider/OpenAIProvider, the package's own provider abstraction)
  • embeddings/__init__.py
  • omop_llm.ModelBackend does the actual model calling now. omop_llm.EmbeddingRole (re-exported from omop_emb.__init__) replaces the local enum.
• EmbeddingWriterInterface now builds and owns a ModelBackend directly at construction time via build_model_backend_from_resolved(resolved_model), where resolved_model is an oa_configurator.ResolvedModel (from Resolver.resolve_model(cfg.embedding_model_name)):
  • Provider, connection, embedding_dim, and document_prefix/query_prefix all come from that resolved model, not from omop-emb's own config fields.
  • There's no separate client object between the interface and the model backend anymore.
• Role-based text prefixing (document vs. query) moves from omop-emb's own document_embedding_prefix/query_embedding_prefix config fields into ModelBackend.embed_texts(role=...):
  • omop_emb.interface.EmbeddingReaderInterface.generate_embeddings (new: a static method, callable without an interface instance) just forwards role and validates the returned shape. It doesn't apply the prefix itself anymore.
• EmbeddingWriterInterface.embedding_dim now resolves lazily via ModelBackend.dimensions()'s three-tier lookup (configured override, then provider fast path, then live probe) instead of the old embedding_dim config-field hint plus EmbeddingClient's own discovery logic.
• CLI (cli_embeddings.py): add_embeddings/create_index's --api-base/--api-key/--provider/--model flags are gone, replaced by a single --model-name/-m flag naming a [models.*] entry (defaults to cfg.embedding_model_name), resolved via a new _resolve_model() helper that turns an unknown model/provider name into an actionable omop-config models add .../providers add ... message instead of a raw KeyError.
• Deleted cli/cli_legacy.py entirely (285 lines). The whole "legacy commands for backward compatibility" subcommand group existed only to import pre-built embeddings using the old ProviderType-based flow; cli_app.py no longer registers it as a subcommand.
• Test suite follows the same shape:
  • deleted: test_embedding_client.py (649 lines), test_ollama_provider_api.py (124), test_providers.py (133)
  • added: test_embedding_generation.py (219 lines), covering the omop_llm-backed path instead
• Docs rewritten throughout to describe the omop_llm-based flow instead of the old client: docs/usage/interface-guide.md, docs/usage/asymmetric-embeddings.md, docs/usage/cli.md, docs/usage/configuration.md, docs/usage/installation.md, docs/usage/backend-selection.md, docs/index.md, README.md.

oa-configurator 1.0 migration (smaller, rides along)

• Migrate OmopEmbConfig off ResourceSpec/ModelFieldSpec/ResourceRef/owned_resources/required_resources/test_resources/referenced_models onto RefTo-marked fields:
  • cdm_db (required)
  • emb_db (deliberately optional, str | None defaulting to None, since it's a pgvector-only requirement; sqlitevec doesn't need a database entry)
  • test_emb_db (optional, RefTo(DatabaseConfig, is_test=True))
  • embedding_model_name (RefTo(ModelConfig), unchanged field name, now properly typed and what the omop-llm rewrite above resolves through)
  • the old, disconnected TEST_DB ClassVar is removed entirely
• Dropped the now-unneeded from omop_alchemy.config import OmopAlchemyConfig import. Cross-package database sharing is naming-convention only under the new model, no typed import needed.
• Cross-package resource resolution now goes through the package's own config rather than a hardcoded import:
  • resolve_omop_cdm_engine()/resolve_omop_emb_engine() now resolve via OmopEmbConfig.get_config().cdm_db/.emb_db instead of OmopAlchemyConfig.CDM_DB.semantic_name/OmopEmbConfig.EMB_DB.semantic_name.
  • Since emb_db is optional, resolve_omop_emb_engine() checks backend first (raising a clear RuntimeError if it's ever called outside the pgvector backend) and then raises if emb_db is unset, rather than passing None through to get_engine().
• tests/conftest.py:
  • resolve_test_resource(OmopEmbConfig.TEST_DB) → resolve_test_database(OmopEmbConfig, "test_emb_db")
  • 3 test files: @pytest.mark.requires_resource → @pytest.mark.requires_database("test_emb_db") (5 call sites across test_cli_pgvector_snapshot.py, test_pgvector_index_manager.py, test_pgvector.py). Same plain-string trade-off as orm-loader's marker fix; see oa-configurator's own PR, "Test-database wiring," for the full reasoning.
  • The now-unused OmopEmbConfig import was dropped from all three of those files.
• pyproject.toml: oa-configurator pins reworded to >=1.0.0,<2.0.0; the already-commented-out omop-llm dependency line's syntax fixed ("omop-llm">=0.2, was invalid TOML if ever uncommented) and its TODO comment clarified to name the eventual target range; openai dropped.

Relationship to open PRs and issues

Config resolution boundary

• Absorbs [v2.0] Stop reading OmopEmbConfig deep inside EmbeddingClient and resolve_backend #50 rather than merged standalone. Its desing is adopted here, extended to use the new vector_stores domain instead of OmopEmbConfig fields directly.

  • Closes Refactor: remove config reads from property getters and factory functions #45

• OmopEmbConfig drops its flat backend/sqlite_path/emb_db/faiss_cache_dir fields entirely, replaced by a single vector_store_name: Annotated[str, RefTo(VectorStoreConfig)] = "vector_store".

• resolve_backend() in backends/base_backend.py becomes a pure function: resolve_backend(backend_type: str | BackendType, *, database: ResolvedDatabase) -> EmbeddingBackend.

  • No internal config read, no sqlite_path string. Every backend, including an in-memory sqlite-vec store, is backed by a real database: dialect and (for sqlitevec) the filesystem path are both read off database.connection.url via sqlalchemy.engine.make_url, not passed as a separate parameter. A sqlitevec store pointed at a non-sqlite connection, or a pgvector store pointed at a non-postgres one, now fails loudly with the actual dialect named, instead of silently misbehaving.

• New resolve_backend_from_resolved(resolved: ResolvedVectorStore) -> EmbeddingBackend is the one construction entry point for a resolved vector store, mirroring omop_llm.build_model_backend_from_resolved(resolved: ResolvedModel) exactly.

  • Takes oa-configurator's plain resolved data, dispatches to SQLiteVecEmbeddingBackend/PGVectorEmbeddingBackend as before.

• All CLI call sites updated: Resolver.from_active_config().resolve_vector_store(cfg.vector_store_name) → resolve_backend_from_resolved(resolved).

• resolve_omop_emb_engine() (the pgvector database engine, previously keyed off cfg.emb_db) now resolves through resolved.database on the same ResolvedVectorStore. resolve_omop_emb_engine() itself was later deleted outright: git log -S confirmed its one historical caller was removed when resolve_backend() became a pure function, and it had zero callers left anywhere in the codebase.

• search()'s (cli_embeddings.py) --faiss-cache-dir fallback reads resolved_vector_store.faiss_cache_dir directly, now that oa-configurator's VectorStoreConfig/ResolvedVectorStore carry it as a dedicated field rather than a configuration["faiss_cache_dir"] dict key.

Concept-similarity queries

• Absorbs Add joint/centroid embedding and concept-to-concept similarity queries #40 rather than merged standalone: its branch predates this session's interface.py rework and no longer merges cleanly. The feature itself is re-ported by hand onto the current interface, not applied as a patch.

  • Closes Add joint/centroid embedding query to EmbeddingReaderInterface #31
  • Closes Add single-call concept-to-concept similarity query #32

• EmbeddingReaderInterface (so EmbeddingWriterInterface inherits it too) gains two new methods, both read-only over already-stored embeddings, no ModelBackend/generation involved:

  • get_joint_embedding(concept_ids, weights=None) -> np.ndarray: the centroid (unweighted or weighted mean) of several concepts' stored embeddings, e.g. for querying by a combination of conditions. Explicitly not L2-normalized even under MetricType.COSINE, since backends compute cosine distance directly off raw stored vectors, not off this method's output. Raises on empty concept_ids, mismatched weights length, or weights summing to zero (a real bug in the original branch, fixed there and preserved here, with a new dedicated test since the original never actually had one).
  • get_similar_concepts(concept_ids, k=None, *, concept_filter=None, faiss_index_config=None) -> Tuple[Tuple[NearestConceptMatch, ...], ...]: neighbours of concepts that are already embedded, without the caller fetching and passing a raw vector themselves. Accepts a bare int or a sequence; one result row per input, self-excluded, in input order. Over-fetches by one neighbour internally so k real results remain even when the self-match isn't guaranteed present.

• EmbeddingConceptFilter already matches the reverted, single-class design, and get_similar_concepts resolves its effective k the same simple way get_nearest_concepts already does -> Deprecation behaviour outlined in Add joint/centroid embedding and concept-to-concept similarity queries #40 is removed.

• Docstring note that EmbeddingConceptFilter's field shape duplicates omop_graph.graph.constraints.SearchConstraintConcept (since omop_emb can't import omop_graph), tracked in OMOP_Alchemy#11, folded into EmbeddingConceptFilter's existing docstring.

• docs/usage/interface-guide.md: two new subsections, "Query similar concepts" and "Combine embeddings (joint/centroid queries)", ported from the branch. Note that get_joint_embedding returns a 1-D (D,) vector but get_nearest_concepts expects (Q, D), so callers reshape it themselves (joint_vec[None, :])

Loosen versioning constraints

• Absorbs Widen omop-alchemy and oa-configurator pins to semver ranges #52