Open providers over closed capabilities: the provider contract and plugin architecture

[JArmandoAnaya]

Why this shape

The standing invariant — zero orphaned capabilities, none declared without a surface that
renders it — is what closes the capability vocabulary. If plugins could add capabilities, every
installed plugin would create orphans by definition. Closing capabilities is also what lets a
single generic dashboard — the Inference screen organized by what a connection enables (#499) —
render any provider without plugins shipping frontend code.

So: closed capabilities, open providers. ModelCapability
(src/visionset/kernel/domain/inference.py) grows only by a deliberate kernel change shipped
alongside the surface that consumes it. What a plugin adds is a driver for a capability the
application already defines.

The provider contract (to be designed here)

A provider must be able to:

• identify itself and the contract version it targets;
• declare the families it serves and its curated entries (pinned revisions, sizes);
• price a download for an exact snapshot;
• fetch weights into the content-addressed cache;
• run predictions over domain types for the capabilities it maps to;
• refuse what it cannot do, in prose, under the unknown-type refusal rule established when
  sam2_video was recognised as a point-prompt model and an unknown type made to refuse (fix(inference): sam2_video is a point-prompt model, and an unknown type refuses #458):
  never guess, and name what is supported.

Capability mapping is derived from declared families through the same mechanism
src/visionset/inference/families.py uses today: CAPABILITY_BY_FAMILY is built from
SEGMENTER_FAMILIES and DETECTOR_FAMILIES with dict.fromkeys, never written out a second
time, so adding a family and declaring its capability are one edit. A provider never hand-writes
its capability map; the change that made connections declare what their model can be asked for
(#486) proves that derivation with a mutation test.

Layering

ModelProvider (src/visionset/kernel/ports/model_provider.py) stays a pure kernel port.
Provider authors get a toolkit layer above it — working name visionset.inference.sdk — offering
the kernel's device and precision rules (precisions_for), cache access
(src/visionset/inference/cache.py), the digest discipline in
src/visionset/inference/integrity.py that came with the weights integrity check (#475),
integration with the download job that reports its bytes (#493), and refusal prose helpers.

The layer carries an explicit contract version. The contract is experimental until
declared stable in a later release: everything is free before first publication and breaking
after. That is the same lesson taught by the suggest route's detail field, which was accepted,
published and then ignored (#463) — applied deliberately this time.

Discovery

An entry-point group, mirroring the exporter precedent: pyproject.toml already registers nine
exporters under [project.entry-points."visionset.formats"].

The group name is an open naming decision (candidate: visionset.providers) — flagged
deferred-needs-input on this issue, to be decided before the first implementation PR that
registers one.

Degradation and trust

A connection whose provider is not installed does not break and is not deleted: it renders
with prose naming the missing provider and coherent allowed_actions — the missing-format
treatment. Documentation states plainly that installing a plugin is trusting its author with code
execution in the workers, exactly as any pip package.

0.1.0 scope vs. later

In 0.1.0: the contract exists internally; both shipped adapters —
src/visionset/inference/sam_provider.py (point-prompted, SAM 2 family) and
src/visionset/inference/transformers_provider.py (text-prompted, Grounding DINO family),
resolved by provider_for in src/visionset/inference/providers.py — are migrated to consume it;
the discovery mechanism works for in-tree providers; nothing is documented as a public API.

Later (its own release, its own decision): contract declared stable, public docs, external
plugin support announced.

The contract is not published as a public extension API in 0.1.0, but the shipped local adapters
must run on it — the contract is proven by our own drivers before any publication. Plugins are
backend driver + declarative metadata only; plugin-supplied UI is out of scope. Nothing here
touches the http connection type or precludes hosted execution.

Sub-decisions this epic must produce

Each recorded on this issue before its slice:

1. Contract version scheme.
2. Entry-point group name (deferred-needs-input).
3. Exact toolkit surface.
4. How a provider's curated entries merge into the create-form dropdown, meaning
   frontend/ui-core/src/screens/inferenceCatalog.ts and the grouped model select that shipped
   with the curated model list and download lifecycle (feat(inference): curated model list, closed device and precision fields, download lifecycle #470).
5. The conformance test suite a provider must pass; the two shipped adapters are its first
   subjects.

[JArmandoAnaya]

Decision (Armando, 2026-08-10): Closed capabilities, open providers. Plugins may add
providers (drivers) for capabilities the application already defines; a plugin can
never introduce a capability. A new capability arrives only with a VisionSet release,
together with the surface that renders it. The provider architecture is part of
milestone 0.1.0: it is not published as a public extension API yet, but the shipped
local adapters must run on it, so the contract is proven by our own drivers before any
publication. Plugins are backend driver + declarative metadata only; plugin-supplied UI
is out of scope.

[JArmandoAnaya]

Designing this epic's discovery mechanism surfaced a contradiction in the wire's forward-compatibility rules: the generated client tolerates an unknown response key but refuses an unknown enum member. That is now tracked as its own sub-decision (#504), because an open provider set is exactly the case where a vocabulary grows without a coordinated release on both sides.