test(inference): a CI job runs the inference suite with the runtime installed

.github/workflows/ci.yml

@@ -473,6 +473,80 @@ jobs:
           cd /tmp
           /tmp/flow-venv/bin/python "$GITHUB_WORKSPACE/examples/thirty_minute_flow.py" /tmp/flow
 
+  # The other half of the inference matrix, on the `format-smoke` mould below: a
+  # job of its own because the `local-inference` extra is roughly two gigabytes of
+  # CUDA wheels, which is a cost nobody in the `python` job is paying for.
+  #
+  # That job installs the base distribution *deliberately* and is untouched by
+  # this one. It is the half that proves a machine without the runtime imports
+  # none of it and refuses with the install command rather than an `ImportError`.
+  # What it cannot do is exercise the code that only exists when the runtime is
+  # present — and until this job, nothing did: `tests/architecture/
+  # test_optional_runtime.py` asserts that importing the server, the CLI, the job
+  # registry and `visionset.inference` leaves torch, torchvision, transformers,
+  # accelerate and huggingface_hub out of `sys.modules`, which is true by
+  # construction on a machine where none of them is installed. A module-level
+  # `import torch` under `visionset/inference` — or a lock bump that broke the
+  # adapter, family resolution or the download path — would have passed green.
+  #
+  # `uv sync --locked --extra local-inference`: from the lockfile, never from a
+  # side index, for the reason the `python` job gives — a resolution happening
+  # here would be one nobody applied the cool-down to. The locked torch wheels
+  # carry CUDA and run perfectly well on a runner that has no device.
+  #
+  # It runs whole directories, unlike `format-smoke`, and the difference is the
+  # shadowing that job's comment describes: none of these five distributions ships
+  # a top-level `tests` package, so this repository's namespace one is intact and
+  # `tests.fixtures` imports normally here.
+  inference-smoke:
+    name: inference smoke (transformers, torch on cpu)
+    runs-on: ubuntu-latest
+    env:
+      # A missing runtime is an error rather than a skip, the
+      # VISIONSET_REQUIRE_FFMPEG rule: a job that exists to exercise the runtime
+      # and quietly exercised nothing looks exactly like a passing one.
+      #
+      # It says nothing about a missing *GPU*. This runner has no CUDA device and
+      # never will, so the one test that reproduces the half-precision finding on
+      # real tensors asks for the runtime and the device as two separate
+      # questions and skips here on the second — see `tests/inference/test_fp16.py`.
+      VISIONSET_REQUIRE_LOCAL_INFERENCE: "1"
+      # Every hub call in these tests is faked, and this is what keeps that true:
+      # a test that regressed into a real fetch fails here instead of downloading
+      # gigabytes or hanging on somebody else's uptime.
+      HF_HUB_OFFLINE: "1"
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          enable-cache: true
+
+      - name: Sync environment with the local-inference extra
+        run: uv sync --locked --extra local-inference
+
+      # The inference surface, named directory by directory rather than left to
+      # `testpaths`: the rest of the suite already runs in the `python` job, and
+      # what is wanted here is the code that reads a config, resolves a family,
+      # drives the hub client and converts tensors — with the real libraries
+      # loaded rather than a stand-in.
+      #
+      # No `-q`: `addopts` in pyproject.toml already carries one, and a second
+      # suppresses the count line this job exists to put on the record.
+      - name: The inference surface, with the runtime present
+        run: |
+          uv run pytest \
+            tests/inference \
+            tests/architecture/test_optional_runtime.py \
+            tests/server/test_inference.py \
+            tests/server/test_suggest.py \
+            tests/cli/test_inference_commands.py \
+            tests/jobs/test_weights_job.py \
+            -rs
+
   # #62's and #63's second acceptance criteria: the tools the exporters exist to
   # feed actually load what they wrote. Its own job because `ultralytics` brings
   # torch — roughly two gigabytes — and putting that in front of every `uv sync`

CONTRIBUTING.md

@@ -133,7 +133,7 @@ file and read it (`uv run pytest -q > /tmp/out.log 2>&1; echo $?`), or use the s
 above.
 
 The table below is the full list, and it is still wider than the script: the wheel build,
-the 30-minute flow, the format smoke tests and the annotator benchmark are left to CI or to
+the 30-minute flow, the two smoke suites and the annotator benchmark are left to CI or to
 a deliberate manual run, because each costs minutes or needs its own install.
 
 | Check | Command | In `check.sh` |
@@ -153,6 +153,7 @@ a deliberate manual run, because each costs minutes or needs its own install.
 | Wire action rosters | part of `pnpm test` — `tests/scripts/wire_rosters.test.mjs` compares the two transcriptions of `allowed_actions` (`ui-core`'s `testing/wire.fixtures.ts` and the e2e suite's `_wire.ts`) in both directions. Only the first is typed against the generated union, so the second can drift silently; when it did, the failure surfaced as every gallery spec timing out. It proves the two agree with each other, not that either agrees with the kernel — see #358 | part of `frontend` |
 | Docs links | part of `pnpm test` — `tests/scripts/docs_links.test.mjs` resolves every internal link and every `#anchor` in all 46 tracked Markdown files (266 links, 551 headings), naming the file, line and dead fragment. External URLs are ignored on purpose: a gate that fails for somebody else's rate limit is one people re-run rather than read. Renaming a heading breaks inbound anchors *silently* — the link just lands at the top of the page — which was a near miss during the `visionset ui` → `visionset server` rename (#329) | part of `frontend` |
 | Format smoke (ultralytics, pycocotools) | `uv sync --group yolo --group coco && uv run pytest tests/formats/test_*_smoke.py` — their own groups because ultralytics brings torch **and its wheel ships a top-level `tests` package that shadows this repo's**, so run only those files and `uv sync` again afterwards; skips without them, and CI sets `VISIONSET_REQUIRE_ULTRALYTICS=1` / `VISIONSET_REQUIRE_PYCOCOTOOLS=1` so a broken install goes red | — CI |
+| Inference smoke (local-inference extra) | `uv sync --extra local-inference` then `VISIONSET_REQUIRE_LOCAL_INFERENCE=1 uv run pytest tests/inference tests/architecture/test_optional_runtime.py tests/server/test_inference.py tests/server/test_suggest.py tests/cli/test_inference_commands.py tests/jobs/test_weights_job.py -rs`, and `uv sync` again afterwards. The **with-runtime** half of the matrix — see [the two halves](#the-two-halves-of-the-inference-matrix) below. Roughly two gigabytes of CUDA wheels, which is why it is opt-in locally; CI's `inference-smoke` job runs it | — CI |
 | Wheel (build, install, serve) | `bash scripts/build_dist.sh && VISIONSET_REQUIRE_WHEEL=1 uv run pytest tests/packaging` — builds the UI into `_static/`, builds the wheel, installs it in a fresh venv and serves `/app/` from it. Opt-in locally (it costs about a minute); CI's `wheel` job runs it and uploads the artifact | — CI |
 | The 30-minute flow | `uv run python examples/thirty_minute_flow.py` — the vision document's success metric end to end. CI's `30-minute flow (wheel, end to end)` job runs it from the **installed wheel** in an empty venv, with `ultralytics` required there | — CI |
 | Version sync | `pnpm version:check` | `generated` |
@@ -327,3 +328,29 @@ automation.
   which turns that skip into a hard failure so a broken install cannot pass unnoticed. The
   container route needs nothing on the host — `docker/api.Dockerfile` installs it into the image,
   and CI's `docker` job builds that image and runs the video tests inside it.
+
+### The two halves of the inference matrix
+
+`visionset.inference` is tested twice, in two environments, and both halves are deliberate.
+
+The **without-runtime** half is the ordinary `uv run pytest` and CI's `python` job: no
+`local-inference` extra installed. It is what proves a base install is a working install —
+that importing the server, the CLI, the job registry and `visionset.inference` pulls in none
+of torch, torchvision, transformers, accelerate or huggingface_hub, and that a machine
+without them refuses with the install command rather than an `ImportError` from a library the
+caller never named. Do not "fix" those skips by installing the extra into the default
+environment; they are the test.
+
+The **with-runtime** half is the table row above and CI's `inference-smoke` job: the extra
+installed from `uv.lock`, on a CPU-only runner. It is the only place the lazy-import contract
+means anything — on a machine where torch is not installed, "importing the product did not
+load torch" is true by construction — and the only place family resolution, capability
+derivation, the download path and the tensor conversions meet the real libraries at their
+locked versions.
+
+`VISIONSET_REQUIRE_LOCAL_INFERENCE=1` turns a missing runtime from a skip into an error, so a
+broken install goes red instead of quietly shrinking the suite. It says nothing about a
+missing **GPU**: no runner has a CUDA device, so the one test that reproduces the
+half-precision finding on real tensors asks for the runtime and the device separately and
+keeps skipping on the second. Anything that needs a GPU must be written the same way —
+`tests/fixtures/local_inference.py` says why.

docs/architecture/backend/inference.md

@@ -86,6 +86,15 @@ with the optional runtime absent, and
 [`tests/architecture/test_optional_runtime.py`](../../../tests/architecture/test_optional_runtime.py)
 proves it in a fresh interpreter.
 
+**That proof needs an environment where the libraries are actually installed.**
+On a machine without them, "importing the product left torch out of `sys.modules`"
+is true whatever the code does, so the assertion passes and says nothing. CI's
+`inference-smoke` job is where it means something: it installs the extra from the
+lockfile and runs this file among the rest of the inference surface. The two
+halves of that matrix, and the environment variable that keeps the with-runtime
+half honest, are described in
+[CONTRIBUTING](../../../CONTRIBUTING.md#the-two-halves-of-the-inference-matrix).
+
 The optional dependency group is `local-inference`; `_extra.py` names what it
 brings and turns a missing one into a sentence carrying the install command.
 

tests/architecture/test_optional_runtime.py

@@ -18,6 +18,12 @@
 interpreter has already been filled by the rest of the suite, so the only honest
 place to ask "what did importing this load?" is an interpreter that has imported
 nothing else.
+
+**And an environment where the five are installed, or this proves nothing.** Where
+they are absent the assertion holds whatever the code does — a module-level
+`import torch` would raise rather than register — so the run that gives this file
+its meaning is CI's `inference-smoke` job, which installs the extra from the
+lockfile and runs it there.
 """
 
 from __future__ import annotations

tests/cli/test_inference_commands.py

@@ -7,18 +7,17 @@
 
 from __future__ import annotations
 
-import importlib.util
 import json
 from pathlib import Path
 from typing import Any
 
 import pytest
 from click.testing import Result
 from tests.cli._flow import ok, payload, run, runner, workspace
+from tests.fixtures.local_inference import without_the_extra
 
 from visionset.cli import inference as cli_inference
 from visionset.cli.main import app
-from visionset.inference import MODULES
 from visionset.inference import weights as weights_module
 from visionset.inference.integrity import IntegrityReport
 from visionset.kernel.errors import WeightsDamaged
@@ -244,17 +243,6 @@ def test_declining_the_prompt_keeps_the_connection(root: Path) -> None:
 # --- download -----------------------------------------------------------------
 
 
-def _extra_is_installed() -> bool:
-    """Whether this environment actually carries the local runtime.
-
-    The base development environment does not, and CI's does not either, so the
-    unstubbed refusal test is the one that runs for real. Guarded rather than
-    assumed: a contributor who has the extra installed must not see a red suite
-    for having it.
-    """
-    return all(importlib.util.find_spec(name) is not None for name in MODULES)
-
-
 @pytest.fixture()
 def fetched(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> list[str]:
     """Record what would have been downloaded, and download nothing.
@@ -344,7 +332,7 @@ def test_downloading_an_unknown_connection_exits_one(root: Path, fetched: list[s
     assert "Error:" in result.stderr
 
 
-@pytest.mark.skipif(_extra_is_installed(), reason="the local runtime is installed here")
+@without_the_extra
 def test_a_missing_local_runtime_exits_one_with_the_install_command(root: Path) -> None:
     """Unstubbed. A sentence naming what to run, never a traceback.
 
@@ -441,7 +429,7 @@ def test_the_size_command_prints_the_document_on_json(monkeypatch: pytest.Monkey
     }
 
 
-@pytest.mark.skipif(_extra_is_installed(), reason="the local runtime is installed here")
+@without_the_extra
 def test_a_size_without_the_runtime_exits_one_with_the_install_command() -> None:
     """``size`` opens no workspace, so it carries ``domain_errors`` itself.
 

tests/fixtures/local_inference.py

@@ -0,0 +1,84 @@
+# usage: from tests.fixtures.local_inference import require_local_inference, without_the_extra
+"""Which half of the inference matrix this environment can run, decided once.
+
+`visionset.inference` is written so that a base install is a working install: it
+can create a local connection, list it, and be told exactly what to install
+instead of raising an `ImportError` from a library the caller never named. Both
+sides of that are tested — the refusals a machine without the runtime gives, and
+the code that only exists when the runtime is there — so every such test needs to
+know which environment it is in. That predicate lives here rather than in each
+module that asks, because five of them used to spell it out and a sixth would
+have spelled it slightly differently.
+
+CI runs the two halves as two jobs. The `python` job installs the base
+distribution deliberately, which is what makes the without-runtime tests real
+there; `inference-smoke` installs the extra from the lock and sets
+`VISIONSET_REQUIRE_LOCAL_INFERENCE=1`.
+
+**What the variable does is turn a missing runtime from a skip into an error**,
+the rule `VISIONSET_REQUIRE_FFMPEG` states for the video suite: a job that exists
+to exercise the runtime and quietly exercised nothing looks exactly like a
+passing one. It says nothing about a missing *GPU*, which is a separate and
+permanent fact about a CI runner — see `require_local_inference`.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import os
+from typing import Final
+
+import pytest
+
+from visionset.inference import INSTALL_COMMAND, MODULES
+
+EXTRA_REQUIRED_ENV: Final = "VISIONSET_REQUIRE_LOCAL_INFERENCE"
+
+EXTRA_MISSING_HINT: Final = (
+    "the local-inference runtime is not installed here. Install it to run VisionSet's "
+    "with-runtime tests: `uv sync --extra local-inference` in this repository, or "
+    f"`{INSTALL_COMMAND}` against an installed distribution."
+)
+
+EXTRA_INSTALLED: Final[bool] = all(importlib.util.find_spec(name) is not None for name in MODULES)
+"""Whether all five modules of the extra import here.
+
+`find_spec` rather than an import: this is read at collection, and importing
+torch to find out whether torch is installed would put two gigabytes on the
+startup path of every test run that does not need it.
+"""
+
+without_the_extra: Final = pytest.mark.skipif(
+    EXTRA_INSTALLED, reason="the local runtime is installed here"
+)
+"""For a test whose subject is the *refusal* a base install gives.
+
+Never an error under `EXTRA_REQUIRED_ENV`, and that is the whole asymmetry:
+these tests are the `python` job's, and their skipping in `inference-smoke` is
+the job doing its job. A contributor who installs the extra must not get a red
+suite for having it either.
+"""
+
+
+def require_local_inference() -> None:
+    """Skip locally, fail where the runtime was supposed to be installed.
+
+    `tests.fixtures.media.require_ffmpeg`, called from inside a test rather than
+    at module level: the modules holding these tests hold the without-runtime
+    half too, and a module-level skip would take the tests the `python` job
+    exists to run along with them.
+
+    **Missing runtime, not missing GPU.** A CI runner has no CUDA device and
+    never will, so a test that needs one keeps skipping honestly under this
+    variable — it asks for the runtime through this function and for the device
+    separately, afterwards. A single condition covering both would make the job
+    permanently red for the one reason nobody can fix.
+    """
+    if EXTRA_INSTALLED:
+        return
+    if os.environ.get(EXTRA_REQUIRED_ENV) == "1":
+        raise RuntimeError(
+            f"{EXTRA_MISSING_HINT} "
+            f"({EXTRA_REQUIRED_ENV}=1 is set, so a missing runtime is an error, not a skip.)"
+        )
+    pytest.skip(EXTRA_MISSING_HINT)

tests/inference/test_download_size.py

@@ -12,18 +12,16 @@
 
 from __future__ import annotations
 
-import importlib.util
 from dataclasses import dataclass
 from typing import Any
 
 import pytest
+from tests.fixtures.local_inference import without_the_extra
 
-from visionset.inference import MODULES, DownloadSizes, download_size, known_sizes, measure
+from visionset.inference import DownloadSizes, download_size, known_sizes, measure
 from visionset.inference import weights as weights_module
 from visionset.kernel.errors import LocalInferenceUnavailable
 
-EXTRA_INSTALLED = all(importlib.util.find_spec(name) is not None for name in MODULES)
-
 
 @dataclass(frozen=True)
 class FakeSibling:
@@ -177,7 +175,7 @@ def model_info(*_: object, **__: object) -> FakeInfo:
     assert "Repository Not Found" in str(raised.value)
 
 
-@pytest.mark.skipif(EXTRA_INSTALLED, reason="the local runtime is installed here")
+@without_the_extra
 def test_a_missing_hub_client_names_the_install_command() -> None:
     """Unstubbed: the size is read with the client that would do the fetching, so
     a machine without the extra is refused here too — with the remedy."""