Skip to content

Commit 4d2e8dd

Browse files
authored
feat(standard): check pinned images against registry tags (#87)
Closes #31 ## What Add `nullius standard pins BENCHMARK [--tag TAG] [--limit N]` to compare a benchmark's pinned images with registry tag digests. It reads images.json or string literals in driver.py without executing the driver, then reports same, moved (both digests), gone, or not checked. Exit codes are 0 for all matching, 1 for moved or missing tags, and 2 if any check could not complete. The command reads only registry manifest headers, supports anonymous bearer authentication, and leaves images, pins and user storage unchanged. An explicit tag overrides embedded tags; pins without a tag compare latest. ## Touches A stdlib-only standard/pins.py helper, the standard CLI, regression coverage in the existing unified CLI tests, and the existing CLI command reference. ## Verified - 37 focused CLI tests passed: both discovery sources, no driver execution or file writes, tags/registry ports, anonymous authentication, missing tags versus authentication/rate-limit/network failures, limits, moved digest output, and exit-code precedence. - Real public-registry checks, one selected pin each: SWE-bench Verified astropy__astropy-12907 matched latest (exit 0); Terminal-Bench bn-fit-modify had no latest tag (exit 1); a deliberately absent tag returned gone (exit 1). Absence of a tag is not disappearance of its pinned digest. - Source pin file hashes and the temporary user profile were unchanged after the real checks. No image pulls/builds, benchmark tasks or model calls; cost $0. - Full backend suite: 1734 tests across 107 modules passed. - Branch starts from main d1d4308, including #83 task environment changes. No dependency on PR #85. - git diff --check passed. ## Risk Checks require registry connectivity. Private credentials are not loaded; authentication, rate limits, unsupported redirects or unavailable digest headers report not checked with exit 2. Each HTTP request times out after 15 seconds. The comparison uses the tag's manifest digest and does not select a platform. For tagless pins, maintainers must supply --tag when the original tag was not latest. Revert to remove the command; no migration or pin refresh.
1 parent f1a57eb commit 4d2e8dd

5 files changed

Lines changed: 269 additions & 4 deletions

File tree

‎app/nullius/standard/cli.py‎

Lines changed: 34 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
PYTHONPATH=app python3 -m nullius.standard.cli list
44
PYTHONPATH=app python3 -m nullius.standard.cli check [PATH]
55
PYTHONPATH=app python3 -m nullius.standard.cli fit BENCHMARK[@VERSION]
6+
PYTHONPATH=app python3 -m nullius.standard.cli pins BENCHMARK[@VERSION] [--tag TAG] [--limit N]
67
PYTHONPATH=app python3 -m nullius.standard.cli new benchmark|agent|capability KEY [--family F] [--port P]
78
PYTHONPATH=app python3 -m nullius.standard.cli new version KEY VERSION
89
PYTHONPATH=app python3 -m nullius.standard.cli conform AGENT|CAPABILITY --model M [--provider P] [--port PORT ...] [--repeat K] [--option NAME=VALUE ...] [--unsandboxed | --backend docker]
@@ -69,7 +70,7 @@
6970
if __name__ == "__main__" and "--from-run" not in sys.argv[1:]:
7071
settings.load(paths.settings_file())
7172

72-
from nullius.standard import conform, definitions, fidelity, job, match, merge, ports, prove, run, shard
73+
from nullius.standard import conform, definitions, fidelity, job, match, merge, pins, ports, prove, run, shard
7374

7475
ENV_FILE = paths.settings_file()
7576

@@ -138,6 +139,33 @@ def skills(d: dict, ports_=None) -> list:
138139
return out
139140

140141

142+
def cmd_pins(a):
143+
if a.limit is not None and a.limit < 1:
144+
print("--limit must be a positive whole number", file=sys.stderr)
145+
return 2
146+
if a.tag is not None and not pins.TAG.fullmatch(a.tag):
147+
print("invalid image tag", file=sys.stderr)
148+
return 2
149+
bench = definitions.get("benchmark", a.benchmark)
150+
if bench is None:
151+
print("no such benchmark", file=sys.stderr)
152+
return 2
153+
try:
154+
refs = pins.references(bench["_here"])
155+
except (OSError, ValueError, SyntaxError):
156+
print("not checked: cannot read image pins from images.json or driver.py", file=sys.stderr)
157+
return 2
158+
if not refs:
159+
print("not checked: no image pins found", file=sys.stderr)
160+
return 2
161+
code = 0
162+
for name, ref in refs[:a.limit]:
163+
state, detail = pins.check(ref, a.tag)
164+
print(f"{name}: {state}: {detail}", flush=True)
165+
code = max(code, 2 if state == "not checked" else 0 if state == "same" else 1)
166+
return code
167+
168+
141169
def cmd_list(a):
142170
got, bad = _all()
143171
for kind in definitions.KINDS:
@@ -576,6 +604,10 @@ def main(argv=None):
576604
c.add_argument("path", nargs="?")
577605
f = sub.add_parser("fit", help="what can run a benchmark, and why the rest cannot")
578606
f.add_argument("benchmark")
607+
p = sub.add_parser("pins", help="compare image pins with registry tags without changing them")
608+
p.add_argument("benchmark")
609+
p.add_argument("--tag", help="tag to compare (default: embedded tag, or latest)")
610+
p.add_argument("--limit", type=int, help="check only the first N pins")
579611
n = sub.add_parser("new", help="a definition to fill in, in its catalogue folder")
580612
n.add_argument("kind", type=definitions.names.kind, choices=[*definitions.KINDS, "version"])
581613
n.add_argument("key")
@@ -662,7 +694,7 @@ def main(argv=None):
662694
progress.ECHO = sys.stderr # each stage as a line (nullius/progress.py): stdout stays the job and its units
663695
if not getattr(a, "from_run", False):
664696
settings.load(ENV_FILE)
665-
return {"list": cmd_list, "check": cmd_check, "fit": cmd_fit, "new": cmd_new, "conform": cmd_conform, "run": cmd_run,
697+
return {"list": cmd_list, "check": cmd_check, "fit": cmd_fit, "pins": cmd_pins, "new": cmd_new, "conform": cmd_conform, "run": cmd_run,
666698
"fidelity": cmd_fidelity, "prove": cmd_prove, "merge": cmd_merge, "traces": cmd_traces}[a.cmd](a)
667699

668700

‎app/nullius/standard/pins.py‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
"""Compare a benchmark's image pins with registry tags, without pulling or
2+
changing anything. Read images.json, or string literals in driver.py without
3+
executing it. Registry V2 HEAD requests read manifest digests; anonymous
4+
Bearer challenges are supported, private credentials are never loaded.
5+
"""
6+
from __future__ import annotations
7+
8+
import ast
9+
import json
10+
import re
11+
from http.client import HTTPException
12+
from pathlib import Path
13+
from urllib.error import HTTPError, URLError
14+
from urllib.parse import urlencode, urlsplit
15+
from urllib.request import HTTPRedirectHandler, Request, build_opener
16+
17+
TAG = re.compile(r"[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}")
18+
DIGEST = re.compile(r"sha256:[0-9a-f]{64}")
19+
REPOSITORY = re.compile(r"[a-z0-9]+(?:[._-]+[a-z0-9]+)*(?:/[a-z0-9]+(?:[._-]+[a-z0-9]+)*)*")
20+
ACCEPT = ", ".join(("application/vnd.oci.image.index.v1+json", "application/vnd.oci.image.manifest.v1+json",
21+
"application/vnd.docker.distribution.manifest.list.v2+json",
22+
"application/vnd.docker.distribution.manifest.v2+json"))
23+
24+
25+
def references(folder: str) -> list:
26+
"""(name, reference) in file order. images.json takes precedence."""
27+
root = Path(folder)
28+
if (root / "images.json").exists():
29+
data = json.loads((root / "images.json").read_text())
30+
if not isinstance(data, dict) or not all(isinstance(v, str) for v in data.values()):
31+
raise ValueError("images.json must map names to pinned image references")
32+
return list(data.items())
33+
tree = ast.parse((root / "driver.py").read_text())
34+
strings = sorted((n for n in ast.walk(tree) if isinstance(n, ast.Constant) and isinstance(n.value, str)
35+
and "@sha256:" in n.value), key=lambda n: (n.lineno, n.col_offset))
36+
return [(s, s) for s in dict.fromkeys(n.value for n in strings)]
37+
38+
39+
def target(reference: str, tag: str | None = None) -> tuple:
40+
"""Registry, repository, tag and pinned digest. Docker Hub is implicit;
41+
--tag overrides an embedded tag, otherwise an untagged pin uses latest."""
42+
name, _, digest = reference.rpartition("@")
43+
if not DIGEST.fullmatch(digest):
44+
raise ValueError("expected an image pinned by sha256 digest")
45+
embedded = None
46+
if ":" in name.rsplit("/", 1)[-1]:
47+
name, embedded = name.rsplit(":", 1)
48+
tag = tag if tag is not None else embedded or "latest"
49+
if not TAG.fullmatch(tag):
50+
raise ValueError("invalid image tag")
51+
first, slash, rest = name.partition("/")
52+
if slash and ("." in first or ":" in first or first == "localhost"):
53+
registry, repository = first, rest
54+
else:
55+
registry, repository = "docker.io", name
56+
if not re.fullmatch(r"[A-Za-z0-9.-]+(?::[0-9]+)?", registry) or not REPOSITORY.fullmatch(repository):
57+
raise ValueError("invalid registry or repository name")
58+
if registry in ("docker.io", "index.docker.io", "registry-1.docker.io"):
59+
registry = "registry-1.docker.io"
60+
if "/" not in repository:
61+
repository = "library/" + repository
62+
return registry, repository, tag, digest
63+
64+
65+
class _NoRedirect(HTTPRedirectHandler):
66+
def redirect_request(self, req, fp, code, msg, headers, newurl):
67+
# A registry's bearer token must not follow a redirect to another service.
68+
raise ValueError("registry redirect not followed")
69+
70+
71+
def _token(challenge: str, repository: str, open_) -> str:
72+
kind, _, fields = challenge.partition(" ")
73+
params = dict(re.findall(r'(\w+)="([^"\r\n]*)"', fields))
74+
realm = params.get("realm", "")
75+
url = urlsplit(realm)
76+
if kind.lower() != "bearer" or url.scheme != "https" or not url.hostname or url.username or url.password or url.fragment:
77+
raise ValueError("registry requires unsupported authentication")
78+
query = urlencode({"service": params.get("service", ""), "scope": f"repository:{repository}:pull"})
79+
try:
80+
with open_(Request(realm + ("&" if url.query else "?") + query), timeout=15) as response:
81+
data = json.loads(response.read(65537))
82+
except HTTPError as e:
83+
e.close()
84+
raise ValueError(f"token service returned HTTP {e.code}; anonymous access unavailable") from None
85+
except (ValueError, UnicodeError):
86+
raise ValueError("invalid token service response") from None
87+
token = (data.get("token") or data.get("access_token")) if isinstance(data, dict) else None
88+
if not isinstance(token, str) or not token or any(not 33 <= ord(c) <= 126 for c in token):
89+
raise ValueError("token service returned no usable bearer token")
90+
return token
91+
92+
93+
def current(registry: str, repository: str, tag: str, open_=None) -> str:
94+
"""Ask only for manifest headers. A missing tag raises HTTPError(404);
95+
auth-service failures must not be mistaken for a missing manifest."""
96+
open_ = open_ or build_opener(_NoRedirect()).open
97+
url = f"https://{registry}/v2/{repository}/manifests/{tag}"
98+
headers = {"Accept": ACCEPT}
99+
try:
100+
response = open_(Request(url, headers=headers, method="HEAD"), timeout=15)
101+
except HTTPError as e:
102+
if e.code != 401:
103+
raise
104+
challenge = e.headers.get("WWW-Authenticate", "")
105+
e.close()
106+
headers["Authorization"] = "Bearer " + _token(challenge, repository, open_)
107+
response = open_(Request(url, headers=headers, method="HEAD"), timeout=15)
108+
with response:
109+
digest = response.headers.get("Docker-Content-Digest", "")
110+
if not DIGEST.fullmatch(digest):
111+
raise ValueError("registry returned no valid sha256 manifest digest")
112+
return digest
113+
114+
115+
def check(reference: str, tag: str | None = None) -> tuple:
116+
"""(state, detail): same, moved, gone, or not checked. Never include
117+
response bodies, authentication URLs or credentials in diagnostics."""
118+
try:
119+
registry, repository, tag, pinned = target(reference, tag)
120+
found = current(registry, repository, tag)
121+
return ("same", f"{tag}: {pinned}") if found == pinned else ("moved", f"{tag}: {pinned} -> {found}")
122+
except HTTPError as e:
123+
e.close()
124+
if e.code == 404:
125+
return "gone", f"{tag}: registry returned HTTP 404"
126+
return "not checked", f"registry returned HTTP {e.code}"
127+
except (URLError, OSError, HTTPException):
128+
return "not checked", "registry connection failed or timed out"
129+
except ValueError as e:
130+
return "not checked", str(e)

‎app/tests/test_unified_cli.py‎

Lines changed: 95 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,11 @@
1010
import sys
1111
import tempfile
1212
import unittest
13+
from urllib.error import HTTPError, URLError
1314
from unittest.mock import patch
1415

1516
from nullius import cli
16-
from nullius.standard import cli as standard_cli, definitions
17+
from nullius.standard import cli as standard_cli, definitions, pins
1718

1819
APP = Path(__file__).resolve().parents[1]
1920

@@ -173,5 +174,98 @@ def test_new_creates_only_a_user_entry_and_unknown_version_has_a_clear_error(sel
173174
standard_cli.new_version("missing", "2")
174175

175176

177+
class ImagePins(unittest.TestCase):
178+
OLD, NEW = "sha256:" + "a" * 64, "sha256:" + "b" * 64
179+
REF = "docker.io/example/image@" + OLD
180+
181+
def test_both_pin_sources_are_read_without_executing_or_writing(self):
182+
with tempfile.TemporaryDirectory() as folder:
183+
root = Path(folder)
184+
driver = root / "driver.py"
185+
driver.write_text(f'raise RuntimeError("must not execute")\nIMAGES = {{"task": "{self.REF}"}}\n')
186+
self.assertEqual(pins.references(folder), [(self.REF, self.REF)])
187+
(root / "images.json").write_text(json.dumps({"task": self.REF}))
188+
self.assertEqual(pins.references(folder), [("task", self.REF)])
189+
before = {p.name: p.read_bytes() for p in root.iterdir()}
190+
with patch.object(definitions, "get", return_value={"_here": folder}), \
191+
patch.object(pins, "current", return_value=self.OLD), contextlib.redirect_stdout(io.StringIO()):
192+
self.assertEqual(standard_cli.main(["pins", "fixture", "--limit", "1"]), 0)
193+
self.assertEqual({p.name: p.read_bytes() for p in root.iterdir()}, before)
194+
195+
def test_tags_registry_ports_and_docker_hub_shorthand(self):
196+
for reference, override, expected in (
197+
("ubuntu@" + self.OLD, None, ("registry-1.docker.io", "library/ubuntu", "latest", self.OLD)),
198+
("registry.example:5443/team/image:stable@" + self.OLD, None,
199+
("registry.example:5443", "team/image", "stable", self.OLD)),
200+
(self.REF, "v2", ("registry-1.docker.io", "example/image", "v2", self.OLD))):
201+
self.assertEqual(pins.target(reference, override), expected)
202+
for ref in ("not-pinned", "https://example/image@" + self.OLD, "user:password@host/image@" + self.OLD):
203+
with self.assertRaises(ValueError):
204+
pins.target(ref)
205+
206+
@staticmethod
207+
def response(body=b"", **headers):
208+
response = io.BytesIO(body)
209+
response.headers = headers
210+
return response
211+
212+
def test_registry_head_and_anonymous_bearer_challenge(self):
213+
challenge = HTTPError("https://registry.example/v2/example/image/manifests/latest", 401, "auth", {
214+
"WWW-Authenticate": 'Bearer realm="https://auth.example/token",service="registry.example",scope="repository:other:push"'}, None)
215+
with patch.object(pins, "build_opener") as opener:
216+
open_ = opener.return_value.open
217+
open_.side_effect = [challenge, self.response(b'{"token":"fixture-token"}'),
218+
self.response(**{"Docker-Content-Digest": self.OLD})]
219+
self.assertEqual(pins.current("registry.example", "example/image", "latest"), self.OLD)
220+
requests = [call.args[0] for call in open_.call_args_list]
221+
self.assertEqual([r.get_method() for r in requests], ["HEAD", "GET", "HEAD"])
222+
self.assertIn("scope=repository%3Aexample%2Fimage%3Apull", requests[1].full_url)
223+
self.assertEqual(requests[2].get_header("Authorization"), "Bearer fixture-token")
224+
self.assertIn("application/vnd.oci.image.index.v1+json", requests[0].get_header("Accept"))
225+
226+
def test_missing_manifest_is_distinct_from_auth_rate_limits_and_connection_failures(self):
227+
for failure, state in ((HTTPError("https://registry.example", 404, "gone", {}, None), "gone"),
228+
(HTTPError("https://registry.example", 429, "limited", {}, None), "not checked"),
229+
(HTTPError("https://registry.example", 403, "denied", {}, None), "not checked"),
230+
(URLError("unreachable"), "not checked"), (TimeoutError(), "not checked")):
231+
with self.subTest(failure=failure), patch.object(pins, "current", side_effect=failure):
232+
self.assertEqual(pins.check(self.REF)[0], state)
233+
challenge = 'Bearer realm="https://auth.example/token"'
234+
with patch.object(pins, "build_opener") as opener:
235+
opener.return_value.open.side_effect = [HTTPError("https://registry.example", 401, "auth", {"WWW-Authenticate": challenge}, None),
236+
HTTPError("https://auth.example/token", 404, "gone", {}, None)]
237+
state, detail = pins.check(self.REF)
238+
self.assertEqual(state, "not checked")
239+
self.assertIn("token service returned HTTP 404", detail)
240+
241+
def test_missing_digest_and_unsafe_auth_are_not_used(self):
242+
with self.assertRaisesRegex(ValueError, "no valid sha256"):
243+
pins.current("registry.example", "example/image", "latest", open_=lambda *a, **k: self.response())
244+
for challenge in ('Basic realm="private"', 'Bearer realm="http://auth.example/token"'):
245+
with self.assertRaisesRegex(ValueError, "unsupported authentication"):
246+
pins._token(challenge, "example/image", lambda *a, **k: self.fail("must not request"))
247+
248+
def test_cli_limits_and_exit_precedence(self):
249+
refs = [("first", self.REF), ("second", self.REF)]
250+
with patch.object(definitions, "get", return_value={"_here": "unused"}), patch.object(pins, "references", return_value=refs), \
251+
patch.object(pins, "current") as current:
252+
for results, expected in (([self.OLD, self.OLD], 0), ([self.OLD, self.NEW], 1),
253+
([self.NEW, URLError("down")], 2)):
254+
current.side_effect = results
255+
out = io.StringIO()
256+
with contextlib.redirect_stdout(out):
257+
self.assertEqual(standard_cli.main(["pins", "fixture"]), expected)
258+
if expected == 1:
259+
self.assertIn(f"{self.OLD} -> {self.NEW}", out.getvalue())
260+
current.reset_mock(side_effect=True)
261+
current.return_value = self.OLD
262+
with contextlib.redirect_stdout(io.StringIO()):
263+
self.assertEqual(standard_cli.main(["pins", "fixture", "--tag", "stable", "--limit", "1"]), 0)
264+
current.assert_called_once_with("registry-1.docker.io", "example/image", "stable")
265+
with contextlib.redirect_stderr(io.StringIO()):
266+
self.assertEqual(standard_cli.main(["pins", "fixture", "--limit", "0"]), 2)
267+
self.assertEqual(standard_cli.main(["pins", "fixture", "--tag", "bad/tag"]), 2)
268+
269+
176270
if __name__ == "__main__":
177271
unittest.main()

‎docs/cli/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ also works.
1717
| [Launch](../getting-started.md) | `nullius serve [roots...]` |
1818
| [Read jobs](read-jobs.md) | `nullius runs`, `nullius compare`, `nullius workspace`, `nullius standard traces` |
1919
| [Import a run](import.md) | `nullius import` |
20-
| [Definitions](definitions.md) | `nullius standard list`, `check`, `fit`, `new` |
20+
| [Definitions](definitions.md) | `nullius standard list`, `check`, `fit`, `pins`, `new` |
2121
| [Run a benchmark](run.md) | `nullius standard run` |
2222
| [Prove a benchmark](prove.md) | `nullius standard prove` |
2323
| [Conform an agent or a capability](conform.md) | `nullius standard conform` |

‎docs/cli/definitions.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ Every definition, whether it is sound, which pairs fit, and a new one to fill in
66
nullius standard list
77
nullius standard check [path]
88
nullius standard fit <benchmark>
9+
nullius standard pins <benchmark> [--tag TAG] [--limit N]
910
nullius standard new {benchmark,agent,capability} <key> [--family {agent,capability}] [--port PORT]
1011
nullius standard new version <key> <version>
1112
```
@@ -15,6 +16,7 @@ nullius standard new version <key> <version>
1516
| `list` | Every definition, which pairs fit, each `proven` record (kit job, date, model, score, kit version, skills) |
1617
| `check [path]` | The definitions, or one file, against the catalogue and the kits |
1718
| `fit <benchmark>` | What can run it, and why the rest cannot |
19+
| `pins <benchmark>` | Compare `images.json`, or literal image pins in `driver.py`, with registry tags; report same, moved (both digests), gone, or not checked |
1820
| `new benchmark KEY` | `standard.json` and `driver.py` under `benchmarks/<key>/` in the user catalogue; `--family capability --port PORT` for a capability benchmark |
1921
| `new agent KEY`, `new capability KEY` | A definition to fill in, in its user catalogue folder |
2022
| `new version KEY VERSION` | The default's block repinned, nothing proven; a file without `versions` becomes one, proofs kept |
@@ -24,6 +26,13 @@ nullius standard new version <key> <version>
2426
whole folder. A user entry replaces the bundled entry of the same kind and
2527
key, including its local resources.
2628

29+
`pins` reads registry manifest headers without downloading images or changing
30+
pins. It compares each embedded tag, or `latest` when absent; `--tag` overrides
31+
it and `--limit` checks only the first N pins. Exit codes: 0 all checked pins
32+
match, 1 a tag moved or is gone, 2 a check could not complete (including auth
33+
or rate limits). Anonymous registry authentication is supported; private
34+
credentials are not loaded. Refreshing pins and reproving controls stays manual.
35+
2736
!!! note "A new version runs nothing"
2837
Until `conform KEY@VERSION` passes its kits.
2938

0 commit comments

Comments
 (0)