Give every published row a real identity, and cut the site down to what it can support

.github/workflows/ci.yml

@@ -25,6 +25,22 @@ jobs:
       - name: Test
         run: cargo test --workspace
 
+  site:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      # pages.yml only builds on push, so without this a change that breaks site
+      # generation lands on main and is discovered after deploy.
+      - name: Generate site
+        run: cargo run -q -p willitcall -- site --results results/ --out site/
+      - name: Check expected pages exist
+        run: |
+          for page in index.html submit.html style.css site.js; do
+            test -s "site/$page" || { echo "site/$page missing or empty"; exit 1; }
+          done
+
   results:
     runs-on: ubuntu-latest
     steps:
@@ -48,38 +64,28 @@ jobs:
           from pathlib import Path
           from jsonschema import Draft202012Validator
 
-          # Pre-amendment-6 rows scheduled for re-measurement.
-          PRE_AMENDMENT_6_RESULTS = {
-              "llamacpp-granite3.1-dense-8b.json",
-              "llamacpp-phi4-mini.json",
-              "llamacpp-qwen2.5-1.5b-instruct-q4_k_m.json",
-              "llamacpp-watt-tool-8b-q4_k_m.json",
-              "ollama-gemma3-12b.json",
-              "ollama-gemma3-4b.json",
-              "ollama-granite3.1-dense-8b.json",
-              "ollama-hermes3-8b.json",
-              "ollama-llama3-groq-tool-use-8b.json",
-              "ollama-llama3.1-8b.json",
-              "ollama-mistral-7b.json",
-              "ollama-phi4-mini.json",
-              "ollama-qwen2.5-7b-instruct.json",
-              "ollama-qwen3-0.6b.json",
-              "ollama-qwen3-1.7b.json",
-              "ollama-qwen3-14b.json",
-              "ollama-qwen3-4b.json",
-              "ollama-qwen3-8b.json",
-          }
+          # Rows exempt from the amendment-6 environment requirement, which only
+          # ever applied to schema v2. Every row it covered is now v3, so it is
+          # empty; the staleness check below keeps it that way.
+          PRE_AMENDMENT_6_RESULTS = set()
 
+          # Every schema the repository ships must be loaded here. Pinning this
+          # list behind the current schema version is what kept CI red for two
+          # milestones once already; a version we cannot validate is reported as
+          # a failure, so the list going stale is loud rather than silent.
           validators = {}
-          for version in (1, 2):
+          for version in (1, 2, 3):
               with open(f"schemas/result-v{version}.schema.json") as handle:
                   validators[version] = Draft202012Validator(json.load(handle))
 
           failed = False
+          exemptions_used = set()
           for path in sys.argv[1:]:
               with open(path) as handle:
                   document = json.load(handle)
               version = document.get("schema_version", 1)
+              if version == 2 and Path(path).name in PRE_AMENDMENT_6_RESULTS:
+                  exemptions_used.add(Path(path).name)
               validator = validators.get(version)
               if validator is None:
                   failed = True
@@ -98,5 +104,15 @@ jobs:
                       print(f"{path}: {location}: {error.message}")
               else:
                   print(f"{path}: ok")
+
+          # The exemption list is hand maintained, so it has to say when it stops
+          # describing reality. A name that no longer needs exempting is debt that
+          # would otherwise sit here indefinitely looking load bearing.
+          stale = sorted(PRE_AMENDMENT_6_RESULTS - exemptions_used)
+          if stale:
+              failed = True
+              print("PRE_AMENDMENT_6_RESULTS is stale; remove these entries:")
+              for name in stale:
+                  print(f"  {name}")
           sys.exit(1 if failed else 0)
           PY

CONTRIBUTING.md

@@ -59,8 +59,14 @@ record_id = "rec-17"
 ## Submitting a result file
 
 1. Run the full scenario corpus against one loaded model at a time.
-2. Run `willitcall validate results/<file>.json`; it must pass against `schemas/result-v1.schema.json`.
-3. Open a pull request adding the file under `results/`, and state the hardware and server version used.
+2. Add the exact model selector to `registry/models-v1.json`. Every claimed
+   identity field needs a provenance reference. If the provenance cannot be
+   recovered, an explicit unresolved entry is acceptable and will be displayed
+   as `unresolved`; do not infer identity from the selector or result filename.
+3. Run `willitcall validate results/<file>.json`; new runs must pass as schema v3
+   against `schemas/result-v3.schema.json`. The CLI continues to accept schema v1
+   and v2 files without upgrading them during `annotate` or `rescore`.
+4. Open a pull request adding the file under `results/`, and state the hardware and server version used.
 
 Never hand-edit a result file. Each scenario record carries an evidence hash, so edited results are not comparable.
 

README.md

@@ -56,6 +56,12 @@ Check a result file against the published schema:
 cargo run -p willitcall -- validate willitcall-result.json
 ```
 
+New runs use result schema v3 (`schemas/result-v3.schema.json`). The CLI resolves
+the model selector through `registry/models-v1.json` and embeds the resolved
+model and artifact metadata in the result. Legacy schema v1 and v2 files remain
+valid inputs to `validate`, `site`, `annotate`, and `rescore`; editing a legacy
+file preserves its original schema version.
+
 `--server` selects a preset (`llamacpp`, `ollama`, `mlx-lm`, `lmstudio`,
 `vllm`, `custom`). The preset only supplies request defaults; the preset name is
 recorded in the result file so results stay comparable.
@@ -126,11 +132,13 @@ model. So:
   nearly right that the parser then rejected. The `unparsed_tool_call` failure
   class exists to mark exactly that case, and the transcript shows the bytes.
 
-Each result records which side of this line its server sits on, in
-`server.quirk_flags`: `grammar_constrained_decoding` for llama.cpp,
-`unconstrained_post_hoc_parse` for Ollama and mlx-lm. LM Studio and vLLM are
-unflagged because their decode path has not been verified here; absence of a
-flag means unverified, not unconstrained.
+V3 results can record which side of this line their server sits on in
+`metadata.server.decode_mode`, while retaining any corresponding
+`server.quirk_flags`: `grammar_constrained_decoding` for llama.cpp and
+`unconstrained_post_hoc_parse` for Ollama and mlx-lm. When a historical run did
+not record the mode, the site consults the cited preset mapping in
+`registry/decode-modes-v1.json`; presets absent from that mapping remain
+`unknown` rather than being guessed.
 
 This was established the hard way. An earlier version of this project published
 a claim that Ollama discarded valid tool calls. Recovering the discarded bytes
@@ -180,10 +188,13 @@ that back it. Seed results so far cover `qwen3` at 0.6b/1.7b/4b/8b,
 1. Run the full corpus against your endpoint, one model loaded at a time.
    Running two models at once produces spurious `error` outcomes from resource
    contention, not real measurements.
-2. Run `willitcall validate` on the output. Current results are schema
-   version 2 (`schemas/result-v2.schema.json`); version 1 files are still
-   accepted.
-3. Open a pull request adding the result file **and its `evidence/` directory**
+2. Ensure the exact model selector has an entry in `registry/models-v1.json`,
+   with evidence references for every claimed identity field. An unresolved
+   entry is acceptable when provenance cannot be recovered; the result and site
+   show that status as `unresolved` rather than inferring identity from a name.
+3. Run `willitcall validate` on the schema v3 output. Legacy schema v1 and v2
+   files remain accepted, but must not be upgraded by editing them.
+4. Open a pull request adding the result file **and its `evidence/` directory**
    under `results/`, and say what hardware and server version produced it.
 
 Every scenario writes a full request/response transcript to