feat(model): make HF smoke latency comparisons output-length aware

model/EVALUATION_PROTOCOL.md

@@ -274,28 +274,39 @@ and a checksum says nothing about code usefulness, correctness,
 legality or safety. The real-model and human-review gates remain.
 
 
-## 14. Local HF synthetic generation timing (v4 current protocol)
+## 14. Local HF synthetic generation timing and output length (v5 current protocol)
 
 For actual reviewed local `smoke.py --run` trials, protocol
-`lineloom-hf-synthetic-smoke-v4` adds nearest-rank **p50/p95 generation
+`lineloom-hf-synthetic-smoke-v5` reports nearest-rank **p50/p95 generation
 latency in milliseconds**, overall and per target language. Measurement
 uses a monotonic clock surrounding only the `model.generate()` call,
-with explicit GPU synchronization before and after when using CUDA.
-Model loading, prompt tokenization, decoding, VS Code integration and all
-editor-side delays are **outside this timing**. Unselected languages report
-zero attempts and null percentiles.
-
-Offline `compare_smoke.py` now requires the same fixed timing scheme and
-matching fixture/request/checkpoint scope and reports base, adapter and
-adapter-minus-base latency differences. A **positive difference** is slower
-generation in that particular run, but is not an independently verified
-regression; compare only with controlled hardware, load, thermal state,
-execution order and reproducible conditions. With only six examples per
-language, these p95 measurements are highly unstable.
-
-This v4 contract intentionally rejects older v1–v3 HF reports in the
-comparator; rerun both from the same reviewed checkout. The separate
-Ollama benchmark uses `lineloom-synthetic-v4` and remains incomparable.
-Offline unit tests use invented timings, without GPU, training or any
-model-generation measurements. No real HF inference has been executed
-or claimed by this development change.
+with explicit CUDA synchronization on both sides. Loading, prompt
+tokenization, decoding, VS Code integration and all editor-side delays
+are **outside this timing**. Unselected languages report zero attempts
+and null latency percentiles.
+
+V5 additionally counts **actual generated token IDs** (including a stop
+token when generated), aggregated globally and per language, with at
+most `maxNewTokens` IDs per attempt. It does **not** print tokens, decoded
+text, fixture identities or individual token counts. The source-free
+comparator validates count bounds and five-language sums, then reports
+base, adapter and adapter-minus-base generation-token totals. If token
+totals differ, raw generation-duration deltas are especially unsuitable
+for a speed comparison: one model may simply produce more code. Even
+equal totals do **not** establish matching per-probe output lengths.
+The runner intentionally does not claim throughput, first-token latency,
+identical hardware, compiler correctness or editor acceptance.
+
+Offline `compare_smoke.py` requires identical fixture/request/checkpoint
+scope and reports only descriptive base-versus-adapter differences. A
+**positive duration difference** is slower in that particular run, not
+an independently verified regression. Compare only on controlled
+hardware, load, thermal state, quantization and execution order.
+With six examples per language, p95 is highly unstable.
+
+V5 rejects previous v1–v4 HF smoke protocols in the comparator; rerun
+both base and adapter from the same reviewed checkout. The separate
+Ollama benchmark uses `lineloom-synthetic-v4` and is not directly
+comparable. Offline tests use invented timing and token counts without
+GPU, model inference, training, generated private source or network
+access. No real HF inference is claimed by this development change.

model/train/README.md

@@ -271,7 +271,7 @@ Python standard library. They check seeded repeatability, distribution,
 bucket coverage, rejection of validation/test contamination, nonfinite
 losses and independence from the process-global PRNG.
 
-## Comparing local HF base-versus-adapter smoke reports (protocol v4)
+## Comparing local HF base-versus-adapter smoke reports (protocol v5)
 
 After **independently approved** local base and adapter runs on the same
 machine with the *same* base revision, tokenizer, fixtures and generation
@@ -293,7 +293,7 @@ than 32 KiB. Only one of the fixed checked-in synthetic scopes
 (`all`, `rust`, `c`, `cpp`, `python`, `typescript`) is valid.
 
 Real local `smoke.py --run` reports now include the
-`lineloom-hf-synthetic-smoke-v4` protocol, deterministic
+`lineloom-hf-synthetic-smoke-v5` protocol, deterministic
 `fixtureSetSha256`, `requestSpecSha256`, checked-in fixture scope,
 greedy generation contract, fixed local base config/tokenizer SHA-256,
 and the **user-declared** 40-character base revision. The comparator
@@ -302,14 +302,20 @@ and refuses plans, legacy reports, corrupted counts, mixed base revisions,
 different settings, or comparisons without a local base report followed by
 a local adapter report.
 
-The comparison output contains **allowlisted strict-match counts and
-generation-only latency p50/p95** by language and total, including
-adapter-minus-base deltas, plus fixed caveats. The timer uses a monotonic
-clock around `model.generate()` only (and synchronizes CUDA on each side).
+The comparison output contains **allowlisted strict-match counts,
+generation-only latency p50/p95, and generated-token totals** by language
+and overall, with adapter-minus-base deltas. Generated-token totals
+count actual output token IDs, including a stop token when present, but
+never include raw tokens, private source, individual probe identities
+or decoded text. The timer uses a monotonic clock around
+`model.generate()` only (and synchronizes CUDA on each side).
 It excludes model loading, tokenization, decoding, and VS Code/editor time.
 Positive latency deltas mean the adapter was slower in the recorded run;
-negative deltas mean faster. Six probes per language produce unstable p95
-values. Compare only on the same reviewed machine in controlled conditions:
+negative deltas mean faster. If the models produce **different numbers of
+generated tokens**, those latency differences do not represent a
+same-length throughput comparison. Even identical aggregate token totals
+cannot prove matching lengths for individual probes. Six probes per
+language produce unstable p95 values. Compare only on controlled conditions:
 the report hashes do **not** prove hardware or runtime equivalence.
 It never prints prompts, completions, model checkpoint paths, private
 metadata, filenames or user-supplied report fields.
@@ -321,7 +327,7 @@ GPU settings, tokenizer runtime behavior, hidden sampling defaults,
 compiler correctness, editor acceptance, equivalent timing conditions or whether reports were
 truthfully generated. **The six probes per language are not a production
 model evaluation**, and a strict exact-match improvement does not establish
-quality. Prior HF v1–v3 reports must be rerun; HF protocol v4 remains
+quality. Prior HF v1–v4 reports must be rerun; HF protocol v5 remains
 incomparable with the Ollama `lineloom-synthetic-v4` protocol.
 
 Tests are run automatically by `npm run test:model-training` using Python
@@ -352,7 +358,8 @@ checkpoint weights cannot now be compared as if they were the same base.
 receipts and HF smoke `lineloom-hf-synthetic-smoke-v2` reports are
 not accepted by the new versioned verification/comparator workflow.
 New receipts use `lineloom-local-adapter-receipt-v2`; new HF reports use
-`lineloom-hf-synthetic-smoke-v4` with generation-only timing. Do not alter old receipts to make them
+`lineloom-hf-synthetic-smoke-v5` with generation-only timing and
+generated-token totals. Do not alter old receipts to make them
 pass or imply that a local SHA-256 digest authenticates a declared
 upstream revision, provenance, permissions or loaded model state. Independent
 checkpoint review, safe isolation, and actual inference/quality checks

model/train/compare_smoke.py

@@ -141,6 +141,29 @@ def checked_latency(value) -> bool:
                 raise SmokeReportRejected("invalid-generation-language-latency")
         elif group["p50Ms"] is not None or group["p95Ms"] is not None:
             raise SmokeReportRejected("unexpected-generation-language-latency")
+    if report.get("generatedTokenAccounting") != "new-token-ids-including-stop-v1":
+        raise SmokeReportRejected("invalid-generated-token-method")
+    total = report.get("generatedTokenCount")
+    if (type(total) is not int or
+            not attempts <= total <= attempts * tokens):
+        raise SmokeReportRejected("invalid-generated-token-total")
+    token_groups = report.get("generatedTokensByLanguage")
+    if not isinstance(token_groups, dict) or set(token_groups) != set(LANGUAGES):
+        raise SmokeReportRejected("invalid-generated-token-groups")
+    counted_tokens = 0
+    for name in LANGUAGES:
+        group = token_groups[name]
+        expected_count = groups[name]["attempts"]
+        if (not isinstance(group, dict) or
+                set(group) != {"attempts", "totalTokens"} or
+                type(group["attempts"]) is not int or
+                group["attempts"] != expected_count or
+                type(group["totalTokens"]) is not int or
+                not expected_count <= group["totalTokens"] <= expected_count * tokens):
+            raise SmokeReportRejected("invalid-generated-token-language")
+        counted_tokens += group["totalTokens"]
+    if counted_tokens != total:
+        raise SmokeReportRejected("mismatched-generated-token-total")
     return report
 
 
@@ -163,11 +186,16 @@ def compare_reports(base: dict, adapter: dict) -> dict:
         after = right["perLanguage"][language]["strictExactMatches"]
         left_timing = left["generationLatencyByLanguage"][language]
         right_timing = right["generationLatencyByLanguage"][language]
+        left_tokens = left["generatedTokensByLanguage"][language]["totalTokens"]
+        right_tokens = right["generatedTokensByLanguage"][language]["totalTokens"]
         by_language[language] = {
             "attemptsPerModel": left["perLanguage"][language]["attempts"],
             "baseStrictExactMatches": before,
             "adapterStrictExactMatches": after,
             "adapterMinusBase": after - before,
+            "baseGeneratedTokens": left_tokens,
+            "adapterGeneratedTokens": right_tokens,
+            "generatedTokenDelta": right_tokens - left_tokens,
             "baseGenerationP50Ms": left_timing["p50Ms"],
             "adapterGenerationP50Ms": right_timing["p50Ms"],
             "generationP50DeltaMs": (
@@ -182,13 +210,22 @@ def compare_reports(base: dict, adapter: dict) -> dict:
             ),
         }
     return {
-        "comparisonProtocol": "lineloom-hf-synthetic-comparison-v3",
+        "comparisonProtocol": "lineloom-hf-synthetic-comparison-v4",
         "benchmarkProtocol": PROTOCOL,
         "scope": left["selectedLanguage"],
         "syntheticAttemptsPerModel": left["syntheticAttempts"],
         "baseStrictExactMatches": left["strictExactMatches"],
         "adapterStrictExactMatches": right["strictExactMatches"],
         "adapterMinusBase": delta,
+        "generatedTokenAccounting": "new-token-ids-including-stop-v1",
+        "baseGeneratedTokens": left["generatedTokenCount"],
+        "adapterGeneratedTokens": right["generatedTokenCount"],
+        "generatedTokenDelta": right["generatedTokenCount"] - left["generatedTokenCount"],
+        "generatedTokenCountStatus": (
+            "AGGREGATE_TOKEN_TOTALS_MATCH_ONLY"
+            if left["generatedTokenCount"] == right["generatedTokenCount"]
+            else "DIFFERENT_GENERATED_TOKEN_TOTALS"
+        ),
         "latencyMeasurement": "monotonic-generation-only-v1",
         "baseGenerationP50Ms": left["generationP50Ms"],
         "adapterGenerationP50Ms": right["generationP50Ms"],
@@ -202,7 +239,9 @@ def compare_reports(base: dict, adapter: dict) -> dict:
                        "do not verify real weights, compute environment, code "
                        "correctness, hardware parity, editor latency, acceptance "
                        "or release readiness. Generation-only timings are "
-                       "descriptive, and small samples are unstable."),
+                       "descriptive, and small samples are unstable. "
+                       "Different generated token counts confound absolute latency; "
+                       "even equal totals do not prove per-probe length parity."),
     }
 
 

model/train/smoke.py

@@ -184,7 +184,8 @@ def run_local(fixture_set: list[dict], args) -> dict:
             completion = tokenizer.decode(tokens, skip_special_tokens=False)
             exact = normalize_completion(completion) == fixture["expected"]
             result.append({"language": fixture["language"], "exact": exact,
-                           "generationMs": generation_ms})
+                           "generationMs": generation_ms,
+                           "generationTokens": len(tokens)})
     verify_smoke_inputs_unchanged(
         base_dir, base_identity,
         Path(args.adapter_dir) if args.adapter_dir else None,
@@ -207,7 +208,7 @@ def run_local(fixture_set: list[dict], args) -> dict:
         "baseWeightFiles": base_identity["baseWeightFiles"],
         "maxNewTokens": args.max_new_tokens,
         **summarize(result),
-        **generation_latency_summary(result),
+        **generation_latency_summary(result, args.max_new_tokens),
         "limitation": "Strict original-synthetic equality only; no human, compiler, editor or deployment QA."
     }
 

model/train/smoke_report.py

@@ -11,7 +11,7 @@
 from pathlib import Path
 
 LANGUAGES = ("rust", "c", "cpp", "python", "typescript")
-PROTOCOL = "lineloom-hf-synthetic-smoke-v4"
+PROTOCOL = "lineloom-hf-synthetic-smoke-v5"
 GENERATION_POLICY = "greedy-fim-v2"
 PUBLIC_FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "completion-v1.json"
 
@@ -97,24 +97,32 @@ def fingerprints(fixtures: list[dict], max_new_tokens: int) -> dict:
 
 
 
-def generation_latency_summary(results: list[dict]) -> dict:
-    """Aggregate completed local generations; never expose per-probe timings.
+def generation_latency_summary(results: list[dict], max_new_tokens: int = 128) -> dict:
+    """Source-free generation timing and actual output-length accounting.
 
-    Duration covers model.generate only. It excludes model loading, prompt
-    tokenization and decoding, so it is NOT end-to-end editor latency.
-    CUDA callers must synchronize around generation before recording times.
+    Count actual generated token IDs, including a terminal stop token when
+    present, not UTF-8 characters or decoded text. This avoids interpreting
+    a slower but longer completion as a like-for-like latency regression.
+    Duration covers model.generate only; it is NOT editor latency.
     """
-    if not isinstance(results, list) or not 1 <= len(results) <= 30:
+    if (not isinstance(results, list) or not 1 <= len(results) <= 30 or
+            type(max_new_tokens) is not int or not 1 <= max_new_tokens <= 128):
         raise SmokeReportRejected("invalid-generation-measurements")
     buckets: dict[str, list[int]] = {language: [] for language in LANGUAGES}
+    token_buckets: dict[str, list[int]] = {language: [] for language in LANGUAGES}
     all_times: list[int] = []
     for item in results:
         if not isinstance(item, dict) or item.get("language") not in buckets:
             raise SmokeReportRejected("invalid-generation-language")
         duration = item.get("generationMs")
+        generated_tokens = item.get("generationTokens")
         if type(duration) is not int or not 0 <= duration <= 3_600_000:
             raise SmokeReportRejected("invalid-generation-duration")
+        if (type(generated_tokens) is not int or
+                not 1 <= generated_tokens <= max_new_tokens):
+            raise SmokeReportRejected("invalid-generation-token-count")
         buckets[item["language"]].append(duration)
+        token_buckets[item["language"]].append(generated_tokens)
         all_times.append(duration)
 
     def nearest_rank(values: list[int], fraction: float) -> int | None:
@@ -134,6 +142,14 @@ def nearest_rank(values: list[int], fraction: float) -> int | None:
                 "p95Ms": nearest_rank(buckets[language], 0.95),
             } for language in LANGUAGES
         },
+        "generatedTokenAccounting": "new-token-ids-including-stop-v1",
+        "generatedTokenCount": sum(sum(values) for values in token_buckets.values()),
+        "generatedTokensByLanguage": {
+            language: {
+                "attempts": len(token_buckets[language]),
+                "totalTokens": sum(token_buckets[language]),
+            } for language in LANGUAGES
+        },
     }