fix(ci) make manual speed profiles trustworthy

[davide221]

Summary

• keep automatic PR speed profiles advisory
• make manual promotion runs fail when models, baseline, or profile artifacts are missing
• reject noisy manual measurements and performance regressions beyond the configured threshold
• document the runner-staged baseline and keep ambiguous losslessness divergences advisory

Why

The workflow defaulted to /opt/models, which was absent on the RTX runner, so profiles could finish green after skipping all measurements. It also had job-level continue-on-error: true, which made manual runs unable to act as promotion evidence.

The repository now points LUCEBOX_MODELS_DIR at the staged GGUF files and LUCEBOX_SPEED_BASELINE at a verified main profile stored outside the PR checkout.

Validation

• workflow YAML parses and git diff --check passes
• stable main profile passes the extracted validation step
• simulated regression plus noisy profile fails manual mode and remains advisory in PR mode
• simulated missing profile.json fails manual mode
• real RTX 3090 branch run passed with artifacts, stable noise, and no performance regression: https://github.com/Luce-Org/lucebox/actions/runs/33175970072

Correctness caveat

Main and this branch reproducibly diverge from greedy AR on has_close_elements at token 44 and sum_product at token 28. The profiler cannot yet classify those as near-tie floating-point flips versus an engine bug because the binaries do not expose the logit gap. The workflow keeps that signal visible as a warning rather than pretending it is resolved.

[cubic-dev-ai]

1 issue found across 2 files

_{You’re at about 90% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/workflows/speed-profile.yml">

<violation number="1" location=".github/workflows/speed-profile.yml:224">
P2: When `profile.json` contains a truthy `regression` object but lacks `summary.noise` or headline metrics, this manual gate accepts it as valid evidence. Validate the required report schema and metric values before applying the regression and noise checks.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When profile.json contains a truthy regression object but lacks summary.noise or headline metrics, this manual gate accepts it as valid evidence. Validate the required report schema and metric values before applying the regression and noise checks.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/speed-profile.yml, line 224:

<comment>When `profile.json` contains a truthy `regression` object but lacks `summary.noise` or headline metrics, this manual gate accepts it as valid evidence. Validate the required report schema and metric values before applying the regression and noise checks.</comment>

<file context>
@@ -197,6 +219,17 @@ jobs:
                     "Treat small deltas as below the profiler detection threshold.")
+          if os.environ.get("GITHUB_EVENT_NAME") == "workflow_dispatch":
+              failures = []
+              if not reg:
+                  failures.append("the baseline comparison is missing")
+              elif reg.get("regressed"):
</file context>