Skip to content

fix(runner): validate step templates before provider calls - #40

Merged
sturlese merged 1 commit into
mainfrom
fix/bughunt-preflight-step-templates
Aug 29, 2026
Merged

fix(runner): validate step templates before provider calls#40
sturlese merged 1 commit into
mainfrom
fix/bughunt-preflight-step-templates

Conversation

@sturlese

@sturlese sturlese commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Summary

This bughunt fix validates every workflow step dependency before execution, so a malformed later prompt cannot cause an earlier provider call that leaves no evidence.

Bug

A workflow whose second or later step referenced an unavailable placeholder was accepted until that step rendered. Earlier steps had already called the provider by then, but the resulting VariableError remained intentionally unrecorded as a configuration error.

Root cause

Prompt templates were rendered lazily inside the provider-call loop. Only declared external variables were checked before execution; dependencies on prior step outputs were not preflighted.

Fix

  • Preflight all prompt templates with the same renderer used at execution time.
  • Make only supplied variables and outputs from earlier steps available during validation.
  • Reserve steps.* for runner-generated outputs so callers cannot spoof future dependencies.
  • Preserve the existing VariableError behavior while guaranteeing it occurs before provider resolution or calls.

Testing

  • Regression test confirmed the previous source calls the provider once before failing.
  • .venv/bin/python -m pytest tests/test_runner.py -q — 17 passed.
  • .venv/bin/python -m pytest --cov=flightdeck --cov-fail-under=85 — 248 passed, 94.23% coverage.
  • .venv/bin/ruff check src tests — passed.
  • Build Squad audit and contract re-review — approved with no remaining Medium/High findings.
  • Strong-model adversarial refutation — approved with no Medium/High findings.
  • Offline demo, ledger verification, and dashboard artifact check — passed.

Preflight every prompt dependency before execution so a malformed later step cannot trigger an unrecorded earlier provider call.
@sturlese
sturlese force-pushed the fix/bughunt-preflight-step-templates branch from 4010602 to 60aaee3 Compare August 29, 2026 22:21
@sturlese

Copy link
Copy Markdown
Owner Author

Adversarial gate: PASS

The first review found that caller-controlled steps.* variables could spoof future or nonexistent step outputs. The final patch reserves that namespace for runner-generated outputs and adds regressions for spoofing and provider resolution.

Final evidence for commit 60aaee3c3fea2bb643cfcf007da42ac6bd0a4f79:

  • Focused runner suite: 17 passed.
  • Full suite: 248 passed with 94.23% coverage.
  • Ruff, diff checks, offline demo, ledger verification, and dashboard artifact check: passed.
  • Build Squad audit re-review: APPROVE, no remaining Medium/High findings.
  • Build Squad contract review: APPROVE, including self/future/unknown references, valid chaining, redaction, and budget/policy precedence.
  • Strong-model final refutation: APPROVE, no Medium/High correctness findings.

UX review was skipped because the change preserves the existing error text and command flow; it only moves rejection ahead of provider resolution and calls.

@sturlese
sturlese merged commit de1d1cc into main Aug 29, 2026
5 checks passed
@sturlese
sturlese deleted the fix/bughunt-preflight-step-templates branch August 29, 2026 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant