diff --git a/NOTICE b/NOTICE index 2f234bd..6129875 100644 --- a/NOTICE +++ b/NOTICE @@ -16,11 +16,22 @@ license text for CC-BY-4.0 is available at: What this means in practice --------------------------- -Our spec-driven workflow files in this repository are an original, -lean reimplementation written in our own words. They are inspired by -the ideas above, not a copy of the original skill text. No content -was pasted from the original SKILL.md or its reference files. +Our prompt prose (plugin/prompts/_partials/spec-gates.md, +plugin/prompts/spec-workflow.md, and the orchestrator wording that cites +them) is an original, lean reimplementation written in our own words. No +content was pasted from the original SKILL.md or its reference files. -Where the CC-BY-4.0 attribution duty applies (the original skill -text itself), credit stays with Felipe Rodrigues as stated above. +The gate scripts in scripts/spec-gates/ (validate_spec.py, +validate_tasks.py, check_commit.py, validate_state.py) are adapted from +the scripts shipped with that skill. Adaptations: required sections and +commit types fitted to this repo's conventions (capitalized subjects, +ref/meta/license/revert types), the traceability-ID check dropped, the +CLI takes feature names only (scripts resolve paths from directory +listings and never open caller-supplied paths), and invocation from the +repo root. The originals remain CC-BY-4.0 by Felipe +Rodrigues as stated above; our adaptations are released under the MIT +License in LICENSE. + +Where the CC-BY-4.0 attribution duty applies (the original skill text +and scripts), credit stays with Felipe Rodrigues as stated above. Where our own files apply, the MIT License in LICENSE governs. diff --git a/plugin/agents/orchestrator.md b/plugin/agents/orchestrator.md index a0b783a..fad426f 100644 --- a/plugin/agents/orchestrator.md +++ b/plugin/agents/orchestrator.md @@ -22,10 +22,15 @@ tools: Bash ## Spec gates -- Ask for testable specs: each acceptance criterion holds one behavior, names a SHALL, and uses the shape that fits (WHEN trigger THEN response, WHILE state, WHERE flag, IF fault THEN handling, or a plain invariant). Send back criteria that bundle two behaviors or use vague words with no measurable outcome. -- Ask for gated tasks: each task points to its spec requirement and fills Tests plus Gate. Tests ship inside the task that writes the code, never parked in a later task. Tests none holds only for a layer the coverage matrix marks none. -- Ask for a short coverage matrix before Execute: one row per code layer touched, with test type, where the tests live, and the command that runs them. Treat the confirmed matrix as the authority for the run. -- Close each slice with proof: the spec named tests pass, plus one behavior fault in a scratch copy that the tests catch. Discard the scratch. Log kills and survivors in the closing report; survivors turn into fix slices. +These gates are fixed. They run on every feature slice, on every harness, whether or not any skill is available. No model judgment exempts them. + +- Spec gate: no Execute without a spec holding a goal, acceptance criteria with SHALL (one behavior each), and out-of-scope. Run `python3 scripts/spec-gates/validate_spec.py ` from the repo root before confirming the spec. Non-zero exit means fix first. +- Task gate: every task points to its spec requirement and fills Tests plus Gate. Tests ship inside the task that writes the code, never parked in a later task. Tests none holds only when every touched layer is marked none in the matrix; otherwise test to the strongest type among the touched layers. Run `python3 scripts/spec-gates/validate_tasks.py ` before approving tasks. +- Coverage matrix before Execute: one row per code layer touched, with test type, where the tests live, and the command that runs them. Treat the confirmed matrix as the authority for the run. +- Slice close: the spec-named tests pass, plus one behavior fault in a scratch copy that the tests catch. Discard the scratch. Log kills and survivors in the closing report; survivors turn into fix slices. A done feature carries a validation report with PASS and file:line evidence, checked by `python3 scripts/spec-gates/validate_state.py `. +- Decisions: record what you sized, what you scoped out, and what the probes killed. They land in the closing report in one batch, never as questions mid-run. + +Shapes, tables, and the verifier procedure live in `plugin/prompts/spec-workflow.md`. Read it when writing specs, tasks, or validation reports. ## Reports @@ -92,7 +97,7 @@ You are the CodeDeck orchestrator, and you run on the most capable and most expe - Drive the whole run without being asked for each phase. The human asked for the outcome once. Phase transitions are your call, so never pause between them for confirmation. - Size it from the request, then commit to the size. Trivial (a couple of files, an obvious change): straight to implement plus verify plus the final review round. Anything shaped like a feature: the full loop below. - Specify: dispatch a worker to write `.specs/features//spec.md` with the goal, the acceptance criteria, and what is out of scope. Design and Tasks go the same way when the work needs them: `design.md` for architecture calls, `tasks.md` for atomic tasks that each carry their Tests and Gate. You cannot write files, so workers write every artifact and you track each one in the registry. -- Execute: dispatch the tasks in dependency order. Every briefing names the spec and task files as the source of truth, and tells the worker to activate the `tlc-spec-driven` skill by name when its harness offers it, otherwise to follow the lean briefing steps exactly without pasting skill text. +- Execute: dispatch the tasks in dependency order. Every briefing names the spec and task files as the source of truth, and tells the worker to activate the `tlc-spec-driven` skill by name when its harness offers it, otherwise to follow the lean briefing steps exactly without pasting skill text. Gates are code, not memory: the worker runs `python3 scripts/spec-gates/validate_spec.py` before confirming a spec and `validate_tasks.py` before approving tasks, and reads `plugin/prompts/spec-workflow.md` for shapes. - Verify: a slice is done only when its spec-named tests pass and a bounded mutation probe passes with them. The probe: the worker injects a handful of behavior-level faults in scratch copies, confirms the tests kill each one, discards the scratch, and reports kills plus survivors. Survivors become fix slices, not excuses. - Review: run the final round yourself with `codedeck run --role reviewer --no-worktree "" --bg --json` over the finished scope. Slice self-review never replaces it. Remediate every confirmed finding as a new slice, then at most one re-review. After that, report whatever still stands instead of looping. - Record decisions as you go: what you sized, what you scoped out, what the probes killed. They land in the closing report in one batch, never as questions mid-run. diff --git a/plugin/prompts/_partials/spec-gates.md b/plugin/prompts/_partials/spec-gates.md index 40a0e45..7f039c6 100644 --- a/plugin/prompts/_partials/spec-gates.md +++ b/plugin/prompts/_partials/spec-gates.md @@ -1,6 +1,11 @@ ## Spec gates -- Ask for testable specs: each acceptance criterion holds one behavior, names a SHALL, and uses the shape that fits (WHEN trigger THEN response, WHILE state, WHERE flag, IF fault THEN handling, or a plain invariant). Send back criteria that bundle two behaviors or use vague words with no measurable outcome. -- Ask for gated tasks: each task points to its spec requirement and fills Tests plus Gate. Tests ship inside the task that writes the code, never parked in a later task. Tests none holds only for a layer the coverage matrix marks none. -- Ask for a short coverage matrix before Execute: one row per code layer touched, with test type, where the tests live, and the command that runs them. Treat the confirmed matrix as the authority for the run. -- Close each slice with proof: the spec named tests pass, plus one behavior fault in a scratch copy that the tests catch. Discard the scratch. Log kills and survivors in the closing report; survivors turn into fix slices. +These gates are fixed. They run on every feature slice, on every harness, whether or not any skill is available. No model judgment exempts them. + +- Spec gate: no Execute without a spec holding a goal, acceptance criteria with SHALL (one behavior each), and out-of-scope. Run `python3 scripts/spec-gates/validate_spec.py ` from the repo root before confirming the spec. Non-zero exit means fix first. +- Task gate: every task points to its spec requirement and fills Tests plus Gate. Tests ship inside the task that writes the code, never parked in a later task. Tests none holds only when every touched layer is marked none in the matrix; otherwise test to the strongest type among the touched layers. Run `python3 scripts/spec-gates/validate_tasks.py ` before approving tasks. +- Coverage matrix before Execute: one row per code layer touched, with test type, where the tests live, and the command that runs them. Treat the confirmed matrix as the authority for the run. +- Slice close: the spec-named tests pass, plus one behavior fault in a scratch copy that the tests catch. Discard the scratch. Log kills and survivors in the closing report; survivors turn into fix slices. A done feature carries a validation report with PASS and file:line evidence, checked by `python3 scripts/spec-gates/validate_state.py `. +- Decisions: record what you sized, what you scoped out, and what the probes killed. They land in the closing report in one batch, never as questions mid-run. + +Shapes, tables, and the verifier procedure live in `plugin/prompts/spec-workflow.md`. Read it when writing specs, tasks, or validation reports. diff --git a/plugin/prompts/roles/orchestrator.md b/plugin/prompts/roles/orchestrator.md index bb39f4d..5714e4d 100644 --- a/plugin/prompts/roles/orchestrator.md +++ b/plugin/prompts/roles/orchestrator.md @@ -69,7 +69,7 @@ You are the CodeDeck orchestrator, and you run on the most capable and most expe - Drive the whole run without being asked for each phase. The human asked for the outcome once. Phase transitions are your call, so never pause between them for confirmation. - Size it from the request, then commit to the size. Trivial (a couple of files, an obvious change): straight to implement plus verify plus the final review round. Anything shaped like a feature: the full loop below. - Specify: dispatch a worker to write `.specs/features//spec.md` with the goal, the acceptance criteria, and what is out of scope. Design and Tasks go the same way when the work needs them: `design.md` for architecture calls, `tasks.md` for atomic tasks that each carry their Tests and Gate. You cannot write files, so workers write every artifact and you track each one in the registry. -- Execute: dispatch the tasks in dependency order. Every briefing names the spec and task files as the source of truth, and tells the worker to activate the `tlc-spec-driven` skill by name when its harness offers it, otherwise to follow the lean briefing steps exactly without pasting skill text. +- Execute: dispatch the tasks in dependency order. Every briefing names the spec and task files as the source of truth, and tells the worker to activate the `tlc-spec-driven` skill by name when its harness offers it, otherwise to follow the lean briefing steps exactly without pasting skill text. Gates are code, not memory: the worker runs `python3 scripts/spec-gates/validate_spec.py` before confirming a spec and `validate_tasks.py` before approving tasks, and reads `plugin/prompts/spec-workflow.md` for shapes. - Verify: a slice is done only when its spec-named tests pass and a bounded mutation probe passes with them. The probe: the worker injects a handful of behavior-level faults in scratch copies, confirms the tests kill each one, discards the scratch, and reports kills plus survivors. Survivors become fix slices, not excuses. - Review: run the final round yourself with `codedeck run --role reviewer --no-worktree "" --bg --json` over the finished scope. Slice self-review never replaces it. Remediate every confirmed finding as a new slice, then at most one re-review. After that, report whatever still stands instead of looping. - Record decisions as you go: what you sized, what you scoped out, what the probes killed. They land in the closing report in one batch, never as questions mid-run. diff --git a/plugin/prompts/spec-workflow.md b/plugin/prompts/spec-workflow.md new file mode 100644 index 0000000..cead4d1 --- /dev/null +++ b/plugin/prompts/spec-workflow.md @@ -0,0 +1,97 @@ +# Spec Workflow Reference + +On-demand companion to the Spec gates partial. The partial holds the fixed +gates; this file holds the shapes. Read the section you need when writing +specs, tasks, or validation reports. This file is not inlined into any prompt. + +## EARS acceptance criteria + +One behavior per criterion, always with SHALL, concrete values instead of +vague words (a status code, a message, a bound; never "quickly" or +"gracefully"). + +| Shape | Template | Use for | +| ----- | -------- | ------- | +| Invariant | The [system] SHALL [response] | Always-on constraints | +| Event | WHEN [trigger] THEN the [system] SHALL [response] | Response to a discrete trigger | +| State | WHILE [state] the [system] SHALL [response] | Behavior that holds during a state | +| Optional | WHERE [flag] the [system] SHALL [response] | Behavior behind a flag or capability | +| Fault | IF [bad condition] THEN the [system] SHALL [handling] | Errors, invalid input, timeouts | + +Send back criteria that bundle two behaviors or use vague words with no +measurable outcome. `validate_spec.py` flags any criterion without SHALL. + +## Assumptions table + +Every ambiguity is resolved with the human or recorded here. Nothing proceeds +unmarked. + +| Assumption / decision | Chosen default | Rationale | +| --------------------- | -------------- | --------- | +| [ambiguity] | [what we do] | [why] | + +An empty Chosen default or Rationale cell fails `validate_spec.py`. + +## Task granularity + +One task is one deliverable: one component, one function, one endpoint, one +file change. Two or three cohesive things in one file are acceptable; multiple +files or components mean split. A `Where` naming several files is a smell. + +Each task carries: What (one sentence), Where (file), Depends on (task ids or +None), Requirement (spec id or story), Done when (binary checkboxes), Tests +(unit, integration, e2e, or none per the matrix), Gate (quick, full, or +build), and the planned commit message. + +Dependencies point backward or within the same phase, never to a later phase. +The execution diagram must match every `Depends on` and vice versa. +`validate_tasks.py` checks fields, direction, and diagram parity. + +No tasks.md yet and more than 5 steps ahead: stop and write tasks.md first. +Three or fewer obvious steps may stay inline as an execution plan. + +## Coverage matrix and gates + +Built from the repo before Execute: sample existing tests for style and +location, read the real commands from package manifests and CI config, never +invent them. + +| Code Layer | Required Test Type | Location Pattern | Run Command | +| ---------- | ------------------ | ---------------- | ----------- | +| [layer] | [unit/integration/e2e/none] | [glob or path] | [command] | + +Quick gate after unit-only tasks, full gate after integration or e2e tasks, +build gate (build plus lint plus tests) after phases and config-only tasks. +`Tests: none` is valid only for a layer the matrix marks none. + +## Verifier procedure + +Validation is the closing step of Execute, never a prompt away. A fresh pair +of eyes re-derives coverage from the spec; the author never verifies alone. + +1. Re-anchor every AC to its spec-defined outcome and confirm the test asserts + that exact outcome, citing `file:line` plus the assertion. No citation + means not covered. Vague spec outcomes get flagged, never silently passed. +2. Run the build-level gate. Non-zero exit stops everything. +3. Inject 1 to 3 behavior faults (flipped condition, wrong return, off-by-one, + removed side effect) in scratch copies only, never the real tree (`git + stash` is forbidden here), confirm the tests kill each one, discard the + scratch, and confirm the real tree matches its pre-sensor baseline. + Survivors become fix tasks. +4. Write `.specs/features//validation.md` with PASS or FAIL, per-AC + evidence, sensor kills plus survivors, and the diff range. Gaps become fix + tasks; after 3 fix and re-verify rounds, escalate to the human. +5. Run `validate_state.py`. It demands a filled PASS plus file:line evidence. + +## Decision log + +Batch into the closing report: what you sized, what you scoped out, what the +probes killed, with the reason each time. One batch at the end, no questions +mid-run. + +## Batch rule + +More than about 8 tasks means offering split workers: consecutive whole +phases per worker, sequential batches, each reporting tasks done, commit +hashes, test counts, and deviations before the next starts. Eight or fewer +runs inline. diff --git a/scripts/spec-gates/_gate_io.py b/scripts/spec-gates/_gate_io.py new file mode 100644 index 0000000..8308b03 --- /dev/null +++ b/scripts/spec-gates/_gate_io.py @@ -0,0 +1,28 @@ +"""_gate_io.py - shared file reader for the spec-gate scripts. + +Every gate opens only paths built from directory-listing entries, never +from CLI text; this helper adds the runtime backstop. Pure standard +library, zero dependencies. Imported by sibling scripts, so run them as +files (`python3 scripts/spec-gates/validate_spec.py ...`) with the repo +root as cwd. +""" + +import os +import sys + + +def read_text_file(path, root): + """Read a UTF-8 text file, refusing paths that escape root. + + Exits 2 on refusal or when the target is not a file. + """ + base = os.path.realpath(root) + target = os.path.realpath(path) + if os.path.commonpath([base, target]) != base: + print(f"refusing to read outside project root {base}: {path}", file=sys.stderr) + raise SystemExit(2) + if not os.path.isfile(target): + print(f"not a file: {path}", file=sys.stderr) + raise SystemExit(2) + with open(target, "r", encoding="utf-8", errors="replace") as f: + return f.read() diff --git a/scripts/spec-gates/check_commit.py b/scripts/spec-gates/check_commit.py new file mode 100755 index 0000000..e98f676 --- /dev/null +++ b/scripts/spec-gates/check_commit.py @@ -0,0 +1,120 @@ +#!/usr/bin/env python3 +"""check_commit.py - deterministic commit-message validation. + +Adapted for CodeDeck from the tlc-spec-driven skill (Felipe Rodrigues, +github.com/felipfr, CC-BY-4.0, https://creativecommons.org/licenses/by/4.0/). +Changes from the original: commit types extended to the ones this repo uses +(ref, meta, license, revert); the description must start UPPERCASE (our +convention capitalizes it); the length warning follows our 70-char subject +rule; the message arrives via --message or stdin, never via a file path, +so no caller-controlled text reaches the filesystem. Pure standard +library, zero dependencies. + +It reads the message from --message or stdin. As a git `commit-msg` hook, +wrap the call: + + #!/bin/sh + exec python3 scripts/spec-gates/check_commit.py --message "$(cat "$1")" + +What it checks: + ERROR - header does not match type(scope): description + ERROR - type is not one of the repo's commit types + ERROR - description is empty, starts lowercase, or ends with a period + ERROR - `!` breaking marker present but no `BREAKING CHANGE:` footer + WARN - header longer than 70 characters + +Usage: + python3 scripts/spec-gates/check_commit.py --message "feat(auth): Add email validation" + echo "fix(cart): Prevent negative quantity" | python3 scripts/spec-gates/check_commit.py + +Exit codes: 0 pass, 1 violation, 2 usage error. +""" + +import argparse +import re +import sys + +TYPES = ["feat", "fix", "ref", "perf", "docs", "test", "style", "build", "ci", "chore", "meta", "license", "revert"] +HEADER_RE = re.compile(r"^(?P\w+)(?:\((?P[^)]+)\))?(?P!)?: (?P.+)$") + + +def read_message(args): + # Text only, never a file path: the message arrives via --message or + # stdin, so no caller-controlled text reaches the filesystem. A git + # hook wraps the call, e.g.: + # exec python3 scripts/spec-gates/check_commit.py --message "$(cat "$1")" + if args.message is not None: + return args.message + if args.msgfile: + print("check_commit: pass the message via --message or stdin, not a file path.", file=sys.stderr) + raise SystemExit(2) + if not sys.stdin.isatty(): + return sys.stdin.read() + return "" + + +def check(message): + errors, warnings = [], [] + # Ignore comment lines (git puts '#' comments in the message file). + lines = [ln for ln in message.splitlines() if not ln.lstrip().startswith("#")] + while lines and not lines[0].strip(): + lines.pop(0) + if not lines: + return (["empty commit message"], warnings) + + header = lines[0].rstrip() + if len(header) > 70: + warnings.append(f"header is {len(header)} chars (>70): {header[:60]}...") + + m = HEADER_RE.match(header) + if not m: + errors.append(f"header does not match 'type(scope): description': {header!r}") + return (errors, warnings) + + ctype = m.group("type") + desc = m.group("desc") + bang = m.group("bang") + + if ctype not in TYPES: + errors.append(f"type '{ctype}' is not one of: {', '.join(TYPES)}") + if not desc.strip(): + errors.append("description is empty") + else: + if desc[:1].islower(): + errors.append(f"description should start uppercase: '{desc[:30]}'") + if desc.rstrip().endswith("."): + errors.append("description should not end with a period") + + body = "\n".join(lines[1:]) + breaking_footer = bool(re.search(r"^BREAKING CHANGE:", body, re.MULTILINE)) + if bang and not breaking_footer: + errors.append("'!' breaking marker present but no 'BREAKING CHANGE:' footer") + + return (errors, warnings) + + +def main(argv=None): + p = argparse.ArgumentParser(prog="check_commit.py", description="Validate a commit message.") + p.add_argument("msgfile", nargs="?", default=None, help="path to a commit message file (as git passes to commit-msg)") + p.add_argument("--message", default=None, help="the commit message as a string") + args = p.parse_args(argv) + + message = read_message(args) + if not message.strip(): + print("check_commit: no message provided (pass a file, --message, or pipe via stdin).", file=sys.stderr) + return 2 + + errors, warnings = check(message) + for w in warnings: + print(f" WARN {w}") + for e in errors: + print(f" ERROR {e}") + if errors: + print("\ncheck_commit: FAIL") + return 1 + print("check_commit: OK") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/spec-gates/validate_spec.py b/scripts/spec-gates/validate_spec.py new file mode 100755 index 0000000..1755be2 --- /dev/null +++ b/scripts/spec-gates/validate_spec.py @@ -0,0 +1,235 @@ +#!/usr/bin/env python3 +"""validate_spec.py - deterministic closure-gate checks for a feature spec.md. + +Adapted for CodeDeck from the tlc-spec-driven skill (Felipe Rodrigues, +github.com/felipfr, CC-BY-4.0, https://creativecommons.org/licenses/by/4.0/). +Changes from the original: required sections fitted to the CodeDeck fixed +core (goal, acceptance criteria, out-of-scope); the Requirement +Traceability check was dropped (our specs do not all carry IDs); the +assumptions section matches either "Assumptions" or "Open questions"; +invoked from the repo root as `python3 scripts/spec-gates/validate_spec.py`; +opened paths are built from directory listings, never from CLI text. + +What it checks (heuristic markdown inspection, not a full parser): + ERROR - no goal section (Goal/Goals/Problem Statement) + ERROR - no acceptance-criteria section, or no numbered criteria in it + ERROR - an acceptance criterion has no SHALL (not testable) + ERROR - an Out of Scope section is missing + ERROR - an Assumptions row has an empty "Chosen default" or "Rationale" + WARN - an AC has SHALL but no recognizable EARS lead keyword + WARN - no assumptions/open-questions section at all + WARN - open questions are not explicitly resolved + +Usage: + python3 scripts/spec-gates/validate_spec.py [feature] [--strict] + + Run from the repo root. feature is a bare feature name under + .specs/features/ (never a path). Omitted -> auto-detect the single + feature. + --strict Treat warnings as errors. + +Exit codes: 0 pass, 1 errors found (or warnings under --strict), 2 usage error. +""" + +import argparse +import os +import re +import sys + + +from _gate_io import read_text_file as _read_text_file + + +GOAL_TITLES = {"goal", "goals", "problem statement"} +AC_TITLE = "acceptance criteria" +OUT_OF_SCOPE = "out of scope" + +PLACEHOLDER_RE = re.compile(r"^\s*\[.+\]\s*$") + + +NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_-]*$") +FEATURES_DIR = os.path.join(".specs", "features") + + +def resolve_spec(target): + """Return the spec.md path for a feature NAME. Run from the repo root. + + Names only, never paths: the returned path is built from a + directory-listing entry, so no caller-controlled text reaches the + filesystem. + """ + if target is not None and not NAME_RE.match(target): + print(f"validate_spec: pass a feature name (letters, digits, '-' and '_'), not a path: {target!r}", file=sys.stderr) + raise SystemExit(2) + if not os.path.isdir(FEATURES_DIR): + return None + entries = sorted(os.listdir(FEATURES_DIR)) + if target is None: + with_spec = [e for e in entries if os.path.isfile(os.path.join(FEATURES_DIR, e, "spec.md"))] + if len(with_spec) == 1: + return os.path.join(FEATURES_DIR, with_spec[0], "spec.md") + if not with_spec: + return None + raise SystemExit( + "validate_spec: multiple features found; pass one explicitly:\n " + + "\n ".join(with_spec) + ) + match = next((e for e in entries if e == target), None) + if match is None or not os.path.isfile(os.path.join(FEATURES_DIR, match, "spec.md")): + return None + return os.path.join(FEATURES_DIR, match, "spec.md") + + +def split_row(line): + return [c.strip() for c in line.strip().strip("|").split("|")] + + +def is_separator(line): + return bool(re.match(r"^\s*\|?[\s:|-]+\|?\s*$", line)) and "-" in line + + +def find_section(lines, predicate): + """Return (start, end) line indices for the first ## section matching.""" + start = None + for i, ln in enumerate(lines): + m = re.match(r"^#{1,3}\s+(.*?)\s*$", ln.strip()) + if m and predicate(m.group(1).strip().lower()): + start = i + 1 + break + if start is None: + return None + end = len(lines) + for j in range(start, len(lines)): + if re.match(r"^#{1,3}\s", lines[j]): + end = j + break + return (start, end) + + +def classify_ears(text): + """Return (ok, note). ok requires a SHALL; note records the EARS shape.""" + low = text.strip().lower() + if not re.search(r"\bshall\b", low): + return (False, "no SHALL") + kws = [] + if re.search(r"\bwhile\b", low): + kws.append("WHILE") + if re.search(r"\bwhen\b", low): + kws.append("WHEN") + if re.match(r"^\s*if\b", low) or re.search(r"\bif\b.*\bthen\b", low): + kws.append("IF/THEN") + if re.search(r"\bwhere\b", low): + kws.append("WHERE") + if len(kws) >= 2: + return (True, "complex (" + "+".join(kws) + ")") + if kws: + pattern = { + "WHILE": "state-driven", + "WHEN": "event-driven", + "IF/THEN": "unwanted-behavior", + "WHERE": "optional-feature", + }[kws[0]] + return (True, pattern) + if re.match(r"^\s*the\b", low): + return (True, "ubiquitous") + return (True, "warn: SHALL present but no EARS lead keyword") + + +def check(spec_path, root): + lines = _read_text_file(spec_path, root).splitlines() + errors, warnings = [], [] + + # 1. Goal section. + if find_section(lines, lambda t: t in GOAL_TITLES) is None: + errors.append("missing goal section (## Goal, ## Goals, or ## Problem Statement)") + + # 2. Acceptance criteria are numbered and SHALL-shaped. Criteria live + # in a ## section or under `**Acceptance Criteria**:` labels in stories. + ac = find_section(lines, lambda t: t == AC_TITLE) + ranges = [range(*ac)] if ac is not None else [] + for i, ln in enumerate(lines): + if re.match(r"^\*{0,2}Acceptance Criteria\*{0,2}\s*:?\s*$", ln.strip()): + ranges.append(range(i + 1, len(lines))) + items = [] + for r in ranges: + for i in r: + ln = lines[i] + m = re.match(r"^\s*\d+[.)]\s+(.*)$", ln) + if m: + items.append((i + 1, m.group(1).strip())) + continue + stripped = ln.strip() + if stripped == "": + continue # blank lines often sit between label and items + if re.match(r"^#{1,3}\s", ln) or stripped.startswith("**") or stripped.startswith("```"): + break + # ranges can overlap (a label inside the ## section); keep first hit. + items = list(dict.fromkeys(items)) + if not items: + errors.append("no acceptance criteria found (## Acceptance criteria section or **Acceptance Criteria** blocks with numbered items)") + else: + for lineno, item in items: + if PLACEHOLDER_RE.match(item): + continue + ok, note = classify_ears(item) + if not ok: + errors.append(f"L{lineno}: criterion has no SHALL (not testable): {item[:70]}") + elif note.startswith("warn"): + warnings.append(f"L{lineno}: SHALL but no EARS keyword (WHEN/WHILE/WHERE/IF or 'The ... shall'): {item[:60]}") + + # 3. Out of Scope. + if find_section(lines, lambda t: t == OUT_OF_SCOPE) is None: + errors.append("missing required section: ## Out of Scope") + + # 4. Assumptions / open questions. + b = find_section(lines, lambda t: "assumption" in t or "open question" in t) + if b is None: + warnings.append("no Assumptions / Open questions section (ambiguities have nowhere to land)") + else: + rows = [lines[i] for i in range(*b) if lines[i].strip().startswith("|")] + data = [r for r in rows if not is_separator(r)] + if data: + data = data[1:] # header row + for r in data: + cells = split_row(r) + if len(cells) < 3: + continue + assumption, chosen, rationale = cells[0], cells[1], cells[2] + if PLACEHOLDER_RE.match(assumption) and PLACEHOLDER_RE.match(chosen): + warnings.append("Assumptions table still contains template placeholder rows") + continue + if not chosen or PLACEHOLDER_RE.match(chosen): + errors.append(f"assumption '{assumption[:40]}' has empty 'Chosen default'") + if not rationale or PLACEHOLDER_RE.match(rationale): + errors.append(f"assumption '{assumption[:40]}' has empty 'Rationale'") + oq = [lines[i] for i in range(*b) if "open questions" in lines[i].lower()] + oq_clean = re.sub(r"[*_]", "", " ".join(oq)).lower() + if oq and not re.search(r"open questions.*:\s*none", oq_clean): + warnings.append("open questions do not read as resolved ('Open questions: none')") + + return errors, warnings + + +def main(argv=None): + p = argparse.ArgumentParser(prog="validate_spec.py", description="Closure-gate checks for a feature spec.md.") + p.add_argument("target", nargs="?", default=None) + p.add_argument("--strict", action="store_true") + args = p.parse_args(argv) + + spec = resolve_spec(args.target) + if not spec: + print("validate_spec: could not locate a spec.md. Pass a feature name and run from the project root.", file=sys.stderr) + return 2 + + errors, warnings = check(spec, os.path.abspath(".")) + for w in warnings: + print(f" WARN {w}") + for e in errors: + print(f" ERROR {e}") + fail = errors or (warnings and args.strict) + print(f"\nvalidate_spec: {len(errors)} error(s), {len(warnings)} warning(s) in {spec}") + return 1 if fail else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/spec-gates/validate_state.py b/scripts/spec-gates/validate_state.py new file mode 100755 index 0000000..0715f3a --- /dev/null +++ b/scripts/spec-gates/validate_state.py @@ -0,0 +1,154 @@ +#!/usr/bin/env python3 +"""validate_state.py - deterministic completion gate for a feature. + +Adapted for CodeDeck from the tlc-spec-driven skill (Felipe Rodrigues, +github.com/felipfr, CC-BY-4.0, https://creativecommons.org/licenses/by/4.0/). +Changes from the original: invocation from the repo root as +`python3 scripts/spec-gates/validate_state.py`; gate logic unchanged +(a done feature must have a real PASS validation.md with file:line +evidence, because our validation.md already follows that shape). + +It does NOT merely check that validation.md exists: a report that is +empty, still holds the template placeholder, or has no evidence fails. + +Operates only on the .specs/ markdown artifacts. No dependencies. Run +from the project root (the dir that contains .specs/). +Meant as the closing gate of Execute - not a manual step. + +Usage: + python3 scripts/spec-gates/validate_state.py [feature] + +Run from the repo root. feature is a bare feature name (never a path). + +Exit codes: 0 ok, 1 a completed feature is missing a real PASS report, + 2 usage error. +""" + +import argparse +import os +import re +import sys + +# A file:line citation: a path with an extension, then :. e.g. src/a.ts:42 +EVIDENCE_RE = re.compile(r"[\w./-]+\.[A-Za-z0-9]+:\d+") + + +from _gate_io import read_text_file as _read_text_file + + +def _verdict(text): + """Return 'pass', 'fail', 'unfilled', or None from a validation report.""" + lines = text.splitlines() + candidates = [ + ln for ln in lines + if re.search(r"^#{1,4}\s*validation\b", ln.strip(), re.IGNORECASE) + or re.search(r"\*{0,2}result\*{0,2}\s*:", ln.strip(), re.IGNORECASE) + ] + hay = " ".join(candidates) if candidates else text + has_pass = re.search(r"\bPASS\b", hay) is not None + has_fail = re.search(r"\bFAIL\b", hay) is not None + if has_pass and has_fail: + # Both present on the verdict line = unfilled template "[PASS | FAIL]". + return "unfilled" + if has_pass: + return "pass" + if has_fail: + return "fail" + return None + + +def _appears_complete(fdir, root): + """Conservative completeness heuristic for the cross-check mode.""" + if os.path.exists(os.path.join(fdir, "validation.md")): + return True + tasks = os.path.join(fdir, "tasks.md") + if not os.path.exists(tasks): + return False + body = _read_text_file(tasks, root) + if not re.search(r"^#{2,4}\s+T\d+\s*:", body, re.MULTILINE): + return False + if re.search(r"^\s*-\s*\[\s\]", body, re.MULTILINE): + return False # unchecked box remains -> still in progress + return True + + +def _check_feature(fdir, name, root): + """Return list of error strings for one feature (empty = pass).""" + errors = [] + vpath = os.path.join(fdir, "validation.md") + if not os.path.exists(vpath): + errors.append( + f"{name}: no validation.md - Execute is not done until validation " + f"is written and independent. Dispatch validation before marking done." + ) + return errors + text = _read_text_file(vpath, root) + verdict = _verdict(text) + if verdict is None: + errors.append(f"{name}: validation.md has no PASS/FAIL verdict (a prose-only report does not count)") + elif verdict == "unfilled": + errors.append(f"{name}: validation.md verdict is still the template placeholder '[PASS | FAIL]' - not filled") + elif verdict == "fail": + errors.append(f"{name}: validation.md verdict is FAIL - route the ranked gaps to fix tasks, then re-verify (feature is not done)") + if verdict == "pass" and not EVIDENCE_RE.search(text): + errors.append(f"{name}: validation.md is PASS but cites no file:line evidence - evidence-or-zero not satisfied") + return errors + + +NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_-]*$") +FEATURES_DIR = os.path.join(".specs", "features") + + +def _resolve(feature): + base = FEATURES_DIR + if feature is not None and not NAME_RE.match(feature): + print(f"validate_state: pass a feature name (letters, digits, '-' and '_'), not a path: {feature!r}", file=sys.stderr) + raise SystemExit(2) + if not os.path.isdir(base): + print(f"validate_state: no {base} directory - nothing to check.") + return [] + # Names only: directories below come from the listing, so no + # caller-controlled text reaches the filesystem. + entries = sorted(os.listdir(base)) + dirs = [e for e in entries if os.path.isdir(os.path.join(base, e))] + if feature: + # An explicit target is always honored; otherwise the gate would + # silently pass ("nothing to check") on exactly the feature named. + match = next((e for e in dirs if e == feature), None) + if match is None: + print(f"validate_state: feature not found: {feature}", file=sys.stderr) + raise SystemExit(2) + return [(os.path.join(base, match), match)] + if len(dirs) == 1: + return [(os.path.join(base, dirs[0]), dirs[0])] + if not dirs: + print("validate_state: no features under .specs/features/ - nothing to check.") + return [] + root = os.path.abspath(".") + picked = [(os.path.join(base, d), d) for d in dirs if _appears_complete(os.path.join(base, d), root)] + if not picked: + print("validate_state: no completed feature detected (all in progress) - nothing to gate.") + return picked + + +def main(argv=None): + p = argparse.ArgumentParser(prog="validate_state.py", description="Deterministic completion gate: a done feature must have a real PASS validation report.") + p.add_argument("feature", nargs="?", default=None, help="Feature name (default: sole feature, else cross-check all completed)") + args = p.parse_args(argv) + root = os.path.abspath(".") + + targets = _resolve(args.feature) + all_errors = [] + for fdir, name in targets: + all_errors += _check_feature(fdir, name, root) + + for e in all_errors: + print(f" ERROR {e}") + n = len(all_errors) + checked = ", ".join(name for _, name in targets) or "(none)" + print(f"\nvalidate_state: {n} error(s) across [{checked}]") + return 1 if n else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/spec-gates/validate_tasks.py b/scripts/spec-gates/validate_tasks.py new file mode 100755 index 0000000..d85cbea --- /dev/null +++ b/scripts/spec-gates/validate_tasks.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python3 +"""validate_tasks.py - deterministic pre-approval checks for a feature tasks.md. + +Adapted for CodeDeck from the tlc-spec-driven skill (Felipe Rodrigues, +github.com/felipfr, CC-BY-4.0, https://creativecommons.org/licenses/by/4.0/). +Changes from the original: invocation from the repo root as +`python3 scripts/spec-gates/validate_tasks.py`; check logic unchanged +because our tasks.md already follows the same shape (Test Coverage Matrix, +Gate Check Commands, Execution Plan, Task Breakdown, T-tasks with Tests +plus Gate, Depends on, fenced phase diagrams); every file read is confined +under the repo root (path-injection guard). + +What it checks (heuristic markdown inspection, not a full parser): + ERROR - a required section is missing + ERROR - a task is missing its `Tests` or `Gate` field + ERROR - a task depends on a task in a LATER phase (dependencies point back only) + ERROR - a dependency edge shown in the diagram has no matching `Depends on` + (and vice-versa) when both sides are parseable + WARN - a task's `Where` names multiple files (granularity smell -> split it) + WARN - a task says `Tests: none` (confirm the coverage matrix agrees) + WARN - the diagram could not be parsed confidently (cross-check skipped) + +Usage: + python3 scripts/spec-gates/validate_tasks.py [feature] [--strict] + + Run from the repo root. feature is a bare feature name under + .specs/features/ (never a path). Omitted -> auto-detect the single + feature. + --strict Treat warnings as errors. + +Exit codes: 0 pass, 1 errors found (or warnings under --strict), 2 usage error. +""" + +import argparse +import os +import re +import sys + + +from _gate_io import read_text_file as _read_text_file + + +REQUIRED_SECTIONS = ["Test Coverage Matrix", "Gate Check Commands", "Execution Plan", "Task Breakdown"] +TASK_RE = re.compile(r"^#{2,4}\s+(T\d+)\s*:", re.IGNORECASE) +EDGE_RE = re.compile(r"\bT\d+\b") +FILE_HINT_RE = re.compile(r"[\w./-]+\.\w{1,6}\b") + + +NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_-]*$") +FEATURES_DIR = os.path.join(".specs", "features") + + +def resolve_tasks(target): + """Return the tasks.md path for a feature NAME. Run from the repo root. + + Names only, never paths: the returned path is built from a + directory-listing entry, so no caller-controlled text reaches the + filesystem. + """ + if target is not None and not NAME_RE.match(target): + print(f"validate_tasks: pass a feature name (letters, digits, '-' and '_'), not a path: {target!r}", file=sys.stderr) + raise SystemExit(2) + if not os.path.isdir(FEATURES_DIR): + return None + entries = sorted(os.listdir(FEATURES_DIR)) + if target is None: + with_tasks = [e for e in entries if os.path.isfile(os.path.join(FEATURES_DIR, e, "tasks.md"))] + if len(with_tasks) == 1: + return os.path.join(FEATURES_DIR, with_tasks[0], "tasks.md") + if not with_tasks: + return None + raise SystemExit( + "validate_tasks: multiple features found; pass one explicitly:\n " + + "\n ".join(with_tasks) + ) + match = next((e for e in entries if e == target), None) + if match is None or not os.path.isfile(os.path.join(FEATURES_DIR, match, "tasks.md")): + return None + return os.path.join(FEATURES_DIR, match, "tasks.md") + + +def section_present(lines, name): + return any(re.match(r"^#{1,4}\s+" + re.escape(name) + r"\b", ln.strip()) for ln in lines) + + +def parse_tasks(lines): + """Return a dict: task_id -> {'deps': set, 'tests': str|None, 'gate': str|None, 'where': str}.""" + tasks = {} + current = None + for ln in lines: + m = TASK_RE.match(ln.strip()) + if m: + current = m.group(1).upper() + tasks[current] = {"deps": set(), "tests": None, "gate": None, "where": ""} + continue + if current is None: + continue + stripped = ln.strip() + dm = re.match(r"^\*{0,2}Depends on\*{0,2}\s*:\s*(.*)$", stripped, re.IGNORECASE) + if dm: + body = dm.group(1) + if "none" not in body.lower(): + for e in EDGE_RE.findall(body.upper()): + tasks[current]["deps"].add(e) + wm = re.match(r"^\*{0,2}Where\*{0,2}\s*:\s*(.*)$", stripped, re.IGNORECASE) + if wm: + tasks[current]["where"] = wm.group(1) + tm = re.match(r"^\*{0,2}Tests\*{0,2}\s*:\s*(.*)$", stripped, re.IGNORECASE) + if tm: + tasks[current]["tests"] = tm.group(1).strip() + gm = re.match(r"^\*{0,2}Gate\*{0,2}\s*:\s*(.*)$", stripped, re.IGNORECASE) + if gm: + tasks[current]["gate"] = gm.group(1).strip() + return tasks + + +def parse_phase_membership(lines): + """Map task_id -> phase index, read from '### Phase N' headers.""" + membership = {} + phase_idx = 0 + in_phase = False + for ln in lines: + pm = re.match(r"^#{2,4}\s+Phase\s+(\d+)", ln.strip(), re.IGNORECASE) + if pm: + phase_idx = int(pm.group(1)) + in_phase = True + continue + if in_phase: + # Map only on task headers (### Tn:), never on mere references. + hm = TASK_RE.match(ln.strip()) + if hm: + membership[hm.group(1).upper()] = phase_idx + return membership + + +def parse_diagram_edges(lines): + """Best-effort: parse 'Tx -> Ty' arrow chains from fenced blocks.""" + edges = set() + in_fence = False + found_any_arrow = False + for ln in lines: + if ln.strip().startswith("```"): + in_fence = not in_fence + continue + if not in_fence: + continue + norm = ln.replace("→", "->").replace("──", "-").replace("-", "-") + if "->" not in norm: + continue + segments = [s for s in re.split(r"->", norm)] + seq = [] + for seg in segments: + ids = EDGE_RE.findall(seg.upper()) + seq.append(ids[-1] if ids else None) + for a, b in zip(seq, seq[1:]): + if a and b: + edges.add((a, b)) + found_any_arrow = True + return edges, found_any_arrow + + +def check(tasks_path, root): + lines = _read_text_file(tasks_path, root).splitlines() + errors, warnings = [], [] + + for name in REQUIRED_SECTIONS: + if not section_present(lines, name): + errors.append(f"missing required section: ## {name}") + + tasks = parse_tasks(lines) + if not tasks: + warnings.append("no tasks (### T1: ...) parsed - is this file filled in?") + return errors, warnings + + for tid, t in tasks.items(): + if t["tests"] is None: + errors.append(f"{tid}: missing `Tests` field") + elif t["tests"].lower().startswith("none"): + warnings.append(f"{tid}: Tests: none - confirm the Test Coverage Matrix says 'none' for this layer") + if t["gate"] is None: + errors.append(f"{tid}: missing `Gate` field") + files = FILE_HINT_RE.findall(t["where"]) + if len(set(files)) > 1: + warnings.append(f"{tid}: `Where` names multiple files {sorted(set(files))} - granularity smell, consider splitting") + + membership = parse_phase_membership(lines) + for tid, t in tasks.items(): + p_here = membership.get(tid) + if p_here is None: + continue + for dep in t["deps"]: + p_dep = membership.get(dep) + if p_dep is not None and p_dep > p_here: + errors.append(f"{tid} (phase {p_here}) depends on {dep} (phase {p_dep}) - dependencies must point backward or within the same phase") + + edges, parsed = parse_diagram_edges(lines) + if not parsed: + warnings.append("diagram arrows not parsed confidently - diagram/definition cross-check skipped (verify by hand)") + else: + def intra_phase(a, b): + pa, pb = membership.get(a), membership.get(b) + if pa is None or pb is None: + return True + return pa == pb + + dep_edges = set() + for tid, t in tasks.items(): + for dep in t["deps"]: + dep_edges.add((dep, tid)) + only_in_diagram = {(a, b) for (a, b) in (edges - dep_edges) if intra_phase(a, b)} + only_in_defs = {(a, b) for (a, b) in (dep_edges - edges) if intra_phase(a, b)} + for a, b in sorted(only_in_diagram): + if a in tasks and b in tasks: + errors.append(f"diagram shows {a} -> {b} but {b} has no matching `Depends on: {a}`") + for a, b in sorted(only_in_defs): + errors.append(f"{b} declares `Depends on: {a}` but the diagram has no {a} -> {b} arrow") + + return errors, warnings + + +def main(argv=None): + p = argparse.ArgumentParser(prog="validate_tasks.py", description="Pre-approval checks for a feature tasks.md.") + p.add_argument("target", nargs="?", default=None) + p.add_argument("--strict", action="store_true") + args = p.parse_args(argv) + + tasks_path = resolve_tasks(args.target) + if not tasks_path: + print("validate_tasks: could not locate a tasks.md. Pass a feature name and run from the project root.", file=sys.stderr) + return 2 + + errors, warnings = check(tasks_path, os.path.abspath(".")) + for w in warnings: + print(f" WARN {w}") + for e in errors: + print(f" ERROR {e}") + fail = errors or (warnings and args.strict) + print(f"\nvalidate_tasks: {len(errors)} error(s), {len(warnings)} warning(s) in {tasks_path}") + return 1 if fail else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/__snapshots__/orchestrator-agents.test.ts.snap b/tests/__snapshots__/orchestrator-agents.test.ts.snap index 98bf335..337211e 100644 --- a/tests/__snapshots__/orchestrator-agents.test.ts.snap +++ b/tests/__snapshots__/orchestrator-agents.test.ts.snap @@ -25,10 +25,15 @@ tools: Bash ## Spec gates -- Ask for testable specs: each acceptance criterion holds one behavior, names a SHALL, and uses the shape that fits (WHEN trigger THEN response, WHILE state, WHERE flag, IF fault THEN handling, or a plain invariant). Send back criteria that bundle two behaviors or use vague words with no measurable outcome. -- Ask for gated tasks: each task points to its spec requirement and fills Tests plus Gate. Tests ship inside the task that writes the code, never parked in a later task. Tests none holds only for a layer the coverage matrix marks none. -- Ask for a short coverage matrix before Execute: one row per code layer touched, with test type, where the tests live, and the command that runs them. Treat the confirmed matrix as the authority for the run. -- Close each slice with proof: the spec named tests pass, plus one behavior fault in a scratch copy that the tests catch. Discard the scratch. Log kills and survivors in the closing report; survivors turn into fix slices. +These gates are fixed. They run on every feature slice, on every harness, whether or not any skill is available. No model judgment exempts them. + +- Spec gate: no Execute without a spec holding a goal, acceptance criteria with SHALL (one behavior each), and out-of-scope. Run \`python3 scripts/spec-gates/validate_spec.py \` from the repo root before confirming the spec. Non-zero exit means fix first. +- Task gate: every task points to its spec requirement and fills Tests plus Gate. Tests ship inside the task that writes the code, never parked in a later task. Tests none holds only when every touched layer is marked none in the matrix; otherwise test to the strongest type among the touched layers. Run \`python3 scripts/spec-gates/validate_tasks.py \` before approving tasks. +- Coverage matrix before Execute: one row per code layer touched, with test type, where the tests live, and the command that runs them. Treat the confirmed matrix as the authority for the run. +- Slice close: the spec-named tests pass, plus one behavior fault in a scratch copy that the tests catch. Discard the scratch. Log kills and survivors in the closing report; survivors turn into fix slices. A done feature carries a validation report with PASS and file:line evidence, checked by \`python3 scripts/spec-gates/validate_state.py \`. +- Decisions: record what you sized, what you scoped out, and what the probes killed. They land in the closing report in one batch, never as questions mid-run. + +Shapes, tables, and the verifier procedure live in \`plugin/prompts/spec-workflow.md\`. Read it when writing specs, tasks, or validation reports. ## Reports @@ -95,7 +100,7 @@ You are the CodeDeck orchestrator, and you run on the most capable and most expe - Drive the whole run without being asked for each phase. The human asked for the outcome once. Phase transitions are your call, so never pause between them for confirmation. - Size it from the request, then commit to the size. Trivial (a couple of files, an obvious change): straight to implement plus verify plus the final review round. Anything shaped like a feature: the full loop below. - Specify: dispatch a worker to write \`.specs/features//spec.md\` with the goal, the acceptance criteria, and what is out of scope. Design and Tasks go the same way when the work needs them: \`design.md\` for architecture calls, \`tasks.md\` for atomic tasks that each carry their Tests and Gate. You cannot write files, so workers write every artifact and you track each one in the registry. -- Execute: dispatch the tasks in dependency order. Every briefing names the spec and task files as the source of truth, and tells the worker to activate the \`tlc-spec-driven\` skill by name when its harness offers it, otherwise to follow the lean briefing steps exactly without pasting skill text. +- Execute: dispatch the tasks in dependency order. Every briefing names the spec and task files as the source of truth, and tells the worker to activate the \`tlc-spec-driven\` skill by name when its harness offers it, otherwise to follow the lean briefing steps exactly without pasting skill text. Gates are code, not memory: the worker runs \`python3 scripts/spec-gates/validate_spec.py\` before confirming a spec and \`validate_tasks.py\` before approving tasks, and reads \`plugin/prompts/spec-workflow.md\` for shapes. - Verify: a slice is done only when its spec-named tests pass and a bounded mutation probe passes with them. The probe: the worker injects a handful of behavior-level faults in scratch copies, confirms the tests kill each one, discards the scratch, and reports kills plus survivors. Survivors become fix slices, not excuses. - Review: run the final round yourself with \`codedeck run --role reviewer --no-worktree "" --bg --json\` over the finished scope. Slice self-review never replaces it. Remediate every confirmed finding as a new slice, then at most one re-review. After that, report whatever still stands instead of looping. - Record decisions as you go: what you sized, what you scoped out, what the probes killed. They land in the closing report in one batch, never as questions mid-run. diff --git a/tests/prompt-layers.test.ts b/tests/prompt-layers.test.ts index bbc9bb1..3a723f6 100644 --- a/tests/prompt-layers.test.ts +++ b/tests/prompt-layers.test.ts @@ -83,7 +83,7 @@ const ROLES = Object.keys(EXPECTED); const BUDGETS: Record = { general: 16384, - orchestrator: 14336, + orchestrator: 16384, "orchestrator-read": 8192, "orchestrator-edit": 8192, reviewer: 8192, diff --git a/tests/spec-gates.test.ts b/tests/spec-gates.test.ts new file mode 100644 index 0000000..8a34919 --- /dev/null +++ b/tests/spec-gates.test.ts @@ -0,0 +1,200 @@ +import { execFileSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const gatesDir = path.join(root, "scripts", "spec-gates"); + +function run(script: string, args: string[], cwd: string): { rc: number; out: string } { + try { + const out = execFileSync("python3", [path.join(gatesDir, script), ...args], { + cwd, + encoding: "utf8", + }); + return { rc: 0, out }; + } catch (e) { + const err = e as { status?: number; stdout?: string; stderr?: string }; + return { rc: err.status ?? 1, out: `${err.stdout ?? ""}${err.stderr ?? ""}` }; + } +} + +// A skeleton project root: .specs/features/demo with the given files. +// Gate scripts take feature NAMES and run with cwd at the project root; +// paths below are built from directory listings, never from CLI text. +function featureRoot(files: Record): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "spec-gates-")); + const fdir = path.join(dir, ".specs", "features", "demo"); + fs.mkdirSync(fdir, { recursive: true }); + for (const [name, body] of Object.entries(files)) { + fs.writeFileSync(path.join(fdir, name), body); + } + return dir; +} + +const GOOD_SPEC = `# Demo Specification + +## Problem Statement + +Something hurts. + +## Out of Scope + +| Feature | Reason | +| ------- | ------ | +| X | Later | + +## Assumptions & Open Questions + +| Assumption / decision | Chosen default | Rationale | +| --------------------- | -------------- | --------- | +| Store | SQLite | Zero deps | + +**Open questions:** none + +## User Stories + +### P1: Thing + +**Acceptance Criteria**: + +1. WHEN the user clicks THEN the system SHALL respond +2. The system SHALL stay up +`; + +describe("spec gates: validate_spec", () => { + it("passes a SHALL-shaped spec", () => { + const dir = featureRoot({ "spec.md": GOOD_SPEC }); + const r = run("validate_spec.py", ["demo"], dir); + expect(r.rc).toBe(0); + }); + + it("fails a criterion without SHALL", () => { + const bad = GOOD_SPEC.replace( + "1. WHEN the user clicks THEN the system SHALL respond", + "1. Clicking should work nicely", + ); + const dir = featureRoot({ "spec.md": bad }); + const r = run("validate_spec.py", ["demo"], dir); + expect(r.rc).toBe(1); + expect(r.out).toMatch(/no SHALL/); + }); +}); + +describe("spec gates: validate_tasks", () => { + const GOOD_TASKS = `# Demo Tasks + +## Test Coverage Matrix + +| Code Layer | Required Test Type | Location Pattern | Run Command | +| ---------- | ------------------ | ---------------- | ----------- | +| Service | unit | tests/*.test.ts | npx vitest run tests/a.test.ts | + +## Gate Check Commands + +| Gate Level | When to Use | Command | +| ---------- | ----------- | ------- | +| Quick | Unit-only tasks | npx vitest run tests/a.test.ts | + +## Execution Plan + +### Phase 1: Build + +\`\`\` +T1 -> T2 +\`\`\` + +## Task Breakdown + +### T1: Create service + +**What**: One service +**Where**: \`src/a.ts\` +**Depends on**: None +**Tests**: unit +**Gate**: quick + +### T2: Wire service + +**What**: Wiring +**Where**: \`src/b.ts\` +**Depends on**: T1 +**Tests**: unit +**Gate**: quick +`; + + it("passes well-formed tasks", () => { + const dir = featureRoot({ "tasks.md": GOOD_TASKS }); + const r = run("validate_tasks.py", ["demo"], dir); + expect(r.rc).toBe(0); + }); + + it("fails a task missing its Tests field", () => { + const bad = GOOD_TASKS.replace("**Tests**: unit\n**Gate**: quick\n\n### T2", "**Gate**: quick\n\n### T2"); + const dir = featureRoot({ "tasks.md": bad }); + const r = run("validate_tasks.py", ["demo"], dir); + expect(r.rc).toBe(1); + expect(r.out).toMatch(/missing `Tests` field/); + }); +}); + +describe("spec gates: check_commit", () => { + it("accepts a repo-style message", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "spec-gates-")); + const r = run("check_commit.py", ["--message", "feat(prompts): Add lean spec gates"], dir); + expect(r.rc).toBe(0); + }); + + it("rejects lowercase subjects and missing types", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "spec-gates-")); + const lower = run("check_commit.py", ["--message", "fix(x): add thing"], dir); + expect(lower.rc).toBe(1); + expect(lower.out).toMatch(/uppercase/); + const notype = run("check_commit.py", ["--message", "just some words"], dir); + expect(notype.rc).toBe(1); + }); + + it("refuses a file path instead of message text", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "spec-gates-")); + const r = run("check_commit.py", ["/tmp/anything.txt"], dir); + expect(r.rc).toBe(2); + }); +}); + +describe("spec gates: validate_state", () => { + it("passes a PASS report with file:line evidence", () => { + const dir = featureRoot({ + "validation.md": "# Demo Validation\n\n**Result**: PASS\n\nCovered by `src/a.ts:42`.", + }); + const r = run("validate_state.py", ["demo"], dir); + expect(r.rc).toBe(0); + }); + + it("fails a missing report", () => { + const dir = featureRoot({}); + const r = run("validate_state.py", ["demo"], dir); + expect(r.rc).toBe(1); + expect(r.out).toMatch(/no validation\.md/); + }); +}); + +describe("spec gates: names only, never paths", () => { + it("rejects traversal and absolute targets with usage errors", () => { + const dir = featureRoot({ "spec.md": GOOD_SPEC }); + for (const script of ["validate_spec.py", "validate_tasks.py", "validate_state.py"]) { + for (const target of ["../evil", "/abs/path", "a/b"]) { + const r = run(script, [target], dir); + expect(r.rc).toBe(2); + } + } + }); + + it("rejects unknown feature names without touching the disk", () => { + const dir = featureRoot({ "spec.md": GOOD_SPEC }); + const r = run("validate_state.py", ["nope"], dir); + expect(r.rc).toBe(2); + expect(r.out).toMatch(/not found/); + }); +});