diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index a19e6b606..7eeccfad0 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -18766,6 +18766,113 @@ backlog alloc records before allocating** -- `workflow` 0, `concurren` 0, `fan-o `workflow` 191 lines in the ledger, an impossible string 0. **The `watch.py` margin history above is the filer's own read**, and it is the part that changes the row's ask. +**PROGRESS 2026-09-03 (builder) -- THE SELF-COUNT INSTRUMENT IS BUILT; THE AGGREGATE LIMB IS MEASURED +AND STILL OPEN.** Scope taken: the cost estimator this row's worked example demands, plus a measured +design note on the denominator. Not taken: the gate itself, which is not in this repository, and +lowering the threshold, which this row rules out by name. + +**THE GATE IS NOT IN THIS TREE, AND THE ZERO CARRIES ITS CONTROLS.** `git grep WORKFLOW_GATE_AT -- +':!docs/'` exits 1, and no file named `watch.py` exists anywhere on disk, tracked or untracked. A zero +from a grep is worthless without a positive control, so two ran on the same instrument over the same +path exclusion: `PUBLIC_BACKLOG_FLOOR` exits 0, an impossible string exits 1. The instrument can see a +real match outside `docs/` and does return 1 on a genuine absence, so the gate's absence is a finding +rather than a broken pattern. **The exit codes were re-taken unpiped after the first attempt read `$?` +through a `head`,** which reports the pager's status and not the grep's -- the CLAUDE.md section 11 +trap, hit while checking for it. + +**BUILT: `scripts/coord/workflow_cost.py`, tested by `tests/test_coord_workflow_cost.py`.** It reads a +workflow script statically and prices it as **fan width times depth** across every `parallel()`, +`pipeline()` and `.map()`, summing across phases. `pipeline(items, s1, s2, s3)` is the same arithmetic +said out loud -- every item runs every stage, so both factors sit outside the `agent()` calls. On this +row's worked example it returns **13 where the naive instrument returns 3**, reproducing the 4.3x +under-report from the numbers published above. + +**THE ORIGINAL SCRIPT WAS NOT RECOVERABLE, so the fixture is the SHAPE this row's three numbers +force**, and it is labelled as such in the source: two six-wide fans over a `const ROWS` of six, plus a +lone synthesis agent, giving 6 + 6 + 1 = 13 behind 3 call sites. Nothing here claims to be the original +bytes. + +**IT DISCRIMINATES, WHICH IS THE PART A BIGGER NUMBER DOES NOT PROVE.** A tool that always inflates +would satisfy the headline test, so a script with no fan-out is pinned to price at **exactly** the naive +count, and a script whose `agent(` appears only in a comment and a prompt string prices at 1 where the +naive grep returns 3. The arithmetic is mutation-proved three ways: delete the `* width` and the worked +example collapses back to 3 -- this row's own failure, reproduced on demand -- force every literal array +to one element and an inline fan drops to 1, and disable the honesty residue and an unrecognised +construct goes quiet. Each mutant also fails the tool's own `--self-test`. **31 tests, all green**, +alongside the three neighbouring gates the new files enter (tooling partition, cp1252 console safety, +licence header): 70 more, green. + +**FIVE SILENT UNDER-REPORTS WERE FOUND AND FIXED IN THIS BUILD, EVERY ONE OF THIS ROW'S EXACT CLASS, +and that is the finding worth carrying forward.** Masking string contents made `['a','b','c']` +indistinguishable from a trailing comma, counting three as two. An inline `[...].map(cb)` fan read as a +list of thunks and priced at **zero**. `Promise.all(ROWS.map(r => agent(r)))` priced at **1 and reported +EXACT**, because `.map()` was recognised only as a direct argument of `parallel()` -- the tool +committing this row's headline failure, in the tool built to refuse it. One space after `parallel(` +turned a known array into a runtime width, through an offset that double-counted leading whitespace. And +a `for (const x of ROWS)` fan over a literal array was filed as an unknown loop. **Each passed a read as +correct before it was measured**, which is the general lesson: this class is not caught by review. + +**SO THE HONESTY GUARANTEE IS A RESIDUE, NOT AN ENUMERATION.** After pricing, every `agent` token the +pricer did not attribute to a site is reported as `unattributed`. Without that catch-all, *unrecognised +construct* and *costs nothing* are the same state, and the floor would be sound only for the shapes its +author happened to think of -- `ROWS.map(agent)` passes the hook as a value and matches no call pattern +anywhere in the scanner. An unbalanced construct is likewise reported rather than skipped, so one mask +desync cannot truncate the scan while the report still says exact. A computed width, a `while` loop that +runs until dry, and a nested `workflow()` each print as an UNPRICED TERM with their per-unit cost where +knowable, and the total reads `known floor` instead of `TOTAL (exact)` whenever any exist. A test pins +that a floor is never rendered with the words *"the naive count is correct here"*, because a floor of 0 +against a naive 1 would otherwise print exactly that. `--strict` exits 1 on any unpriced term; the +default does not, since a report that fails on a finding gets muted. + +**THE DENOMINATOR, MEASURED RATHER THAN DESIGNED -- and the gap is a UNIT gap, not a reader gap.** +`lane.ps1` already records `statedInFlight`, and the storage is **already fleet-wide**: one JSON per box +under `/mefor-coord/lanes/`, atomically written through a sibling `.tmp` directory and +keyed by a derived box key (`lane.ps1:71`). Only the READ is first-person, and deliberately so -- the +header at `:11` states there is no `-Lane` and no `-BoxKey` parameter and that adding one defeats the +design, and `-Show` prints the single record for the box derived from `$PWD`. **Every toucher of that +directory is first-person: exactly two, `scripts/coord/lane.ps1:71` and `scripts/hooks/lane-level.ps1:181`, +the latter reading `$bk.json` for the current box only to nag its own session.** `fleet.ps1` does not +read it at all -- it is a pure reader over `seats/` and projects live seats. So a census would add a +reader, not a store. What it could not do is sum what is there: `statedInFlight` counts **items** for +one box, human-stated and stale by design (`turnsSinceStated`, `LANE_PROMPT_MIN`), while the gate needs +**agents remaining** fleet-wide. Those are different units at different granularity, so the existing +number cannot be added up into the one the gate lacks. + +**WHAT SUCH A CENSUS WOULD HAVE TO RECORD, as constraints rather than a design.** It must be written +**before** the first agent starts, not after, or it is the broadcast-after-the-fact this row already +rejects -- and it must decrement as agents finish, since the row asks for agents REMAINING. Release has +to survive a crash: a session that dies mid-fan-out leaves a claim that never releases, and a census +counting it forever would refuse every later launch, which is worse than no census. `claim.ps1` has +this problem already and answers it with a liveness fence, and `fleet.ps1`'s own header records why a +census must reuse that fence rather than grow a second one -- a private `Get-HolderLiveness` accreted +beside it is how the copy nobody tests becomes the copy in use. `fleet.ps1` also already models the +right discipline for the missing half: `liveSessionsWithoutRecord` starts from the fence and subtracts, +so a dead writer shows as a positive count instead of as silence. A census needs that mirror, or a +launcher that publishes nothing reads as free capacity. **And the ordering is the crux: if the census +is read to authorise a launch and written after it, two launchers read the same answer and both pass, +and this row's failure reappears one level up.** That is why this is a claim with release semantics and +not a gauge, and it is the part left open. + +**FIRST-HAND, FROM INSIDE THE FAILURE MODE.** This builder is a subagent of a Console that ran five +concurrent Builders earlier tonight, and it then ran its own four-agent fan-out for the `/simplify` +pass. **At each of those four spawn points the usage-headroom hook returned `UNKNOWN -- no data`**, +because nothing has published to this config root's `mefor-usage/latest.json`; the hook says plainly +that UNKNOWN is neither zero nor full headroom. That is a third blindness worth separating from the two +above: limb 1 is a launcher unable to see other launchers and limb 3 is a launcher unable to see its +own later phases, but here the **per-launch number itself was absent on this root**, so the launcher was +blind to the aggregate and to its own pool reading at once. The four agents were self-counted as four +and that was right only because the fan was four flat calls with nothing nested; the same honest +self-count over a `.map()` would have been wrong by this row's ratio. **No seat is being called +careless, here or above** -- the reading each seat could take was the reading it took. + +**WHAT STAYS OPEN.** The aggregate limb, unbuilt by choice: it needs new shared coordination state with +claim and release semantics, which is not a one-turn change and which this row deliberately leaves as +options rather than a design. Limb 3's second half is also only half-served -- the estimator prices a +script a launcher HAS, and the PM's 37-refuter case was a phase that appeared in no number the gate +checked; pricing that script would have surfaced it, but nothing yet makes reading the price a step in +launching. Wiring this into a launch path is a separate ask and is not filed under a number here, since +citing one this seat has not allocated would resolve to nothing today and to unrelated work later. + ## 1397. the Bash tool unescapes backslashes inside a QUOTED heredoc, so a Windows path or a doubled-backslash regex silently changes meaning > 🔢 **Filed 2026-08-29 - measured, not fixable here.** In the Claude Code Bash tool a quoted heredoc diff --git a/scripts/coord/workflow_cost.py b/scripts/coord/workflow_cost.py new file mode 100644 index 000000000..299a6cf5b --- /dev/null +++ b/scripts/coord/workflow_cost.py @@ -0,0 +1,935 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 MessageFoundry Organization and contributors +"""Workflow cost estimator -- price a fan-out BEFORE launching it (BACKLOG #1400). + +**The failure this exists to prevent, measured.** A launcher sizing its own fan-out reaches for +``grep -c 'agent('`` because that is the call that starts an agent. It under-reports, because **fan +width lives in the array, not at the call site.** From the filing session's own run:: + + verify-six-rows-by-their-own-instrument + grep -c 'agent(' -> 3 call sites, at lines 83, 100 and 130 + const ROWS = [...] -> 6 at line 76, nowhere near any of them + agents actually started -> 13 + +**A 4.3x under-report from the instrument a launcher would reach for first.** The rule this tool +implements is the one that example forces: read every ``parallel()`` and ``.map()``, multiply by the +array length, sum across phases -- **a phase's cost is fan width times depth, and neither number +appears at the ``agent()`` call.** + +``pipeline(items, s1, s2, s3)`` is the same arithmetic said out loud: every item runs through every +stage, so its cost is ``len(items) * stages`` and BOTH factors sit outside the ``agent()`` calls. + +**WHAT MULTIPLIES, and each is scanned as a construct in its own right rather than as an argument of +``parallel()``:** ``EXPR.map()`` / ``.flatMap()`` / ``.forEach()``, ``Array.from({length: N}, cb)``, +``pipeline(items, ...stages)``, and ``for (const x of EXPR)``. Recognising ``.map()`` only inside +``parallel()`` priced ``Promise.all(ROWS.map(r => agent(r)))`` at one agent and called it exact, +which is this tool's own headline failure committed by the tool. ``parallel()`` itself multiplies +nothing -- it reports what its argument costs. + +**WHAT THIS IS NOT.** It is not the 90-percent Workflow gate, which lives in another repository and +answers a different question -- *is the pool low enough for MY launch* -- correctly, per launcher. +This prices one script. It has no view of what any other seat is doing, so it cannot and does not +speak to the aggregate. See BACKLOG #1400 for that limb and for why lowering a per-launcher +threshold cannot fix a failure that is in the sum. + +**IT REPORTS A FLOOR AND SAYS SO.** A static read cannot see a width computed at runtime, a +``while`` loop that runs until it goes dry, or a ``workflow()`` whose body is another file. Every one +of those is printed as an UNPRICED TERM with its per-unit cost, and the total is labelled a FLOOR +rather than a count whenever any exist. **A cost estimator that under-reports silently reproduces +#1400 one layer down**, which is the whole reason that item exists -- so the one thing this must +never do is return a small number with a straight face. + +**THE HONESTY GUARANTEE IS A RESIDUE, NOT AN ENUMERATION.** After pricing, every ``agent`` token the +pricer did not attribute to a site is reported as ``unattributed``. Without that catch-all, +*unrecognised construct* and *costs nothing* are the same state, and the floor would be trustworthy +only for the shapes this author happened to think of -- ``ROWS.map(agent)`` passes the hook as a +value and matches no call pattern anywhere in the scanner. + +Usage:: + + python scripts/coord/workflow_cost.py path/to/script.js + python scripts/coord/workflow_cost.py --stdin < script.js + python scripts/coord/workflow_cost.py path/to/script.js --json + python scripts/coord/workflow_cost.py --self-test + +Exit 0 when the read completes -- **a large number is a finding, not an error**, and a tool that +exits non-zero on one gets muted. Exit 1 on a broken instrument (unreadable source, failed +self-test) or, with ``--strict``, when any term could not be priced. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from dataclasses import dataclass, field, replace +from pathlib import Path + +# The script hooks that can start an agent. `log` and `phase` cost nothing and are not read. +# `workflow()` is priced as unpriceable on purpose: it shares this run's agent counter, but its body +# is a different file that this tool was not handed. STATED ONCE -- `_TOKEN_RE` is built from this +# tuple, and a hook added here with no pricer becomes a loud `unknown-hook` rather than a silent +# mis-price. +_HOOKS = ("agent", "parallel", "pipeline", "workflow") + +# The array methods that fan a callback across a receiver. These carry the multiplier, and they are +# scanned as first-class constructs rather than only as arguments of `parallel()`. +_FAN_METHODS = ("map", "flatMap", "forEach") + +# `(?" + "|".join(_HOOKS) + r")\s*\(" + r"|(?Array\s*\.\s*from)\s*\(" + r"|\.\s*(?P" + "|".join(_FAN_METHODS) + r")\s*\(" + r"|(?while|for|do)\b" +) +_AGENT_WORD_RE = re.compile(r"(? Unknown: + return replace(self, per_unit=self.per_unit * width) + + +@dataclass(frozen=True) +class Site: + """One place in the script that starts agents, priced as width times depth.""" + + line: int + form: str + width: int + depth: int # agents one unit of the fan starts; width x depth == agents + agents: int + + +@dataclass(frozen=True) +class Cost: + agents: int = 0 + sites: tuple[Site, ...] = () + unknowns: tuple[Unknown, ...] = () + + @property + def exact(self) -> bool: + """True when nothing was left unpriced, so ``agents`` is a count and not a floor.""" + return not self.unknowns + + def __add__(self, other: Cost) -> Cost: + return Cost( + self.agents + other.agents, + self.sites + other.sites, + self.unknowns + other.unknowns, + ) + + def scaled(self, width: int) -> Cost: + """Multiply this cost by a fan width -- the arithmetic the naive count omits. + + Sites are dropped rather than scaled: the caller replaces them with one aggregate Site for + the whole fan, so a scaled copy was built and thrown away on every fan. + """ + return Cost(self.agents * width, (), tuple(u.scaled(width) for u in self.unknowns)) + + +@dataclass +class _Ctx: + raw: str + masked: str + widths: dict[str, int] = field(default_factory=dict) + # Offsets of every `agent` token the pricer actually attributed to a site. `_residue` reports + # the rest, so an unrecognised construct cannot contribute zero in silence. + attributed: set[int] = field(default_factory=set) + + def line_of(self, index: int) -> int: + return self.masked.count("\n", 0, index) + 1 + + def snippet(self, start: int, end: int, limit: int = 46) -> str: + text = " ".join(self.raw[start:end].split()) + return text if len(text) <= limit else text[: limit - 3] + "..." + + +# -------------------------------------------------------------------------------------------- +# Lexical masking -- so `agent(` inside a prompt string or a comment is not a call site +# -------------------------------------------------------------------------------------------- + + +def _is_regex_position(prev: str) -> bool: + """Decide whether a `/` opens a regex literal or divides. + + The standard heuristic: a `/` following a VALUE (identifier, literal, closing bracket) is + division; anywhere else it opens a regex. Imperfect at the margins -- `return /x/` is right, + a keyword like `typeof` ending in a letter reads as a value -- so `_mask` additionally refuses + any "regex" that does not close on its own line, which keeps a misread from desynchronising + the rest of the scan. + """ + return prev == "" or not (prev.isalnum() or prev in "_$)]}") + + +def _mask(src: str) -> str: + """Blank out comments, strings, template literals and regexes, PRESERVING length and newlines. + + Positions and line numbers in the masked text therefore address the original file exactly, so + findings can be reported against real line numbers without a second mapping. + """ + out = list(src) + n = len(src) + i = 0 + prev = "" + + def blank(start: int, stop: int) -> None: + for k in range(start, min(stop, n)): + if out[k] != "\n": + out[k] = " " + + while i < n: + c = src[i] + + if c == "/" and i + 1 < n and src[i + 1] == "/": + end = src.find("\n", i) + end = n if end == -1 else end + blank(i, end) + i = end + continue + + if c == "/" and i + 1 < n and src[i + 1] == "*": + end = src.find("*/", i + 2) + end = n if end == -1 else end + 2 + blank(i, end) + i = end + continue + + if c in "'\"": + j = i + 1 + while j < n: + if src[j] == "\\": + j += 2 + continue + if src[j] == c or src[j] == "\n": + break + j += 1 + end = min(j + 1, n) + blank(i, end) + i, prev = end, "x" + continue + + if c == "`": + # A template literal masks whole, `${...}` interpolations included. A hook call inside + # an interpolation would be missed; none has ever been written and the alternative is + # re-entering code mode mid-string, which desynchronises far more often than it helps. + j = i + 1 + depth = 0 + while j < n: + if src[j] == "\\": + j += 2 + continue + if src[j] == "$" and j + 1 < n and src[j + 1] == "{": + depth += 1 + j += 2 + continue + if src[j] == "}" and depth: + depth -= 1 + j += 1 + continue + if src[j] == "`" and depth == 0: + break + j += 1 + end = min(j + 1, n) + blank(i, end) + i, prev = end, "x" + continue + + if c == "/" and _is_regex_position(prev): + j = i + 1 + closed = -1 + while j < n and src[j] != "\n": + if src[j] == "\\": + j += 2 + continue + if src[j] == "[": + while j < n and src[j] != "]" and src[j] != "\n": + j += 2 if src[j] == "\\" else 1 + if src[j] == "/": + closed = j + break + j += 1 + if closed != -1: + blank(i, closed + 1) + i, prev = closed + 1, "x" + continue + # Did not close on its line: treat as division and keep the scan in sync. + + if not c.isspace(): + prev = c + i += 1 + + return "".join(out) + + +# -------------------------------------------------------------------------------------------- +# Bracket walking +# -------------------------------------------------------------------------------------------- + + +def _match_bracket(masked: str, open_idx: int) -> int: + """Index of the bracket closing the one at ``open_idx``, or -1 if the source is unbalanced.""" + stack: list[str] = [] + for i in range(open_idx, len(masked)): + c = masked[i] + if c in _PAIRS: + stack.append(_PAIRS[c]) + elif c in _CLOSERS: + if not stack or stack[-1] != c: + return -1 + stack.pop() + if not stack: + return i + return -1 + + +def _split_top_level(ctx: _Ctx, start: int, end: int) -> list[tuple[int, int]]: + """Split ``[start, end)`` on commas at bracket depth zero. + + Depth is read from the MASKED text, so a comma inside a prompt string is not a separator. The + final segment's emptiness is read from the RAW text, and that asymmetry is load-bearing: a + masked ``['a','b','c']`` is ``[ , , ]``, whose last element is indistinguishable from a + trailing comma. Testing the mask there counted three strings as two -- a silent under-report of + exactly the kind this tool exists to refuse. + """ + parts: list[tuple[int, int]] = [] + depth = 0 + cur = start + for i in range(start, end): + c = ctx.masked[i] + if c in _PAIRS: + depth += 1 + elif c in _CLOSERS: + depth -= 1 + elif c == "," and depth == 0: + parts.append((cur, i)) + cur = i + 1 + if ctx.raw[cur:end].strip(): + parts.append((cur, end)) + return parts + + +def _collect_array_widths(ctx: _Ctx) -> dict[str, int]: + """Record the element count of every array literal bound to a name. + + This is the number the naive instrument never reaches: ``const ROWS = [...]`` sits at line 76 + and the ``agent()`` calls it multiplies are at 83, 100 and 130. + """ + widths: dict[str, int] = {} + for m in _DECL_RE.finditer(ctx.masked): + open_idx = ctx.masked.index("[", m.start()) + close = _match_bracket(ctx.masked, open_idx) + if close == -1: + continue + widths[m.group(1)] = len(_split_top_level(ctx, open_idx + 1, close)) + return widths + + +# -------------------------------------------------------------------------------------------- +# Width resolution -- the single authority on "how many" +# -------------------------------------------------------------------------------------------- + + +def _width_of(ctx: _Ctx, start: int, end: int) -> tuple[int | None, str]: + """How many elements the expression in ``[start, end)`` has, and why we believe it. + + ONE authority, called from every construct that multiplies. Two readers of "how wide is this" + drift: an earlier draft had a second copy that resolved bare identifiers while the first did + not, so the same array read as 6 in a ``pipeline()`` and as unknown in a ``parallel()``. + """ + seg = ctx.masked[start:end] + if not seg.strip(): + return 0, "empty" + lead = start + (len(seg) - len(seg.lstrip())) + + if ctx.masked[lead] == "[": + close = _match_bracket(ctx.masked, lead) + if close != -1 and not ctx.masked[close + 1 : end].strip(): + return len(_split_top_level(ctx, lead + 1, close)), "inline array" + + m = _ARRAY_FROM_RE.match(seg) + if m: + open_idx = lead + (m.end() - m.start()) - 1 + close = _match_bracket(ctx.masked, open_idx) + if close != -1: + args = _split_top_level(ctx, open_idx + 1, close) + if args: + spec = _LENGTH_RE.match(ctx.masked[args[0][0] : args[0][1]]) + if spec: + return int(spec.group(1)), "Array.from length" + return None, "Array.from length is not a literal" + + name = ctx.raw[start:end].strip() + if name in ctx.widths: + return ctx.widths[name], f"len({name})" + if _IDENT_RE.fullmatch(name): + return None, f"{name} is not a literal array in this file" + return None, "width is computed at runtime" + + +def _receiver_span(ctx: _Ctx, dot_idx: int, floor: int) -> tuple[int, int]: + """Span of the expression a ``.map(`` hangs off, found by walking LEFT from the dot. + + The receiver carries the multiplier, and it sits to the left of the token the scan matched -- + which is the whole reason a launcher's forward-reading grep never reaches it. + """ + i = dot_idx - 1 + while i >= floor and ctx.masked[i].isspace(): + i -= 1 + if i < floor: + return dot_idx, dot_idx + end = i + 1 + + if ctx.masked[i] in _CLOSERS: + opener = _match_bracket_back(ctx.masked, i, floor) + if opener == -1: + return dot_idx, dot_idx + i = opener - 1 + while i >= floor and ctx.masked[i].isspace(): + i -= 1 + # A bracketed receiver may itself hang off a chain (`fresh.filter(x)`), so keep walking + # back through identifier and dot characters to take the whole chain, which resolves to an + # unknown width -- correctly, because a filter can change the count. + while i >= floor and (ctx.masked[i].isalnum() or ctx.masked[i] in "_$."): + i -= 1 + return i + 1, end + + while i >= floor and (ctx.masked[i].isalnum() or ctx.masked[i] in "_$"): + i -= 1 + return i + 1, end + + +def _match_bracket_back(masked: str, close_idx: int, floor: int) -> int: + """Index of the bracket opening the one at ``close_idx``, or -1.""" + stack: list[str] = [] + for i in range(close_idx, floor - 1, -1): + c = masked[i] + if c in _CLOSERS: + stack.append(c) + elif c in _PAIRS: + if not stack or stack[-1] != _PAIRS[c]: + return -1 + stack.pop() + if not stack: + return i + return -1 + + +# -------------------------------------------------------------------------------------------- +# The pricer +# -------------------------------------------------------------------------------------------- + + +def _loop_head(ctx: _Ctx, after_kw: int) -> tuple[tuple[int, int] | None, int, int, int]: + """Split a loop into (of-expression, body start, body end, resume index). + + The of-expression is the span after ``of``/``in`` in a ``for (const x of ROWS)``, which is a + real multiplier and must be resolved rather than shrugged at. + """ + masked = ctx.masked + i, n = after_kw, len(masked) + while i < n and masked[i].isspace(): + i += 1 + subject: tuple[int, int] | None = None + if i < n and masked[i] == "(": + close = _match_bracket(masked, i) + if close == -1: + return None, after_kw, after_kw, n + head = _OF_RE.search(masked, i + 1, close) + if head: + subject = (head.end(), close) + i = close + 1 + while i < n and masked[i].isspace(): + i += 1 + if i < n and masked[i] == "{": + close = _match_bracket(masked, i) + if close == -1: + return subject, i, n, n + return subject, i + 1, close, close + 1 + stop = masked.find(";", i) + stop = n if stop == -1 else stop + return subject, i, stop, stop + + +def _price(ctx: _Ctx, start: int, end: int) -> Cost: + """Price every agent-starting construct in ``[start, end)``. + + Nested constructs are consumed by the recursion, so nothing is counted twice: a ``.map()`` + inside a ``parallel()`` is priced once, by the ``.map()``, and ``parallel()`` merely reports + what its argument costs. + """ + total = Cost() + pos = start + while pos < end: + m = _TOKEN_RE.search(ctx.masked, pos, end) + if not m: + break + + if m.group("hook"): + open_idx = m.end() - 1 + close = _match_bracket(ctx.masked, open_idx) + if close == -1 or close > end: + # Unreadable rather than absent. Skipping it silently would let one mask desync + # truncate the scan while the report still said "exact". + total = total + Cost(unknowns=(_unreadable(ctx, m.start(), m.group("hook")),)) + pos = m.end() + continue + total = total + _price_hook(ctx, m.group("hook"), m.start(), open_idx, close) + pos = close + 1 + continue + + if m.group("fan") or m.group("from"): + total = total + _price_fan_call(ctx, m, start, end) + pos = _resume_after_call(ctx, m.end() - 1, end) + continue + + total = total + _price_loop(ctx, m) + pos = _loop_head(ctx, m.end())[3] + return total + + +def _resume_after_call(ctx: _Ctx, open_idx: int, end: int) -> int: + close = _match_bracket(ctx.masked, open_idx) + return open_idx + 1 if close == -1 or close > end else close + 1 + + +def _unreadable(ctx: _Ctx, at: int, form: str) -> Unknown: + return Unknown( + kind="unreadable", + line=ctx.line_of(at), + form=f"{form}(...) does not close -- the scan could not read past it", + ) + + +def _fan_cost(line: int, width: int | None, why: str, inner: Cost, label: str) -> Cost: + """Turn a per-unit cost and a fan width into either a priced Site or a named unpriced term. + + ONE tail for every multiplying construct. Written out per construct it was three near-identical + blocks that grew by one with each new shape, and the branch that must never be got wrong -- the + unknown-width one, which decides whether a number is a count or a floor -- was the part being + copied. + """ + if not inner.agents and not inner.unknowns: + return Cost() + if width is None: + return Cost( + unknowns=( + Unknown( + kind="dynamic-width", line=line, form=f"{label}: {why}", per_unit=inner.agents + ), + ) + + inner.unknowns + ) + scaled = inner.scaled(width) + site = Site(line, f"{label} ({why})", width, inner.agents, scaled.agents) + return Cost(agents=scaled.agents, sites=(site,), unknowns=scaled.unknowns) + + +def _price_fan_call(ctx: _Ctx, m: re.Match[str], region_start: int, region_end: int) -> Cost: + """Price ``EXPR.map(cb)`` and ``Array.from(spec, cb)`` -- the constructs carrying fan width. + + A ``.map()`` is priced HERE rather than only as an argument of ``parallel()``, because + ``Promise.all(ROWS.map(r => agent(r)))`` is an equally ordinary fan-out and an earlier draft + priced it at 1 agent and called the answer exact. + """ + open_idx = m.end() - 1 + line = ctx.line_of(m.start()) + is_map = m.group("fan") is not None + label = f".{m.group('fan')}(...)" if is_map else "Array.from(...)" + + close = _match_bracket(ctx.masked, open_idx) + if close == -1 or close > region_end: + return Cost(unknowns=(_unreadable(ctx, m.start(), label),)) + args = _split_top_level(ctx, open_idx + 1, close) + + if is_map: + # The `fan` alternative of _TOKEN_RE begins with `\.`, so the match already starts AT the + # dot -- no search needed. + width, why = _width_of(ctx, *_receiver_span(ctx, m.start(), region_start)) + body = args[0] if args else (open_idx + 1, close) + else: + spec = _LENGTH_RE.match(ctx.masked[args[0][0] : args[0][1]]) if args else None + width = int(spec.group(1)) if spec else None + why = "length" if spec else "length is not a literal" + body = args[1] if len(args) > 1 else (open_idx + 1, close) + + return _fan_cost(line, width, why, _price(ctx, body[0], body[1]), label) + + +def _price_loop(ctx: _Ctx, m: re.Match[str]) -> Cost: + """Price a loop, resolving ``for (const x of ROWS)`` rather than calling every loop unknown. + + A loop over a literal array IS a fan of known width. Reporting it as unknown put real fan-out + in the unpriced list, which trains a reader to skim the list that matters most. + """ + subject, body_start, body_end, _ = _loop_head(ctx, m.end()) + inner = _price(ctx, body_start, body_end) + if not inner.agents and not inner.unknowns: + return Cost() + + line = ctx.line_of(m.start()) + keyword = m.group("loop") + if subject is not None: + width, why = _width_of(ctx, subject[0], subject[1]) + if width is not None: + return _fan_cost(line, width, why, inner, f"{keyword}...of") + + return Cost( + unknowns=( + Unknown( + kind="loop", + line=line, + form=f"{keyword} loop, repetitions unknown", + per_unit=inner.agents, + ), + ) + + inner.unknowns + ) + + +def _price_hook(ctx: _Ctx, hook: str, name_start: int, open_idx: int, close: int) -> Cost: + if hook == "agent": + ctx.attributed.add(name_start) + return Cost(agents=1, sites=(Site(ctx.line_of(name_start), "agent(...)", 1, 1, 1),)) + + if hook == "workflow": + # Shares this run's agent counter, but its body is another file. Naming it is the only + # honest answer; returning zero and staying quiet is the #1400 failure one layer down. + return Cost( + unknowns=( + Unknown( + kind="nested-workflow", + line=ctx.line_of(name_start), + form=ctx.snippet(name_start, min(close + 1, len(ctx.raw))), + ), + ) + ) + + if hook == "parallel": + # No multiplier of its own. Its argument is either a `.map()` (priced by _price_map) or a + # literal list of thunks (priced element by element by the ordinary scan), so parallel() + # simply reports what its argument costs. Multiplying here as well would double-count. + return _price(ctx, open_idx + 1, close) + + if hook == "pipeline": + return _price_pipeline(ctx, name_start, open_idx, close) + + return Cost( + unknowns=( + Unknown( + kind="unknown-hook", + line=ctx.line_of(name_start), + form=f"{hook}(...) has no pricer registered", + ), + ) + ) + + +def _price_pipeline(ctx: _Ctx, name_start: int, open_idx: int, close: int) -> Cost: + """``pipeline(items, s1, s2, s3)`` -- every item runs every stage. + + The multiplication is in the call's SEMANTICS rather than written as a ``.map()``, so both + factors sit outside the ``agent()`` calls just as the item describes. + """ + args = _split_top_level(ctx, open_idx + 1, close) + if not args: + return Cost() + width, why = _width_of(ctx, args[0][0], args[0][1]) + per_item = Cost() + for lo, hi in args[1:]: + per_item = per_item + _price(ctx, lo, hi) + label = f"pipeline(...) x {len(args) - 1} stages" + return _fan_cost(ctx.line_of(name_start), width, why, per_item, label) + + +def _residue(ctx: _Ctx) -> tuple[Unknown, ...]: + """Every ``agent`` the scan did NOT attribute to a site, reported as unpriced. + + THE CATCH-ALL IS THE HONESTY GUARANTEE. Without it, "unrecognised construct" and "costs + nothing" are the same state, and the tool's floor is only trustworthy for the shapes its author + happened to enumerate. ``ROWS.map(agent)`` passes the hook as a VALUE and matches no call + pattern at all; this is what catches it. + """ + out: list[Unknown] = [] + for m in _AGENT_WORD_RE.finditer(ctx.masked): + if m.start() in ctx.attributed: + continue + line_end = ctx.raw.find("\n", m.start()) + out.append( + Unknown( + kind="unattributed", + line=ctx.line_of(m.start()), + form=ctx.snippet(m.start(), len(ctx.raw) if line_end == -1 else line_end), + ) + ) + return tuple(out) + + +# -------------------------------------------------------------------------------------------- +# Public entry point +# -------------------------------------------------------------------------------------------- + + +def naive_agent_grep(source: str) -> int: + """What ``grep -c 'agent('`` returns: matching LINES of the raw file, blind to everything. + + Reproduced exactly, comments and strings included, because it is the instrument being compared + against -- not a better version of it. + """ + return sum(1 for line in source.split("\n") if "agent(" in line) + + +def workflow_name(source: str) -> str | None: + idx = source.find("export const meta") + if idx == -1: + return None + m = _META_NAME_RE.search(source, idx, idx + 2000) + return m.group(1) if m else None + + +def price_script(source: str) -> Cost: + """Price one workflow script. The returned cost is EXACT only when ``.exact`` is true.""" + ctx = _Ctx(raw=source, masked=_mask(source)) + ctx.widths = _collect_array_widths(ctx) + cost = _price(ctx, 0, len(ctx.masked)) + # The residue is appended LAST, after every pricer has had its chance to attribute an `agent`. + # It is what turns "the shapes this author enumerated" into "everything", which is the + # difference between an honest floor and a confident one. + return Cost(cost.agents, cost.sites, cost.unknowns + _residue(ctx)) + + +# -------------------------------------------------------------------------------------------- +# Reporting +# -------------------------------------------------------------------------------------------- + + +def render(cost: Cost, source: str, label: str) -> str: + naive = naive_agent_grep(source) + name = workflow_name(source) + out: list[str] = [] + out.append(f"workflow cost: {name or '(no meta.name)'}") + out.append(f" source: {label}") + out.append(f" scanned: {len(source.splitlines())} lines") + out.append("") + + if cost.sites: + out.append(" LINE FAN WIDTH x DEPTH AGENTS") + for site in sorted(cost.sites, key=lambda s: s.line): + out.append( + f" {site.line:>4} {site.form:<48} {site.width:>5} x {site.depth:<5} {site.agents:>6}" + ) + out.append(f" {'':>4} {'':<48} {'':>5} {'':<5} {'------':>6}") + label_total = "known floor" if cost.unknowns else "TOTAL (exact)" + out.append(f" {'':>4} {label_total:<48} {'':>5} {'':<5} {cost.agents:>6}") + out.append("") + + out.append(f" naive `grep -c 'agent('` -> {naive}") + if cost.unknowns: + # Never compare a floor against the naive count as though it were a total. A floor BELOW + # the naive count would otherwise print as "the naive count is correct here", which is the + # #1400 failure verbatim: an instrument reassuring a launcher with a number it cannot back. + note = "a FLOOR -- see the unpriced terms below" + elif naive and cost.agents > naive: + note = f"{cost.agents / naive:.1f}x the naive count" + elif cost.agents == naive: + note = "the naive count is correct here -- no fan-out to miss" + else: + note = "exact" + out.append(f" this read -> {cost.agents} ({note})") + out.append("") + + if cost.unknowns: + out.append( + f" UNPRICED TERMS ({len(cost.unknowns)}) -- THE TOTAL ABOVE IS A FLOOR, NOT A COUNT:" + ) + for u in sorted(cost.unknowns, key=lambda x: x.line): + if u.kind == "nested-workflow": + per = "the whole cost is in another script" + elif u.kind == "unattributed": + per = "an `agent` reference no priced call accounts for" + elif u.kind == "unreadable": + per = "the scan stopped here, so anything past it is uncounted" + elif u.per_unit: + per = f"{u.per_unit} agents per unit, multiplier unknown" + else: + per = "neither the per-unit cost nor the multiplier is statically known" + out.append(f" line {u.line:<5} {u.kind:<16} {u.form}") + out.append(f" {'':<10} {'':<16} {per}") + out.append("") + out.append(" A static read cannot resolve these. Price them by hand before you launch.") + else: + out.append(" UNPRICED TERMS: none. Every fan width in this script is a literal.") + return "\n".join(out) + + +def to_json(cost: Cost, source: str, label: str) -> dict[str, object]: + return { + "source": label, + "name": workflow_name(source), + "agents": cost.agents, + "exact": cost.exact, + "naive_agent_grep": naive_agent_grep(source), + "sites": [ + { + "line": s.line, + "form": s.form, + "width": s.width, + "depth": s.depth, + "agents": s.agents, + } + for s in sorted(cost.sites, key=lambda s: s.line) + ], + "unpriced": [ + {"line": u.line, "kind": u.kind, "form": u.form, "per_unit": u.per_unit} + for u in sorted(cost.unknowns, key=lambda u: u.line) + ], + } + + +# -------------------------------------------------------------------------------------------- +# Self-test -- the positive control, printed rather than assumed +# -------------------------------------------------------------------------------------------- + +# The item's worked example, reconstructed. The original script is not in this repository and was +# not recoverable, so this is the SHAPE its three measured numbers force: three `agent(` call +# sites, a six-element `const ROWS` declared far above them, and 13 agents actually started +# (6 + 6 + 1). Every figure below is from BACKLOG #1400. +SELF_TEST_SCRIPT = """ +export const meta = { + name: 'verify-six-rows-by-their-own-instrument', + description: 'Verify six rows, then synthesise', + phases: [{ title: 'Read' }, { title: 'Refute' }, { title: 'Synthesise' }], +} + +const ROWS = [ + { id: 1301 }, { id: 1302 }, { id: 1303 }, + { id: 1304 }, { id: 1305 }, { id: 1306 }, +] + +phase('Read') +const reads = await parallel(ROWS.map(r => () => agent(`read row ${r.id}`))) + +phase('Refute') +const refutes = await parallel(ROWS.map(r => () => agent(`refute row ${r.id}`))) + +phase('Synthesise') +return await agent('synthesise the two passes') +""" + +# The discriminating control: a script with no fan-out at all, where the naive count is RIGHT. +# Without it, "this tool reports a bigger number" is satisfied by a tool that always reports a +# bigger number, and a pass would prove nothing about the arithmetic. +SELF_TEST_FLAT = """ +export const meta = { name: 'flat', description: 'no fan-out' } +phase('One') +const a = await agent('first') +const b = await agent('second') +return [a, b] +""" + + +def _self_test() -> int: + failures: list[str] = [] + + fan = price_script(SELF_TEST_SCRIPT) + naive = naive_agent_grep(SELF_TEST_SCRIPT) + print(f"SELF-TEST fan-out script: priced {fan.agents}, naive grep {naive}, exact={fan.exact}") + if fan.agents != 13: + failures.append(f"the worked example must price at 13 agents, got {fan.agents}") + if naive != 3: + failures.append(f"the naive grep control must return 3, got {naive}") + if not fan.exact: + failures.append("the worked example has only literal widths and must price exactly") + + flat = price_script(SELF_TEST_FLAT) + naive_flat = naive_agent_grep(SELF_TEST_FLAT) + print(f"SELF-TEST flat script: priced {flat.agents}, naive grep {naive_flat}") + if flat.agents != naive_flat: + failures.append( + f"with no fan-out the two instruments must AGREE: {flat.agents} vs {naive_flat}" + ) + + for problem in failures: + print(f"SELF-TEST FAILED: {problem}", file=sys.stderr) + if failures: + return 1 + print("SELF-TEST OK: the 4.3x under-report is reproduced and the flat case still agrees.") + return 0 + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="workflow_cost.py", + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument("script", nargs="?", help="path to the workflow script (.js)") + parser.add_argument("--stdin", action="store_true", help="read the script from stdin") + parser.add_argument("--json", action="store_true", help="emit machine-readable output") + parser.add_argument( + "--strict", + action="store_true", + help="exit 1 when any term could not be priced statically", + ) + parser.add_argument("--self-test", action="store_true", help="prove the tool can see a fan-out") + args = parser.parse_args(argv) + + if args.self_test: + return _self_test() + + if args.stdin: + source, label = sys.stdin.read(), "" + elif args.script: + path = Path(args.script) + try: + source = path.read_text(encoding="utf-8") + except OSError as exc: + print(f"cannot read {path}: {exc}", file=sys.stderr) + return 1 + label = str(path) + else: + parser.error("give a script path, --stdin, or --self-test") + + cost = price_script(source) + if args.json: + print(json.dumps(to_json(cost, source, label), indent=2)) + else: + print(render(cost, source, label)) + + return 1 if args.strict and cost.unknowns else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_coord_workflow_cost.py b/tests/test_coord_workflow_cost.py new file mode 100644 index 000000000..10d7b5f12 --- /dev/null +++ b/tests/test_coord_workflow_cost.py @@ -0,0 +1,469 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +# Copyright (C) 2026 MessageFoundry Organization and contributors +"""Tests for the workflow cost estimator (BACKLOG #1400). + +**What these pin, and why the negative controls carry the weight.** The tool exists because +``grep -c 'agent('`` under-reported a fan-out by 4.3x. A replacement that merely *returns a bigger +number* would satisfy a naive test while being just as wrong, so the suite is built around cases +that discriminate: + +- the item's own worked example must price at **13**, where the naive instrument returns 3 +- a script with **no fan-out** must price at exactly the naive count, so a pass is not explained by + "this tool always reports more" +- a script whose width is **computed at runtime** must report an UNPRICED TERM, never a quiet low + number -- an estimator that under-reports silently reproduces #1400 one layer down +- the width multiplication itself is **mutation-proved**: break it and the worked example collapses + back to 3, which is the very failure the item measured + +**Several cases are regressions from defects found while building the tool, all of the same +silent-under-report class it exists to refuse** -- and they are the tests most worth keeping, +because each one passed review as correct before it was measured: + +- an array of string literals counted one element short, because masking blanks string content and + made the last element indistinguishable from a trailing comma +- an inline ``[...].map()`` fan read as a list of thunks and priced at zero +- ``Promise.all(ROWS.map(...))`` priced at 1 and reported EXACT, because ``.map()`` was recognised + only as a direct argument of ``parallel()`` +- one space after ``parallel(`` turned a known array into a runtime width +- a ``for (const x of ROWS)`` fan reported as an unknown loop + +The residue check that catches the general case is itself mutation-proved, since an honesty guard +nobody can see fail is not a guard. +""" + +from __future__ import annotations + +import importlib.util +import re +import subprocess +import sys +from pathlib import Path +from types import ModuleType + +import pytest + +_ROOT = Path(__file__).resolve().parents[1] +_SCRIPT = _ROOT / "scripts" / "coord" / "workflow_cost.py" + + +def _load(source: str | None = None, name: str = "coord_workflow_cost") -> ModuleType: + """Import the tool, optionally from MUTATED source, so a mutant can be driven like the real one.""" + if source is None: + spec = importlib.util.spec_from_file_location(name, _SCRIPT) + assert spec is not None and spec.loader is not None + mod = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = mod + spec.loader.exec_module(mod) + return mod + mod = ModuleType(name) + mod.__file__ = str(_SCRIPT) + # Register BEFORE exec: the tool uses `from __future__ import annotations`, so @dataclass + # resolves its field annotations as strings by looking the module up in sys.modules. Without + # this the mutant dies in dataclasses rather than in the assertion under test. + sys.modules[name] = mod + exec(compile(source, str(_SCRIPT), "exec"), mod.__dict__) # noqa: S102 + return mod + + +@pytest.fixture(scope="module") +def mod() -> ModuleType: + return _load() + + +# ------------------------------------------------------------------------------------------------ +# The item's worked example +# ------------------------------------------------------------------------------------------------ + + +def test_the_worked_example_prices_at_thirteen_not_three(mod: ModuleType) -> None: + """BACKLOG #1400's own measurement: 3 call sites, a 6-element array, 13 agents started. + + The original script is not in this repository and was not recoverable, so ``SELF_TEST_SCRIPT`` + reconstructs the SHAPE its three published numbers force: two six-wide fans plus a lone + synthesis agent. Every figure asserted here is quoted from the item. + """ + cost = mod.price_script(mod.SELF_TEST_SCRIPT) + assert cost.agents == 13 + assert mod.naive_agent_grep(mod.SELF_TEST_SCRIPT) == 3 + assert cost.exact, "every width in the worked example is a literal, so the total is a count" + + +def test_the_under_report_ratio_is_the_one_the_item_measured(mod: ModuleType) -> None: + naive = mod.naive_agent_grep(mod.SELF_TEST_SCRIPT) + priced = mod.price_script(mod.SELF_TEST_SCRIPT).agents + assert round(priced / naive, 1) == 4.3 + + +def test_the_fan_width_comes_from_the_array_not_the_call_site(mod: ModuleType) -> None: + """The two six-wide fans must be reported at the ``.map()`` lines, not at ``const ROWS``. + + This is the property the naive instrument lacks: it reads the call site and never reaches the + declaration that multiplies it. Asserted on ``width``/``depth``, which are structured data -- + ``form`` is display prose and pinning a sentence would redden this leg on a wording change. + """ + script = mod.SELF_TEST_SCRIPT + decl_line = script[: script.index("const ROWS")].count("\n") + 1 + + fans = [s for s in mod.price_script(script).sites if s.width > 1] + assert [s.width for s in fans] == [6, 6] + assert all(s.depth == 1 for s in fans) + assert all(s.line > decl_line for s in fans), ( + "the fan is reported where the agents start, while the width comes from a declaration " + "further up -- the separation the naive instrument cannot cross" + ) + + +# ------------------------------------------------------------------------------------------------ +# The discriminating control -- a pass must not be explained by "always reports more" +# ------------------------------------------------------------------------------------------------ + + +def test_with_no_fan_out_the_naive_count_is_correct_and_this_tool_agrees(mod: ModuleType) -> None: + """The negative control. Without it, a tool that always inflates would pass every other test.""" + cost = mod.price_script(mod.SELF_TEST_FLAT) + assert cost.agents == mod.naive_agent_grep(mod.SELF_TEST_FLAT) == 2 + assert cost.exact + + +def test_a_comment_or_a_prompt_string_is_not_a_call_site(mod: ModuleType) -> None: + """The naive grep counts both. This tool must not, or it trades one wrong number for another.""" + script = ( + "export const meta = { name: 'x', description: 'y' }\n" + "// a comment mentioning agent( which starts nothing\n" + "const note = 'a prompt mentioning agent( which starts nothing'\n" + "const only = await agent('the one real call')\n" + ) + assert mod.naive_agent_grep(script) == 3 + assert mod.price_script(script).agents == 1 + + +# ------------------------------------------------------------------------------------------------ +# pipeline() -- width times depth, said out loud +# ------------------------------------------------------------------------------------------------ + + +def test_a_pipeline_costs_items_times_stages(mod: ModuleType) -> None: + """Every item runs every stage, so BOTH factors sit outside the ``agent()`` calls.""" + script = ( + "const ROWS = [1, 2, 3, 4, 5, 6]\n" + "const out = await pipeline(ROWS, r => agent('one'), r => agent('two'), r => agent('three'))\n" + ) + cost = mod.price_script(script) + assert cost.agents == 18 + assert mod.naive_agent_grep(script) == 1 + site = cost.sites[0] + assert (site.width, site.depth) == (6, 3) + + +def test_a_nested_fan_multiplies_rather_than_adds(mod: ModuleType) -> None: + """A 3-lens panel inside a 4-wide fan is 12 agents, not 7.""" + script = ( + "const ITEMS = [1, 2, 3, 4]\n" + "const LENSES = ['a', 'b', 'c']\n" + "await parallel(ITEMS.map(i => () => parallel(LENSES.map(l => () => agent('judge')))))\n" + ) + assert mod.price_script(script).agents == 12 + + +# ------------------------------------------------------------------------------------------------ +# The two silent under-reports found while building this +# ------------------------------------------------------------------------------------------------ + + +def test_an_array_of_string_literals_counts_every_element(mod: ModuleType) -> None: + """Regression: masking blanks string CONTENT, so ``['a','b','c']`` becomes ``[ , , ]``. + + Testing the masked text for a trailing element cannot tell the third string from a trailing + comma, and the width came back 2. The count must be taken from the raw text. + """ + script = "const DIMS = ['a', 'b', 'c']\nawait parallel(DIMS.map(d => () => agent(d)))\n" + assert mod.price_script(script).agents == 3 + + +def test_a_trailing_comma_does_not_invent_an_element(mod: ModuleType) -> None: + """The other side of that fix: reading the raw text must not turn ``[1, 2, 3,]`` into four.""" + script = "const ROWS = [1, 2, 3,]\nawait parallel(ROWS.map(r => () => agent(r)))\n" + assert mod.price_script(script).agents == 3 + + +def test_an_inline_literal_fanned_by_map_is_not_read_as_a_thunk_list(mod: ModuleType) -> None: + """Regression: ``parallel(['a','b','c'].map(cb))`` was priced at 0. + + Reading only the leading ``[`` classified it as a literal list of three THUNKS, each of which + contains no ``agent()`` call because the real call lives in the map callback. It reported zero + and said nothing, which is the exact failure mode this tool exists to refuse. + """ + script = "await parallel(['correctness', 'security', 'repro'].map(l => () => agent(l)))\n" + cost = mod.price_script(script) + assert cost.agents == 3 + assert cost.exact + + +def test_a_map_fan_is_priced_wherever_it_appears_not_only_inside_parallel( + mod: ModuleType, +) -> None: + """Regression: ``Promise.all(ROWS.map(r => agent(r)))`` priced at 1 and reported EXACT. + + ``.map()`` was recognised only as the direct argument of ``parallel()``, so the most ordinary + plain-JavaScript fan-out reproduced this tool's own headline failure with a straight face. The + fix scans ``.map`` as a construct in its own right and resolves its receiver by walking left. + """ + script = "const ROWS = [1, 2, 3, 4, 5, 6]\nawait Promise.all(ROWS.map(r => agent(r)))\n" + cost = mod.price_script(script) + assert cost.agents == 6 + assert cost.exact + assert mod.naive_agent_grep(script) == 1 + + +def test_whitespace_before_the_receiver_does_not_lose_the_width(mod: ModuleType) -> None: + """Regression: one space after ``parallel(`` made a known array read as a runtime width. + + The old resolver added the argument's leading whitespace twice while computing the callback + offset, so it worked only when there was none. + """ + script = "const ROWS = [1, 2, 3]\nawait parallel( ROWS.map(r => () => agent(r)))\n" + cost = mod.price_script(script) + assert cost.agents == 3 + assert cost.exact + + +def test_a_for_of_over_a_literal_array_is_a_fan_not_an_unknown(mod: ModuleType) -> None: + """A loop over a known array has a known width; calling it unknown buries the real findings.""" + script = "const ROWS = [1, 2, 3, 4]\nfor (const r of ROWS) { await agent(r) }\n" + cost = mod.price_script(script) + assert cost.agents == 4 + assert cost.exact + + +def test_array_from_supplies_a_width(mod: ModuleType) -> None: + """The refuter-panel shape from the workflow docs: N identical agents, width in the spec.""" + script = "await parallel(Array.from({length: 3}, () => () => agent('refute')))\n" + cost = mod.price_script(script) + assert cost.agents == 3 + assert cost.exact + + +def test_a_literal_list_of_thunks_sums_rather_than_multiplies(mod: ModuleType) -> None: + """``parallel([t1, t2])`` is two distinct bodies, not one body twice.""" + script = "await parallel([() => agent('a'), () => agent('b'), () => agent('c')])\n" + cost = mod.price_script(script) + assert cost.agents == 3 + assert cost.exact + + +def test_a_filtered_chain_is_unknown_because_a_filter_changes_the_count(mod: ModuleType) -> None: + """The receiver walk must not mistake ``fresh.filter(f).map(cb)`` for a known-width fan.""" + script = "const R = [1, 2, 3]\nawait parallel(R.filter(x => x.ok).map(b => () => agent(b)))\n" + cost = mod.price_script(script) + assert not cost.exact + assert [u.kind for u in cost.unknowns] == ["dynamic-width"] + + +# ------------------------------------------------------------------------------------------------ +# Honesty about what a static read cannot see +# ------------------------------------------------------------------------------------------------ + + +def test_an_agent_no_priced_call_accounts_for_is_reported(mod: ModuleType) -> None: + """The catch-all. Without it, "unrecognised construct" and "costs nothing" are one state. + + ``ROWS.map(agent)`` passes the hook as a VALUE, so it matches no call pattern anywhere in the + scanner. An enumeration of known shapes cannot catch it by construction; only a residue check + over what the pricer failed to attribute can. + """ + script = "const ROWS = [1, 2, 3]\nconst handles = ROWS.map(agent)\n" + cost = mod.price_script(script) + assert not cost.exact + assert [u.kind for u in cost.unknowns] == ["unattributed"] + + +def test_an_unbalanced_call_stops_the_scan_loudly(mod: ModuleType) -> None: + """A truncated read must never render as a clean one. + + If an unreadable construct were skipped silently, one mask desync would cut the scan short + while the report still said "exact" over a file it never finished. + """ + script = "await parallel(ROWS.map(r => () => agent(r))\n" + cost = mod.price_script(script) + assert not cost.exact + assert "unreadable" in {u.kind for u in cost.unknowns} + + +def test_every_priced_agent_is_attributed_so_the_residue_stays_quiet(mod: ModuleType) -> None: + """The catch-all's negative control: it must not fire on scripts the pricer fully understood. + + A residue check that always fires is a residue check nobody reads. + """ + for script in (mod.SELF_TEST_SCRIPT, mod.SELF_TEST_FLAT): + assert not [u for u in mod.price_script(script).unknowns if u.kind == "unattributed"] + + +def test_a_runtime_width_is_named_rather_than_silently_priced_low(mod: ModuleType) -> None: + script = "await parallel(discovered.map(d => () => agent('work ' + d)))\n" + cost = mod.price_script(script) + assert not cost.exact + assert [u.kind for u in cost.unknowns] == ["dynamic-width"] + assert cost.unknowns[0].per_unit == 1, "the per-item cost IS known and must be reported" + + +def test_a_loop_is_named_with_its_per_iteration_cost(mod: ModuleType) -> None: + script = ( + "const FINDERS = [1, 2, 3, 4]\n" + "while (dry < 2) {\n" + " await parallel(FINDERS.map(f => () => agent('find')))\n" + "}\n" + ) + cost = mod.price_script(script) + assert not cost.exact + loops = [u for u in cost.unknowns if u.kind == "loop"] + assert len(loops) == 1 + assert loops[0].per_unit == 4 + + +def test_a_loop_that_starts_no_agents_is_not_reported(mod: ModuleType) -> None: + """Ordinary loops must stay silent, or the real findings drown in noise and get ignored.""" + script = "for (const x of xs) { total += x }\nawait agent('one')\n" + cost = mod.price_script(script) + assert cost.agents == 1 + assert cost.exact + + +def test_a_nested_workflow_is_named_as_living_in_another_script(mod: ModuleType) -> None: + script = "const r = await workflow('some-other-workflow', {x: 1})\n" + cost = mod.price_script(script) + assert [u.kind for u in cost.unknowns] == ["nested-workflow"] + assert cost.agents == 0 + + +def test_a_floor_is_never_presented_as_a_count(mod: ModuleType) -> None: + """The rendered report must say FLOOR, and must not claim the naive count was right. + + A floor of 0 against a naive count of 1 would otherwise print "the naive count is correct + here" -- an instrument reassuring a launcher with a number it cannot back, which is #1400 + verbatim. + """ + script = "await parallel(discovered.map(d => () => agent('work')))\n" + text = mod.render(mod.price_script(script), script, "") + assert "FLOOR" in text + assert "UNPRICED TERMS (1)" in text + assert "the naive count is correct here" not in text + + +def test_an_exact_read_says_so(mod: ModuleType) -> None: + text = mod.render(mod.price_script(mod.SELF_TEST_SCRIPT), mod.SELF_TEST_SCRIPT, "") + assert "TOTAL (exact)" in text + assert "UNPRICED TERMS: none" in text + assert "FLOOR" not in text + + +# ------------------------------------------------------------------------------------------------ +# Mutation proof -- the multiplication is the whole tool, so breaking it must go red +# ------------------------------------------------------------------------------------------------ + + +def _mutate(pattern: str, replacement: str, name: str) -> ModuleType: + """Load the tool with one expression rewritten, refusing to run if the seam moved. + + Matched by REGEX rather than by an indented literal, so a re-indent or a move between functions + does not silently turn a negative control into a no-op. The count assertion is the guard: a + pattern matching zero times, or more than once, fails here with a message naming the cause + instead of failing later as a confusing arithmetic mismatch. + """ + source = _SCRIPT.read_text(encoding="utf-8") + mutant_source, count = re.subn(pattern, replacement, source) + assert count == 1, f"the mutation seam {pattern!r} matched {count} times; re-point this test" + return _load(mutant_source, name=name) + + +def test_breaking_the_width_multiplication_collapses_the_example_to_the_naive_count( + mod: ModuleType, +) -> None: + """Drop the ``* width`` and the worked example prices at 3 -- the failure the item measured. + + This is the negative control for every assertion above. Without it, "13" is a number the suite + asserts rather than a number the arithmetic produces. + """ + mutant = _mutate(r"self\.agents \* width", "self.agents", "coord_workflow_cost_mutant") + assert mutant.price_script(mutant.SELF_TEST_SCRIPT).agents == 3, ( + "a mutant that ignores fan width must reproduce the 4.3x under-report" + ) + assert mutant._self_test() == 1, "the tool's own self-test must fail on this mutant" + assert mod.price_script(mod.SELF_TEST_SCRIPT).agents == 13 + + +def test_breaking_the_array_element_count_is_caught(mod: ModuleType) -> None: + """The other half of the arithmetic: the width itself must come from the literal's length.""" + mutant = _mutate( + r'return len\(_split_top_level\(ctx, lead \+ 1, close\)\), "inline array"', + 'return 1, "inline array"', + "coord_workflow_cost_mutant2", + ) + script = "await parallel(['a', 'b', 'c'].map(l => () => agent(l)))\n" + assert mutant.price_script(script).agents == 1 + assert mod.price_script(script).agents == 3 + + +def test_disabling_the_residue_check_is_caught(mod: ModuleType) -> None: + """The catch-all is itself mutation-proved, because a silent honesty guard is no guard. + + If ``_residue`` returned nothing, an unrecognised construct would price at zero and the report + would call it exact -- which is the failure BACKLOG #1400 describes, one layer down. + """ + mutant = _mutate(r"cost\.unknowns \+ _residue\(ctx\)", "cost.unknowns", "cwc_mutant3") + script = "const ROWS = [1, 2, 3]\nconst handles = ROWS.map(agent)\n" + assert mutant.price_script(script).exact, "the mutant must go quiet, or this proves nothing" + assert not mod.price_script(script).exact + + +# ------------------------------------------------------------------------------------------------ +# The tool as a process -- what a launcher actually runs +# ------------------------------------------------------------------------------------------------ + + +def test_self_test_passes_as_a_subprocess() -> None: + """If the built-in positive control cannot pass, no number this tool prints is evidence.""" + proc = subprocess.run( + [sys.executable, str(_SCRIPT), "--self-test"], + capture_output=True, + text=True, + check=False, + ) + assert proc.returncode == 0, proc.stderr + # Both numbers, and NOT `"3" in stdout` -- that is a substring of "13" and would pass on the + # priced figure alone, reading as a two-sided check while testing one side. + assert "priced 13" in proc.stdout + assert "naive grep 3" in proc.stdout + + +def test_strict_exits_non_zero_only_when_a_term_is_unpriced( + mod: ModuleType, tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + """Driven in-process: ``main`` returns the code and only ``__main__`` raises SystemExit. + + Four subprocesses cost 3.6 seconds here and measured almost entirely as interpreter startup. + The one process-level control above is what proves the entry point works when spawned. + """ + exact = tmp_path / "exact.js" + exact.write_text("const R = [1, 2]\nawait parallel(R.map(r => () => agent(r)))\n", "utf-8") + fuzzy = tmp_path / "fuzzy.js" + fuzzy.write_text("await parallel(found.map(r => () => agent(r)))\n", encoding="utf-8") + + assert mod.main([str(exact)]) == 0 + assert mod.main([str(exact), "--strict"]) == 0 + assert mod.main([str(fuzzy)]) == 0, "a report is not a gate -- an unpriced term must not fail" + assert mod.main([str(fuzzy), "--strict"]) == 1 + capsys.readouterr() + + +def test_json_output_carries_the_floor_flag(mod: ModuleType, tmp_path: Path) -> None: + """``to_json`` is a public entry point for a programmatic caller and must pin its contract.""" + payload = mod.to_json(mod.price_script(mod.SELF_TEST_SCRIPT), mod.SELF_TEST_SCRIPT, "") + assert payload["agents"] == 13 + assert payload["exact"] is True + assert payload["naive_agent_grep"] == 3 + assert payload["unpriced"] == [] + + script = "await parallel(found.map(r => () => agent(r)))\n" + fuzzy = mod.to_json(mod.price_script(script), script, "") + assert fuzzy["exact"] is False + assert [u["kind"] for u in fuzzy["unpriced"]] == ["dynamic-width"] diff --git a/tests/tooling_manifest.txt b/tests/tooling_manifest.txt index bb5029a6f..7116f5155 100644 --- a/tests/tooling_manifest.txt +++ b/tests/tooling_manifest.txt @@ -75,6 +75,7 @@ tests/test_coord_recoverable.py tests/test_coord_dispatch_gate.py tests/test_coord_seat_clock_alarm.py tests/test_coord_throughput.py +tests/test_coord_workflow_cost.py tests/test_doc_guards_lane.py tests/test_install_gate_records_the_install.py tests/test_quality_expiry_audit.py