fix(mt#4972): Resolve the evaluation-log defaults through the writer's own resolver

[minsky-ai]

Summary

mt#4748 moved evaluation streams to <state dir>/projects/<key>/. Two live-corpus readers kept a
.minsky/-rooted default, so with no --log they addressed a directory that has held no evaluation
stream since 2026-08-30.

The two failed differently, and the spec asserted otherwise until this pass measured it. The
spec read "each … reports an empty corpus rather than an error"; that is true of one site and
false of the other. Both were run with no override before any code changed:

Site	Before	After
measure-causal-premise-fn-rate.ts	exit 1, loud: FAIL: no evaluation stream at .minsky/causal-premise-evaluations.jsonl	exit 0, raw records: 647
measure-negative-existence-recall.ts	exit 0, a complete report of zeros: log records: 0 … joined: 0	log records: 331, joined: 227

The asymmetry is the priority argument. The first is a broken default with a working error path — it
costs a confused run and then says what to do. The second is a can't-fail probe (mem#704): it
recovered 3,284 turns, joined zero, and printed a full stratified recall analysis in which every
cell was 0, including a row captioned "a claim-shape fix could flip these". Nothing in that
output says the corpus was not found. A recall measurement that silently reports no misses is the
one that gets believed. The Summary now records each site's measured behaviour separately.

Key changes

• Both sites call evaluationLogPath(<stream>, { fallbackCwd: REPO_ROOT }) — the same shape mt#4971
  (DONE) established for the ten calibration readers. fallbackCwd rather than projectDir keeps
  the resolver's CLAUDE_PROJECT_DIR tier ahead of the checkout, which matters when these run from
  a session workspace.
• --log overrides preserved, including measure-negative-existence-recall.ts's
  resolve(REPO_ROOT, …) wrapper — with an absolute default, resolve returns it unchanged while a
  relative --log still resolves against the repo root, which is the easy thing to break here.
• A stale docblock in measure-negative-existence-recall.ts naming the old path is corrected, so
  the file no longer contradicts its own default.
• New scripts/lib/evaluation-log-defaults.test.ts. Separate file rather than another list inside
  mt#4971's calibration-log-defaults.test.ts: the two families have different resolvers and
  different filename suffixes, so one combined list would need a per-entry family tag to assert
  either property.

Scope correction carried from planning: this task was filed against five sites. mt#4978 retired
the stop-at-decision detector and deleted all three of its replay scripts, so it is two.

SC coverage

• SC1 / SC2 — both sites resolve through evaluationLogPath with fallbackCwd; both overrides
  verified live under AT2 below.
• SC3 / SC4 — the regression test and its negative control, below.
• SC5 — run as a counted enumeration rather than a truncated listing (mem#1012 R3: a
  glob-filtered tree search is a completeness question and must not be positionally cut). 6 hits
  total, all six dispositioned: 3 fixed (the two defaults plus the stale docblock); 3 out of scope
  and named — consolidate-evaluation-stream-logs.test.ts (asserts the consolidator reads the OLD
  location, its job), backfill-precutover-telemetry.test.ts (asserts legacyPathFor produces the
  legacy path by construction), probe-mt3743-evaluation-stream.ts (writes to a scratch dir, never
  the repo). No remaining repo-rooted evaluation default under scripts/.

Testing

Execution evidence:

AT4 — bun scripts/run-related-tests.ts over the three changed files:

 39 pass
 0 fail
 72 expect() calls
Ran 39 tests across 3 files. [95.00ms]
run-related-tests.ts: 3 related test file(s) passed: scripts/lib/evaluation-log-defaults.test.ts,
scripts/measure-causal-premise-fn-rate.test.ts, scripts/measure-negative-existence-recall.test.ts

SC3 — the new test file itself:

(pass) every live-corpus reader calls evaluationLogPath [1.08ms]
(pass) no live-corpus reader carries a repo-rooted evaluation literal [0.70ms]
(pass) the patterns match the shapes they are meant to catch [0.11ms]
(pass) a docblock mentioning the old path is not a violation [0.02ms]
 4 pass  0 fail

AT1 — both sites, no --log, CLAUDE_PROJECT_DIR at the main checkout. The causal-premise
figures reproduce the planning-time positive control (taken with an explicit --log) exactly:

raw records:              647
= denominator:            582
fired within denominator: 1          <- exit 0; was exit 1

log records: 331   in window (--until none): 331
joined on (session_id, proseChars): 227   unjoined: 104     <- was 0 and 0

AT2 — both override shapes. A RELATIVE --log tmp-at2-relative.jsonl read 331 records against a
file wc -l independently counted at 331, confirming repo-root-relative resolution survives. An
ABSOLUTE --log at a different project key read raw records: 1 rather than the default's 647,
confirming the override still outranks the new default.

AT3 — negative control: site 1 reverted and the test observed FAILING.

The revert was FULL — the literal, the evaluationLogPath import, and REPO_ROOT — not just the
one line, per mem#4512: a partial revert leaves a state that is neither pre- nor post-fix, and a
control that fails to fire is then two hypotheses rather than one.

(fail) every live-corpus reader calls evaluationLogPath
(fail) no live-corpus reader carries a repo-rooted evaluation literal
+   "scripts/measure-causal-premise-fn-rate.ts: \".minsky/causal-premise-evaluations.jsonl\"",
 2 pass  2 fail

The failure names the literal, as SC4 requires. Fix restored and re-verified 4/4.

What this control does not buy: it proves the scan can fail on a literal. It is a source-text
scan and inherits that bound — it sees literals, not computed paths — which both its docblock and
mt#4971's state. The manual reader list is the residual: a future script omitted from it is
uncovered. That is the tier the repo chose deliberately (repo-rooted-telemetry-paths.ts excludes
scripts/ with a stated reason), and both files' docblocks instruct authors to add new readers.

Typecheck clean across 8 projects (including tsconfig.scripts.json, which covers these files);
lint clean, 4,389 files.

Deploy verification

Not applicable — isDeploySurfaceFile returns false for all three changed files. Run as the
predicate over the actual changed-file list rather than recalled from a pattern list:

false scripts/measure-causal-premise-fn-rate.ts
false scripts/measure-negative-existence-recall.ts
false scripts/lib/evaluation-log-defaults.test.ts

Parallel work

No collision. git_log --path over both sites for 7 days: pathMatched: true, no commits — a real
negative, not an unmatched pathspec. All 17 open PRs enumerated; the one candidate, PR #3625
(mt#4954, IN-REVIEW), was read via get_files — it adds only scripts/verify-hook-root-resolution.ts
and its test, touching neither site nor .minsky/hooks/dispatcher.ts, so evaluationLogPath's
signature is not in flight. mt#4976 has no PR, so a file-level claim is unavailable; recorded as
task-level adjacency — it converges the writer/reader on one root, this fixes readers that consulted
no resolver at all, and a tier change inside evaluationLogPath leaves the call site intact.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TPNyYTreM4F7PRbvh4xA5b

[minsky-reviewer]

Minsky Reviewer Status

Verdict: APPROVED — no blocking findings
Review: View review
Model: openai/gpt-5 | Tokens: 560K prompt, 6K completion | Duration: 162s
Mode: normal

Commands

• /review — request a fresh review

[minsky-reviewer]

Independent adversarial review (Chinese-wall)
Reviewer: minsky-reviewer[bot] via openai:gpt-5
Tier: 2

---

Overall, the PR implements the spec cleanly: both scripts now resolve defaults via evaluationLogPath with fallbackCwd, preserve the intended --log override semantics (including repo-root-relative resolution for the negative-existence script), and add a focused regression test guarding against repo-rooted evaluation-path literals. Dispatcher semantics support the intended tiering. I raised a few non-blocking robustness nits: (1) the test’s evaluationLogPath detection is a brittle .includes check, (2) the literal-detection regexes could broaden and enumerate multiple matches, and (3) the two scripts differ in relative --log resolution semantics — confirm the divergence is intentional. No blocking issues found. I reviewed the changed files fully; I did not sweep the entire repo for other evaluation-path usages beyond what the new test names. Ready to merge after optional nits.

Findings

• [NON-BLOCKING] scripts/lib/evaluation-log-defaults.test.ts:73 — Fragile detection of evaluationLogPath usage via plain string .includes
  The test asserts that every live-corpus reader calls evaluationLogPath by checking source.includes("evaluationLogPath(") (scripts/lib/evaluation-log-defaults.test.ts:73-77). This is brittle: formatting like evaluationLogPath /*comment*/ ( or an alias assignment (e.g., const r = evaluationLogPath; r(...)) will evade detection while still satisfying the intent — or conversely cause a false negative if spacing/newline occurs. Prefer a regex like /\bevaluationLogPath\s*\(/ or a lightweight AST parse to avoid whitespace/comment sensitivity per testing-standards.mdc §Testable Design robustness guidelines.
• [NON-BLOCKING] scripts/lib/evaluation-log-defaults.test.ts:54 — Repo-rooted literal patterns risk blind spots (single match, narrow name charset)
  REPO_ROOTED_PATTERNS uses two regexes to catch *.minsky/*-evaluations.jsonl literals but (a) each uses .match() without global flags, returning only the first hit, and (b) the filename part is constrained to [a-z0-9-]+ (scripts/lib/evaluation-log-defaults.test.ts:41-66). While a single hit is enough to fail the test, multiple distinct literals in one file would not be enumerated (which the failure message aims to do), and names with underscores or capitals would be missed. Consider: (1) using matchAll with /gi to enumerate all occurrences for clearer failure output, and (2) broadening the filename pattern to [-A-Za-z0-9_.]+ or sourcing the allowed character set from the dispatcher’s own name constraints if codified.
• [NON-BLOCKING] scripts/measure-causal-premise-fn-rate.ts:215 — Relative --log handling differs from sibling script; confirm intended semantics
  measure-causal-premise-fn-rate.ts still takes a relative --log path as-is (resolved by the process CWD) via readArg(argv, "--log") ?? DEFAULT_LOG (scripts/measure-causal-premise-fn-rate.ts:212-217), whereas the sibling measure-negative-existence-recall.ts intentionally wraps resolve(REPO_ROOT, flag("--log") ?? DEFAULT_LOG) to preserve repo-root-relative semantics for a relative override. The spec singles out the latter as important to preserve; if parity is desired across both tools, consider adopting the same resolve(REPO_ROOT, …) here or document the difference explicitly. If divergence is intentional, no change needed.

Spec verification

Criterion	Status	Evidence
Both sites resolve their default through the writer's own evaluationLogPath (.minsky/hooks/dispatcher.ts), passing fallbackCwd rather than projectDir so the CLAUDE_PROJECT_DIR tier still wins.	Met	Both scripts import evaluationLogPath from .minsky/hooks/dispatcher and construct DEFAULT_LOG via evaluationLogPath(<name>, { fallbackCwd: REPO_ROOT }): scripts/measure-causal-premise-fn-rate.ts:23-36 and :49 and scripts/measure-negative-existence-recall.ts:25-38 and :43-48. No projectDir is passed; dispatcher code shows fallbackCwd sits below CLAUDE_PROJECT_DIR (.minsky/hooks/dispatcher.ts:... evaluationLogPath).
Each site's existing --log <path> override reads that path unchanged, including measure-negative-existence-recall.ts's repo-root-relative resolution of a relative argument.	Met	measure-negative-existence-recall.ts computes LOG_PATH = resolve(REPO_ROOT, flag("--log") ?? DEFAULT_LOG) preserving repo-root resolution for relative --log (scripts/measure-negative-existence-recall.ts:44-52). measure-causal-premise-fn-rate.ts reads --log via readArg(argv, "--log") ?? DEFAULT_LOG and then uses it directly (scripts/measure-causal-premise-fn-rate.ts:206-217), preserving prior behavior (no resolve wrapper existed before per spec). The criterion requires overrides to still read the provided path; both continue to do so. The special relative-resolution for the negative-existence script is preserved.
A regression test in the shape of scripts/lib/calibration-log-defaults.test.ts (shipped by mt#4971) covers the evaluation family, including its own can't-fail guard: a case asserting the patterns match the shapes they are meant to catch.	Met	New scripts/lib/evaluation-log-defaults.test.ts mirrors the prior style: it lists live-corpus readers, scans sources via findRepoRootedEvaluationLiterals, and includes a guard test that the regexes match sample literals (scripts/lib/evaluation-log-defaults.test.ts:85-107).
A negative control: reverting either covered site to a repo-rooted literal fails the test, with the failure naming the literal.	Met	The test assembles violation messages as <path>: <literal> (scripts/lib/evaluation-log-defaults.test.ts:80-84). Replacing DEFAULT_LOG with a .minsky/...-evaluations.jsonl literal would be caught and reported by the test. The PR description includes AT3 evidence asserting the failure output format. Code support is in place.
Any remaining .minsky/*-evaluations.jsonl default under scripts/ is either fixed or recorded with a reason naming why the old location is correct for it.	Met	The PR updates both live-corpus readers and the test file's header documents out-of-scope intentional old-location usages (consolidate-evaluation-stream-logs.ts, migrate-*, backfill-precutover-telemetry.test.ts, scratch writers). A repo sweep is not automated here, but within the diff's scope, both relevant defaults are fixed and the test guards future regressions. No additional .minsky/*-evaluations.jsonl defaults were changed in this PR.

Documentation impact

• no-update-needed — This PR modifies internal script defaults and adds a regression test. It does not introduce or change public APIs, CLI flags, or documented behavior surfaces. I searched the changed script headers and they already document the new default locations inline (updated docblock in measure-negative-existence-recall.ts:6-14). No files under docs/ were touched by the diff, and no external user-facing contract appears changed.

[minsky-reviewer]

Independent adversarial review (Chinese-wall)
Reviewer: minsky-reviewer[bot] via openai:gpt-5
Tier: 2

---

Verified the R1 non-blocking observations remain as nits and no blocking issues were present then or now. The fix commits route both scripts’ defaults through evaluationLogPath with fallbackCwd as required, preserve each script’s existing --log semantics (including repo-root-relative resolution for the negative-existence script), and introduce a focused regression test guarding against repo-rooted evaluation-path literals with an explicit guard case. I find no new critical defects introduced by the changes. Tests and inline doc updates align with the spec. Event is APPROVE.

Spec verification

Criterion	Status	Evidence
Both sites resolve their default through the writer's own evaluationLogPath (.minsky/hooks/dispatcher.ts), passing fallbackCwd rather than projectDir so the CLAUDE_PROJECT_DIR tier still wins.	Met	scripts/measure-causal-premise-fn-rate.ts:33-41 imports evaluationLogPath and sets DEFAULT_LOG = evaluationLogPath("causal-premise", { fallbackCwd: REPO_ROOT }). scripts/measure-negative-existence-recall.ts:28-36 and :53-61 import and set DEFAULT_LOG = evaluationLogPath("negative-existence-claim", { fallbackCwd: REPO_ROOT }). No projectDir is passed.
Each site's existing --log <path> override reads that path unchanged, including measure-negative-existence-recall.ts's repo-root-relative resolution of a relative argument.	Met	scripts/measure-causal-premise-fn-rate.ts:206-212 computes logPath = readArg(argv, "--log") ?? DEFAULT_LOG (relative paths left to CWD as before). scripts/measure-negative-existence-recall.ts:66-75 computes LOG_PATH = resolve(REPO_ROOT, flag("--log") ?? DEFAULT_LOG) preserving repo-root resolution for relative arguments; absolute DEFAULT_LOG remains unchanged by resolve.
A regression test in the shape of scripts/lib/calibration-log-defaults.test.ts (shipped by mt#4971) covers the evaluation family, including its own can't-fail guard: a case asserting the patterns match the shapes they are meant to catch.	Met	New scripts/lib/evaluation-log-defaults.test.ts enumerates readers (:26-36), asserts each uses evaluationLogPath (:88-94), asserts no repo-rooted literal (:96-104), and includes a guard test validating the regex patterns (:106-120).
A negative control: reverting either covered site to a repo-rooted literal fails the test, with the failure naming the literal.	Met	The test reports violations as <path>: <literal> (scripts/lib/evaluation-log-defaults.test.ts:98-103). Reverting a site's DEFAULT_LOG to a .minsky/...-evaluations.jsonl literal would be captured and named in the failure output.
Any remaining .minsky/*-evaluations.jsonl default under scripts/ is either fixed or recorded with a reason naming why the old location is correct for it.	Met	Both live-corpus readers now resolve via evaluationLogPath (see above). The test header documents intentional old-location uses outside this scope (scripts/lib/evaluation-log-defaults.test.ts:14-25), aligning with the spec's carve-outs.

Documentation impact

• no-update-needed — The PR updates two internal maintenance scripts to resolve evaluation-log defaults via the existing dispatcher resolver and adds a regression test. No public API, CLI surface, or documented user workflow changes. No files under docs/ were modified, and the updated inline docblock in scripts/measure-negative-existence-recall.ts reflects the already-documented migration (mt#4748).

[minsky-reviewer]

Independent adversarial review (Chinese-wall)
Reviewer: minsky-reviewer[bot] via openai:gpt-5
Tier: 2

---

Verification pass: there were no prior BLOCKING findings to address, and this commit cleanly implements the spec. Both scripts now resolve their defaults via evaluationLogPath with fallbackCwd, each preserves its existing --log semantics (including repo-root-relative resolution for the negative-existence script), and a focused regression test guards against repo-rooted evaluation literals and includes a can’t-fail guard. I find no new critical defects introduced by these changes. Non-blocking robustness nits noted in R1 (regex breadth, .includes detection) remain advisory and do not block merge. Event is APPROVE.

Spec verification

Criterion	Status	Evidence
Both sites resolve their default through the writer's own evaluationLogPath (.minsky/hooks/dispatcher.ts), passing fallbackCwd rather than projectDir so the CLAUDE_PROJECT_DIR tier still wins.	Met	scripts/measure-causal-premise-fn-rate.ts:33-49 imports evaluationLogPath and defines DEFAULT_LOG = evaluationLogPath("causal-premise", { fallbackCwd: REPO_ROOT }). scripts/measure-negative-existence-recall.ts:63-81 does the same for "negative-existence-claim". Neither passes projectDir; both pass fallbackCwd.
Each site's existing --log <path> override reads that path unchanged, including measure-negative-existence-recall.ts's repo-root-relative resolution of a relative argument.	Met	scripts/measure-causal-premise-fn-rate.ts:206-214 computes logPath = readArg(argv, "--log") ?? DEFAULT_LOG and uses it directly (preserving prior CWD-relative behavior). scripts/measure-negative-existence-recall.ts:87-95 computes LOG_PATH = resolve(REPO_ROOT, flag("--log") ?? DEFAULT_LOG), preserving repo-root-relative resolution for a relative --log.
A regression test in the shape of scripts/lib/calibration-log-defaults.test.ts (shipped by mt#4971) covers the evaluation family, including its own can't-fail guard: a case asserting the patterns match the shapes they are meant to catch.	Met	New file scripts/lib/evaluation-log-defaults.test.ts lists the live-corpus readers, asserts they call evaluationLogPath, and checks for repo-rooted literals via findRepoRootedEvaluationLiterals. It includes a guard test that the regexes match sample literals (:68-108).
A negative control: reverting either covered site to a repo-rooted literal fails the test, with the failure naming the literal.	Met	The test assembles violations as <path>: <literal> when findRepoRootedEvaluationLiterals finds a match (scripts/lib/evaluation-log-defaults.test.ts:98-104). Reverting a site's DEFAULT_LOG to a .minsky/...-evaluations.jsonl literal would be caught and reported with the literal string.
Any remaining .minsky/*-evaluations.jsonl default under scripts/ is either fixed or recorded with a reason naming why the old location is correct for it.	Met	Within this PR, both live-corpus readers are fixed. The test file’s header documents intentional old-location usages (consolidator, migrations, scratch writer) as out of scope (scripts/lib/evaluation-log-defaults.test.ts:20-36). No other scripts/ defaults are changed by this diff, and the new test guards future regressions.

Documentation impact

• no-update-needed — The PR updates two internal measurement scripts to resolve evaluation-log defaults via the dispatcher and adds a regression test. No public API, CLI flags, or user-facing commands changed. Inline docblocks in the scripts were updated to reflect the new default path. No files under docs/ were touched, and no existing documentation appears invalidated by these internal path-resolution changes.