Skip to content

test(dynamic-workflow): mechanical claims table and reflection formation example - #45

Merged
hetaoBackend merged 4 commits into
MiniMax-AI:mainfrom
modacker:community/claims-and-examples
Sep 18, 2026
Merged

hetaoBackend merged 4 commits into
MiniMax-AI:mainfrom
modacker:community/claims-and-examples

Conversation

@modacker

@modacker modacker commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

What changes

  1. Mechanical claims table (VERIFICATION.md + scripts/verify-claims.mjs): the claims a machine can re-check are tabulated in a ```verify fenced block. The runner executes each row via /bin/sh -c, prints one PASS/FAIL line per claim, and exits 0 only when every row matches its expected exit status (1 on the first mismatch, 2 when the tool itself cannot run). Rows needing development dependencies say so (npm ci first); non-mechanical claims (dashboard acceptance, real-agent calls, platform coverage) intentionally stay prose.
  2. Reflection formation example (examples/reflection.js): draft → independent critique → revision, three phases with separate agents, a checkpoint snapshot of the draft, and explicit open-issue handling — styled after examples/audit.js.

Verification

  • node scripts/verify-claims.mjs: 4/4 PASS on a committed tree. V-04 failing on a dirty tree is the feature: it detects drifted committed assets.
  • examples/reflection.js passes validateScript
  • Repository npm run check: green

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@modacker

Copy link
Copy Markdown
Contributor Author

Related: #47 (zh/en term-consistency audit) — same spirit as the claims table here, extending mechanical checks from behavior to documentation. No action needed on this PR.

@modacker
modacker force-pushed the community/claims-and-examples branch from a950f42 to 8d4a56e Compare September 18, 2026 01:57

@hetaoBackend hetaoBackend left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes for exact current head 8d4a56e.

Blocking issues:

  1. The PR changes the maintained plugins/hetaoBackend/mcode-dynamic-workflows/ tree, while the PR author/commits are modacker / moc <moc@sgmov.com> and the manifests still identify hetaoBackend as owner (plugin.json:6, .claude-plugin/plugin.json:5). Under CONTRIBUTING.md:28,80, provide explicit owner authorization or move the change to the contributor's own plugin scope before merge.
  2. scripts/verify-claims.mjs:10-17 extracts Markdown cells and passes each cell directly to /bin/sh -c. This turns a modified verification document into arbitrary local command execution, with no allowlist for commands/argv or shell metacharacters. Replace it with structured, fixed commands/argv (or an explicit documented trusted-only policy plus strict validation); do not use an unrestricted shell for a claims verifier.
  3. The parser only collects rows matching one regex (verify-claims.mjs:12) and silently ignores malformed/unexpected/duplicate rows. It can exit 0 after verifying a subset while claiming “every row”. Validate the header, exact expected IDs/order, uniqueness, columns, and that every table row is consumed; add focused parser tests.
  4. The new verifier and examples/reflection.js are not executed by the current CI workflow. The green checks cover the existing 73 source checks and package test, but not node scripts/verify-claims.mjs or validateScript on the reflection example. Add both checks to CI and provide exact-head output.
  5. examples/reflection.js:12-15 gives the revision agent the draft and critique but not the original task/material passed to the critic. The runtime contract says agents are self-contained and do not inherit the parent conversation; the revision cannot independently verify the critique evidence. Pass the original material (or a bounded equivalent) to revision and add a regression assertion.

The current [code]smith check is SKIPPED and is not evidence. Reconcile the ownership issue and add the missing executable CI evidence before approval.

- VERIFICATION.md gains a verify-fenced claims table; the new
  scripts/verify-claims.mjs executes each row and exits 0 only when all
  match (1 first mismatch, 2 tool failure)
- examples/reflection.js: draft -> independent critique -> revision
  formation with a checkpoint snapshot, styled after audit.js
@modacker
modacker force-pushed the community/claims-and-examples branch from 8d4a56e to b3b8acc Compare September 18, 2026 03:26
@modacker

Copy link
Copy Markdown
Contributor Author

Note on the failing validate (ubuntu-latest): the two failures (new-branch attribution considers baselines beyond the first 256 tips, a workspace fetch makes commit attribution explicitly unavailable) are both in cli-agent-bridge's git-attribution tests — the same intermittent family documented with three runs of evidence in #43 (comment). This PR touches only dynamic-workflow docs/examples and a verification script; zero overlap with the failing plugin. Evidence thread on #43 has the details.

@modacker

Copy link
Copy Markdown
Contributor Author

Thank you for the exact-head review — all five blockers are actionable. Point-by-point:

  1. Ownership (CONTRIBUTING.md:28,80): fix(dynamic-workflow): deterministic wait polling and checkpoint lineage in repair reuse #44 — same maintained plugin tree, same commit authorship, reviewed against exact head and approved+merged — was our basis for treating contributor fixes to mcode-dynamic-workflows as authorized. If that reading is wrong: this is the explicit request for the owner's authorization for this PR's changes to your plugin scope (and for feat(dynamic-workflow): tamper-evident integrity ledger for repair_cache and events #48/feat(dynamic-workflow): opt-in content-addressed cross-run reuse #49, already under your review). If you prefer a different authorization form, say the word.
  2. verify-claims shell execution: agreed — /bin/sh -c over document cells is an execution hazard. Reworking to your preferred first shape: claims as fixed argv data in the script (no shell, no markdown on the execution path; node resolved to process.execPath), with the markdown table demoted to a validated mirror (a consistency test asserts the mirror equals the executable claims — malformed or smuggled rows fail loudly instead of being silently skipped).
  3. Parser strictness: subsumed by (2); the mirror-consistency test covers exact ID/column/uniqueness/consumption parity.
  4. CI coverage: we cannot push .github/workflows/** (OAuth token lacks workflow scope). Two options — (a) maintainer adds two steps (snippet will be in the rework: node scripts/verify-claims.mjs and an examples demo-smoke); or (b) we complete a one-time workflow-scope authorization and add them ourselves. Exact-head outputs of both commands will be posted on this PR with the rework.
  5. reflection.js: agreed — agents are self-contained; the revision node will receive the original task/material alongside draft+critique, the example will be made demo-executor-safe, and a demo-mode end-to-end regression check is being added.

Rework incoming on this branch; will re-request review against the exact new head.

@hetaoBackend

Copy link
Copy Markdown
Collaborator

Follow-up review of current head c7387e5f7245cb52f5da7a8e59c344e912923316: changes are still needed before merge.

  1. [P2] Malformed verification rows are silently omitted. scripts/verify-claims.mjs:12-13 collects only regex matches, rather than validating every data row. In a disposable fixture, a valid V-01 followed by V-02 with exit zero instead of exit 0 produced exit status 0 and 1 claims verified; the second claim was never checked. This contradicts the documented “every row” guarantee. Validate all table rows (including malformed rows and duplicate IDs), fail nonzero for invalid input, and add parser regression tests.
  2. [P2] The independent revision agent lacks the original task and evidence. examples/reflection.js:14-15 passes only draft and critique. I ran the example through the actual Engine with a deterministic executor and confirmed that the revision input contains neither task nor material. It cannot independently check whether a critique is supported by the source or whether the revision meets the original request. Pass the original task/material (or an explicitly bounded equivalent) to revision; also provide the task to the critic when assessing task coverage. Add an input-contract regression check.
  3. Cover the new behavior in CI. The current workflow does not exercise the new claims parser or run this reflection example. Wire their focused regression checks into the existing test suite so green checks cover these additions.

Validation boundary: the reflection probe checks orchestration and delivered inputs, not real-model output quality. The current public CI failure in run 35304360094 is in CLI Agent Bridge; I am not attributing it to this PR.

Clarification of the earlier review: external contributor authorship is not itself a blocker, and executing explicitly invoked, trusted repository verification commands through a shell is not by itself a demonstrated security vulnerability. The POSIX-only dependency should nevertheless be disclosed in VERIFICATION.md or removed for portability.

@modacker

Copy link
Copy Markdown
Contributor Author

Thank you for the follow-up and the two clarifications. Rework pushed at exact head bdf7717 — point-by-point:

  1. Parser strictness (your exit zero repro): the execution path no longer parses markdown at all. Claims are fixed argv data in scripts/verify-claims.mjs, spawned directly (no shell; node resolves to process.execPath). The VERIFICATION.md table is demoted to a human-readable mirror, and checks/claims.check.mjs strictly validates it: exact header, ordered equality with the executable claims, ID uniqueness, column shape, and full consumption — your exit zero row shape now fails the suite loudly (unparseable row), as do duplicate IDs, reordered rows, and smuggled extra rows (six negative parser tests, each asserting the specific failure).
  2. Reflection input contract: critique now receives task + material alongside the draft (task-coverage assessment), and revise receives task + material + draft + critique — the revision can independently check critique evidence against the source. checks/examples.check.mjs runs the example end-to-end through the real Engine with a deterministic executor and asserts every agent's delivered inputs (the same probe you ran, now as a permanent regression).
  3. CI coverage: both new checks live in checks/*.check.mjs, so the existing npm run check validate job executes them on every run — no workflow changes needed (which we cannot push without workflow scope).
  4. POSIX dependency: removed rather than disclosed — no /bin/sh anywhere; the runner spawns node/npm/git directly and VERIFICATION.md states the POSIX/macOS portability boundary.

Ownership: understood and noted for future submissions — thank you for the explicit clarification that external authorship is not itself a blocker.

Full suite at this head: 82/82 plugin checks, packaged MCP smoke 1/1, verify-claims 4/4. Re-requesting review.

@modacker

Copy link
Copy Markdown
Contributor Author

Additional evidence at exact head bdf7717: the complete 493-test repository gate also passes on a local Ubuntu 24.04 box (Node 24) — 482 pass / 0 fail / 11 platform skips — including the two new checks (claims mirror/parser suite, examples input-contract regression) on the Linux path. Local macOS full suite: 82/82, packaged smoke 1/1, verify-claims 4/4.

…nput contract

- scripts/verify-claims.mjs: claims are fixed argv data spawned directly
  (no shell, node resolved to process.execPath); the markdown table is a
  human-readable mirror, never an execution source
- checks/claims.check.mjs: mirror equals the executable claims exactly
  (header, order, uniqueness, columns, full consumption); six negative
  parser tests prove malformed/duplicate/smuggled/reordered rows fail
  loudly instead of being silently skipped
- examples/reflection.js: critique and revision agents each receive the
  original task and material (agents are self-contained; the revision can
  independently verify critique evidence against the source)
- checks/examples.check.mjs: end-to-end input-contract regression through
  the real Engine (deterministic executor, no model calls); both new
  checks run in the standard suite, so CI covers them via npm test
- VERIFICATION.md: mirror semantics + POSIX/macOS portability note
@modacker
modacker force-pushed the community/claims-and-examples branch from bdf7717 to e931a3a Compare September 18, 2026 06:28
@modacker

Copy link
Copy Markdown
Contributor Author

Self-audit correction: exact head is now e931a3a (was bdf7717). The runner's direct-invocation guard relied on require (undefined in ESM, silently caught), which could make a direct run no-op with exit 0 on Node versions where argv[1] stays relative — the same silent-skip class your parser review targets. Rewritten to an explicit pathToFileURL(realpathSync(argv[1])) === import.meta.url comparison; both entry paths re-verified (direct run 4/4, import path silent). Full suite 82/82, packaged smoke 1/1.

@modacker

Copy link
Copy Markdown
Contributor Author

For the record on the chore: retrigger commit: the red validate at e931a3a (run 35315014641) was the known intermittent cli-agent-bridge process-tree flake — two tests, zero overlap with this PR's files. The evidence thread with four runs, the constant error family, and candidate root causes lives on #43 (comment); this retrigger came back green, same code.

@hetaoBackend hetaoBackend left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at ee3964c. The previous findings are resolved: verification uses fixed argv with strict mirror validation; critique and revise receive the original task/material; platform limitations are documented. Focused claims/example tests pass (9/9), and the latest-head CI checks are green. Approved for merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants