Skip to content

feat(heroku-to-aws): make report validation blocking + harden it - #288

Merged
leon1418 merged 20 commits into
awslabs:mainfrom
herosjourney:feat/heroku-report-validator-blocking
Sep 23, 2026
Merged

leon1418 merged 20 commits into
awslabs:mainfrom
herosjourney:feat/heroku-report-validator-blocking

Conversation

@herosjourney

@herosjourney herosjourney commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Heroku migrations produce a migration-report.html — the summary a stakeholder reads. There's a checker that's supposed to make sure that report is complete and sound, but it wasn't actually enforcing anything: it was optional (a bad report didn't stop anything), it had no tests, and it only looked at a handful of the report's sections. On top of that, two of the skill's own instruction files disagreed about whether the check even blocks — one said "optional, never halt," the other said "fail the gate."

The practical result: a Heroku user could ship a report that's missing its required cost-optimization section, or shows a verdict signalled by color alone (a problem for colorblind readers), or has an empty section where content should be — and nothing would catch it. This PR makes the check actually block, hardens what it inspects, gives it a test suite, and resolves the contradiction so the two files tell one story.

In code, for reviewers

validate-heroku-migration-report.py was optional/non-blocking, had zero tests, and checked only three section IDs + the footer. generate-report.md said "optional / never-halt" while generate-assemble.md said "GATE_FAIL" — a direct contradiction. A report could ship missing cost-optimization, carrying a color-only verdict pill, or with an empty section, with no gate. Review item P1-B.

Solution

Harden the Heroku report validator, make it blocking, and add the test suite it never had —
scoped to what the intentionally-thin one-pager actually emits.

Validator (validate-heroku-migration-report.py), Python-stdlib only:

  • Requires cost-optimization and enforces it is non-empty (a table or the explicit
    no-eligible-commitment sentence, never a blank section — matching the skill's Step 3 contract).
  • Typography-first verdict: decision-summary must not contain any badge-verdict-*
    pill; when estimation-infra.json declares recommendation.outcome, it must render a
    verdict-headline.
  • Accessibility (safe subset): <html lang>; every <th> declares scope=col|row; any
    <figure> carries aria-label + <figcaption>.

Emitter and gate now agree (a11y). generate-report.md was updated so the report template
instructs the a11y attributes the validator requires — every <th scope>, and figure labels if a
figure is emitted. Without this the blocking gate would have false-failed a report that followed
the template (the tables emit <th> with no scope). <html lang> was already in the skeleton.

Where the validator runs (round-2 fix — mirrors #287). Generate dispatches under
_exec._agent: rw, which excludes shell/Bash — the dispatched worker (fragments and
assembler) cannot run the validator. An earlier revision put the python3 invocation in
generate-assemble.md and called it "the main window," but the assembler is dispatched, so
"blocking" was prose on a path that can't execute. Corrected to the three-step split:

  • Worker (assembler + generate-report.md, dispatched, no shell): authors
    migration-report.html and does the in-fragment Step 3 self-check + one-repair; it never runs
    the validator.
  • "Finish Generate" (main window, BEFORE the gate): generate.md runs
    validate-heroku-migration-report.py ($PLUGIN_ROOT-anchored) over the report. On REPORT_OK
    it stamps a durable report-validation-status.json (temp status sidecar, not a _produces
    artifact); on REPORT_FAIL it emits GATE_FAIL and pastes the validator's errors[]
    verbatim
    so recovery is actionable. This step edits nothing else.
  • _postconditions gate (main window, READ-ONLY): the section-id _assert plus a new
    _assert that report-validation-status.json has report_status == REPORT_OK. Runs no
    validator, edits nothing (per INTERPRETER.md § _postconditions). The durable stamp means the
    assert reads an artifact, not conversation memory — parity with feat(heroku-to-aws): wire tf-best-practices policy gate into Generate #287's validation-report.json.

Recovery on GATE_FAIL is a hand-edit of the report from the pasted errors + a direct
validator re-run (or a maintainer re-running Generate for a clean rebuild) — not a blind "re-run
Generate," which re-dispatches the shell-less worker and re-authors the report.

Blocking wiring resolved the contradiction. generate-report.md and generate-assemble.md
now agree the gate is required/blocking and read-only; generate.md _postconditions gained
cost-optimization in the section-id _assert plus the durable report-validation-status.json
REPORT_OK _assert.

Tests: tests/test_validate_heroku_migration_report.py (both trees, 14 cases) — compliant
report (with and without a declared recommendation) passes; failures fire for missing/empty
cost-optimization, missing footer, badge-verdict-* pill, <th> without scope, missing
<html lang>, missing verdict-headline when an outcome is declared, figure missing aria-label,
and (new) figure with aria-label but no <figcaption>; plus the figure-happy-path and
≥2-scenario paths.

Deliberately NOT ported (would false-fail the thin report): decision-before-TOC, single-<h1>,
per-table <caption>, glossary/appendix set.

Type of Change

  • Bug fix
  • Enhancement to existing content
  • Guardrail/CI update

Team Folder

  • advisor/
  • migrate/
  • solution-architecture/
  • Other: ___

Verification

  • cross-plugin-drift.ts: OK (273 identical, 25 allowlisted — no new drift)
  • test_validate_heroku_migration_report.py (both trees): 13 passed each
  • frontmatter-validator (both trees): OK
  • markdownlint-cli2: 0 errors; dprint check: clean; tsc --noEmit: OK both trees
  • False-fail probe: a template-shaped report (with <th scope>, non-empty cost-optimization) →
    exit 0; a bare <th> → exit 1 with the actionable message (gate matches emitter)
  • mise run build (full composite incl. bandit/semgrep/gitleaks/checkov/grype):
    PASS, exit 0 — run against a fresh detached worktree at this branch's commit (not my working
    checkout, which has an untracked, unrelated migration-watchdog/ directory failing an unrelated
    markdownlint check). checkov 276/0; gitleaks no leaks; grype no vulnerabilities.

Not run locally: no browser a11y tooling (Vale/Pa11y/axe) — the validator is a dependency-free
static HTML reader, matching the plan's "Python 3 only" constraint.

Checklist

  • I have read the CONTRIBUTING.md guidelines
  • My changes do not include hardcoded secrets, credentials, or internal-only content
  • I have run mise run build locally and it passes — verified against a fresh worktree at this branch's commit (see Verification)
  • I have updated documentation if needed (generate-report.md a11y authoring rules + blocking wiring; generate-assemble.md main-window gate)
  • My changes are scoped to my team's folder only (advisor/ + migrate/ migration skills)

@herosjourney
herosjourney requested review from a team as code owners September 12, 2026 21:52
@herosjourney
herosjourney force-pushed the feat/heroku-report-validator-blocking branch from c9ae7fa to b8f4260 Compare September 12, 2026 22:21
@herosjourney

Copy link
Copy Markdown
Contributor Author

Follow-up review + fixes (pushed as b8f4260)

A review pass over this PR found the blocking flip had real problems — a gate stricter than the
emitter, the same interpreter conflicts as the tf-policy-gate PR, and test gaps. All fixed.
Structured by what the review found and how it was addressed:

Blocking issues found

1. The a11y rules would false-fail a report that follows the template. The validator required
scope=col|row on every <th> and aria-label+<figcaption> on every <figure>, but
generate-report.md never told the agent to emit them — and the report emits tables
(exec-costs, cost-optimization, what-if-scenarios). An agent following the template would
produce <th>Tier</th> with no scope and the now-blocking validator would REPORT_FAIL. Same
"gate stricter than emitter" mistake I'd consciously avoided for <caption>.
→ Fix: added the a11y requirements to the emitter (generate-report.md Step 2 skeleton + the
Step 3 self-check): every <th> declares scope, any <figure> carries aria-label +
<figcaption>. Now the gate matches what the template produces — verified with a probe: a
template-shaped report passes, a bare <th> fails.

2. The completion gate said "GATE_FAIL … and repair," which the interpreter forbids.
INTERPRETER.md § _postconditions: a gate must not modify artifacts. And the paragraph above
the new blocking gate still said "repair the HTML and continue," reading as a second
continue-on-fail path.
→ Fix: the one-repair-then-rewrite lives only in the generate-report.md in-fragment Step 3
self-check; the assembler now emits GATE_FAIL and halts (tells the user to re-run Generate),
editing nothing. Step 3 was renamed and split into "in-fragment self-check (repair here)" vs
"blocking validation (main window)."

3. The required validator couldn't run under _exec: rw, and the path was cwd-relative. Same
host gap as the tf-policy-gate PR — the dispatched worker has no shell, so a "required, blocking"
validator invoked in the fragment is just prose. And generate-report.md used
python3 scripts/… (cwd-relative, misses the plugin script even on inline hosts).
→ Fix: the validator runs in the main window completion gate (generate-assemble.md +
generate.md _postconditions, never dispatched — INTERPRETER.md § _exec step 4), the only
place with a shell; the worker only writes the HTML. Path corrected to
"$PLUGIN_ROOT/scripts/validate-heroku-migration-report.py".

Should-fix issues found

4. Docstring vs code on badge-verdict-*. The docstring said "as the sole verdict carrier" but
the code (correctly) fails on any badge-verdict-*. Three sentences disagreed.
→ Fix: aligned docstring, code comment, and generate-report.md Step 3 to "must not contain
badge-verdict-*"; Step 3 now also requires a verdict-headline when recommendation.outcome is
set, matching the validator.

5. Test gaps for rules the PR made blocking. No figure-a11y test, no happy-path test with a
declared recommendation.outcome, and empty cost-optimization passed (only the id was checked).
→ Fix: the validator now enforces non-empty cost-optimization; added tests for
figure-without-aria-label (fails), figure-with-label (passes), good-report-with-recommendation
(passes), and empty-cost-optimization (fails). 9 → 13 cases. Renamed Step 3 from "Soft gate."

6. Corrupt estimation-infra.json fails open silently.
→ Fix: added a one-line comment documenting the intentional fail-open-on-ambiguity.

Verification (b8f4260, fresh detached worktree)

mise run build PASS (checkov 276/0, gitleaks no leaks, grype no vulns), cross-plugin-drift.ts
OK (273 identical), test_validate_heroku_migration_report.py 13 passed both trees, all touched
files twin-identical, false-fail probe confirms the gate matches the emitter. Not run locally:
browser a11y tooling — the validator is a dependency-free static reader.

herosjourney pushed a commit to herosjourney/startups that referenced this pull request Sep 12, 2026
…(P1-C)

The report validators checked structure, decision-UX, and a11y but never
confirmed the rendered dollar figures matched the estimate — the HTML could
drift from estimation-infra.json silently. This adds a numeric cross-check.

- validate-migration-report.py (GCP) + validate-heroku-migration-report.py:
  new _validate_cost_figures — every data-cost-key-anchored dollar figure must
  equal the matching estimation-infra.json value (GCP: aws_monthly_balanced,
  current_monthly->current_costs.gcp_monthly, optional premium/optimized;
  Heroku: aws_monthly_balanced only — current-spend deferred, key unsettled).
  Skip-on-absent (no anchor / no JSON / no estimate); FAIL only on a real
  mismatch, citing JSON path vs HTML excerpt. Untagged figures are never checked.
- Emitters (generate-artifacts-report.md rule 21, heroku generate-report.md):
  instruct emitting the data-cost-key anchors (attribute, not reader text).
- migration-report-reference.html: anchors the balanced + current figures.
- tests: mismatch->FAIL, match->PASS, no-estimate->skip, untagged->skip.
- docs/report-quality.md: numeric-check scope note.

Both plugin trees updated byte-identically. Heroku validator + generate-report.md
also touched by the report-blocking PR (awslabs#288); land order noted in the PR.
@herosjourney
herosjourney force-pushed the feat/heroku-report-validator-blocking branch from b8f4260 to 2868f78 Compare September 13, 2026 00:39
@herosjourney

herosjourney commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Verified additional issues and each point against the code at b8f4260 before changing anything — all four hold, and #1 is the same host-gap as the first #287 pass, one file over. Fixed in 2868f78 (both trees, byte-identical). Structured as finding → what I changed.

1. Should-fix — validator was still in the worker, not the main window (agreed, blocking)

Confirmed: the python3 validate-heroku-migration-report.py call lived only in generate-report.md (fragment) and generate-assemble.md (assembler) — both dispatched to the shell-less rw worker. generate.md _postconditions only asserted section-ids + footer prose; it never ran the script or asserted REPORT_OK. So on _exec: rw the new checks (no badge-verdict-*, verdict-headline, <th scope>, non-empty cost-optimization, <html lang>) never executed, and a report with the four section-ids + footer passed the only main-window check. The assembler even labelled itself "the main window," which is wrong — the assembler is dispatched. Mirrored the #287 split exactly:

  • generate.md — new main-window "Finish Generate in the main window (report validation)" step that runs validate-heroku-migration-report.py after the worker returns and before _postconditions. Added a read-only _assert that this step printed REPORT_OK (not just that the section-ids exist). Scope Boundary carves the main-window validate out of "Nothing else."
  • _postconditions is read-only: the section-id _assert + the REPORT_OK _assert, no script run, no edits.
  • generate-assemble.md — dropped the false "In the main window … run the report validator" claim; it now states the assembler is dispatched and points at the finish step; halts read-only on GATE_FAIL.
  • generate-report.md — points at the finish step for authoritative execution.

2. Nit — GATE_FAIL should quote the validator errors (agreed)

Right — the fragment has no shell to re-run the validator, so a bare "re-run Generate" is not actionable. The finish step now emits GATE_FAIL and pastes the validator's errors[] verbatim (missing scope id, empty cost-optimization, banned pill class, <th> without scope, missing <html lang>, etc.). I also fixed the recovery copy the same way #287 was fixed: recovery is a hand-edit from the pasted errors + a direct validator re-run, not a full Generate re-dispatch (which re-authors the report under the shell-less worker and wipes the fix).

3. Nit — skeleton had no <th scope="col"> example (agreed)

The HTML skeleton showed <section id="exec-costs">…</section>. Agents copy skeletons, so <html lang="en"> got followed but <th scope> might not. The exec-costs skeleton section now shows a sample table with <th scope="col"> header cells and a comment to copy the pattern for cost-optimization / what-if-scenarios.

4. Nit — no test for figure-without-<figcaption> (agreed)

Added test_figure_without_figcaption_fails — a <figure aria-label="…"> with no <figcaption> now asserts exit 1 + a figcaption error, symmetric to the existing missing-aria-label case. Test count 13 → 14 (both trees).

Verification (2868f78, both trees)

  • cross-plugin-drift.ts: OK (273 identical, 25 allowlisted)
  • test_validate_heroku_migration_report.py: 14 passed advisor, 14 passed migrate
  • mise run build against a fresh detached worktree at this commit: PASS, exit 0 (checkov 276/0, gitleaks no leaks, grype no vulnerabilities, fail 0 across DSL suites)
  • Fail-closed preserved: if the finish step can't run the validator (no shell), it does not record REPORT_OK, so the read-only _assert fails closed.

No change to the a11y/emitter alignment, the assembler-no-longer-edits-HTML property, the $PLUGIN_ROOT path, the badge-verdict-* wording, or the not-ported set — those stay as you approved.

The heroku-to-aws report validator was optional/non-blocking with zero tests,
and generate-report.md contradicted generate-assemble.md on whether it gates.
This makes it a required, blocking gate and hardens it to the report's own
decision-UX + accessibility contract (ported subset of the gcp-to-aws awslabs#223
checks, scoped to what the thin Heroku one-pager emits).

Round-2 review (host-placement): the validator invocation was in the dispatched
assembler/fragment (no shell under _exec: rw), so 'blocking' was prose on a path
that can't execute. Mirrors the awslabs#287 split: the python3 validator run + a
REPORT_OK assert now live in a main-window 'Finish Generate (report validation)'
step in generate.md that runs BEFORE the read-only _postconditions gate. On
REPORT_FAIL the step emits GATE_FAIL and pastes the validator errors[] verbatim
(so recovery is actionable, not a blind 're-run Generate'). The gate is
read-only; the assembler no longer claims to be 'the main window'.

- validate-heroku-migration-report.py: require cost-optimization; forbid
  badge-verdict-* pills; require verdict-headline when recommendation.outcome
  present; a11y subset (html lang, th scope, figure aria-label+figcaption)
- generate.md: 'Finish Generate' main-window validator step + read-only REPORT_OK
  _assert; Scope-Boundary carve-out for the main-window validate
- generate-report.md: points at the finish step; GATE_FAIL pastes errors[];
  skeleton gains a <th scope=col> example so agents copy the a11y pattern
- generate-assemble.md: read-only gate, no validator call, recovery = hand-edit
  + targeted re-run (not full Generate re-dispatch, which wipes the fix)
- tests: 13 -> 14 cases (add figure-with-aria-label-without-figcaption FAIL)
Deliberately not ported (would false-fail the thin report): decision-before-TOC,
single-h1, per-table caption, glossary/appendix set. Both plugin trees updated
byte-identically.
@herosjourney
herosjourney force-pushed the feat/heroku-report-validator-blocking branch from 2868f78 to 2bce3f5 Compare September 13, 2026 00:50
@herosjourney

herosjourney commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Fixed 3 additional nits.

1. REPORT_OK was conversation-only (agreed — added parity)

Correct: #287 stamps a durable policy_status into validation-report.json, but #288's finish step only "recorded" a print and the _assert read prose. Since the validator only emits to stdout/stderr, I added the one-line stamp you suggested: the Finish Generate step now writes report-validation-status.json ({"report_status": "REPORT_OK"}, or REPORT_FAIL with errors[]), and the read-only _postconditions _assert now checks that file's report_status == REPORT_OK. So the assert reads an artifact, not conversation memory — parity with #287. It's a temp status sidecar (not in _produces), and if the validator can't run the stamp is absent/not_run, so it still fails closed. generate-report.md and generate-assemble.md reference the stamp consistently, and recovery now says re-run + re-stamp.

2. Assembler header was stale (agreed)

Right — the opener still said the assembler "enforces the completion handoff gate." Rewrote it: the assembler runs the cross-artifact checks and emits the handoff signal, but the report validator runs later in the main-window "Finish Generate" step (the assembler is dispatched and has no shell). Body was already correct; this aligns the opener.

3. Two GATE_FAIL emitters (agreed it's redundant — leaving it)

Agreed it's fine and fail-closed. The redundancy is deliberate defense-in-depth: the finish step emits GATE_FAIL on REPORT_FAIL, and the read-only _postconditions _assert independently blocks on a non-REPORT_OK stamp — so a skipped/half-run finish step still can't complete. I'd rather keep both than rely on either alone, so no change here.

Verification (2bce3f5, both trees)

  • cross-plugin-drift.ts: OK (273 identical, 25 allowlisted)
  • test_validate_heroku_migration_report.py: 14 passed (both trees)
  • mise run build against a fresh detached worktree at this commit: PASS, exit 0 (checkov 276/0, gitleaks no leaks, grype no vulnerabilities)

No validator-code or test change in this revision — it's the durable-stamp wiring + the assembler-opener wording.

herosjourney pushed a commit to herosjourney/startups that referenced this pull request Sep 13, 2026
…(P1-C)

The report validators checked structure, decision-UX, and a11y but never
confirmed the rendered dollar figures matched the estimate — the HTML could
drift from estimation-infra.json silently. This adds a numeric cross-check.

- validate-migration-report.py (GCP) + validate-heroku-migration-report.py:
  new _validate_cost_figures — every data-cost-key-anchored dollar figure must
  equal the matching estimation-infra.json value (GCP: aws_monthly_balanced,
  current_monthly->current_costs.gcp_monthly, optional premium/optimized;
  Heroku: aws_monthly_balanced only — current-spend deferred, key unsettled).
- Required keys fail CLOSED: when a required figure's JSON value exists and
  exec-costs is rendered, a MISSING anchor is a FAIL (an un-anchored wrong
  figure must not pass — the bug P1-C exists to catch). Gated on require_anchors
  so minimal --no-require-toc unit fixtures are not forced to anchor. Only
  untagged *illustrative* numbers and absent optional per-tier keys are skipped.
- Safe int-coerce (non-whole-dollar JSON -> named FAIL, never a crash); nested
  markup stripped before compare; mismatch/no-$ = FAIL citing JSON path vs HTML.
- Emitters (generate-artifacts-report.md rule 21, heroku generate-report.md) +
  docs describe the REQUIRED-anchor rule (round-2: the three author-facing lines
  that still said 'untagged skipped / missing anchor skipped' now state required
  anchors are mandatory; illustrative extras stay skipped).
- migration-report-reference.html: anchors the balanced + current figures.
- tests: mismatch->FAIL, match->PASS, no-estimate->skip (asserts code==0),
  untagged->skip, nested-markup, required-absent FAIL for BOTH balanced and
  current_monthly, non-numeric->named FAIL. 83 passed (77 GCP + 6 Heroku cost).
Both plugin trees updated byte-identically. Heroku validator + generate-report.md
also touched by the report-blocking PR (awslabs#288); land order noted in the PR.
@herosjourney herosjourney changed the title feat(heroku-to-aws): make report validation blocking + harden it (P1-B) feat(heroku-to-aws): make report validation blocking + harden it Sep 13, 2026

@leon1418 leon1418 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖]

Reviewed head d35f4b8bb65d673d541ed03ab9f7f0346abc32d0, including the follow-up explanations and both plugin copies. Requesting changes for the three inline findings:

  • P1: The newly mandatory validator is not included in the documented skill-only installation.
  • P2: Heading-only cost-optimization sections and tables without data rows still pass.
  • P2: The blocking accessibility checks reject valid HTML attribute syntax.

Validation: all 28 added tests passed across both trees; all five changed mirror pairs are byte-identical; cross-plugin drift and both Heroku frontmatter checks passed; current GitHub checks are green. Additional probes reproduced the validator gaps in both copies, and a skill-directory packaging probe confirmed the missing script. These cases are not covered by the added tests.

Comment thread migrate/plugins/migration-to-aws/scripts/validate-heroku-migration-report.py Outdated
Comment thread migrate/plugins/migration-to-aws/scripts/validate-heroku-migration-report.py Outdated
@herosjourney

Copy link
Copy Markdown
Contributor Author

Thanks — reproduced all three before replying:

  1. Validator doesn't ship with a standalone skill install. Checked a real --skill heroku-to-aws install on disk: no scripts/ directory, no validator file anywhere under it, and $PLUGIN_ROOT is never defined inside the skill. This is a real blocker for that documented install path, not a hypothetical.
  2. Cost-optimization content check accepts decorative-only markup. Confirmed through the full CLI: a heading-only section and a table with headers but an empty <tbody> both pass REPORT_OK; a control case with an actual opportunity row behaves correctly once the section's other structural rules are met.
  3. Attribute regexes reject valid HTML syntax. Confirmed: scope="col" passes, scope = "col" and scope=col (both valid, equivalent HTML) fail the now-blocking validator.

All three will be fixed in both trees: bundle the validator inside the skill and resolve its path from the skill root, exclude headings/table headers from the content check and require real rows or the explicit no-eligible statement, and parse attributes properly instead of matching literal syntax. Replied inline on each with the specific fix.

Logan Kleier and others added 3 commits September 16, 2026 18:20
Three issues found in review, fixed in both plugin trees:

1. The validator lived at the plugin root (scripts/validate-heroku-migration-report.py)
   but Generate resolves it via `$PLUGIN_ROOT/scripts/...`. A standalone
   `npx skills add --skill heroku-to-aws` install (the documented single-skill
   install path in README.md) copies only the skill's own directory tree, not
   the plugin root's scripts/ — confirmed against a real prior install on disk
   with no scripts/ dir and no validator anywhere in it. Since this PR makes
   the validator mandatory and blocking, that install path could no longer
   complete Generate. Moved the validator (and its test) into
   skills/heroku-to-aws/scripts/, matching the existing agent-advisor /
   llm-to-bedrock precedent, and resolve it relative to `<SKILL_BASE>`
   (the harness-provided skill base directory) instead of `$PLUGIN_ROOT`
   in SKILL.md, generate.md, and generate-report.md.

2. The cost-optimization content check stripped all tags and asserted the
   remainder was non-empty, so a heading alone
   ("<h2>Cost Optimization Opportunities</h2>") or a table with column headers
   but an empty <tbody> both satisfied it — confirmed via the full CLI on both
   probes, including with a populated optimization_opportunities input.
   Strip headings and <th> cells before the emptiness check, so only real
   body content (an opportunity row, or the explicit no-eligible-commitment
   sentence) can pass it.

3. The `<th scope>`, `<html lang>`, and `<figure aria-label>` checks used
   literal-syntax regexes (`name="value"`), so valid-but-differently-spelled
   HTML — `scope = "col"` (spaces around =), `scope=col` (unquoted) — failed
   the now-blocking validator even though both are legal, equivalent HTML.
   Replaced the regexes with a stdlib HTMLParser-based scan that accepts any
   legal attribute syntax.

Added regression tests for all three: a packaging-location assertion, the
heading-only / headers-only-table content cases (plus a real-row control),
and the scope/lang/aria-label attribute-syntax variants.

Verified: 23/23 tests pass in each tree (14 original + 9 new), drift 277
identical / 25 allowlisted (the 2 moved files now count as identical twins),
fixtures/lint/frontmatter/model-id/types all clean, bandit/checkov/gitleaks
clean (added the same `# nosec B404` marker the equivalent llm-to-bedrock
test files already carry, now that this test moved out of the tests/
directory bandit's config excludes).

@leon1418 leon1418 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖]

Re-reviewed exact head 8ddd56436b48e1c5d166e9f6bb1d7d150b916188 against base 0f067dfe8b4da3f9cca969f1c2022a75b4b7f154, including all author replies, the full PR diff, both plugin trees, and the Generate worker/main-window handoff.

The standalone validator packaging P1 is fixed. I replied in all three original threads without creating duplicate findings. Two P2 concerns remain:

  • A table with no opportunity rows still passes when accompanied by the credits disclaimer that the report template requires.
  • The accessibility attributes are normalized, but verdict class attributes still produce false failures for valid syntax and miss an unquoted banned badge.

Local validation: 46/46 tests passed; isolated skill-only validator execution passed for both copies; all six changed surviving mirror pairs are byte-identical and the old script is removed in both trees; cross-plugin drift (277 identical, 25 allowlisted), both Heroku frontmatter checks, both vendored-shared checks, and git diff --check passed. I also inspected the Heroku allowlisted interpreter/schema differences and ran 40 targeted CLI probes; the remaining failures are reproduced in both copies. No full agent-driven migration replay was run.

Current GitHub CI is green, separately from those local results. Retaining Request changes until the two remaining validator gaps are addressed.

Logan Kleier added 2 commits September 17, 2026 22:32
…e gaps

Two P2s remained open on the report validator after the packaging P1 fix:

- cost-optimization content check accepted an empty <tbody> paired with the
  credits disclaimer prose (or any other explanatory text), because it only
  checked whether *any* text remained after stripping headings and <th>
  cells. Replaced with a positive check: require either a populated
  opportunity row (a <td> with text inside a <tbody>) or the exact
  no-eligible-commitment sentence — nothing else counts.

- _validate_verdict() matched class attributes with a literal
  class="value" regex, so class = "verdict-headline" or
  class=verdict-headline (both valid HTML) failed the verdict-headline
  requirement, and class=badge-verdict-go (unquoted) bypassed the pill ban
  entirely. Switched to the same HTMLParser-based tokenized-attribute
  approach already used for scope/lang/aria-label elsewhere in the file, via
  a new _class_tokens() helper.

Added regression tests for both, plus symmetric coverage for the
unquoted/spaced class-attribute variants on both the pill ban and the
headline requirement. 14 -> 29 tests, both plugin trees stay byte-identical.

Verification: pytest 29/29 passed in both trees; cross-plugin-drift.ts OK
(277 identical, 25 allowlisted); dprint check clean; frontmatter validator
OK. Manually reproduced both reviewer repro cases against the pre-fix
validator to confirm they now fail as expected.
…lidator-blocking

# Conflicts:
#	advisor/plugins/aws-startup-advisor/skills/heroku-to-aws/references/phases/generate/generate.md
#	migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/generate/generate.md

@herosjourney herosjourney left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up fixes (pushed as 4dddc4a1)

Both remaining P2s from the last review round are fixed in both plugin trees:

1. Cost-optimization content check accepted decorative-only markup. _validate_optimization_content() checked for "any text left after stripping headings/<th>", so an empty <tbody> paired with the credits disclaimer (or any other prose) satisfied it. Replaced with a positive check: pass only on a real <tbody> row with non-empty <td> text, or the exact no-eligible-commitment sentence. Nothing else counts. Replied inline with the specific repro coverage.

2. Verdict checks weren't using tokenized class attributes. _validate_verdict() still matched class="..." literally, unlike the scope/lang/aria-label checks that already went through HTMLParser. Added _class_tokens() (same _collect_tags() machinery) and switched both the badge-verdict-* ban and the verdict-headline requirement to set-membership checks. Replied inline with the specific repro coverage.

Also merged current main (this PR's branch had fallen behind — main since gained the rwx capability tier + Terraform policy gate on this same phase, and the resulting generate.md conflict was resolved by combining both main-window exceptions in the Scope Boundary paragraph: the policy checker's scoped shell during assembly, and the report validator's separate main-window "Finish Generate" step).

Verification (4dddc4a1, both trees)

  • test_validate_heroku_migration_report.py: 29 passed (14 → 29; added 6 regression tests for the two P2s plus retained coverage)
  • cross-plugin-drift.ts: OK (283 identical, 27 allowlisted)
  • mise run lint:frontmatter: OK (both trees)
  • mise run fmt:check (dprint): clean
  • mise run lint:md (markdownlint-cli2): 0 errors across 896 files
  • Manually reproduced both reviewer repro cases against the pre-fix validator to confirm they failed as described, then confirmed the fix resolves them

@leon1418 please re-review the current head when convenient.

@leon1418 leon1418 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖]

Re-reviewed exact head 4dddc4a14098ae1e4eb71a46cd8342215e2280ce against base b30cb39be02d2eae78cb8c482f4018435d4dc988, including the author's latest replies, the current diff, both plugin copies, and the merged rwx policy-checker/main-window report-validation integration.

The standalone packaging P1 remains fixed. The verdict-class P2 is now fixed, and the earlier credits-disclaimer repro is fixed. I replied in the two existing P2 threads. One P2 content-validation concern remains: the new regex row check accepts blank character-reference cells and commented-out rows, while rejecting a populated table whose source omits optional tbody tags. The original content thread contains the reproductions and suggested correction; no duplicate thread was created.

Local validation: 58/58 tests passed; all eight before/after checks reproduced the earlier bugs and verified the claimed corrections across both mirrors; isolated skill-only validator execution passed in both copies; 64 scoped CLI probes covered original and additional cases. Cross-plugin drift (283 identical, 27 allowlisted), both Heroku frontmatter checks, both vendored-shared checks, and git diff --check passed. The changed validator/tests/Generate files match across mirrors; the allowlisted SKILL.md difference is the existing advisor-only offers block. No full agent-driven migration replay or deployment was run.

Current GitHub CI is green, separately from those local checks. Retaining Request changes for the remaining content-validation gap.

Logan Kleier and others added 3 commits September 18, 2026 20:07
…raw HTML

_has_populated_opportunity_row() used three nested regex passes over raw
HTML source (<tbody>, then <tr>, then <td>) and stripped tags with
re.sub before .strip(). That has three gaps, all found by the same
reviewer repro:

- HTML character references (&nbsp;, &awslabs#160;) were never decoded, so a
  cell containing only "&nbsp;" was treated as non-empty literal text.
- The regex searched inside HTML comments, so a commented-out
  <tr><td>...</td></tr> still matched as a populated row.
- An explicit <tbody> tag was required, so <table><tr><td> with no
  <tbody> (valid HTML — browsers infer an implicit tbody) was never
  matched and a real opportunity row was rejected.

This is the third round on the same underlying requirement (real
opportunity content, not structural noise) and the prior two rounds
patched the regex incrementally. Replacing it with an HTMLParser
subclass (_OpportunityRowParser), mirroring the pattern already used
elsewhere in this file (_AccessibilityParser), fixes the whole class of
bug at once: HTMLParser decodes entities via convert_charrefs=True,
never routes comment text to handle_data, and tracks table/tr/td state
directly without depending on tbody presence.

Verified against all of the reviewer's repro cases plus the two
previously-fixed cases (heading-only, table-headers-only-with-empty-
tbody, empty-tbody-plus-credits-disclaimer, th-only row) and the
control case (real opportunity row with tbody) — all still behave
correctly. Added 4 new regression tests in both plugin trees
(nbsp-only cell, numeric-charref-only cell, commented-out row,
no-explicit-tbody-with-real-data) — 33/33 passing in both trees, up
from 29.
herosjourney pushed a commit to herosjourney/startups that referenced this pull request Sep 19, 2026
_section_html() in both validators matched section boundaries with a
literal-syntax regex:

- GCP's version required a literal `id="value"` (double-quoted only) —
  `<section id='exec-costs'>` (single-quoted, equally valid HTML) was
  never recognized.
- Heroku's version accepted both quote styles for `id=` but matched
  the closing tag as the literal string `</section>` — `</section >`
  (trailing whitespace) or a newline before the `>` was never
  recognized either.

Both cases silently defeated the required-cost-anchor check added in
this PR: `if exec_costs_html is not None:` skipped the whole
requirement whenever `_section_html` failed to extract the section,
even though the section was genuinely present and would render fine
in a browser. An unanchored, wrong cost figure inside a
differently-spelled-but-valid exec-costs section passed validation.

This is the same root cause as the heroku-to-aws report validator's
opportunity-row bug (PR awslabs#288): regex against raw markup instead of
parsing actual structure. Replaced both `_section_html` implementations
with an `HTMLParser` subclass (`_SectionScopeParser`) that finds the
target `<section id="...">` by parsed attributes (any legal spelling)
and its true matching close tag by counting nested same-name tags
(any legal spelling), rather than string-matching a specific
serialization of either.

Verified against both of the reviewer's exact repros (GCP:
single-quoted id; Heroku: whitespace/newline before the closing `>`)
plus every existing call site via the full pytest suite in both
plugin trees — 99/99 passing (up from 97), with 4 new regression
tests (2 GCP, 2 Heroku) that fail against the pre-fix code and pass
against the fix.

@leon1418 leon1418 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖]

Re-reviewed exact head b62d8c67aeddb7c9fa2fd36e7102a60ca71cb8ab against base ade57adaf9d776d18d316d28d3cee3f746582ced, including the author reply, current diff, both mirrors, and Generate's report-validator integration.

The latest entity-only, commented-row, and implicit-tbody cases are fixed. Packaging, substantive-content controls, and parsed verdict classes retain their earlier fixes. One P2 remains: a populated table with a valid omitted td end tag is rejected because the new parser only commits cell content on an explicit </td>. I added the reproduction and suggested correction to the original content thread; no duplicate thread was created.

Local validation: 66/66 tests passed; eight before/after comparisons reproduced and verified the newly claimed fixes in both copies; isolated skill-only validator execution passed in both copies. Of 64 scoped CLI probes, only the omitted-td-ending case remains incorrect (once per mirror). Cross-plugin drift (283 identical, 27 allowlisted), both Heroku frontmatter checks, both vendored-shared checks, and git diff --check passed. I checked the intentional Heroku SKILL.md/interpreter/schema differences and confirmed the main-window report-validation placement. No full agent-driven migration replay or deployment was run.

Current GitHub CI is green, separately from local validation. Recommendation: Request changes for the remaining parser boundary case.

herosjourney pushed a commit to herosjourney/startups that referenced this pull request Sep 19, 2026
…n-basis

Three findings from the latest review round, all involving the
decision-gate re-entry state machine and its validator:

1. Preserve customer state instead of recursively deleting terraform/
   (P1, data-loss risk). workshop-refresh.md's Stale Generate guard
   deleted `terraform/` recursively on re-entry. That directory is not
   purely generated: generate-terraform.md tells users to create
   terraform.tfvars (gitignored, holds real values, never
   regenerated), and never emits *.tfstate/*.tfstate.backup or a
   .terraform/ provider cache -- those come from the user's own
   `terraform apply`/`terraform init` and are often the only local
   record of what's actually deployed. Recursive deletion destroyed
   all of that. Replaced with an explicit, named list of only the
   files each Generate fragment's own `_contributes:` frontmatter
   declares it writes -- every other file under terraform/ (tfvars,
   tfstate, .terraform/, or anything else the customer added) is now
   left untouched.

2. Move the stale-execution-pack cleanup to the single point every
   re-entry path actually passes through. The cleanup was gated behind
   workshop-refresh.md's own `phases.generate == "completed"` check,
   but workshop.md Entry step 2 already resets phases.generate to
   "pending" via reset_downstream_to_pending before workshop-refresh.md
   is ever reached -- so that guard's condition could never fire on the
   very re-entry it was meant to catch. Worse, the "Compare scenarios"
   loop branch never touches workshop-refresh.md at all, so it could
   return to the Decision gate with the old execution pack still on
   disk regardless. Moved the guard (detection + confirm + deletion)
   into workshop.md Entry step 2 itself, which every re-entry path
   (Apply & reprice, Compare scenarios, or exiting workshop without
   entering refresh) passes through before any branch is chosen.
   workshop-refresh.md's own step 2 now just documents that this
   already happened upstream, rather than re-deriving a check that can
   never fire.

3. Parse actual <section> elements for decision-basis, excluding
   comments. _section_counts() (the older, pre-awslabs#288-relocation copy of
   the Heroku report validator) matched <section id="..."> via a raw
   regex, so the unexpanded skeleton placeholder comment from
   generate-report.md -- `<!-- <section id="decision-basis"> when
   recommendation.decision_basis exists -->` -- was counted as a real,
   present section even though it renders nothing. A report that
   declared decision_basis but shipped the template comment verbatim
   returned REPORT_OK. Same root cause as PRs awslabs#288/awslabs#289 in this
   session: regex against raw markup instead of parsed structure.
   Replaced with an HTMLParser subclass (_SectionOpenTagCollector)
   that only counts real start tags -- comments are never re-tokenized
   as tags.

Verified: (1) and (2) are DSL/prose fixes with no executable unit
under test in this repo (the reviewer's own stated verification method
for these files is tracing the documented state transitions, not a
live agent run) -- checked for internal consistency between
workshop.md and workshop-refresh.md and confirmed the golden
heroku-workshop/heroku-decision-gate fixture asserters still pass. (3)
is code: added a regression test using the exact placeholder text from
generate-report.md, confirmed it fails against the pre-fix code
(REPORT_OK when it should fail) and passes against the fix -- 19/19
tests passing in both trees. Full repo verification also green:
fixtures:assert (9 asserters), cross-plugin-drift (282 identical, 27
allowlisted), node test suites, dprint, markdownlint, lint:frontmatter,
git diff --check.
_OpportunityRowParser only registered a populated <td> on an explicit
handle_endtag("td"). The HTML Standard permits a <td> end tag to be
omitted immediately before the next <td>/<th>, or before its parent
<tr>/<table> closes -- so a perfectly valid, compact table like
<table><tr><td>RDS Reserved Instance</tr></table> never fired
handle_endtag("td") at all, and its real content silently registered
as empty, failing an otherwise-valid report.

Root cause: the parser tracked "is a <td> open" as a depth counter
incremented on every <td> start tag, with no concept of implicit
closure -- a new <td>/<th> starting, or the row/table ending, should
each first finalize whatever cell was still open, exactly as a real
HTML parser does, but this parser only checked accumulated text on an
explicit close.

Replaced the depth counter with a single _close_cell() finalization
step, invoked whenever a <td>, <th>, or self-closed cell tag starts
(closing any sibling cell still open in the same row), when a <tr>
opens (closing any cell that leaked from an earlier, malformed row),
and on every actual close of <td>/<th>/<tr>/<table>. This correctly
handles the reviewer's repro plus adjacent edge cases: sibling cells
in the same row sharing omitted end tags, and cross-row leakage
(a cell opened in one row must never be finalized against a later
row's content).

Verified against the reviewer's exact repro
(<table><tr><td>RDS Reserved Instance</tr></table>) plus 8 additional
edge cases (sibling omitted cells, th/td boundary interaction,
cross-row leakage in both directions, blank omitted-end cells still
correctly failing) and all 9 previously-fixed regressions from the
prior three rounds on this same function. Confirmed the two new
positive-case tests fail against the pre-fix code (REPORT_FAIL where
REPORT_OK is expected, matching the reviewer's report) and pass
against the fix. 36/36 tests passing in both trees (up from 33).
fixtures:assert (8 asserters), cross-plugin-drift (283 identical, 27
allowlisted), dprint, markdownlint, git diff --check all clean. No
merge from origin/main needed this round -- already caught up, no new
commits landed since the last fix.

@leon1418 leon1418 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖]

Completed the follow-up review of exact head e3ca4d232e9592c742e40dcaf30903f0e876ca90 against base ade57adaf9d776d18d316d28d3cee3f746582ced. The optional-td-end finding is fixed, and all previously reported packaging, content-validation, and attribute/class findings remain addressed. I found no remaining substantive concerns and replied in the original content thread.

Validation: 72/72 tests passed across both trees; 92/92 scoped CLI probes matched their expected outcomes; ten before/after comparisons reproduced the prior failures and verified the corrected behavior and negative controls. Isolated skill-only validator execution passed for both copies. Cross-plugin drift (283 identical, 27 allowlisted), both Heroku frontmatter checks, both vendored-shared checks, and git diff --check passed. The intentional SKILL.md/interpreter/schema differences and the main-window report-validation integration were checked. No full agent-driven migration replay or deployment was run.

Current GitHub CI is green, separately from local validation. Recommendation: Approve, for the user to submit. This is a COMMENT follow-up only; I have not submitted approval, dismissed prior reviews, or resolved conversations. The existing blocking review state still requires the user's action.

Logan Kleier added 2 commits September 21, 2026 13:18
Resolves a modify/delete conflict on validate-heroku-migration-report.py in
both plugin trees:

- This branch RELOCATED the Heroku report validator from the plugin root
  (scripts/validate-heroku-migration-report.py) into the skill
  (skills/heroku-to-aws/scripts/) and hardened it with a decision-UX + a11y
  contract (verdict-headline, banned badge pills, <html lang>, <th scope>,
  <figure> aria-label/figcaption, populated cost-optimization content).
- origin/main INDEPENDENTLY added a currency-formatting check to the OLD-path
  file (PR awslabs#310 line of work: CENTS_RE / _RATE_SUFFIX_RE / decoded-text +
  block-boundary parsing) plus a currency test suite pointing at the old path.

A naive "keep the delete" would have silently dropped main's
currency-formatting feature and left main's currency test pointing at a
deleted path. Instead:

- Kept the relocation (validator stays in the skill).
- Ported main's currency-formatting block verbatim into the relocated
  validator (helpers + `errors.extend(_validate_currency_formatting(html))`),
  additive to the branch's a11y contract — no name collisions.
- Moved main's currency test into skills/heroku-to-aws/scripts/ alongside the
  validator and the branch's structural test, switching its locator to
  `Path(__file__).resolve().parent`; updated its GOOD fixture to satisfy the
  branch's stricter structural contract (adds the cost-optimization section).
- Mirrored validator + currency test byte-identically across advisor/ and
  migrate/ trees.

Verification: heroku-to-aws/scripts suites 54 pass per tree (currency 18 +
structural 36); shared validate-migration-report suites 92 pass per tree;
emit-plan-json 46 pass; drift:check OK (284 identical, 27 allowlisted);
fixtures:assert PASS; fmt:check clean; lint:md 0 errors. Confirmed the ported
currency check fires on the SF Beach drift case ($25,684.89/mo).
The merge moved the Heroku currency test out of tests/ (which the bandit CI
job excludes via `-x .../tests`) into skills/heroku-to-aws/scripts/, which IS
scanned. Bandit then flagged B404 (import subprocess) on the test's import.
Add `# nosec B404 — test-only, list args, no shell, committed script path` to
the import line, matching the sibling test_validate_heroku_migration_report.py
that already lives in that directory. The subprocess.run call already carries
`# nosec B603`.

Verified: `mise run security:bandit` clean ("No issues identified"); currency
suite 18 pass per tree; both tree copies byte-identical.

@leon1418 leon1418 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖]

Reviewed exact head 2ecdcfcd06c10b0a3ce5e2fd6f6882bc7256e5b9 against base 811fddafced0870173dc9c1a66749a9143baa040, including the currency-validation merge, relocated tests, standalone skill layout, both plugin copies, and the current discussions. The previously reported packaging, content, optional-td-ending, and attribute/class issues remain fixed.

One integration follow-up is inline: the now-mandatory currency check requires whole-dollar monthly values, but the dispatched Heroku report writer and its self-check do not carry that rule. Please align the producer instructions with the gate. Recommendation: Request changes for this P2 instruction/validator mismatch.

Validation: 158 focused tests passed; all 122 Heroku CLI probes and four isolated skill-packaging checks passed. Of 74 GCP currency probes, two expose an implicit-cell-boundary behavior already present in the byte-identical merged-base GCP validator; I am not attributing it to this PR. The relocated Heroku currency helpers match main, and the prior structural/verdict helpers are unchanged. Cross-plugin drift (284 identical, 27 allowlisted), both Heroku frontmatter checks, both vendored-shared checks, and git diff --check passed. GitHub CI is green separately from local validation. No full agent-driven migration replay or deployment was performed.

This is a COMMENT review; the existing human approval and historical reviews are left untouched.

@leon1418 leon1418 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖]

Completed startups-hybrid-v1 review of exact head 571f7daaaa419e221e6d4c9fa5404ea9d5c19778 against base ec681ba5d49b039dbf1ef9d87d3304205b824f3d: actual OCR v1.12.5 delegation, one fresh general reviewer, the parent repository review, and independent candidate verification. All 18 Git inventory entries were covered, including Markdown and deletions. The canonical evidence gate passed.

Two verified P2 concerns remain, updated in the original threads:

  1. The dispatched report writer still lacks the monthly rounding instruction required by the blocking currency gate.
  2. The no-commitment fallback rejects valid encoded text and can accept content present only in comments/inert templates.

Local validation: the 108 authored tests passed in both independent perspectives; the parent ran 34 CLI/packaging checks, the general reviewer ran 48 focused base/head CLI probes and four packaging checks, and the parent independently ran 14 fallback verification cases. Drift, frontmatter, vendored-source, and diff checks passed. Current GitHub CI is green separately from those results. No full agent-driven migration replay, browser-accessibility run, or deployment is claimed.

Recommendation: Request changes for the two P2s. No approval, dismissal, merge, or conversation-resolution action was performed.

…ort worker the currency rule

Closes the two remaining P2s on the blocking Heroku report validator:

1. No-commitment fallback rejected valid encoded text and accepted invisible
   text. `_validate_optimization_content` matched the sentence with a raw
   `re.sub("<[^>]+>", " ")` tag-strip that never decoded entities or excluded
   inert/comment content — so `No 1&awslabs#45;year/3&awslabs#45;year …` and `&nbsp;`-separated
   forms FAILED, while the sentence present only inside a <template> passed.
   Now extracts RENDERED text via the existing _DecodedTextParser (entities
   decoded, comments + script/style/template excluded), normalizes whitespace
   incl. NBSP, then checks — so the same parser the currency gate uses decides
   what "rendered" means. Encoded/nbsp sentences pass; comment/template-only fail.

2. The report worker was never told the (now blocking) whole-dollar rule.
   generate-report.md's authoring rules and Step-3 self-check never required
   whole-dollar monthly figures, so a worker could emit `$112.34/mo` and only
   discover the failure at the blocking currency gate (which cannot re-author in
   the shell-less worker). Added a "Whole-dollar monthly figures (blocking)"
   authoring rule and a self-check item 8 so the worker rounds monthly-scale
   figures before returning.

Four regression tests added (entity-hyphen sentence passes, nbsp sentence
passes, comment-only fails, template-only fails); the entity/nbsp/template ones
verified to fail against the pre-fix parser. Both plugin trees byte-identical;
heroku validator suite 40 pass per tree; drift:check OK (284 identical);
fmt:check clean; lint:md 0 errors; security:bandit clean.

@leon1418 leon1418 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖]

Completed startups-hybrid-v1 at exact head e538f368cb388fa36e6dbbb0c4e8cd52c58650d2 against base ec681ba5d49b039dbf1ef9d87d3304205b824f3d: OCR v1.12.5 delegation, one fresh general review, parent repository review, independent candidate verification, and the canonical gate. All 18 Git entries were covered. The actual child's initial-context disclosure and recorded tool audit passed the isolation checks.

The currency-authoring and decoded-fallback fixes are verified. Two P2 concerns remain: inert-only opportunity rows can satisfy the content check, and an inert-only verdict headline can satisfy the declared-outcome check. Both were independently reproduced with supported Estimate inputs and updated in the original threads. The base lacked these positive-content requirements; this is incomplete enforcement of the newly introduced guarantees.

Validation: 116 head tests passed in both perspectives; the general reviewer also ran 36 base currency tests. The parent ran 54 CLI/packaging checks and 26 authoritative candidate-verification executions. Drift, frontmatter, vendored-source, and diff checks passed. GitHub CI is green separately from local results. No full agent-driven migration replay, browser-accessibility run, or deployment was performed.

Recommendation: Request changes for the inert-content cases. No approval, dismissal, merge, or conversation-resolution action was performed.

…nt check

Close the inert/hidden-content blind spot as a CLASS, not one parser per
review round. The decoded-fallback check already excluded inert subtrees
(<script>/<style>/<template>); its sibling checks in the same file did not,
so unrendered content could still satisfy a blocking gate:

- _TagAttrCollector (feeds _class_tokens -> _validate_verdict): skip tags
  inside inert subtrees, so a `verdict-headline` declared only inside a
  <template> is no longer read as a rendered headline when an Estimate
  declares recommendation.outcome.
- _OpportunityRowParser: track inert ancestors, so a <template>/<script>-only
  table cell (or a whole table nested in a <template>) no longer registers as
  a populated opportunity row.
- _AccessibilityParser: skip inert subtrees, so a <th>/<figure> that exists
  only inside a <template> is not audited for scope/aria-label (removes a
  false-positive symmetric to the same blind spot).

All three now mirror _DecodedTextParser's inert handling, so every
"is this rendered?" check in the file agrees on the answer.

Adds inert-only negative controls (inert opportunity table, inert-only cell,
verdict-headline-only-in-template, th-in-template-not-audited); each fails
against the pre-fix script and passes after. Mirrored byte-identical across
the advisor and migrate trees.

@leon1418 leon1418 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🤖 AI review 🤖]

Completed startups-hybrid-v1 with convergence v1 at exact head ab95346fcad2bc3f432489d09a7c1696a5b9f7e3, base ec681ba5d49b039dbf1ef9d87d3304205b824f3d: OCR v1.12.5 delegation, one fresh general reviewer, separate parent review and impact maps, candidate verification, and the canonical evidence gate. All 18 Git entries were covered.

The two previously reported inner-inert cases are fixed. Packaging, currency guidance, decoded fallback text, optional cell endings and class handling retain their fixes. Two consolidated P2 corrections remain:

  1. Section parsing: in both skill-local validators and structural test copies, use actual parsed section IDs with document/comment/inert context. Invisible sections and data-id must not satisfy the gate; real quoted/spaced/unquoted IDs must pass; commented duplicates and the what-if skeleton comment must not count. The original content and verdict threads carry the reproductions.
  2. Missing-Python instructions: update the tooling fallback in both SKILL.md copies so it no longer promises Generate completion without successful report validation. Earlier phases can continue; the existing missing/not_run gate stays blocking. This is inline.

Both gaps already existed at e538f368; I missed them in the prior review. Neither is attributed to the latest inert-parser fix.

Local evidence: 124 authored tests passed in each perspective; the parent ran 54 CLI/packaging checks and 76 authoritative before/after boundary executions with preserved inputs. The fresh reviewer separately reproduced the section issue. Drift, frontmatter, vendored-source and diff checks passed. Current GitHub CI reports nine successful checks, separately from local results. No full agent-driven migration, missing-Python host replay, browser accessibility suite or deployment was performed.

Recommendation: Request changes. GitHub also reports merge conflicts, a separate landing blocker. Approval remains for the user; no approval or review dismissal was performed.

Comment thread migrate/plugins/migration-to-aws/skills/heroku-to-aws/SKILL.md Outdated
Logan Kleier added 3 commits September 22, 2026 22:11
# Conflicts:
#	advisor/plugins/aws-startup-advisor/scripts/validate-heroku-migration-report.py
#	advisor/plugins/aws-startup-advisor/skills/heroku-to-aws/references/phases/generate/generate-assemble.md
#	advisor/plugins/aws-startup-advisor/skills/heroku-to-aws/references/phases/generate/generate-report.md
#	migrate/plugins/migration-to-aws/scripts/validate-heroku-migration-report.py
#	migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/generate/generate-assemble.md
#	migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/generate/generate-report.md
…ated validator

The heroku-decision-gate golden asserters (check_expected_decide.py, and
check_expected_decide_with_retained_pack.py which imports its main) still resolved
the validator at PLUGIN_ROOT/scripts/validate-heroku-migration-report.py — the
pre-relocation path. This PR moved the validator into the skill
(skills/heroku-to-aws/scripts/), so both goldens failed fixtures:assert with
"No such file or directory" when validating decision-report.html --mode decision.

Point VALIDATOR at the skill-relocated path. Both plugin copies byte-identical;
fixtures:assert now PASS (10 asserters, 6 golden + 4 smoke) on both trees;
drift/fmt/lint:md/bandit green.
@leon1418
leon1418 merged commit 96bdb94 into awslabs:main Sep 23, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants