feat(heroku-to-aws): make report validation blocking + harden it - #288
Conversation
c9ae7fa to
b8f4260
Compare
Follow-up review + fixes (pushed as
|
…(P1-C) The report validators checked structure, decision-UX, and a11y but never confirmed the rendered dollar figures matched the estimate — the HTML could drift from estimation-infra.json silently. This adds a numeric cross-check. - validate-migration-report.py (GCP) + validate-heroku-migration-report.py: new _validate_cost_figures — every data-cost-key-anchored dollar figure must equal the matching estimation-infra.json value (GCP: aws_monthly_balanced, current_monthly->current_costs.gcp_monthly, optional premium/optimized; Heroku: aws_monthly_balanced only — current-spend deferred, key unsettled). Skip-on-absent (no anchor / no JSON / no estimate); FAIL only on a real mismatch, citing JSON path vs HTML excerpt. Untagged figures are never checked. - Emitters (generate-artifacts-report.md rule 21, heroku generate-report.md): instruct emitting the data-cost-key anchors (attribute, not reader text). - migration-report-reference.html: anchors the balanced + current figures. - tests: mismatch->FAIL, match->PASS, no-estimate->skip, untagged->skip. - docs/report-quality.md: numeric-check scope note. Both plugin trees updated byte-identically. Heroku validator + generate-report.md also touched by the report-blocking PR (awslabs#288); land order noted in the PR.
b8f4260 to
2868f78
Compare
|
Verified additional issues and each point against the code at 1. Should-fix — validator was still in the worker, not the main window (agreed, blocking)Confirmed: the
2. Nit — GATE_FAIL should quote the validator errors (agreed)Right — the fragment has no shell to re-run the validator, so a bare "re-run Generate" is not actionable. The finish step now emits 3. Nit — skeleton had no
|
The heroku-to-aws report validator was optional/non-blocking with zero tests, and generate-report.md contradicted generate-assemble.md on whether it gates. This makes it a required, blocking gate and hardens it to the report's own decision-UX + accessibility contract (ported subset of the gcp-to-aws awslabs#223 checks, scoped to what the thin Heroku one-pager emits). Round-2 review (host-placement): the validator invocation was in the dispatched assembler/fragment (no shell under _exec: rw), so 'blocking' was prose on a path that can't execute. Mirrors the awslabs#287 split: the python3 validator run + a REPORT_OK assert now live in a main-window 'Finish Generate (report validation)' step in generate.md that runs BEFORE the read-only _postconditions gate. On REPORT_FAIL the step emits GATE_FAIL and pastes the validator errors[] verbatim (so recovery is actionable, not a blind 're-run Generate'). The gate is read-only; the assembler no longer claims to be 'the main window'. - validate-heroku-migration-report.py: require cost-optimization; forbid badge-verdict-* pills; require verdict-headline when recommendation.outcome present; a11y subset (html lang, th scope, figure aria-label+figcaption) - generate.md: 'Finish Generate' main-window validator step + read-only REPORT_OK _assert; Scope-Boundary carve-out for the main-window validate - generate-report.md: points at the finish step; GATE_FAIL pastes errors[]; skeleton gains a <th scope=col> example so agents copy the a11y pattern - generate-assemble.md: read-only gate, no validator call, recovery = hand-edit + targeted re-run (not full Generate re-dispatch, which wipes the fix) - tests: 13 -> 14 cases (add figure-with-aria-label-without-figcaption FAIL) Deliberately not ported (would false-fail the thin report): decision-before-TOC, single-h1, per-table caption, glossary/appendix set. Both plugin trees updated byte-identically.
2868f78 to
2bce3f5
Compare
|
Fixed 3 additional nits. 1.
|
…(P1-C) The report validators checked structure, decision-UX, and a11y but never confirmed the rendered dollar figures matched the estimate — the HTML could drift from estimation-infra.json silently. This adds a numeric cross-check. - validate-migration-report.py (GCP) + validate-heroku-migration-report.py: new _validate_cost_figures — every data-cost-key-anchored dollar figure must equal the matching estimation-infra.json value (GCP: aws_monthly_balanced, current_monthly->current_costs.gcp_monthly, optional premium/optimized; Heroku: aws_monthly_balanced only — current-spend deferred, key unsettled). - Required keys fail CLOSED: when a required figure's JSON value exists and exec-costs is rendered, a MISSING anchor is a FAIL (an un-anchored wrong figure must not pass — the bug P1-C exists to catch). Gated on require_anchors so minimal --no-require-toc unit fixtures are not forced to anchor. Only untagged *illustrative* numbers and absent optional per-tier keys are skipped. - Safe int-coerce (non-whole-dollar JSON -> named FAIL, never a crash); nested markup stripped before compare; mismatch/no-$ = FAIL citing JSON path vs HTML. - Emitters (generate-artifacts-report.md rule 21, heroku generate-report.md) + docs describe the REQUIRED-anchor rule (round-2: the three author-facing lines that still said 'untagged skipped / missing anchor skipped' now state required anchors are mandatory; illustrative extras stay skipped). - migration-report-reference.html: anchors the balanced + current figures. - tests: mismatch->FAIL, match->PASS, no-estimate->skip (asserts code==0), untagged->skip, nested-markup, required-absent FAIL for BOTH balanced and current_monthly, non-numeric->named FAIL. 83 passed (77 GCP + 6 Heroku cost). Both plugin trees updated byte-identically. Heroku validator + generate-report.md also touched by the report-blocking PR (awslabs#288); land order noted in the PR.
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Reviewed head d35f4b8bb65d673d541ed03ab9f7f0346abc32d0, including the follow-up explanations and both plugin copies. Requesting changes for the three inline findings:
- P1: The newly mandatory validator is not included in the documented skill-only installation.
- P2: Heading-only cost-optimization sections and tables without data rows still pass.
- P2: The blocking accessibility checks reject valid HTML attribute syntax.
Validation: all 28 added tests passed across both trees; all five changed mirror pairs are byte-identical; cross-plugin drift and both Heroku frontmatter checks passed; current GitHub checks are green. Additional probes reproduced the validator gaps in both copies, and a skill-directory packaging probe confirmed the missing script. These cases are not covered by the added tests.
|
Thanks — reproduced all three before replying:
All three will be fixed in both trees: bundle the validator inside the skill and resolve its path from the skill root, exclude headings/table headers from the content check and require real rows or the explicit no-eligible statement, and parse attributes properly instead of matching literal syntax. Replied inline on each with the specific fix. |
Three issues found in review, fixed in both plugin trees:
1. The validator lived at the plugin root (scripts/validate-heroku-migration-report.py)
but Generate resolves it via `$PLUGIN_ROOT/scripts/...`. A standalone
`npx skills add --skill heroku-to-aws` install (the documented single-skill
install path in README.md) copies only the skill's own directory tree, not
the plugin root's scripts/ — confirmed against a real prior install on disk
with no scripts/ dir and no validator anywhere in it. Since this PR makes
the validator mandatory and blocking, that install path could no longer
complete Generate. Moved the validator (and its test) into
skills/heroku-to-aws/scripts/, matching the existing agent-advisor /
llm-to-bedrock precedent, and resolve it relative to `<SKILL_BASE>`
(the harness-provided skill base directory) instead of `$PLUGIN_ROOT`
in SKILL.md, generate.md, and generate-report.md.
2. The cost-optimization content check stripped all tags and asserted the
remainder was non-empty, so a heading alone
("<h2>Cost Optimization Opportunities</h2>") or a table with column headers
but an empty <tbody> both satisfied it — confirmed via the full CLI on both
probes, including with a populated optimization_opportunities input.
Strip headings and <th> cells before the emptiness check, so only real
body content (an opportunity row, or the explicit no-eligible-commitment
sentence) can pass it.
3. The `<th scope>`, `<html lang>`, and `<figure aria-label>` checks used
literal-syntax regexes (`name="value"`), so valid-but-differently-spelled
HTML — `scope = "col"` (spaces around =), `scope=col` (unquoted) — failed
the now-blocking validator even though both are legal, equivalent HTML.
Replaced the regexes with a stdlib HTMLParser-based scan that accepts any
legal attribute syntax.
Added regression tests for all three: a packaging-location assertion, the
heading-only / headers-only-table content cases (plus a real-row control),
and the scope/lang/aria-label attribute-syntax variants.
Verified: 23/23 tests pass in each tree (14 original + 9 new), drift 277
identical / 25 allowlisted (the 2 moved files now count as identical twins),
fixtures/lint/frontmatter/model-id/types all clean, bandit/checkov/gitleaks
clean (added the same `# nosec B404` marker the equivalent llm-to-bedrock
test files already carry, now that this test moved out of the tests/
directory bandit's config excludes).
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Re-reviewed exact head 8ddd56436b48e1c5d166e9f6bb1d7d150b916188 against base 0f067dfe8b4da3f9cca969f1c2022a75b4b7f154, including all author replies, the full PR diff, both plugin trees, and the Generate worker/main-window handoff.
The standalone validator packaging P1 is fixed. I replied in all three original threads without creating duplicate findings. Two P2 concerns remain:
- A table with no opportunity rows still passes when accompanied by the credits disclaimer that the report template requires.
- The accessibility attributes are normalized, but verdict
classattributes still produce false failures for valid syntax and miss an unquoted banned badge.
Local validation: 46/46 tests passed; isolated skill-only validator execution passed for both copies; all six changed surviving mirror pairs are byte-identical and the old script is removed in both trees; cross-plugin drift (277 identical, 25 allowlisted), both Heroku frontmatter checks, both vendored-shared checks, and git diff --check passed. I also inspected the Heroku allowlisted interpreter/schema differences and ran 40 targeted CLI probes; the remaining failures are reproduced in both copies. No full agent-driven migration replay was run.
Current GitHub CI is green, separately from those local results. Retaining Request changes until the two remaining validator gaps are addressed.
…e gaps Two P2s remained open on the report validator after the packaging P1 fix: - cost-optimization content check accepted an empty <tbody> paired with the credits disclaimer prose (or any other explanatory text), because it only checked whether *any* text remained after stripping headings and <th> cells. Replaced with a positive check: require either a populated opportunity row (a <td> with text inside a <tbody>) or the exact no-eligible-commitment sentence — nothing else counts. - _validate_verdict() matched class attributes with a literal class="value" regex, so class = "verdict-headline" or class=verdict-headline (both valid HTML) failed the verdict-headline requirement, and class=badge-verdict-go (unquoted) bypassed the pill ban entirely. Switched to the same HTMLParser-based tokenized-attribute approach already used for scope/lang/aria-label elsewhere in the file, via a new _class_tokens() helper. Added regression tests for both, plus symmetric coverage for the unquoted/spaced class-attribute variants on both the pill ban and the headline requirement. 14 -> 29 tests, both plugin trees stay byte-identical. Verification: pytest 29/29 passed in both trees; cross-plugin-drift.ts OK (277 identical, 25 allowlisted); dprint check clean; frontmatter validator OK. Manually reproduced both reviewer repro cases against the pre-fix validator to confirm they now fail as expected.
…lidator-blocking # Conflicts: # advisor/plugins/aws-startup-advisor/skills/heroku-to-aws/references/phases/generate/generate.md # migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/generate/generate.md
herosjourney
left a comment
There was a problem hiding this comment.
Follow-up fixes (pushed as 4dddc4a1)
Both remaining P2s from the last review round are fixed in both plugin trees:
1. Cost-optimization content check accepted decorative-only markup. _validate_optimization_content() checked for "any text left after stripping headings/<th>", so an empty <tbody> paired with the credits disclaimer (or any other prose) satisfied it. Replaced with a positive check: pass only on a real <tbody> row with non-empty <td> text, or the exact no-eligible-commitment sentence. Nothing else counts. Replied inline with the specific repro coverage.
2. Verdict checks weren't using tokenized class attributes. _validate_verdict() still matched class="..." literally, unlike the scope/lang/aria-label checks that already went through HTMLParser. Added _class_tokens() (same _collect_tags() machinery) and switched both the badge-verdict-* ban and the verdict-headline requirement to set-membership checks. Replied inline with the specific repro coverage.
Also merged current main (this PR's branch had fallen behind — main since gained the rwx capability tier + Terraform policy gate on this same phase, and the resulting generate.md conflict was resolved by combining both main-window exceptions in the Scope Boundary paragraph: the policy checker's scoped shell during assembly, and the report validator's separate main-window "Finish Generate" step).
Verification (4dddc4a1, both trees)
test_validate_heroku_migration_report.py: 29 passed (14 → 29; added 6 regression tests for the two P2s plus retained coverage)cross-plugin-drift.ts: OK (283 identical, 27 allowlisted)mise run lint:frontmatter: OK (both trees)mise run fmt:check(dprint): cleanmise run lint:md(markdownlint-cli2): 0 errors across 896 files- Manually reproduced both reviewer repro cases against the pre-fix validator to confirm they failed as described, then confirmed the fix resolves them
@leon1418 please re-review the current head when convenient.
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Re-reviewed exact head 4dddc4a14098ae1e4eb71a46cd8342215e2280ce against base b30cb39be02d2eae78cb8c482f4018435d4dc988, including the author's latest replies, the current diff, both plugin copies, and the merged rwx policy-checker/main-window report-validation integration.
The standalone packaging P1 remains fixed. The verdict-class P2 is now fixed, and the earlier credits-disclaimer repro is fixed. I replied in the two existing P2 threads. One P2 content-validation concern remains: the new regex row check accepts blank character-reference cells and commented-out rows, while rejecting a populated table whose source omits optional tbody tags. The original content thread contains the reproductions and suggested correction; no duplicate thread was created.
Local validation: 58/58 tests passed; all eight before/after checks reproduced the earlier bugs and verified the claimed corrections across both mirrors; isolated skill-only validator execution passed in both copies; 64 scoped CLI probes covered original and additional cases. Cross-plugin drift (283 identical, 27 allowlisted), both Heroku frontmatter checks, both vendored-shared checks, and git diff --check passed. The changed validator/tests/Generate files match across mirrors; the allowlisted SKILL.md difference is the existing advisor-only offers block. No full agent-driven migration replay or deployment was run.
Current GitHub CI is green, separately from those local checks. Retaining Request changes for the remaining content-validation gap.
…raw HTML _has_populated_opportunity_row() used three nested regex passes over raw HTML source (<tbody>, then <tr>, then <td>) and stripped tags with re.sub before .strip(). That has three gaps, all found by the same reviewer repro: - HTML character references ( , &awslabs#160;) were never decoded, so a cell containing only " " was treated as non-empty literal text. - The regex searched inside HTML comments, so a commented-out <tr><td>...</td></tr> still matched as a populated row. - An explicit <tbody> tag was required, so <table><tr><td> with no <tbody> (valid HTML — browsers infer an implicit tbody) was never matched and a real opportunity row was rejected. This is the third round on the same underlying requirement (real opportunity content, not structural noise) and the prior two rounds patched the regex incrementally. Replacing it with an HTMLParser subclass (_OpportunityRowParser), mirroring the pattern already used elsewhere in this file (_AccessibilityParser), fixes the whole class of bug at once: HTMLParser decodes entities via convert_charrefs=True, never routes comment text to handle_data, and tracks table/tr/td state directly without depending on tbody presence. Verified against all of the reviewer's repro cases plus the two previously-fixed cases (heading-only, table-headers-only-with-empty- tbody, empty-tbody-plus-credits-disclaimer, th-only row) and the control case (real opportunity row with tbody) — all still behave correctly. Added 4 new regression tests in both plugin trees (nbsp-only cell, numeric-charref-only cell, commented-out row, no-explicit-tbody-with-real-data) — 33/33 passing in both trees, up from 29.
_section_html() in both validators matched section boundaries with a literal-syntax regex: - GCP's version required a literal `id="value"` (double-quoted only) — `<section id='exec-costs'>` (single-quoted, equally valid HTML) was never recognized. - Heroku's version accepted both quote styles for `id=` but matched the closing tag as the literal string `</section>` — `</section >` (trailing whitespace) or a newline before the `>` was never recognized either. Both cases silently defeated the required-cost-anchor check added in this PR: `if exec_costs_html is not None:` skipped the whole requirement whenever `_section_html` failed to extract the section, even though the section was genuinely present and would render fine in a browser. An unanchored, wrong cost figure inside a differently-spelled-but-valid exec-costs section passed validation. This is the same root cause as the heroku-to-aws report validator's opportunity-row bug (PR awslabs#288): regex against raw markup instead of parsing actual structure. Replaced both `_section_html` implementations with an `HTMLParser` subclass (`_SectionScopeParser`) that finds the target `<section id="...">` by parsed attributes (any legal spelling) and its true matching close tag by counting nested same-name tags (any legal spelling), rather than string-matching a specific serialization of either. Verified against both of the reviewer's exact repros (GCP: single-quoted id; Heroku: whitespace/newline before the closing `>`) plus every existing call site via the full pytest suite in both plugin trees — 99/99 passing (up from 97), with 4 new regression tests (2 GCP, 2 Heroku) that fail against the pre-fix code and pass against the fix.
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Re-reviewed exact head b62d8c67aeddb7c9fa2fd36e7102a60ca71cb8ab against base ade57adaf9d776d18d316d28d3cee3f746582ced, including the author reply, current diff, both mirrors, and Generate's report-validator integration.
The latest entity-only, commented-row, and implicit-tbody cases are fixed. Packaging, substantive-content controls, and parsed verdict classes retain their earlier fixes. One P2 remains: a populated table with a valid omitted td end tag is rejected because the new parser only commits cell content on an explicit </td>. I added the reproduction and suggested correction to the original content thread; no duplicate thread was created.
Local validation: 66/66 tests passed; eight before/after comparisons reproduced and verified the newly claimed fixes in both copies; isolated skill-only validator execution passed in both copies. Of 64 scoped CLI probes, only the omitted-td-ending case remains incorrect (once per mirror). Cross-plugin drift (283 identical, 27 allowlisted), both Heroku frontmatter checks, both vendored-shared checks, and git diff --check passed. I checked the intentional Heroku SKILL.md/interpreter/schema differences and confirmed the main-window report-validation placement. No full agent-driven migration replay or deployment was run.
Current GitHub CI is green, separately from local validation. Recommendation: Request changes for the remaining parser boundary case.
…n-basis Three findings from the latest review round, all involving the decision-gate re-entry state machine and its validator: 1. Preserve customer state instead of recursively deleting terraform/ (P1, data-loss risk). workshop-refresh.md's Stale Generate guard deleted `terraform/` recursively on re-entry. That directory is not purely generated: generate-terraform.md tells users to create terraform.tfvars (gitignored, holds real values, never regenerated), and never emits *.tfstate/*.tfstate.backup or a .terraform/ provider cache -- those come from the user's own `terraform apply`/`terraform init` and are often the only local record of what's actually deployed. Recursive deletion destroyed all of that. Replaced with an explicit, named list of only the files each Generate fragment's own `_contributes:` frontmatter declares it writes -- every other file under terraform/ (tfvars, tfstate, .terraform/, or anything else the customer added) is now left untouched. 2. Move the stale-execution-pack cleanup to the single point every re-entry path actually passes through. The cleanup was gated behind workshop-refresh.md's own `phases.generate == "completed"` check, but workshop.md Entry step 2 already resets phases.generate to "pending" via reset_downstream_to_pending before workshop-refresh.md is ever reached -- so that guard's condition could never fire on the very re-entry it was meant to catch. Worse, the "Compare scenarios" loop branch never touches workshop-refresh.md at all, so it could return to the Decision gate with the old execution pack still on disk regardless. Moved the guard (detection + confirm + deletion) into workshop.md Entry step 2 itself, which every re-entry path (Apply & reprice, Compare scenarios, or exiting workshop without entering refresh) passes through before any branch is chosen. workshop-refresh.md's own step 2 now just documents that this already happened upstream, rather than re-deriving a check that can never fire. 3. Parse actual <section> elements for decision-basis, excluding comments. _section_counts() (the older, pre-awslabs#288-relocation copy of the Heroku report validator) matched <section id="..."> via a raw regex, so the unexpanded skeleton placeholder comment from generate-report.md -- `<!-- <section id="decision-basis"> when recommendation.decision_basis exists -->` -- was counted as a real, present section even though it renders nothing. A report that declared decision_basis but shipped the template comment verbatim returned REPORT_OK. Same root cause as PRs awslabs#288/awslabs#289 in this session: regex against raw markup instead of parsed structure. Replaced with an HTMLParser subclass (_SectionOpenTagCollector) that only counts real start tags -- comments are never re-tokenized as tags. Verified: (1) and (2) are DSL/prose fixes with no executable unit under test in this repo (the reviewer's own stated verification method for these files is tracing the documented state transitions, not a live agent run) -- checked for internal consistency between workshop.md and workshop-refresh.md and confirmed the golden heroku-workshop/heroku-decision-gate fixture asserters still pass. (3) is code: added a regression test using the exact placeholder text from generate-report.md, confirmed it fails against the pre-fix code (REPORT_OK when it should fail) and passes against the fix -- 19/19 tests passing in both trees. Full repo verification also green: fixtures:assert (9 asserters), cross-plugin-drift (282 identical, 27 allowlisted), node test suites, dprint, markdownlint, lint:frontmatter, git diff --check.
_OpportunityRowParser only registered a populated <td> on an explicit
handle_endtag("td"). The HTML Standard permits a <td> end tag to be
omitted immediately before the next <td>/<th>, or before its parent
<tr>/<table> closes -- so a perfectly valid, compact table like
<table><tr><td>RDS Reserved Instance</tr></table> never fired
handle_endtag("td") at all, and its real content silently registered
as empty, failing an otherwise-valid report.
Root cause: the parser tracked "is a <td> open" as a depth counter
incremented on every <td> start tag, with no concept of implicit
closure -- a new <td>/<th> starting, or the row/table ending, should
each first finalize whatever cell was still open, exactly as a real
HTML parser does, but this parser only checked accumulated text on an
explicit close.
Replaced the depth counter with a single _close_cell() finalization
step, invoked whenever a <td>, <th>, or self-closed cell tag starts
(closing any sibling cell still open in the same row), when a <tr>
opens (closing any cell that leaked from an earlier, malformed row),
and on every actual close of <td>/<th>/<tr>/<table>. This correctly
handles the reviewer's repro plus adjacent edge cases: sibling cells
in the same row sharing omitted end tags, and cross-row leakage
(a cell opened in one row must never be finalized against a later
row's content).
Verified against the reviewer's exact repro
(<table><tr><td>RDS Reserved Instance</tr></table>) plus 8 additional
edge cases (sibling omitted cells, th/td boundary interaction,
cross-row leakage in both directions, blank omitted-end cells still
correctly failing) and all 9 previously-fixed regressions from the
prior three rounds on this same function. Confirmed the two new
positive-case tests fail against the pre-fix code (REPORT_FAIL where
REPORT_OK is expected, matching the reviewer's report) and pass
against the fix. 36/36 tests passing in both trees (up from 33).
fixtures:assert (8 asserters), cross-plugin-drift (283 identical, 27
allowlisted), dprint, markdownlint, git diff --check all clean. No
merge from origin/main needed this round -- already caught up, no new
commits landed since the last fix.
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Completed the follow-up review of exact head e3ca4d232e9592c742e40dcaf30903f0e876ca90 against base ade57adaf9d776d18d316d28d3cee3f746582ced. The optional-td-end finding is fixed, and all previously reported packaging, content-validation, and attribute/class findings remain addressed. I found no remaining substantive concerns and replied in the original content thread.
Validation: 72/72 tests passed across both trees; 92/92 scoped CLI probes matched their expected outcomes; ten before/after comparisons reproduced the prior failures and verified the corrected behavior and negative controls. Isolated skill-only validator execution passed for both copies. Cross-plugin drift (283 identical, 27 allowlisted), both Heroku frontmatter checks, both vendored-shared checks, and git diff --check passed. The intentional SKILL.md/interpreter/schema differences and the main-window report-validation integration were checked. No full agent-driven migration replay or deployment was run.
Current GitHub CI is green, separately from local validation. Recommendation: Approve, for the user to submit. This is a COMMENT follow-up only; I have not submitted approval, dismissed prior reviews, or resolved conversations. The existing blocking review state still requires the user's action.
Resolves a modify/delete conflict on validate-heroku-migration-report.py in both plugin trees: - This branch RELOCATED the Heroku report validator from the plugin root (scripts/validate-heroku-migration-report.py) into the skill (skills/heroku-to-aws/scripts/) and hardened it with a decision-UX + a11y contract (verdict-headline, banned badge pills, <html lang>, <th scope>, <figure> aria-label/figcaption, populated cost-optimization content). - origin/main INDEPENDENTLY added a currency-formatting check to the OLD-path file (PR awslabs#310 line of work: CENTS_RE / _RATE_SUFFIX_RE / decoded-text + block-boundary parsing) plus a currency test suite pointing at the old path. A naive "keep the delete" would have silently dropped main's currency-formatting feature and left main's currency test pointing at a deleted path. Instead: - Kept the relocation (validator stays in the skill). - Ported main's currency-formatting block verbatim into the relocated validator (helpers + `errors.extend(_validate_currency_formatting(html))`), additive to the branch's a11y contract — no name collisions. - Moved main's currency test into skills/heroku-to-aws/scripts/ alongside the validator and the branch's structural test, switching its locator to `Path(__file__).resolve().parent`; updated its GOOD fixture to satisfy the branch's stricter structural contract (adds the cost-optimization section). - Mirrored validator + currency test byte-identically across advisor/ and migrate/ trees. Verification: heroku-to-aws/scripts suites 54 pass per tree (currency 18 + structural 36); shared validate-migration-report suites 92 pass per tree; emit-plan-json 46 pass; drift:check OK (284 identical, 27 allowlisted); fixtures:assert PASS; fmt:check clean; lint:md 0 errors. Confirmed the ported currency check fires on the SF Beach drift case ($25,684.89/mo).
The merge moved the Heroku currency test out of tests/ (which the bandit CI
job excludes via `-x .../tests`) into skills/heroku-to-aws/scripts/, which IS
scanned. Bandit then flagged B404 (import subprocess) on the test's import.
Add `# nosec B404 — test-only, list args, no shell, committed script path` to
the import line, matching the sibling test_validate_heroku_migration_report.py
that already lives in that directory. The subprocess.run call already carries
`# nosec B603`.
Verified: `mise run security:bandit` clean ("No issues identified"); currency
suite 18 pass per tree; both tree copies byte-identical.
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Reviewed exact head 2ecdcfcd06c10b0a3ce5e2fd6f6882bc7256e5b9 against base 811fddafced0870173dc9c1a66749a9143baa040, including the currency-validation merge, relocated tests, standalone skill layout, both plugin copies, and the current discussions. The previously reported packaging, content, optional-td-ending, and attribute/class issues remain fixed.
One integration follow-up is inline: the now-mandatory currency check requires whole-dollar monthly values, but the dispatched Heroku report writer and its self-check do not carry that rule. Please align the producer instructions with the gate. Recommendation: Request changes for this P2 instruction/validator mismatch.
Validation: 158 focused tests passed; all 122 Heroku CLI probes and four isolated skill-packaging checks passed. Of 74 GCP currency probes, two expose an implicit-cell-boundary behavior already present in the byte-identical merged-base GCP validator; I am not attributing it to this PR. The relocated Heroku currency helpers match main, and the prior structural/verdict helpers are unchanged. Cross-plugin drift (284 identical, 27 allowlisted), both Heroku frontmatter checks, both vendored-shared checks, and git diff --check passed. GitHub CI is green separately from local validation. No full agent-driven migration replay or deployment was performed.
This is a COMMENT review; the existing human approval and historical reviews are left untouched.
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Completed startups-hybrid-v1 review of exact head 571f7daaaa419e221e6d4c9fa5404ea9d5c19778 against base ec681ba5d49b039dbf1ef9d87d3304205b824f3d: actual OCR v1.12.5 delegation, one fresh general reviewer, the parent repository review, and independent candidate verification. All 18 Git inventory entries were covered, including Markdown and deletions. The canonical evidence gate passed.
Two verified P2 concerns remain, updated in the original threads:
- The dispatched report writer still lacks the monthly rounding instruction required by the blocking currency gate.
- The no-commitment fallback rejects valid encoded text and can accept content present only in comments/inert templates.
Local validation: the 108 authored tests passed in both independent perspectives; the parent ran 34 CLI/packaging checks, the general reviewer ran 48 focused base/head CLI probes and four packaging checks, and the parent independently ran 14 fallback verification cases. Drift, frontmatter, vendored-source, and diff checks passed. Current GitHub CI is green separately from those results. No full agent-driven migration replay, browser-accessibility run, or deployment is claimed.
Recommendation: Request changes for the two P2s. No approval, dismissal, merge, or conversation-resolution action was performed.
…ort worker the currency rule
Closes the two remaining P2s on the blocking Heroku report validator:
1. No-commitment fallback rejected valid encoded text and accepted invisible
text. `_validate_optimization_content` matched the sentence with a raw
`re.sub("<[^>]+>", " ")` tag-strip that never decoded entities or excluded
inert/comment content — so `No 1&awslabs#45;year/3&awslabs#45;year …` and ` `-separated
forms FAILED, while the sentence present only inside a <template> passed.
Now extracts RENDERED text via the existing _DecodedTextParser (entities
decoded, comments + script/style/template excluded), normalizes whitespace
incl. NBSP, then checks — so the same parser the currency gate uses decides
what "rendered" means. Encoded/nbsp sentences pass; comment/template-only fail.
2. The report worker was never told the (now blocking) whole-dollar rule.
generate-report.md's authoring rules and Step-3 self-check never required
whole-dollar monthly figures, so a worker could emit `$112.34/mo` and only
discover the failure at the blocking currency gate (which cannot re-author in
the shell-less worker). Added a "Whole-dollar monthly figures (blocking)"
authoring rule and a self-check item 8 so the worker rounds monthly-scale
figures before returning.
Four regression tests added (entity-hyphen sentence passes, nbsp sentence
passes, comment-only fails, template-only fails); the entity/nbsp/template ones
verified to fail against the pre-fix parser. Both plugin trees byte-identical;
heroku validator suite 40 pass per tree; drift:check OK (284 identical);
fmt:check clean; lint:md 0 errors; security:bandit clean.
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Completed startups-hybrid-v1 at exact head e538f368cb388fa36e6dbbb0c4e8cd52c58650d2 against base ec681ba5d49b039dbf1ef9d87d3304205b824f3d: OCR v1.12.5 delegation, one fresh general review, parent repository review, independent candidate verification, and the canonical gate. All 18 Git entries were covered. The actual child's initial-context disclosure and recorded tool audit passed the isolation checks.
The currency-authoring and decoded-fallback fixes are verified. Two P2 concerns remain: inert-only opportunity rows can satisfy the content check, and an inert-only verdict headline can satisfy the declared-outcome check. Both were independently reproduced with supported Estimate inputs and updated in the original threads. The base lacked these positive-content requirements; this is incomplete enforcement of the newly introduced guarantees.
Validation: 116 head tests passed in both perspectives; the general reviewer also ran 36 base currency tests. The parent ran 54 CLI/packaging checks and 26 authoritative candidate-verification executions. Drift, frontmatter, vendored-source, and diff checks passed. GitHub CI is green separately from local results. No full agent-driven migration replay, browser-accessibility run, or deployment was performed.
Recommendation: Request changes for the inert-content cases. No approval, dismissal, merge, or conversation-resolution action was performed.
…nt check Close the inert/hidden-content blind spot as a CLASS, not one parser per review round. The decoded-fallback check already excluded inert subtrees (<script>/<style>/<template>); its sibling checks in the same file did not, so unrendered content could still satisfy a blocking gate: - _TagAttrCollector (feeds _class_tokens -> _validate_verdict): skip tags inside inert subtrees, so a `verdict-headline` declared only inside a <template> is no longer read as a rendered headline when an Estimate declares recommendation.outcome. - _OpportunityRowParser: track inert ancestors, so a <template>/<script>-only table cell (or a whole table nested in a <template>) no longer registers as a populated opportunity row. - _AccessibilityParser: skip inert subtrees, so a <th>/<figure> that exists only inside a <template> is not audited for scope/aria-label (removes a false-positive symmetric to the same blind spot). All three now mirror _DecodedTextParser's inert handling, so every "is this rendered?" check in the file agrees on the answer. Adds inert-only negative controls (inert opportunity table, inert-only cell, verdict-headline-only-in-template, th-in-template-not-audited); each fails against the pre-fix script and passes after. Mirrored byte-identical across the advisor and migrate trees.
leon1418
left a comment
There was a problem hiding this comment.
[🤖 AI review 🤖]
Completed startups-hybrid-v1 with convergence v1 at exact head ab95346fcad2bc3f432489d09a7c1696a5b9f7e3, base ec681ba5d49b039dbf1ef9d87d3304205b824f3d: OCR v1.12.5 delegation, one fresh general reviewer, separate parent review and impact maps, candidate verification, and the canonical evidence gate. All 18 Git entries were covered.
The two previously reported inner-inert cases are fixed. Packaging, currency guidance, decoded fallback text, optional cell endings and class handling retain their fixes. Two consolidated P2 corrections remain:
- Section parsing: in both skill-local validators and structural test copies, use actual parsed section IDs with document/comment/inert context. Invisible sections and
data-idmust not satisfy the gate; real quoted/spaced/unquoted IDs must pass; commented duplicates and the what-if skeleton comment must not count. The original content and verdict threads carry the reproductions. - Missing-Python instructions: update the tooling fallback in both SKILL.md copies so it no longer promises Generate completion without successful report validation. Earlier phases can continue; the existing missing/not_run gate stays blocking. This is inline.
Both gaps already existed at e538f368; I missed them in the prior review. Neither is attributed to the latest inert-parser fix.
Local evidence: 124 authored tests passed in each perspective; the parent ran 54 CLI/packaging checks and 76 authoritative before/after boundary executions with preserved inputs. The fresh reviewer separately reproduced the section issue. Drift, frontmatter, vendored-source and diff checks passed. Current GitHub CI reports nine successful checks, separately from local results. No full agent-driven migration, missing-Python host replay, browser accessibility suite or deployment was performed.
Recommendation: Request changes. GitHub also reports merge conflicts, a separate landing blocker. Approval remains for the user; no approval or review dismissal was performed.
# Conflicts: # advisor/plugins/aws-startup-advisor/scripts/validate-heroku-migration-report.py # advisor/plugins/aws-startup-advisor/skills/heroku-to-aws/references/phases/generate/generate-assemble.md # advisor/plugins/aws-startup-advisor/skills/heroku-to-aws/references/phases/generate/generate-report.md # migrate/plugins/migration-to-aws/scripts/validate-heroku-migration-report.py # migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/generate/generate-assemble.md # migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/generate/generate-report.md
…ated validator The heroku-decision-gate golden asserters (check_expected_decide.py, and check_expected_decide_with_retained_pack.py which imports its main) still resolved the validator at PLUGIN_ROOT/scripts/validate-heroku-migration-report.py — the pre-relocation path. This PR moved the validator into the skill (skills/heroku-to-aws/scripts/), so both goldens failed fixtures:assert with "No such file or directory" when validating decision-report.html --mode decision. Point VALIDATOR at the skill-relocated path. Both plugin copies byte-identical; fixtures:assert now PASS (10 asserters, 6 golden + 4 smoke) on both trees; drift/fmt/lint:md/bandit green.
Problem
Heroku migrations produce a
migration-report.html— the summary a stakeholder reads. There's a checker that's supposed to make sure that report is complete and sound, but it wasn't actually enforcing anything: it was optional (a bad report didn't stop anything), it had no tests, and it only looked at a handful of the report's sections. On top of that, two of the skill's own instruction files disagreed about whether the check even blocks — one said "optional, never halt," the other said "fail the gate."The practical result: a Heroku user could ship a report that's missing its required cost-optimization section, or shows a verdict signalled by color alone (a problem for colorblind readers), or has an empty section where content should be — and nothing would catch it. This PR makes the check actually block, hardens what it inspects, gives it a test suite, and resolves the contradiction so the two files tell one story.
In code, for reviewers
validate-heroku-migration-report.pywas optional/non-blocking, had zero tests, and checked only three section IDs + the footer.generate-report.mdsaid "optional / never-halt" whilegenerate-assemble.mdsaid "GATE_FAIL" — a direct contradiction. A report could ship missingcost-optimization, carrying a color-only verdict pill, or with an empty section, with no gate. Review item P1-B.Solution
Harden the Heroku report validator, make it blocking, and add the test suite it never had —
scoped to what the intentionally-thin one-pager actually emits.
Validator (
validate-heroku-migration-report.py), Python-stdlib only:cost-optimizationand enforces it is non-empty (a table or the explicitno-eligible-commitment sentence, never a blank section — matching the skill's Step 3 contract).
decision-summarymust not contain anybadge-verdict-*pill; when
estimation-infra.jsondeclaresrecommendation.outcome, it must render averdict-headline.<html lang>; every<th>declaresscope=col|row; any<figure>carriesaria-label+<figcaption>.Emitter and gate now agree (a11y).
generate-report.mdwas updated so the report templateinstructs the a11y attributes the validator requires — every
<th scope>, and figure labels if afigure is emitted. Without this the blocking gate would have false-failed a report that followed
the template (the tables emit
<th>with no scope).<html lang>was already in the skeleton.Where the validator runs (round-2 fix — mirrors #287). Generate dispatches under
_exec._agent: rw, which excludes shell/Bash — the dispatched worker (fragments andassembler) cannot run the validator. An earlier revision put the
python3invocation ingenerate-assemble.mdand called it "the main window," but the assembler is dispatched, so"blocking" was prose on a path that can't execute. Corrected to the three-step split:
generate-report.md, dispatched, no shell): authorsmigration-report.htmland does the in-fragment Step 3 self-check + one-repair; it never runsthe validator.
generate.mdrunsvalidate-heroku-migration-report.py($PLUGIN_ROOT-anchored) over the report. OnREPORT_OKit stamps a durable
report-validation-status.json(temp status sidecar, not a_producesartifact); on
REPORT_FAILit emitsGATE_FAILand pastes the validator'serrors[]verbatim so recovery is actionable. This step edits nothing else.
_postconditionsgate (main window, READ-ONLY): the section-id_assertplus a new_assertthatreport-validation-status.jsonhasreport_status == REPORT_OK. Runs novalidator, edits nothing (per
INTERPRETER.md§_postconditions). The durable stamp means theassert reads an artifact, not conversation memory — parity with feat(heroku-to-aws): wire tf-best-practices policy gate into Generate #287's
validation-report.json.Recovery on
GATE_FAILis a hand-edit of the report from the pasted errors + a directvalidator re-run (or a maintainer re-running Generate for a clean rebuild) — not a blind "re-run
Generate," which re-dispatches the shell-less worker and re-authors the report.
Blocking wiring resolved the contradiction.
generate-report.mdandgenerate-assemble.mdnow agree the gate is required/blocking and read-only;
generate.md_postconditionsgainedcost-optimizationin the section-id_assertplus the durablereport-validation-status.jsonREPORT_OK_assert.Tests:
tests/test_validate_heroku_migration_report.py(both trees, 14 cases) — compliantreport (with and without a declared recommendation) passes; failures fire for missing/empty
cost-optimization, missing footer,badge-verdict-*pill,<th>withoutscope, missing<html lang>, missingverdict-headlinewhen an outcome is declared, figure missingaria-label,and (new) figure with
aria-labelbut no<figcaption>; plus the figure-happy-path and≥2-scenario paths.
Deliberately NOT ported (would false-fail the thin report): decision-before-TOC, single-
<h1>,per-table
<caption>, glossary/appendix set.Type of Change
Team Folder
advisor/migrate/solution-architecture/Verification
cross-plugin-drift.ts: OK (273 identical, 25 allowlisted — no new drift)test_validate_heroku_migration_report.py(both trees): 13 passed eachfrontmatter-validator(both trees): OKmarkdownlint-cli2: 0 errors;dprint check: clean;tsc --noEmit: OK both trees<th scope>, non-empty cost-optimization) →exit 0; a bare
<th>→ exit 1 with the actionable message (gate matches emitter)mise run build(full composite incl.bandit/semgrep/gitleaks/checkov/grype):PASS, exit 0 — run against a fresh detached worktree at this branch's commit (not my working
checkout, which has an untracked, unrelated
migration-watchdog/directory failing an unrelatedmarkdownlint check). checkov 276/0; gitleaks no leaks; grype no vulnerabilities.
Not run locally: no browser a11y tooling (Vale/Pa11y/axe) — the validator is a dependency-free
static HTML reader, matching the plan's "Python 3 only" constraint.
Checklist
mise run buildlocally and it passes — verified against a fresh worktree at this branch's commit (see Verification)generate-report.mda11y authoring rules + blocking wiring;generate-assemble.mdmain-window gate)advisor/+migrate/migration skills)