Define Heroku application source review contracts - #239
Conversation
herosjourney
left a comment
There was a problem hiding this comment.
Approving -- verified independently, not just read
This is a dormant, data-only contract (no wiring into Discover/Clarify/Design/Estimate/Generate), so the review bar here is: does the schema actually enforce what the doc and PR description claim, and are the security properties real. Checked both directly rather than trusting the prose.
Verified by hand-crafting and validating documents against the schema (Python jsonschema, not just reading it):
- A
runtime_settingsfinding with a literal"value": "postgres://user:pass@host/db"field is correctly rejected (additionalProperties: falseon every record type does its job -- no config-value smuggling path). - A record of the wrong shape for its
question(e.g. aprocessCommandRecordpayload underruntime_framework) is correctly rejected -- the per-questiononeOfbranches infindingare real constraints, not decorative. - The
sourcePathtraversal-blocking regex correctly rejects../,..\, absolute Unix/Windows paths, and double slashes; correctly accepts normal relative paths. The one edge case I flagged myself (root: "."being disallowed by the general pattern) turns out to already be handled --runtimeFrameworkRecord.rootexplicitly carves out{"const": "."}as a sibling alternative, so this was already caught and fixed by the author, not an oversight.
Verified the "UNKNOWN requires a limitation" and "qualified absence" rules exist and are tested, even though they're not expressible cleanly in JSON Schema draft-07 without heavy duplication -- they're correctly implemented as semantic checks in validateSemantics() (test.ts:373-380) with dedicated positive/negative test cases (lines 462-479, 549-561), not silently dropped between the doc and the implementation.
Ran the actual suite, not just read it:
node --test application-source-contract.test.ts-> 8/8 pass, matching the PR body's claim exactly.mise run build-> exit 0.drift:check: 261 identical, 25 allowlisted -- the four new/changed files (contract doc, schema, two SKILL.md touches) are confirmed byte-identical betweenadvisor/andmigrate/, matching the PR's "byte-identical" claim rather than just asserting it.mise.tomldiff is exactly the minimal registration described -- one new line in the existingtesttask, no new task, no new runtime dependency.
Scope claim holds up: grepped the diff for any change to Discover/Clarify/Design/Estimate/Generate/Feedback/Workshop phase files -- none. This really is additive and dormant as described.
No findings. Clean, well-tested, and the security-relevant claims (no secrets, no config values, no path traversal, no cross-question value confusion) are backed by both schema constraints and semantic tests, not just documentation.
|
I've addressed five of the findings you brought up: stricter source-path validation, schema-required explanations for UNKNOWN, clearer secret-redaction requirements, correct validation of partial reviews, and renaming recurring-job entrypoint to command. I did not restrict discovered process or configuration names to the Heroku inventory because differences can identify missing configuration or drift. Reversed line ranges are already rejected by the executable validator and covered by a focused test. |
|
@ayn-builds Thanks for the review. All three findings are addressed in 049fb00, with regression tests added for each case. Local build and security checks pass; GitHub CI is running. |
Problem
The Heroku-to-AWS skill can collect Heroku account metadata, Terraform,
Procfile, andapp.jsoninformation, but it does not yet have a stable format for a future application-code review to report its findings. Without thatboundary, later source-review work could return inconsistent or unbounded data, include information that should not be retained, or mix code observations with destination decisions.
Solution
Add a data-only request and findings contract for 22 application-code questions. The request exposes only selected question names, application identity, and an allowlisted projection of non-secret Heroku inventory. Findings use bounded, question-specific values with explicit present, absent-within-reviewed-scope, unknown, and not-applicable states.
The contract has no fields for configuration values, credentials, connection strings, or source excerpts. Optional source locations must be safe relative paths. Focused tests cover all 22 questions and reject malformed, duplicate,
unrequested, out-of-bounds, secret-bearing, or internally inconsistent data.
Both published Heroku skill surfaces contain byte-identical contract and documentation files.
Broader Context
This is the second incremental change in a larger effort to make the existing Heroku migration workflow use application-code evidence when assessing Elastic Beanstalk. PR #231 removes unsafe generated port and health-check assumptions. This PR defines what a future read-only source reviewer may return. Later, separate PRs will run that review, validate and combine its findings, calculate deterministic Beanstalk compatibility, improve cost and reporting, and gate starter infrastructure generation.
The existing plugin structure and its Elastic Beanstalk, Fargate, and EKS destinations remain in place.
Scope
This PR does not run source review or change Discover, Clarify, Design, Estimate, Generate, Feedback, or Workshop behavior. It does not change destination selection, cost calculations, reports, or generated artifacts. It is independent of #231 and leaves the current customer workflow usable on its own.
The schema remains dormant until a later PR connects source review to the workflow. The test-only schema checker supports the subset used here and does not add a runtime dependency.
Type of Change
Team Folder
advisor/migrate/The
advisor/andmigrate/files are the repository's synchronized copies of the same Heroku skill. The focused contract test is stored once under the canonical migration plugin test tree and verifies both copies.Validation
mise run fmt:checkmise run lintmise run securityGitHub build and security workflows pass. No live Heroku test is included because this PR has no runtime integration or customer-visible behavior.
Checklist
content
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.