[Hackathon] feat(test): preview real run selection and setup blockers without starting tests #657

Verified against (commit SHA or released version)

CLI v0.15.0, commit d61e6dfc400f5e88ee34ac799c5496ac5c5a7751. Proposal based on inspection of the public CLI's run selection, environment resolution, sign-in metadata, preflight, and dry-run paths. This feature has not been implemented or verified against a live account.

Discord Username / User ID

sanaro99

What does this improvement do?

Let a developer or coding agent inspect the intended run using real project metadata before starting tests. Add an opt-in --preview flag to test run that reports selected tests, the effective environment/target, and known setup blockers, with an actionable next step.

For a small team, a useful question is: "Will this command select the checkout tests against staging, and is the saved setup ready?" Existing --dry-run teaches the command through sample responses. It cannot answer that question about the actual project.

Example proposed invocation:

testsprite test run --all --project p_demo --filter checkout --env staging --preview

Illustrative output, assuming the API explicitly reports an expired manual session:

Run preview — no tests started
Project: p_demo
Selection: 3 of 12 tests match "checkout"
Environment: staging
Frontend target: https://staging.example.com (saved environment)

BLOCKED  Saved manual sign-in session has expired
ACTION   Refresh the saved session for staging using the project's sign-in instructions

Server admission, available credits and test outcomes are checked when a run starts.

This helps both a person checking their command and an agent deciding whether it needs human intervention. It uses existing CLI/API information rather than changing test execution or generation in the hosted platform.

Details / implementation notes

Proposed first scope

• Support one test ID, or --all --project / the existing project environment default, with the existing name filter. Include explicit/default environments, target overrides and local targets.
• Share selection, environment resolution and local frontend eligibility with the execution path so preview cannot drift into a second interpretation of the command. Paginate the complete selection; an incomplete listing must be labeled incomplete.
• Read existing identity/scope, test, environment and sign-in metadata through the configured TestSprite API. Only report a blocker when metadata establishes it; absent or unfamiliar metadata remains unknown. Do not attempt a login.
• Show which frontend target is effective and where it came from. Explain backend exclusions: a frontend target override does not rewrite URLs embedded in backend test code.
• Provide text and structured JSON with stable check IDs, status, blocking flag, message and next action. Distinguish ready, blocked and incomplete; ready means no known blocker within the checks performed.
• Keep --dry-run's sample behavior unchanged. Reject combining it with --preview. Require credentials for the live metadata inspection and surface read failures rather than substituting samples.

Boundaries

Preview must not trigger a run, generate/accept plans, mint or delete a tunnel, change project/sign-in settings, mint an idempotency key, poll a run, download artifacts, or write execution/CI reports. Reject execution-only report writers in preview mode before side effects. The initial feature can refuse several explicit test IDs and adopted tunnels with a clear explanation; CI, rerun and testlist preview can follow separately.

Use allowlisted output fields: no API keys, sign-in credential values, environment variables or raw API bodies. Sanitize displayed target URLs. JSON stays on stdout and progress on stderr. Bound metadata reads and the overall inspection, and preserve cancellation.

An unfiltered --all run currently lets the server choose the final batch. Its preview must describe a snapshot of candidate tests, not promise the final admitted set. Preview does not reserve state; removing the flag resolves current state again. Do not predict test success, estimate charges, or assert cloud-runner reachability from a local probe.

Maintainer decisions requested during triage

1. Confirm flag naming/placement as the run commands evolve toward 1.0.
2. Agree the diagnostic JSON/exit-code contract.
3. Decide whether to reuse the existing target/loopback probes. The contribution guide restricts outbound destinations: this proposal does not assume new approval for direct target calls. An API-metadata-only first version is possible, with target reachability explicitly unchecked.

No new runtime dependency is proposed.

Acceptance checks

• A matching test on a later page appears, and filtered preview agrees with execution selection.
• An unknown environment, authoritative missing permission/credentials, expired manual session or empty selection produces a specific next action; unknown metadata never becomes a false readiness guarantee. Preserve --allow-empty semantics.
• No run trigger, tunnel mutation, project mutation or report write is reachable from inspection. Verify request methods/routes and filesystem boundaries with fakes.
• JSON contains no secret canaries; partial reads and cancellation are visible. Existing execution and dry-run contracts remain compatible.
• If probes are approved, test closed local ports and ambiguous reachability without treating a CLI-machine result as proof of cloud reachability.

Related work: #606 concerns unsafe empty filters, #620 / #638 concern protected Vercel targets, #502 concerns DNS preflight deadlines, and #555 concerns CI dry-run side effects. Those fixes remain separate. Targeted public issue and open-PR searches found no matching real-run readiness preview proposal.

AI assistance: Codex assisted with public source inspection, scope analysis and drafting. No feature implementation or live-service validation is claimed.

Confirmations

• I have searched existing issues and this is not a duplicate.
• I have provided my Discord identity above for reward coordination.