Skip to content

test: make run-log scenarios API-safe and stop mock drift - #357

Merged
bradhe merged 1 commit into
developfrom
fix/api-safe-run-log-scenarios
Aug 19, 2026
Merged

test: make run-log scenarios API-safe and stop mock drift#357
bradhe merged 1 commit into
developfrom
fix/api-safe-run-log-scenarios

Conversation

@bradhe

@bradhe bradhe commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

What this changes

Two scenarios in cli_runs.feature asserted on log content that only the mock API server ever produced. First log before run completes and Warning: This run is using a deprecated runtime exist nowhere in the Tower server, so the scenarios could only ever pass against the mock.

That matters because the monorepo runs this exact suite against a real Tower server in its CLI regression job. It clones the tower-cli tag matching the installed PyPI version, so when v0.3.71 shipped these assertions, the monorepo's develop went red and stays red on every push. See tower-monorepo run 32238254228.

This PR makes both scenarios assert on things a real run produces:

  • The post-completion drain scenario now deploys a fixture app that prints those two lines itself, with a delay between them and the second printed right before exit. The assertions are unchanged, but the content is now actual program output rather than something the mock invented.
  • The follow scenario asserts Warning: No new logs available, which is one of only three warnings sendRunLogNotifications can send. It is guaranteed ahead of any stream close, because the terminal-status check sits inside the same branch that emits it.

I also pulled the mock back in line with the server, since the drift is what let this happen in the first place. Its log stream now closes the way the real one does, with the idle warning followed by a terminal end-of-stream event, and its stored log lines match what the hello-world fixture actually prints. Starting application... and Application completed successfully were mock fiction that nothing asserted on.

History

This is the second time around for these assertions. They arrived in #120 in October, and #198 in February removed them for exactly this reason, under the title "align BDD tests and mock server with real API behavior". #353 put them back in August. Keeping the mock honest is the part that stops a third round.

Verification

I ran the suite against a real local Tower server, via test-cli.sh with CLI_BRANCH pointed at this branch, which is the same path the monorepo's CLI regression job takes. Against the same server, develop fails 3 of 8 scenarios and this branch fails 1:

Scenario develop this branch
:35 logs that arrive after run completes FAIL PASS
:49 follow without duplicates FAIL PASS
:58 finished run, stored logs exactly once FAIL FAIL

Both scenarios this PR touches now pass against the real API, including the post-completion drain. The develop run also confirms the warning choice, since the server emitted Warning: No new logs available and then Warning: stream complete, which is what the mock now does too.

:58 fails the same way on both branches, so it is pre-existing and not from this change. It passes in CI and fails locally, because it waits a fixed 2 seconds before reading stored logs and that is not long enough on a cold runner building a uv environment. Worth fixing separately, and I left it alone here to keep this PR to the drift problem.

The full feature also passes against the mock, so both sides of the suite are green.

The two run-log scenarios asserted on strings only the mock API server ever
emitted, so they passed locally and failed in the monorepo's CLI regression
job, which runs this suite against a real Tower server.

The post-completion drain scenario now deploys a fixture app that prints the
two lines itself, and the follow scenario asserts a warning the server
actually sends. The mock's log stream is aligned with the server: it closes
with the idle warning and a terminal end-of-stream event, and its stored log
lines match what the hello-world fixture prints.
@coderabbitai

coderabbitai Bot commented Aug 19, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 49fde1a0-2317-46cd-9f24-acfd66e49ce1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@konstantinoscs konstantinoscs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My human review looks good.

Sol marked these. I leave them up to you:

[P2] Terminal sentinel is emitted while the run is still active. generate_normal_log_streamsends end_of_stream=true after 100 ms, while ordinary mock runs remain running for five seconds. The CLI reconnects and replays the warnings; a diagnostic assertion saw 9 copies of each warning. Gate the sentinel on terminal status and ideally assert it is consumed exactly once.

Two additional robustness concerns:

  1. logs_after_completion.py prints its second line before process exit. A real-server run can therefore pass without exercising the post-completion drain.
  2. The warning assertion uses an instantaneous app. If it finishes before --follow attaches, the CLI takes the stored-log path and emits no SSE warning. A deliberately long-running fixture would remove that race.

@bradhe
bradhe merged commit 638f232 into develop Aug 19, 2026
32 of 33 checks passed
@bradhe
bradhe deleted the fix/api-safe-run-log-scenarios branch August 19, 2026 10:45
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants