feat(browser): circuit-break repeated render timeouts - #17
Conversation
A hung renderer cost a full render_timeout_ms (default 45s) on EVERY attempt — nothing remembered the prior timeout, so a persistently-dead renderer stalled every agent 45s at a time (prod: browser 0/9 ok, each 'waiting on browser → render_failed in 45.0s'). After render_breaker_threshold (default 3) consecutive render timeouts the breaker opens for render_breaker_cooldown_ms (default 60s): a render fails fast with verdict 'render_unavailable' instead of dispatching and waiting. A single success closes it; after the cooldown one real attempt is allowed and re-opens on a fresh timeout. Only :render_timeout trips it — a fast crash/http error is not the 45s stall this guards. Decision logic is pure in Browser.Core (breaker_open?/2, register_render/5), wired into do_render via note_render/2 on state. 6 pure breaker tests; full objects harness 23/23 green. Config knobs render_breaker_threshold / render_breaker_cooldown_ms; both default to sane values so existing consumers get the guard for free.
|
Warning Review limit reached
Next review available in: 14 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9a87f4a748
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Logger.info("browse: from=#{from} url=#{url} verdict=rate_limited") | ||
| {:reply, err("rate_limited"), state} | ||
|
|
||
| Browse.breaker_open?(state.breaker_open_until, state.now_fn.()) -> |
There was a problem hiding this comment.
This breaker check only runs for the render action; existing sessions still route click/back/type/press through do_act, which calls render_sync directly and never checks breaker_open? or records :render_timeout via note_render. When a renderer wedges after an initial successful page load, each interactive action can still block for the full render_timeout_ms, and those repeated timeouts will not open the breaker, leaving the stall loop in place for active browser sessions.
Useful? React with 👍 / 👎.
…cision browser_breaker_test.exs pins Core.breaker_open?/register_render (pure). This drives the browse OBJECT end-to-end with a counting mock renderer: three render timeouts open the breaker and the 4th fails fast with render_unavailable WITHOUT dispatching to the renderer; a success before the threshold resets the streak. Pins the state threading (note_render) and the do_render fast-fail path.
1 participant
From the 2026-07-15 review: prod showed browser 0/9 ok, each attempt
waiting on browser → render_failed in 45.0s. A hung renderer cost a fullrender_timeout_ms(default 45s) on every attempt — nothing remembered the prior timeout, so a persistently-dead renderer stalled every agent 45s at a time.Fix
After
render_breaker_threshold(default 3) consecutive render timeouts, the breaker opens forrender_breaker_cooldown_ms(default 60s): a render fails fast with verdictrender_unavailableinstead of dispatching and waiting the full timeout. A single success closes it; after the cooldown one real attempt is allowed and re-opens on a fresh timeout. Only:render_timeouttrips it — a fast crash / http error is not the 45s stall this guards.Decision logic is pure in
Browser.Core(breaker_open?/2,register_render/5), wired intodo_rendervianote_render/2. Config knobs default to sane values, so existing consumers get the guard for free.Tests
6 pure breaker cases (
browser_breaker_test.exs); full objects harness 23/23 green.Not in this PR
The underlying Chromium/agent-browser cluster hang still needs the in-pod doctor with devops — this only stops it from costing 45s per attempt. Shipping to prod needs a browser package publish + attested pin bumps (wingston + micromarkets); holding for your greenlight (esp. with the genlayerlabs swarmidx token pending rotation).