Add timeout check for smartstacks#463
Open
timbeccue wants to merge 2 commits into
Open
Conversation
timbeccue
force-pushed
the
feature/stacking-timeout
branch
from
ebbdffb to
79b9afd
Compare
timbeccue
force-pushed
the
feature/separate-site-deps
branch
from
adf8300 to
5f760f6
Compare
timbeccue
changed the title
timbeccue
force-pushed
the
feature/stacking-timeout
branch
from
79b9afd to
396f4f8
Compare
timbeccue
force-pushed
the
feature/separate-site-deps
branch
from
84e206a to
c421b9e
Compare
timbeccue
force-pushed
the
feature/stacking-timeout
branch
from
396f4f8 to
69d49a8
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds a fixed “latest reduced subframe” timeout mechanism to smart-stacking so that incomplete stacks don’t remain active indefinitely (which can otherwise stall per-camera stacking workers). The stacking worker now performs a DB sweep after draining Redis notifications to finalize stacks that are either complete or have exceeded the fixed 15-minute threshold.
Changes:
- Add fixed 15-minute timeout detection based on the newest reduced subframe
created_at, with timeout/complete finalization during a per-camera sweep. - Make
get_subframes()return rows ordered bystack_num, and add a helper to fetch active subframes for a camera. - Update unit tests, E2E tests, and architecture docs to reflect the new sweep/timeout behavior.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/smartstacking_architecture.md | Documents the new fallback sweep and fixed timeout behavior (incl. updated flow). |
| banzai/stacking.py | Introduces STACK_TIMEOUT_SECONDS, timeout detection, and per-tick timeout sweep in the worker loop. |
| banzai/dbs.py | Orders get_subframes() by stack_num; adds get_active_subframes_for_camera() for sweep queries. |
| banzai/tests/test_smart_stacking.py | Adds unit tests for ordering and fixed timeout behavior; updates worker-loop resilience tests. |
| banzai/tests/site_e2e/test_site_e2e.py | Adds an E2E test that forces staleness via created_at update and verifies timeout. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+670
to
+672
| def get_active_subframes_for_camera(db_address, camera): | ||
| """Get active subframe records for a camera ordered by stack and arrival time.""" | ||
| with get_session(db_address, site_deploy=True) as session: |
Comment on lines
+146
to
+148
| def check_timeouts(db_address, camera, now=None): | ||
| """Finalize active stacks that are complete or have exceeded the fixed timeout.""" | ||
| active_subframes = dbs.get_active_subframes_for_camera(db_address, camera) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds fixed timeout handling for smart-stack subframes.
The basic behavior is that an in-progress smart stack enters a timeout state if more than 15 minutes elapses between reduced subframes. This will trigger the final stack and cleanup routine for the smartstack (not yet implemented). The purpose is to prevent a hanging stack from indefinitely blocking the stack worker from picking up subsequent stack jobs.
Details
timeoutcompletebefore considering timeoutget_subframes()return rows ordered bystack_num