You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Stage 12 coordinator can outlive its timeout while waiting and leaves workers running #792
The Stage 12 coordinator can be killed before it records a failure, and it leaves workers running after a comparison has failed.
Separate wait timeouts: the coordinator waits up to 3,000 s for the baseline worker, then up to a fresh 3,000 s for the reform worker. Its own Modal timeout is 4,500 s. When workers are queued, the two waits can exceed that, and Modal kills the coordinator before it saves the failure.
Siblings keep running: when one worker's wait fails or times out, the other worker keeps running. That can mean a 32 GB container for up to 50 minutes on a comparison that has already failed.
One production report from 6 to 13 October 2026 waited 3,000.0175 s for baseline and then stored TimeoutError. Its reform worker had already been interrupted after about 305 s.
Expected
Both waits share one deadline that leaves the coordinator time to save the outcome before its Modal timeout.
When one worker fails or times out, the coordinator cancels every unfinished worker and records the cancelled one as failed.
Observing baseline and reform concurrently is out of scope here; it is part of #712.
Activity