feat(orion-server): queue builds when no idle worker for up to 60m - #2163
Conversation
Restore FIFO enqueue when VMs are offline or busy instead of immediately marking builds Interrupted, and re-queue on dispatch races.
Return aggregated latest-task build_status from the list API and render it with the shared Checks status icons.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2430a74a1b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // No idle worker (VM not started or all busy): queue and wait for capacity. | ||
| match state | ||
| .scheduler | ||
| .enqueue_task_v2(task_id, &req.cl_link, req.repo, req.changes, 0) |
There was a problem hiding this comment.
When this new no-idle path enqueues a build, enqueue_task_v2 first inserts a default target with latest_state = Uninitialized and then asks find_initialized_build_targets for targets; that helper returns an empty vector whenever any target for the task is still Uninitialized. Every build accepted through this queue therefore reaches dispatch with no target metadata and falls back to Uuid::nil, so completion updates do not touch the real target and the UI/CL list can remain stuck as queued even after the worker finishes. The retry queue path has the same shape, so the queue needs to carry the inserted target or mark it initialized before dispatch.
Useful? React with 👍 / 👎.
| // Register the build before sending TaskBuild so early TaskBuildOutput | ||
| // lines from the worker are not dropped (ws_handler only publishes logs | ||
| // when active_builds contains the build_id). | ||
| self.claim_worker_for_build(&chosen_id, build_info, msg)?; |
There was a problem hiding this comment.
For queued builds handled by a very fast worker, this sends TaskBuild before the subsequent DB writes set the target to Building. ws_service can process TaskBuildComplete immediately because active_builds already contains the build, write a terminal target state, and then this dispatch function can resume and overwrite build_targets.latest_state back to Building, leaving a completed build looking in progress. Persist the pre-dispatch state before exposing the build to the worker, or make the later Building update conditional on the build still being unfinished.
Useful? React with 👍 / 👎.
| pub async fn reconcile_orphaned_builds_on_startup(&self) { | ||
| let now = chrono::Utc::now().with_timezone(&FixedOffset::east_opt(0).unwrap()); | ||
| self.reconcile_orphaned_builds( | ||
| now, |
There was a problem hiding this comment.
Passing now as the startup cutoff makes reconciliation finalize every unfinished build event created before the process starts. With this commit, no-idle submissions are real queued builds whose only queue state is in memory; after a normal deploy/restart, a build accepted seconds earlier is no longer in active_builds and is immediately marked Interrupted instead of waiting up to the configured 60 minutes. Startup reconciliation should distinguish queued Uninitialized/Pending builds from Building orphans, use the queue wait cutoff, or restore/requeue them.
Useful? React with 👍 / 👎.
2 participants
No description provided.