Repository navigation
fix(ci): report and fail evidence world waits before the job limit - #6473
Open
benjaminshafii wants to merge 1 commit into
Open
benjaminshafii wants to merge 1 commit into
benjaminshafii wants to merge 1 commit into
Conversation
…job limit A dependency change on dev invalidated the shared Daytona image. One PR core-journey job took the image builder lock and stayed silent until the 30-minute job limit cancelled it; another PR job and the dev warm-up waited on that lock with no output (the warm-up then built the same image in about three minutes). Nothing explained the cancellations. The budgets did not nest: job 30 min, prepare step 35, layer wait 45, a stale builder only after 60, build script 25, snapshot 20, snapshot quota retries 16. - packages/daytona-evidence/src/build-budget.ts holds the timing in one place. A preparation gets a deadline (OPENWORK_EVIDENCE_PREPARE_BUDGET_MINUTES from the CI step); every wait (builder lock, snapshot, build script, sandbox start, quota retry) is capped two minutes before it. - Waits print a progress line at most every 30 s: the layer, what it waits on, the builder's state, age, commit and when its controller gives up, or the snapshot state. - When the budget runs out while another controller holds the layer, the job prints the wait summary and the builder's classified log tail (existing bounded diagnostic) and fails with a clear message. - A controller records its deadline on its builder, so a builder left by a cancelled job is taken over minutes after that deadline instead of after an hour. Builders without the label keep the one-hour rule. - pr-proof.yml: core prepare step 13 min with an 11 min budget inside the 30 min job; checkpoint prepare 30 min with a 28 min budget inside 60. .github/scripts/pr-proof-budgets.test.mjs fails if they stop nesting. - Fake-clock unit tests cover the budget, progress throttling, a waiter failing with a diagnosis, taking over an abandoned builder and a build script stopping at the budget. Signed-off-by: Benjamin Shafii <benjamin.shafii@gmail.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
OpenWork evidence
Run details · Open the preview to inspect evidence and sandbox options. |
![diff-warden[bot]](https://avatars.githubusercontent.com/u/118544361?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Warden clearance. Complete security and confidentiality review of 03466b3ce18fd3df5dbdacb6cd648f3f867f9d21 found no high or medium security findings and no confidentiality findings. Any new push dismisses this approval. Analysis run
Contributor
Warden: approvedNo high or medium security findings and no confidentiality findings. Reviewed |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this about?
PR core-journey jobs that prepare a Daytona evidence world now report what they wait on and fail with a reason before the job limit, instead of being cancelled silently.
What problem does it solve?
After a dependency change on dev, core-journey jobs died at the 30-minute job limit with no output and no artifacts, so nobody could tell why.
What was the situation before?
Runs 40049411515 and 40049368045 printed at most
{"stage":"daytona-base","cacheHit":true}, then 30 silent minutes, then "The operation was canceled".How is this implemented?
packages/daytona-evidence/src/build-budget.tsholds the timing in one place.OPENWORK_EVIDENCE_PREPARE_BUDGET_MINUTESsets a deadline, and every wait inbuilder.tsstops 2 minutes before it.pr-proof.yml: core step 13 min with an 11-minute budget, checkpoint step 30 with 28..github/scripts/pr-proof-budgets.test.mjsfails if they stop nesting.What else changed?
Release note
none
Evidence
No E2E; fake-clock unit tests, the workflow budget test (fails on dev's workflow), and a run against real Daytona. A held builder printed progress and failed in 91 s with "still waiting on builder … its controller gives up in 18.5 min". An abandoned one was taken over in 1 s.