fix(build, buildah): stop builds failing when a cached layer vanishes - #327
Draft
reyreavman wants to merge 7 commits into
Draft
reyreavman wants to merge 7 commits into
reyreavman wants to merge 7 commits into
Conversation
RunCommandWithOptions buffered the whole werf output and wrote it to GinkgoWriter only after Wait() returned. When a spec hit the suite timeout mid-build the report contained nothing from werf, so the stage it was stuck on was invisible. Tee every write into GinkgoWriter so a timed-out spec still shows what the subprocess printed. Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
Native Buildah resolves the rootless graph root from XDG_DATA_HOME and
the run root from XDG_RUNTIME_DIR, so all parallel specs of every e2e
job on a host shared one image graph and one mount table. Sibling specs
deleting their project images raced with builds in progress ("layer not
known", "image not known") and every build contended for the same
storage locks.
Point both at a per-process dir under HOME in the build suite and remove
it via buildah unshare after the suite, since rootless storage holds
files owned by mapped sub-UIDs.
Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
The helper listed the whole containers storage and then removed every match one by one; a sibling spec removing its own image between the two steps failed the caller with "image not known". Filter on the buildah side by reference and treat an image that is already gone as removed. Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
The cancel specs waited one minute from process start for the script's first log line and then sent SIGTERM. On a loaded host the image build alone took longer than that, so werf was killed mid-build and the spec failed on a missing "Creating namespace" instead of testing cancel. Add CancelOnOutputAfter to the command helper so the one-minute window opens only after "Executing into pod", and widen SpecTimeout to leave room for a slow build. Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
The complex and extra e2e jobs run Native Buildah under ginkgo -p, which picks 15 procs on the CI host, and several PRs' jobs land on the same machine at once. Under that load a Native Buildah spec that normally finishes in minutes runs past the suite timeout. Put the buildah-heavy jobs in one non-cancelling concurrency group, cap them at 6 procs, allow one flake retry, give the daily suites an explicit ginkgo timeout below the job limit, and log loadavg before the run so a timed-out job still reports it. Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
The daily complex and extra e2e jobs ran a werf built with both coverage instrumentation and -race. The race-instrumented binary runs Native Buildah several times slower, which is what pushed those specs past the one-hour suite timeout. Keep coverage, drop -race for the two buildah-heavy jobs; the simple, integration and unit jobs still run under the race detector. Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
imagebuildah walks every image in the local containers storage while looking for a cache hit. When another process on the host removes one of those images during the walk, the top-layer lookup returns "layer not known" and the whole build failed, although nothing was wrong with the build itself. This is routine on shared build hosts where several werf processes prune their own images concurrently. Treat that storage error from the first attempt as a cache miss and rebuild once with the local cache disabled. Only the error predicate is unit-tested; the retry path needs a Linux CGO build and a concurrent prune to exercise. Signed-off-by: Radmir Khurum <radmir.khurum@flant.com>
Collaborator
Author
Verification
Review focus
Follow-up
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Native Buildah e2e jobs on the shared CI host failed for reasons unrelated to the code under test: builds ran past the suite timeout, sibling specs saw "image not known" / "layer not known" from each other's cleanup, and the kube-run cancel spec killed werf before its image had finished building. Alongside the test and CI changes, a Dockerfile build with the Buildah backend no longer fails when another process removes a layer from the local storage while imagebuildah looks for a cache hit.
What
werf (Buildah backend)
FROM <local-image>.buildah rmiduring a build would settle it.e2e test harness
test/e2e/buildgets its own rootless containers-storage (graph root and run root) under a per-process dir in HOME, removed after the suite; on non-Linux nothing changes.GinkgoWriteras it arrives, so a spec that hits the suite timeout still shows what werf printed.RmiByRepoReflists only images under its own repo and treats an image that vanished beforermias removed.SpecTimeoutis 10 minutes.CI
e2e_complex,e2e_extraand their daily*_per_k8stwins share one non-cancelling concurrency group (buildah-e2e,queue: max), so only one Buildah job runs on the host at a time and the rest wait rather than get cancelled. A PR'se2e_complexande2e_extranow run one after the other.--procs=6(was 15 via-p) and--flake-attempts=2;e2e_simplegets--flake-attempts=2only.--timeout=110mbelow the 120-minute job limit; before, ginkgo's default one-hour timeout applied.e2e_complex_per_k8sande2e_extra_per_k8sbuild werf without-race(coverage kept); the other daily jobs still build with it.loadavgbefore the tests, so a timed-out job still reports host load.Why
Native Buildah resolves the rootless storage from
XDG_DATA_HOME/XDG_RUNTIME_DIR, and the test harness never set them, so 15 ginkgo procs per job, times three or four concurrently scheduled jobs on the same runner host, all worked on one image graph. The observed cost: a spec that takes 14 minutes on a quiet host runs past 85 minutes under that contention, sibling specs'rmiremoved layers underneath running builds, and every red run was retried blindly, hiding real regressions. The daily run made it worse by shipping a-racewerf binary into the slowest jobs with ginkgo's implicit one-hour timeout.A werf-side
--buildah-graph-rootflag was the alternative to setting the XDG variables in the tests; rejected because the tests can isolate themselves without adding user-facing surface.