Merge in buildkit all at once#14
Conversation
containerimage: fix possible race on parallel image create
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
policy: fix ResolveAttestations via policy callback
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
gitsign: support combined public keys for verification
Abort uncommitted ingest on ReaderAt failures and reset resumed writer offset before copy to avoid stale partial state for digest refs. Tests for canceled-read poisoning and resumed-ingest offset handling. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This removes the MigrateV2 function, which was added in BuildKit v0.7.0 (Docker v20.10.0) in 31a9aee. That was in 2019, which is now over 6 Years ago, so it's very unlikely for old files to be still present. Removing this code would impact users migrating from Docker 19.03 or older, which are versions that reached EOL many years ago, so very unlikely. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
docs: add "good" example for SecretsUsedInArgOrEnv check
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
contentutil: fix referrer blob ingest cleanup
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
update binfmt to v10.1.3
Signed-off-by: Kevin L. <kevin.lecouvey@gmail.com>
…t conversion Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
…OVENANCE_V1 Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
…ient Add support for riscv64 architecture in llb client
These requests allow computing additional checksum algorithms for the payload of HTTP sources. Optionally suffix can be passed that is added to payload. This is needed to make validation of PGP signatures possible remotely. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Add detached PGP verification for HTTP sources during metadata resolution and expose LLB options/caps/attrs for signature validation. Extract shared OpenPGP verification/parsing logic into util/pgpsign and reuse it from git signing, plus add integration and source-level tests. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
http: add support for checksum requests and pgp signatures for http sources
worker: remove automatic migration of legacy v1 cache
default to provenance slsa v1
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Dependency update for v0.28
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Image blob source in LLB allows addressing a single blob from a container image registry. The difference from the image source is that image source needs to point to a manifest that internally points to an array of layer blobs that are all extracted on top of each other to form a root FS. Contrary, image blob points to a single blob that is not extracted but downloaded as a single file into an empty snapshot, similarily how the HTTP source works. The main use case for this source is to pin snapshots of HTTP URLs, upload the downloaded blob into container registry, and then use a source policy to map a HTTP URL (whose content might be changed) to the copy of the source as image blob to ensure immutability. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
- Fix Env field incorrectly set to hook.Args instead of hook.Env - Suppress staticcheck SA1019 for intentional legacy prestart support Signed-off-by: Giles Cope <gilescope@gmail.com>
Resolve 47 lint issues introduced by upstream's stricter lint config: - errorlint: use errors.Is/errors.As instead of == and type assertions - forbidigo: use context.WithCancelCause/WithTimeoutCause/Cause - gocritic: fix comment formatting (space after //) - gofmt: reformat 4 files - staticcheck: fix deprecated imports, simplify selectors, nolint pkg names - noctx: use DialContext/Listen with context - unused: remove dead code Signed-off-by: Giles Cope <gilescope@gmail.com>
- Replace interface{} with any
- Use strings.CutPrefix instead of HasPrefix+TrimPrefix
- Use maps.Copy instead of manual k/v loops
- Modernize for loops to range over int
- Remove unnecessary variable copy
- Fix comment formatting (space after //)
Signed-off-by: Giles Cope <gilescope@gmail.com>
9c315bd to
d8293f6
Compare
Sort maps import alphabetically among stdlib imports. Signed-off-by: Giles Cope <gilescope@gmail.com>
Prefix unused function parameters with _ to satisfy the gopls unusedparams analyzer. Signed-off-by: Giles Cope <gilescope@gmail.com>
Earthly's base64 encoding of git stderr prevented upstream code in source/git/source.go from detecting "would clobber existing tag" and "refname conflict" errors via strings.Contains. Include raw redacted stderr alongside the base64 so pattern matching works. Also update TestGitCLIConfigEnv to match earthly's intentional behavior of not isolating git config (HOME is always passed through). Signed-off-by: Giles Cope <gilescope@gmail.com>
go mod tidy removed unused direct deps (containerd v1, docker/docker, gogo/googleapis, go-fuzz-headers) left over from the upstream merge. go mod vendor then cleaned up 149 files of unused vendored code. Signed-off-by: Giles Cope <gilescope@gmail.com>
- Remove gogoproto.stdtime annotations from control.proto (earthly- specific SessionHistory fields) since gogo protoc is no longer used - Fix vendored fsutil wire.proto bare import to use fully-qualified path - Exclude earthly-specific protos (localhost, pullping, socketforward, registry, gateway) from protoc generation as they use old gogo-style codegen incompatible with modern protoc-gen-go - Add M-flag overrides for fsutil types proto go_package paths - Remove hand-written control_earthly.pb.go (now generated by protoc) - Update control.go to use timestamppb.New() for History timestamps Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
- Fix copylocks warning in client_test.go: types.Stat now contains a mutex via protoimpl.MessageState, use pointers instead of copies - Re-vendor after go mod tidy to sync vendor with updated fsutil Signed-off-by: Giles Cope <gilescope@gmail.com>
|
Not looking too shabby. Passes all the tests (except some flaky windows ones that are flaky upstream also). Next question, what does EarthBuild make of it? |
Signed-off-by: Giles Cope <gilescope@gmail.com>
Earthly's multi-BUILD pattern legitimately produces multiple refs without platform mapping. The upstream verifier added in buildkit treats this as a hard error; downgrade to a warning so earthly builds continue to work. Signed-off-by: Giles Cope <gilescope@gmail.com>
Upstream commit 7aaa797 caches gateway mounts for performance, but this causes stale reads when a persistent cache mount is modified by a RUN and then read by a subsequent SAVE ARTIFACT within the same target. Disable the cache lookup so each operation gets a fresh mount. Signed-off-by: Giles Cope <gilescope@gmail.com>
The manual runtime.LockOSThread() in run() was added to stop runc processes intermittently exiting with status -1 (the classic PdeathSignal-vs-Go-scheduler race, golang/go#27505). go-runc v1.1.0 (which the upstream merge pulled in) already handles this: because we set Runc.PdeathSignal (updateRuncFieldsForHostOS), Run() -> startCommand() takes the Monitor.StartLocked() path, which calls runtime.LockOSThread() on the goroutine that actually does cmd.Start() (the fork). Our manual lock was on the *caller* goroutine -- not the forking one -- so it no longer does anything useful here. Removing it restores executor_linux.go to the exact upstream version, shrinking the fork delta with no behaviour change.
docopy() dropped action.ModeStr (string chmod, e.g. COPY --chmod=u+x) and hard-disabled action.AlwaysReplaceExistingDestPaths, with a comment claiming "not supported in this fsutil fork". That comment is stale: the vendored fork (EarthBuild/fsutil, pulled in by 5638371) supports both -- copy.CopyInfo.ModeStr (copy.go:197) and AlwaysReplaceExistingDestPaths (copy.go:208) are honoured. The removals dated from an older fsutil and silently degraded string-mode COPY to the integer-mode fallback. Restores the upstream code path; matches base.
Three changes that diverged from upstream with zero functional effect, restored to reduce future merge conflicts: - diffapply_linux.go: uint64(stat.Dev) -> stat.Dev (Dev is uint64 on all linux arches, the cast was a no-op) and Rdev: 0 -> unix.Mkdev(0, 0) (Mkdev(0,0) == 0). - scheduler_test.go: context.TODO() -> t.Context() in two loop bodies. t.Context() (Go 1.24+) is available under go 1.25.5 and is what upstream uses; the downgrade was an older-toolchain artifact.
Reverts util/archutil to upstream: removes the /proc mount in the arch check and the check_nolinux.go stub, and restores check_unix.go's !windows build constraint (it had been narrowed to linux). The patch worked around Rosetta failing to open /proc/self/exe inside the check's chroot, but it is not needed here and carried two upstream regressions: the !windows->linux narrowing dropped the arch check on FreeBSD/Darwin, and the mount error was fatal, breaking the check on rootless/unprivileged daemons. Restoring upstream shrinks the fork delta and removes both regressions.
Removes the earthly diagnostic scaffolding added to chase the "inconsistent graph state" / "return leaving incoming/outgoing open" scheduler errors: the dgstTracker ring buffer (inconsistent_graph_state_error_tracker.go), the helpMe context strings and bklog error logging in scheduler.go dispatch/NewInputRequest, and the dgstTracker.add call sites in jobs.go. The bug is no longer being actively investigated, so this telemetry is no longer needed. All three files now match upstream exactly, removing ~120 lines of delta from the scheduler/jobs hot path -- the most merge-conflict-prone files in the fork.
maps.Clone(opt.FrontendAttrs) returns nil when opt.FrontendAttrs is nil, and the following maps.Copy(frontendAttrs, cacheOpt.frontendAttrs) then panics with "assignment to entry in nil map" whenever cache attrs are present. This is hit on the Client.Build path (earthly's multiImageSolver passes nil FrontendAttrs) with remote/inline cache configured. Regression from upstream "simplify with maps pkg functions" (dfc3527), which replaced a nil-safe explicit map init with maps.Clone/Copy. Present in moby/buildkit master too -- candidate for upstreaming. Re-add the nil guard so frontendAttrs is always writable.
This reverts commit 80e66c7.
This reverts commit e708358.
The fork existed only to add a per-file VerboseProgressCB (verbose-only transfer logging, 2022). Stock fsutil already provides the equivalent per-file hook on receive via ReceiveOpt.NotifyHashed (ChangeFunc), which carries path + FileInfo. Rewire the earthly filesync target callbacks to NotifyHashed and use stock fsutil's 4-arg Send everywhere, removing every VerboseProgressCB reference. Drop the go.mod replace directive and re-vendor stock fsutil. Client API: ExportEntry.VerboseProgressCB -> OnReceiveFile (fsutil.ChangeFunc).
Resolves the Dockerfile version-baking conflict: keep the branch's worktree-aware heredoc block (skips version check for git worktrees, sets dev stubs) and apply main's rebrand from #18 (EARTHLY_PKG github.com/earthly/buildkit -> github.com/EarthBuild/buildkit).
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
… AS LOCAL panic) (#20) * session/filesync: byte-level output progress, no ContentHasher (fixes AS LOCAL panic) syncTargetDiffCopy set NotifyHashed without a ContentHasher. fsutil wraps every incoming file in a hashedWriter whenever NotifyCb is set and DEREFERENCES the hasher to do it (diskwriter.go, newHashWriter), so a nil hasher is a nil-pointer panic, not a no-op - and `SAVE ARTIFACT ... AS LOCAL` crashed outright: panic: runtime error: invalid memory address or nil pointer dereference fsutil.newHashWriter(0x0, ...) diskwriter.go:285 The session attachables run in the CLI, which is why it showed up there, not in buildkitd. Introduced by the verbose-progress rewire: the forked fsutil had a VerboseProgressCB hook that needed no hasher; stock fsutil offers only NotifyHashed, and the port adopted the hook without its precondition. The real problem was the wrong hook. This is the OUTPUT path (writing built artifacts to the user's local disk); it computes no cache key and reads no digest - the callback was pure progress. But NotifyHashed is fsutil's HASHING per-file hook: it would MultiWriter every received byte through a hasher for a digest nobody reads, on every AS LOCAL, even once the nil-deref is fixed. So switch to ReceiveOpt.ProgressCb: cumulative received bytes, per packet, NO hasher. It gives LIVE transfer feedback - which is what an output copy actually wants - and the panic is structurally impossible because no ContentHasher is ever constructed. The earthly-specific sync-target API carries a `func(int, bool)` progress callback instead of an `fsutil.ChangeFunc` (ExportEntry.OnReceiveFile -> OnReceiveProgress). Not the Filter hook: it is an inclusion filter, applied in two places (receive.go destWalker + doubleWalkDiff) and sometimes with an empty stat, so it double-counts. Not a discard/no-op ContentHasher: it works, but it abuses the hashing hook for a non-hashing need. ProgressCb is the hook fsutil actually provides for this. Trade-off: byte-level aggregate instead of per-file lines. On this path the per-file lines were VerbosePrintf (debug-only) anyway; the normal-level UX is a throttled byte summary, which ProgressCb feeds directly - and byte-level is the better signal for a slow network copy. Verified: SAVE ARTIFACT AS LOCAL of a 60 MiB artifact succeeds, exact bytes on disk, live byte progress, no hashing. It panicked 100% of the time before. Assisted-by: Claude:claude-opus-4.8 claude-code * Update client/solve.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update session/filesync/filesync.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update session/filesync/filesync.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update session/filesync/filesync.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update session/filesync/diffcopy.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * ci: bump docker/bake-action v7.0.0 -> v7.3.0 (fixes transitive SHA-pin failure) The repo ruleset requires all actions pinned to a full-length commit SHA, and enforces it TRANSITIVELY. docker/bake-action@v7.0.0 is correctly pinned by us, but its own subaction/matrix/action.yml uses `actions/github-script@v7` UNPINNED internally - which we cannot pin (it is Docker's action, not ours). Result: every workflow using bake-action failed at "Set up job": The action actions/github-script@v7 is not allowed in EarthBuild/buildkit because all actions must be pinned to a full-length commit SHA. bake-action v7.3.0 pins its internal github-script to 3a2844b7e9c422d3c10d287c895573f7108da1b3 (v9.0.0), so bumping satisfies the ruleset. All 8 refs across 4 workflows (.test, buildkit, test-os, validate), including the /subaction/matrix variants that carry the offending reference. This is why the validate/prepare gate was red on PRs against this branch (and on the branch's own runs). Unrelated to any code change. Assisted-by: Claude:claude-opus-4.8 claude-code Signed-off-by: Giles Cope <gilescope@gmail.com> (cherry picked from commit 1dd49bd) Signed-off-by: Giles Cope <gilescope@gmail.com> * ci: guard empty includes in test-matrix prepare (js-yaml throws on '') The prepare job's github-script does `yaml.load(core.getInput('includes'))`. Current js-yaml throws `YAMLException: expected a document, but the input is empty` on `yaml.load('')`, and empty includes is the NORMAL case - so prepare failed every run, and since `run` has `needs: [prepare]` the whole test matrix was blocked. Masked until now by the transitive SHA-pin ruleset failure (the job died at "Set up job" before reaching the script); unmasked once the bake-action pin bump let it past setup. Guard the empty input, restoring the prior undefined -> `[]` behaviour that the existing `?? []` was already written for. Assisted-by: Claude:claude-opus-4.8 claude-code Signed-off-by: Giles Cope <gilescope@gmail.com> --------- Signed-off-by: Giles Cope <gilescope@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
15 participants
Experimental huge merge of latest moby/buildkit into earthbuild fork's main.
Not planning to merge this as-is, just opening to see what can be learned about the work ahead from attempting the merge
Claude final message after performing the merge
It would be interesting to see how this can be split in to coherent PRs and to use this state of the code to try to determine what needs to be changed on the earthbuild client side
Split out coherent PRs:
Split out coherent PRs to upstream: