build: apply global .dockerignore during context sync

[crazy-max]

• needs session: add per-request filter hook to file sync moby/buildkit#7202
• fixes proposal: add client-side global dockerignore for local build contexts #3776

Buildx now reads a global .dockerignore from its config directory and applies it to local build contexts and local named contexts. The global rules run before repository rules, so a repository ! pattern can restore a globally excluded path. This extends the additive-only behavior originally proposed in #3776. The change uses the per-request file sync hook from moby/buildkit#7202. Remote contexts and Dockerfile mounts are unaffected.

[jsternberg]

I do have a general question for this and we might want to delay this from the current release until we have a longer discussion about it. I know it's been discussed before but the linked issue doesn't talk about it.

It's already fairly common to get issues or experience problems where a file that is expected to be in the context is missing or in the wrong place and the error message is a bit opaque. Are we taking any steps to mitigate that? Maybe we want some way to control the build to ensure that there is a way to enforce a "pure" build (only stuff in the context) versus an "impure" build (which could have some of the more local conveniences) and we could maybe configure that at the builder level or somewhere else?

I also wonder if having a file alongside .dockerignore such as .dockerignore.local would be a better solution than a global ignore file as it would allow the same functionality but keep the functionality in the context and prevent it from applying to things like git checkouts (which I'm not actually sure if git checkouts pay attention to dockerignore now that I think about it).

[jsternberg]

if err != nil {
    if errors.Is(err, os.ErrNotExist) {
        return nil
    }
    return errors.Wrapf(err, "...", ...)
}

You could also just set err = nil and call errors.Wrapf either way since that function will return nil if it is passed a nil error.

[jsternberg]

You can take advantage of that here by doing err != nil || len(patterns) == 0.

[tonistiigi]

where a file that is expected to be in the context is missing or in the wrong place and the error message is a bit opaque. Are we taking any steps to mitigate that?

We do have warning for the common cases where COPY doesn't match because file are ignored. This should work for this case as well. This doesn't look too different from similar issues with global .gitignore. You do need to be careful about using this feature.

[crazy-max]

where a file that is expected to be in the context is missing or in the wrong place and the error message is a bit opaque. Are we taking any steps to mitigate that?

We do have warning for the common cases where COPY doesn't match because file are ignored. This should work for this case as well. This doesn't look too different from similar issues with global .gitignore. You do need to be careful about using this feature.

Was wondering if we could show that in build progress if loaded like:

#0 building with "desktop-linux" instance using docker driver

#1 [internal] load global .dockerignore for context from /Users/kalvarez/docker/buildx/.dev/global-ignore-progress/config/.dockerignore
#1 0.000 *.tmp
#1 0.000 !keep.tmp
#1 0.000 secret
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 86B done
#2 DONE 0.1s

#3 [internal] load .dockerignore
#3 transferring context:
#3 transferring context: 2B done
#3 DONE 0.1s

#4 [internal] load build context
#4 transferring context: 144B done
#4 DONE 0.1s

#5 [1/1] COPY visible.txt keep.tmp /payload/
#5 DONE 0.0s

[thaJeztah]

If we add a global .dockerignore, we probably must match the behavior when using the classic builder as well. I think the feature is useful, but making it visible what's ignored (and why) may be very relevant (also for us if users report issues).

I once wrote down some thoughts here;

• Global .dockerignore moby/moby#12843 (comment)

I agree with most of what's said above, and understand use-cases such as excluding common files (like .DS_Store, thumbs.db etc). While implementing a global ignore file would (likely) not be too difficult from a technical perspective, it's primarily the side-effects of such a feature that need to be looked at (some of which mentioned above);

• reproducibility; the presence of a global .dockerignore can easily lead to "it (doesn't) work on my machine" situations; identical source repositories producing different results on different machines because files were excluded.
• discoverability: docker (currently) doesn't have a way to easily discover what files were excluded (and by which "exclude" rule); this can either lead to cryptic errors (file doesn't exist), or (worse) a build that is "seemingly" successful, but where files were excluded that are needed at runtime.
• in general; .dockerignore is a bit of a two-edged sword. While there are some good reasons to use a .dockerignore, it also results in Dockerfiles not being as "declarative" as they should be; while this is already a road taken (a .dockerignore will be shared by all Dockerfiles in a source repository), adding the concept of a global .dockerignore digs in deeper into that situation.

For the second bullet, a command to debug what's excluded (similar to git check-ignore) could assist (I listed this as an option in an epic I created on improving .dockerignore here: #40319). In addition, when using BuildKit as builder, the error messages could be improved to mention that a file was excluded (see moby/buildkit#1647). Note that improving the error messages would not help with situations where the build is successful, but the image fails at runtime due to files missing.

So, from the above, I think the first focus should be on making excluded files more discoverable / debuggable before (deciding to) adding a global ignore.

One other thing that could be useful (but .. also can be complicated) is multiple .dockerignore files, similar to Git; being able to add a .ignore file in a directory (which, if we support the same rules, could even be a symlink to a .gitignore)