[A-1511] Add OTel spans for repo-checkout git operations with --trace-git-checkout flag

[isaacsu]

Description

New experimental feature that emits additional git.* OpenTelemetry spans when the agent is started with --trace-git-checkout.

This will allow customers to see what goes on behind the "Preparing working directory" / repo-checkout span and figure out ways to speed up the checkout phase of their jobs.

Spans added

Span	Description
repo-checkout (existing)	Added attempt counter; git.* builds off its child context
git.mirror.update	main repo getOrUpdateMirrorDir
└ git.mirror.lock_wait.clone	mirror clonelock
└ git.mirror.clone	mirror gitClone
└ git.mirror.lock_wait.update	mirror updatelock
└ git.mirror.fetch	mirror gitFetch
└ git.mirror.snapshot	snapshot gitClone
git.dissociate	main repo gitRepack if GitMirrorCheckoutMode is dissociate
git.clone	main repo gitClone
git.clean.pre	main repo gitCleanSubmodules and gitClean
git.lfs.install	git lfs install
git.fetch	main repo fetchSource
git.verify_commit	main repo verifyCommit
git.sparse_checkout	setupSparseCheckout
git.checkout	main repo gitCheckout
git.submodules	main repo updateGitSubmodules
└ (for each submodule)
└ git.mirror.update	submodule getOrUpdateMirrorDir (see git.mirror.update above)
└ git.submodule.update	git submodule update
git.lfs.fetch	main repo gitLFSFetchCheckout
git.clean.post	main repo gitClean and gitCleanSubmodules

Screenshot

Changes

• Add --trace-git-checkout flag to CLI
• Add instrumentation to internal/job/checkout*.go paths
• Add traceGit*helper functions to internal/job/tracing.go
• Add/update tests for this change.
• Builds upon
  • [A-1511] Redact repo URL credentials in repo-checkout OTel span #4099
  • [A-1511] Split internal/job/checkout.go into multiple files #4109
  • [A-1511] Extract updateGitSubmodules into its own method #4111
  • [A-1511] Refactor refspec classification in fetchSource #4117

Testing

• Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
• Code is formatted (with go tool gofumpt -extra -w .)

Disclosures / Credits

While this PR was produced with the help of advanced automation tools, I take full credit and responsibility for it.

[zhming0]

More Otel span the better 🙌🏿 , but I wonder if we can make them less invasive in our code, perhaps we can start by thinking if the flag is worth it.

[zhming0]

Just for my understanding, what is the rationale behind adding a CLI flag as the toggle? I think it's a matter of 10 spans per job?

[zhming0]

Once we add a CLI flag it becomes a product surface that we will to support forever.

[zhming0]

These abstractions aren't ideal. Nothing in traceGitOp is about git, they seem mostly do a feature flag check + typical Otel ceremony.

If this is sole place where we check the flag I think it'd be reasonable but I think we have various other places checking the flag, which makes these questionable.

[buildsworth-bk-app]

I didn't find a concrete defect in the latest revision. Making the git.* spans automatic for either tracing backend across the default checkout path is cross-cutting enough to warrant a human sanity-check.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 8013, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.