Balance test_inst CI shards by measured duration instead of project-path hash

[cgcote]

What Does This Do

Each Gradle project used to get its CI shard (-Pslot=X/Y) from abs(project.path.hashCode() % totalSlots) + 1. This PR replaces that with a deterministic, duration-aware greedy LPT (longest-processing-time-first) bin-packing:

• Weights file. buildSrc/src/main/resources/datadog/gradle/plugin/ci/test-shard-weights.json holds per-project test_inst weights for the 36 heaviest projects. Each weight is a mean per pipeline: CI Visibility sum(@duration) at module level, JVM 11, 7 days, divided by 491 pipelines. Any other project gets DEFAULT_WEIGHT_SECONDS = 20.6, the mean of the projects left out.
• TestShardBalancer is pure Kotlin with no Gradle types. It sorts projects by weight, heaviest first, breaking ties by path. Each project goes into the least loaded slot, breaking ties by lowest index. Hash codes are never used, so the result is the same on every JVM and machine.
• isInSelectedSlot computes the assignment once per slot count from the sorted rootProject.allprojects paths, so every shard agrees on it. It's cached on rootProject.extra behind a lock. All early returns and the info log line are unchanged.
• docs/ci-test-shard-weights.md covers how to regenerate the weights (measuring each project individually), when to refresh them, and the known limitation below.
• Muzzle classloader cache fix, which the new shard assignment exposed:
  • MuzzleAction kept one bootstrap/tooling classloader pair per build. It was only rebuilt when the classpath file count grew.
  • It now caches the pair per distinct classpath.
• opentelemetry-annotations-1.20 now declares muzzleBootstrap otel-bootstrap like its sibling OpenTelemetry modules. This is a consistency improvement only: muzzle passes with and without it, and it did not cause the muzzle failure.

The split stays 8-way, and .gitlab-ci.yml is unchanged.

Motivation

Work item: APMCI-45.

test_inst runs as 8 shards, and the tests stage ends when the slowest one does. The hash split balanced project counts, but not durations.

Measured, JVM 11 test_inst. Before is the CI Visibility p50 with the hash split; run 1 is the first pipeline with this PR, using the original 24-entry weights:

• 1/8: before 29m46s, run 1 39m07s
• 2/8: before 27m29s, run 1 24m49s
• 3/8: before 18m03s, run 1 40m58s
• 4/8: before 24m22s, run 1 23m52s
• 5/8: before 40m11s, run 1 27m20s
• 6/8: before 35m38s, run 1 27m04s
• 7/8: before 21m11s, run 1 28m45s
• 8/8: before 35m38s, run 1 28m33s

Run 1 did not improve the slowest shard: ~41 min, against a ~40 min baseline. The max/min spread fell from 2.2x to 1.7x, but:

• lettuce-5.0 was missing from the original weights. It is the second-heaviest project, at about 978 s per pipeline. As an "unknown" project it got the default weight and landed in slot 3 next to spring-webflux-5.0. Shard 3 ran 4105 s of tests, against 1525–2312 s for the others.
• This PR now re-harvests the weights per project (36 entries). That puts jdbc and lettuce-5.0 in separate slots.
  • The balancer's own model: max/min 1.10x (1.08x counting only real projects, without empty parent projects).
  • The hash on the same data: 2.14x / 2.34x.
  • These are modeled sums of test time, not wall-clock predictions.

Known limitation: the heaviest project sets a floor. A project's tests always run in one shard, and tests run in parallel within a shard. The balancer adds up seconds and cannot see this.

• In run 1, shards 1/8 and 4/8 ran the same total test time (2312 s and 2292 s) but took 39 and 24 minutes.
• Shard 1/8 held jdbc. Its tests (20.5 min of summed test time) spanned 27.6 min of wall clock and started about 10 min into the job.
• Under the hash split, jdbc was in slot 5, the old slowest shard at 40m11s.
• So the slowest shard is bounded below at roughly 38 min by jdbc alone, and this PR can only gain a minute or two on merge-ready time. The lettuce-5.0 shard (16.3 min of tests per pipeline) is the next floor.
• The TestShardBalancerTest target of max/min ≤ 1.25 is a check on the model, not a wall-clock guarantee.
• Lowering the floor needs finer-grained sharding than whole projects. Capping weights would make the model look balanced while piling more work onto the jdbc shard.

Why the muzzle fix is needed. On the first pipeline, muzzle: [5/8] failed: every opentelemetry-1.47 check hit NoClassDefFoundError: datadog/trace/bootstrap/otel/metrics/OtelInstrument, although 1.47 declares otel-bootstrap.

• graal-native-image-20.0 adds agent-installer, a larger but different bootstrap classpath, and ran first in the same worker.
• opentelemetry-1.47 then reused graal's loader, which has no otel-bootstrap.
• Under the hash split the two never shared a muzzle shard.
• The same cache also let projects see bootstrap classes they never declared.

Testing

• TestShardBalancerTest (28 cases), using the real project paths from settings.gradle.kts. It covers:
  • every project is selected by exactly one slot, and every slot is within range;
  • the assignment is deterministic and order-independent;
  • the two heaviest projects never share a slot (2–12 slots), and the top 8 are distinct at 8 slots;
  • every weighted path is an included project;
  • the modeled instrumentation ratio is at most 1.25;
  • the default weight, edge cases, and parsing.
• All 36 weighted paths resolve to projects included in settings.gradle.kts with a build file.
• Muzzle:
  • the new MuzzleActionTest fails with the old count-based cache and passes with the fix;
  • a local reproduction of the CI failure (graal, then opentelemetry-1.47, on a single worker) went from 24/24 failing to 24/24 passing;
  • all 1039 executed checks of CI's shard 5 pass;
  • all 8 muzzle shards pass in CI;
  • opentelemetry-annotations-1.20 passes 24/24 with the new declaration.
• ./gradlew :buildSrc:test -PrunBuildSrcTests: all balancer and muzzle tests pass, including MuzzlePluginFunctionalTest 20/20. 16 tests in BuildTimeInstrumentationPluginTest, CallSiteInstrumentationPluginTest, DumpHangedTestIntegrationTest and TestJvmCacheInputsTest fail only in the sandbox, because their nested TestKit builds can't resolve dependencies. They fail the same way without this PR, and check_build_src passes in CI.

Additional Notes

• Test coverage is unchanged. Only the shard a project runs in changes, and no test or muzzle check is skipped, retried or disabled.
• The weights are a snapshot. The window is 7 days ending 2026-10-11. Stale weights degrade balance but never coverage, and the test flags weights for renamed or removed projects.
• The project set must be the same on all shards. akka-http-10.6 is only included with akkaRepositoryToken, which CI sets for every job.
• Rollback: revert the balancing commits, and slot assignment goes back to path hashing. The muzzle cache fix is safe to keep regardless.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: [APMCI-45]

🤖 Generated with Claude Code (link removed)

NOTE: some AI-generated links and images were removed to ensure safety.

---

PR by Bits - View session in Datadog

Comment @DataDog to request changes

[datadog-official]

View session in Datadog

Bits Code status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @DataDog to request changes

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.94 s	13.87 s	[-0.1%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent	12.92 s	12.92 s	[-0.8%; +0.8%] (no difference)
startup:petclinic:appsec:Agent	17.13 s	16.48 s	[-0.6%; +8.4%] (no difference)
startup:petclinic:iast:Agent	16.85 s	15.99 s	[-0.9%; +11.6%] (unstable)
startup:petclinic:profiling:Agent	16.66 s	16.61 s	[-0.8%; +1.4%] (no difference)
startup:petclinic:sca:Agent	16.99 s	16.91 s	[-0.6%; +1.5%] (no difference)
startup:petclinic:tracing:Agent	16.06 s	15.65 s	[-1.6%; +6.9%] (no difference)

Commit: 62125929 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.