Skip to content

Split JDBC tests across CI shards - #12830

Draft
cgcote wants to merge 7 commits into
dd/ci/APMCI-45-balance-test-inst-shards-by-durationfrom
codex/apmci-50
Draft

cgcote wants to merge 7 commits into
dd/ci/APMCI-45-balance-test-inst-shards-by-durationfrom
codex/apmci-50

Conversation

@cgcote

@cgcote cgcote commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Splits the JDBC instrumentation's existing test workload into 15 independently scheduled CI shard projects. The three regular version tasks expose their slow RemoteJDBCInstrumentationV0Test suite separately. The forked suite is divided into eight exhaustive groups, including one group for each slow remote class and a catch-all for future forked tests.

The virtual projects delegate to test tasks in the parent JDBC project. Local allTests and check retain every suite, while sharded CI assigns each group independently.

Motivation

JDBC originally contributed 1116.7 seconds of test execution per JVM 11 test_inst pipeline and set an 18.6-minute scheduling floor.

The first coverage-complete split run, pipeline 144153574, exposed three dominant remote suites, so this revision schedules those suites independently. Refined pipeline 144158285 then reported 46 suites and 1,814 tests, all passing, and supplied direct weights for all 14 nonempty groups.

Across the two refined measurements, the largest JDBC unit was 321.831–346.626 seconds. The green confirmation result is a 68.96% reduction from 1116.7 seconds and is 3.22 times smaller. The modeled eight-shard load ratio is 1.0045.

Measured Results

  • Pipeline 144153574 completed successfully with no failed jobs.
  • Its initial nine JDBC groups had a largest unit of 503.141 seconds, already 54.9% below the baseline.
  • The eight JVM 11 test_inst jobs totaled 13,963 job seconds, versus 14,983 seconds on baseline pipeline 144088181: 6.8% less compute time.
  • Pipeline 144158285 measured the refined layout at 321.831 seconds for its largest unit, with complete passing JDBC coverage.
  • That pipeline's only failures were the six Java-version jobs assigned the intentionally empty future-test catch-all. Commit 98d11d6fd1 disables Gradle's no-tests failure for that task while retaining its include/exclude coverage for future forked classes.
  • A configuration-only inspection confirms the catch-all has failOnNoDiscoveredTests=false and retains the exhaustive forked-test filters.
  • Confirmation pipeline 144164425 passed every check with 46 JDBC suites and 1,814 tests, all passing. Its largest JDBC unit was 346.626 seconds.
  • The confirmation run's eight JVM 11 test_inst jobs totaled 13,768 job seconds, versus 14,983 seconds on baseline pipeline 144088181: 8.11% less compute time.
  • A configuration-only audit across all eight slots confirms that every JDBC task is selected exactly once.
  • Spotless formatting and git diff --check pass. No test suites were executed locally.
  • This is stacked on Balance test_inst CI shards by measured duration instead of project-path hash #12822 and should be rebased onto master after that PR merges.

Contributor Checklist

  • Preserve existing JDBC test coverage
  • Use direct CI Visibility measurements for shard weights
  • Keep a catch-all for newly added forked tests
  • Assign the required labels

Work item: APMCI-50

@cgcote cgcote added tag: no release notes Changes to exclude from release notes type: refactoring inst: jdbc JDBC instrumentation comp: tooling Build & Tooling tag: ai generated Largely based on code generated by an AI or LLM labels Oct 12, 2026
@datadog-prod-us1-6

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)

Suite Status
Startup 🟡 warning

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.91 s 13.84 s [-0.2%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 12.84 s 12.90 s [-1.2%; +0.3%] (no difference)
startup:petclinic:appsec:Agent 17.77 s 17.60 s [-0.0%; +2.0%] (no difference)
startup:petclinic:iast:Agent 17.39 s 17.68 s [-2.6%; -0.8%] (maybe better)
startup:petclinic:profiling:Agent 17.09 s 17.29 s [-2.3%; -0.0%] (maybe better)
startup:petclinic:sca:Agent 17.70 s 17.72 s [-1.2%; +0.9%] (no difference)
startup:petclinic:tracing:Agent 16.56 s 16.73 s [-2.1%; +0.0%] (no difference)

Commit: 98d11d6f · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@cgcote cgcote mentioned this pull request Oct 12, 2026
4 tasks done

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: tooling Build & Tooling inst: jdbc JDBC instrumentation tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant