Skip to content
Merged
114 changes: 110 additions & 4 deletions .github/workflows/mvn-ci-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,19 +29,125 @@ on:
branches:
- "main"

# The build cache extension does not hash the JDK, so it is part of every cache
# key below. Defined once here rather than repeated, so that changing the JDK
# cannot leave a key asserting the old one.
env:
JDK_VERSION: '11'

jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 30
# A cold build is ~27 min, which left no headroom under the previous 30.
timeout-minutes: 40
steps:
- uses: actions/checkout@v7

- name: Set up JDK 11
- name: Set up JDK ${{ env.JDK_VERSION }}
uses: actions/setup-java@v5
with:
java-version: '11'
java-version: ${{ env.JDK_VERSION }}
distribution: 'temurin'
cache: maven

# Restore only. License Check writes this cache, and only on `main`: it
# finishes in a couple of minutes against this job's 27, and it stops at
# `package` so the entry never carries this project's artifacts. It
# resolves the reactor explicitly for this job's benefit; its own license
# steps reach only part of it.
# Previously both jobs used setup-java's `cache: maven`, which gave them
# the same key, and whichever finished first reserved it - this job then
# logged "Unable to reserve cache with key" on every single run. Making
# the reader explicit removes the race rather than winning it.
#
# A warm start, not a guarantee: `install` here runs plugins that a build
# stopping at `package` never resolves, so expect a delta every run.
- name: Restore Maven repository cache
uses: actions/cache/restore@v6
with:
path: ~/.m2/repository
key: m2-repository-${{ runner.os }}-jdk${{ env.JDK_VERSION }}-${{ hashFiles('**/pom.xml', '**/.mvn/extensions.xml') }}
restore-keys: |
m2-repository-${{ runner.os }}-jdk${{ env.JDK_VERSION }}-

- name: Restore Maven wrapper cache
uses: actions/cache/restore@v6
with:
path: ~/.m2/wrapper/dists
key: m2-wrapper-${{ runner.os }}-${{ hashFiles('**/.mvn/wrapper/maven-wrapper.properties') }}

# A separate cache from the repository above: the build cache extension
# configured in .mvn/extensions.xml writes to ~/.m2/build-cache, so
# without this it starts empty every run and never restores a module.
# Only `main` writes an entry, so pull requests warm up from `main` instead
Comment thread
slachiewicz marked this conversation as resolved.
# of each branch filling the repository-wide cache budget.
#
# The modules producing large bundled jars opt out of the cache entirely
# with <maven.build.cache.enabled>false</maven.build.cache.enabled> in their
# own poms, which keeps this entry around 12 MB. Without that opt-out the
# entry is 1.2 GB, 97% of it three shaded jars that account for only ~3 of
# the 27 minutes a cold build takes.
#
# Note: a cache-restored xtable-service does not reproduce
# target/quarkus-app. No step consumes that directory today; anything added
# later that does must not rely on it being present.
- name: Restore Maven build cache
id: build-cache
uses: actions/cache/restore@v6
with:
path: ~/.m2/build-cache
key: maven-build-cache-${{ runner.os }}-jdk${{ env.JDK_VERSION }}-${{ github.sha }}
restore-keys: |
maven-build-cache-${{ runner.os }}-jdk${{ env.JDK_VERSION }}-

- name: Build all module with Maven
run: ./mvnw clean install -ntp -B

# Guard against a module starting to cache large artifacts again: warn and
# skip the save rather than churn the repository-wide 10 GB cache budget,
# which is otherwise mostly the ~/.m2/repository entries above. Checking the
# size rather than specific paths keeps this working across module renames.
- name: Check Maven build cache size
id: build-cache-size
if: success() && github.ref == 'refs/heads/main'
run: |
size_mb=$(du -sm ~/.m2/build-cache 2>/dev/null | cut -f1)
size_mb=${size_mb:-0}
echo "size_mb=${size_mb}" >> "$GITHUB_OUTPUT"
echo "~/.m2/build-cache is ${size_mb} MB"
# The two bounds below are what the save step gates on, so they have to
# cover the whole range between them: 0 means the extension wrote
# nothing and there is no path for the save action to read.
if [ "${size_mb}" -eq 0 ]; then
echo "::warning title=Maven build cache not saved::~/.m2/build-cache is empty or absent. The build cache extension did not write anything; check that .mvn/extensions.xml is still in effect."
elif [ "${size_mb}" -ge 100 ]; then
echo "::warning title=Maven build cache not saved::~/.m2/build-cache is ${size_mb} MB (limit 100 MB). A module is caching large artifacts; opt it out with <maven.build.cache.enabled>false</maven.build.cache.enabled> in its pom."
fi

# Skipped when the exact key already hit on restore - a re-run of the same
# commit would otherwise try to write over an immutable entry and log the
# "Unable to reserve cache with key" that splitting these jobs removed.
- name: Save Maven build cache
if: >
success() && github.ref == 'refs/heads/main'
&& steps.build-cache.outputs.cache-hit != 'true'
&& steps.build-cache-size.outputs.size_mb > 0
&& steps.build-cache-size.outputs.size_mb < 100
uses: actions/cache/save@v6
with:
path: ~/.m2/build-cache
key: maven-build-cache-${{ runner.os }}-jdk${{ env.JDK_VERSION }}-${{ github.sha }}

# `install` above puts this project's own artifacts into ~/.m2/repository,
# including the bundled jars (~1.2 GB, xtable-utilities alone is ~1 GB).
# actions/setup-java caches that whole directory with no way to exclude a
# path, so drop them before its post step runs. They are rebuilt from
# source on every run and can never produce a useful cache hit, and a
# stale SNAPSHOT left in the local repository can shadow a reactor build.
# Source Build Check clears the same path for the same reason.
#
# This job no longer saves that cache, so nothing depends on this today.
# It is kept so the invariant holds by construction rather than by which
# step happens to write last.
- name: Drop this project's own artifacts from the Maven repository
if: always()
run: rm -rf ~/.m2/repository/org/apache/xtable
89 changes: 86 additions & 3 deletions .github/workflows/mvn-license-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,62 @@ on:
branches:
- "main"

# Kept in step with mvn-ci-build.yml: this job writes the cache that job reads,
# and the JDK is part of the key on both sides.
env:
JDK_VERSION: '11'

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7

- name: Set up JDK 11
- name: Set up JDK ${{ env.JDK_VERSION }}
uses: actions/setup-java@v5
with:
java-version: '11'
java-version: ${{ env.JDK_VERSION }}
distribution: 'temurin'
cache: maven

# ~/.m2/repository is cached here rather than via setup-java's
# `cache: maven`, which restores on an exact key match only - it sets no
# restore-keys by design (actions/setup-java#269), so any pom change made
# every job re-download the whole dependency tree from Central. The prefix
# fallback below warms from the previous entry and downloads just the delta.
#
# This job is the writer for that cache, and deliberately the only one:
# it is the short job, so making it the writer costs the least, and it
# stops at `package`, so this project's own artifacts never enter the
# entry. Maven CI Build restores the same key read-only.
#
# Being the only writer means this job has to resolve what the readers
# need, which the license steps below do not do on their own - they build
# the shaded modules and their `-am` closure, which excludes the two
# leaves, xtable-utilities and xtable-service. The explicit resolve step
# after them covers the rest of the reactor.
#
# Restore and save are split so that only `main` writes. The entry is
# ~1 GB against a repository-wide 10 GB budget, so letting every branch
# write one would crowd out the entry they all restore from. Pull requests
# warm from `main` via the prefix and re-download only their own delta.
- name: Restore Maven repository cache
id: m2-repository
uses: actions/cache/restore@v6
with:
path: ~/.m2/repository
key: m2-repository-${{ runner.os }}-jdk${{ env.JDK_VERSION }}-${{ hashFiles('**/pom.xml', '**/.mvn/extensions.xml') }}
restore-keys: |
m2-repository-${{ runner.os }}-jdk${{ env.JDK_VERSION }}-

# Replaces the separate wrapper cache that setup-java's `cache: maven`
# used to provide. Keyed only on the wrapper properties, which change
# rarely, so it survives the pom changes that rotate the key above. Left
# unsplit unlike the repository cache: at ~9 MB it is not worth gating.
- name: Cache Maven wrapper
uses: actions/cache@v6
with:
path: ~/.m2/wrapper/dists
key: m2-wrapper-${{ runner.os }}-${{ hashFiles('**/.mvn/wrapper/maven-wrapper.properties') }}

- name: Apache License Check
run: ./mvnw apache-rat:check -B
Expand All @@ -61,3 +105,42 @@ jobs:

- name: Validate Bundled License Texts
run: python3 release/scripts/validate_bundled_license_texts.py

# Nothing above this line resolves xtable-utilities or xtable-service:
# `apache-rat:check` requires no dependency resolution, `dependency:tree`
# collects poms rather than jars, and the shaded-bundle build reaches only
# the `-am` closure of the three modules it names, which excludes both
# leaves. Without this step the entry never gains the Quarkus stack or
# xtable-utilities' bundled dependencies, and Maven CI Build - which
# cannot write the cache - would re-download them from Central on every
# run, permanently. This job exists to warm the cache for that one, so it
# has to cover the whole reactor even where a license check does not.
#
# Failure here must not fail a license check, hence `|| true`: an
# incomplete cache entry costs download time on the next run and nothing
# else. `go-offline` does not catch plugin dependencies resolved at
# execution time, so expect a small delta regardless.
- name: Resolve the rest of the reactor for the cache
if: success() && github.ref == 'refs/heads/main'
run: ./mvnw -B -ntp -q dependency:go-offline -Dmaven.build.cache.enabled=false || true

# This job writes the repository cache, so the invariant that the entry
# holds third-party dependencies only is enforced here. The steps above
# stop at `package`, so this is a no-op today; it exists so that adding
# an `install` later cannot silently add ~1.2 GB of bundled jars to the
# entry. Must stay ahead of the save below.
- name: Drop this project's own artifacts from the Maven repository
if: always()
run: rm -rf ~/.m2/repository/org/apache/xtable

# Only when the exact key missed - entries are immutable, so saving over
# an existing key just fails. This is what actions/cache does internally
# when restore and save are not split.
- name: Save Maven repository cache
if: >
success() && github.ref == 'refs/heads/main'
&& steps.m2-repository.outputs.cache-hit != 'true'
uses: actions/cache/save@v6
with:
path: ~/.m2/repository
key: m2-repository-${{ runner.os }}-jdk${{ env.JDK_VERSION }}-${{ hashFiles('**/pom.xml', '**/.mvn/extensions.xml') }}
8 changes: 7 additions & 1 deletion .github/workflows/package-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ jobs:
- name: Set Version
run: ./mvnw versions:set -DnewVersion="$VERSION"
- name: Publish package
run: ./mvnw -ntp --batch-mode deploy -DskipTests
# Never build a release from cached module output. `versions:set` above has
# just rewritten the version, while cache restoration is version-agnostic —
# it stamps the current project version onto whatever it restores. This
# runner keeps no build-cache directory today, so the flag is a guard
# against that changing, and it also saves checksumming every module for a
# cache that can never hit. Same reasoning as source-build-check.yml.
run: ./mvnw -ntp --batch-mode deploy -DskipTests -Dmaven.build.cache.enabled=false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
2 changes: 1 addition & 1 deletion .mvn/extensions.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,6 @@
<extension>
<groupId>org.apache.maven.extensions</groupId>
<artifactId>maven-build-cache-extension</artifactId>
<version>1.1.0</version>
<version>1.2.3</version>
</extension>
</extensions>
100 changes: 100 additions & 0 deletions .mvn/maven-build-cache-config.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<cache xmlns="http://maven.apache.org/BUILD-CACHE-CONFIG/1.3.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/BUILD-CACHE-CONFIG/1.3.0 https://maven.apache.org/xsd/build-cache-config-1.3.0.xsd">
<configuration>
<enabled>true</enabled>
<remote enabled="false"/>
<local>
<!--
One generation per module. The default of 3 accumulates generations,
which matters because CI persists this directory between runs; a
single generation keeps the saved entry at one build's size.
-->
<maxBuildsCached>1</maxBuildsCached>
</local>
</configuration>
<executionControl>
<reconcile>
<plugins>
<!--
Command line flags are not part of a module's checksum, so without
this a build that skipped tests and one that ran them are
indistinguishable to the cache. Listing a property makes any
difference between the cached build and the current run a cache
miss, in both directions.

Deliberately no skipValue attribute: skipValue is a relaxation, not
a guard. It tells the cache to tolerate a mismatch when the current
run is the one skipping, which is exactly the exemption we do not
want on a release build.
-->
<plugin artifactId="maven-surefire-plugin" goal="test">

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a similar config for the failsafe plugin?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes — and it turned out to matter more than the question implied. Two things came out of checking, both pushed in 3414da6.

Failsafe itself. It is bound in the parent POM to integration-test and verify with <skip>${skipITs}</skip>, and CI runs install, so integration tests do run on every build and a restored module skips them exactly as it skips unit tests. Added, one entry per goal — reconciles are matched per goal, and the two goals do not expose the same parameters (testFailureIgnore is on verify only; naming a parameter a goal does not have earns a warning on every build, which I checked against help:describe for failsafe 3.3.0 rather than guessing).

The one that actually bites. Writing the failsafe entries made me test the surefire block properly instead of assuming it, and it does not cover -Dmaven.test.skip=true. The parent POM pins <skip>${skipUTs}</skip>, and explicit plugin configuration wins over the user property, so surefire's skip parameter reads identically whether the flag is there or not — and the module checksum comes out byte-identical. What the flag actually suppresses is test compilation.

On xtable-api, Temurin 11, before the fix:

$ ./mvnw clean install -pl xtable-api -Dmaven.test.skip=true   # 0 tests run, entry saved
$ ./mvnw clean install -pl xtable-api
[INFO] Found cached build, restoring org.apache.xtable:xtable-api from cache by checksum 70d4788128782ec7

A plain install restoring a build whose tests never ran, reporting success. After adding a maven-compiler-plugin:testCompile reconcile on skip:

[INFO] Plugin parameter mismatch found. Parameter: skip, expected: true, actual: false
[INFO] Mojo cached parameters mismatch with actual, forcing full project build. Mojo: compiler:testCompile
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 - in org.apache.xtable.spi.extractor.TestExtractFromSource
...

Plain-to-plain still restores, so the guard costs nothing in CI. This is a local-developer hazard rather than a CI one today — CI writes the cache only from the plain install on main, and the release job disables the cache outright — but it is precisely the failure mode the reconcile block exists to prevent, and it was open.

buildinfo.xml now records all nine failsafe parameters as tracked="true" with no Cannot find a Mojo parameter warnings.

<reconciles>
<reconcile propertyName="skip"/>
<reconcile propertyName="skipExec"/>
<reconcile propertyName="skipTests"/>
<reconcile propertyName="testFailureIgnore"/>
</reconciles>
</plugin>
<!--
Failsafe is bound in the parent POM to integration-test and verify,
and CI runs `install`, so integration tests run on every build and a
restored module skips them exactly as it skips unit tests. -DskipITs
is a command line flag, so the same gap applies. Reconciles are
matched per goal, hence both entries; the parameter lists differ
because testFailureIgnore exists on verify only, and naming a
parameter a goal does not have earns a warning on every build.
-->
<plugin artifactId="maven-failsafe-plugin" goal="integration-test">
<reconciles>
<reconcile propertyName="skip"/>
<reconcile propertyName="skipExec"/>
<reconcile propertyName="skipITs"/>
<reconcile propertyName="skipTests"/>
</reconciles>
</plugin>
<plugin artifactId="maven-failsafe-plugin" goal="verify">
<reconciles>
<reconcile propertyName="skip"/>
<reconcile propertyName="skipExec"/>
<reconcile propertyName="skipITs"/>
<reconcile propertyName="skipTests"/>
<reconcile propertyName="testFailureIgnore"/>
</reconciles>
</plugin>
<!--
-Dmaven.test.skip=true never reaches surefire in this build: the
parent POM pins <skip>${skipUTs}</skip>, and explicit plugin
configuration wins over the user property, so surefire's skip
parameter reads identically whether the flag is present or not.
What the flag actually suppresses is test compilation. Without
this entry the module checksum is byte-identical across the two,
and a cache seeded by a `-Dmaven.test.skip=true` build restores
into a plain `install` with the tests never having run.
-->
<plugin artifactId="maven-compiler-plugin" goal="testCompile">
<reconciles>
<reconcile propertyName="skip"/>
</reconciles>
</plugin>
</plugins>
</reconcile>
</executionControl>
</cache>
3 changes: 2 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -953,7 +953,8 @@
</excludes>
<licenseHeader>
<file>style/xml-license-header</file>
<delimiter>^&lt;project|^&lt;configuration|^&lt;Configuration|^&lt;extensions|^&lt;component</delimiter>
<!-- ^&lt;cache is the root element of .mvn/maven-build-cache-config.xml -->
<delimiter>^&lt;project|^&lt;configuration|^&lt;Configuration|^&lt;extensions|^&lt;component|^&lt;cache</delimiter>
<skipLinesMatching>^&lt;.*xml.+?$</skipLinesMatching>
</licenseHeader>
</pom>
Expand Down
11 changes: 11 additions & 0 deletions xtable-aws/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,17 @@

<artifactId>xtable-aws</artifactId>
<name>XTable AWS</name>
<properties>
<!--
This module produces a large bundled jar (~42 MB). Keep it out of the
maven-build-cache entirely: the property disables both restore and save
for this project, so the artifact never enters the cache directory that
CI persists between runs. Shading is cheap relative to the test suites
the cache is there to skip, so there is nothing worth caching here.
See .mvn/maven-build-cache-config.xml.
-->
<maven.build.cache.enabled>false</maven.build.cache.enabled>
</properties>

<dependencies>

Expand Down
Loading
Loading