RAT-573: Make RAT safe for parallel Maven builds - #704
Conversation
… Maven builds Replace the plain static fields in DefaultLog and DeprecationReporter with ThreadLocal storage so that each thread (e.g. parallel Maven reactor threads using `mvn -T`) gets its own logger and deprecation reporter instance. Previously, every Mojo constructor overwrote the JVM-wide DefaultLog.instance singleton, causing log messages to be silently routed to the wrong module's logger during parallel builds. The same race condition affected DeprecationReporter.consumer. The public API (getInstance/setInstance, getLogReporter/setLogReporter) is preserved so all existing callers continue to work unchanged. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claudenw
left a comment
There was a problem hiding this comment.
This is a great start on an issue that has been in the back of my mind. However, these changes will not make RAT thread safe for multi threaded Maven.
There is a class called the MatcherBuilderTracker that also contains statics. This class tracks any matchers that have been declared. Basically you can write your own matcher for some special condition and add it to the run. Those are tracked in as static variable in the MatcherBuilderTracker and are loaded from the configuration file declaration.
The SPDXMatcher factory has a static instance of the matcher factory. The structure is used to ensure that the SPDX matchers checks are clustered. Calling one matcher check causes all the matcher checks to run at once. This is because all the matchers use the same regular expression pattern and it is far more efficient to run the match check once across the file and extract the SPDX id and then verify that its accepted rather than run the expression find for each SPDX id that is accepted.
There may also be issues with reading files, but as I recall that was just an idea that we could speed up the processing by using multiple threads to check the files in parallel.
I would suggest that you change the definition of RAT-573 to make it read that it will make RAT safe for parallel builds, and address the other issues as well.
Finally, there is a massive change coming wherein we split the UIs off into their own projects. This will not be impacted by the changes I see here. However, it does move the conversion from RAT command line options to Maven options into a Maven specific section and Maven will need to track the mappings. I don't think there are any statics in the prototype code, but I will keep an eye out.
Thank you for this contribution. It is greatly appreciated.
…el builds OptionCollection.parseCommands() uses shared mutable state: the Arg enum's OptionGroup instances (whose 'selected' field is mutated by DefaultParser.parse()) and Converters.FILE_CONVERTER (whose workingDirectory field is set during argument processing). When multiple Maven reactor threads call parseCommands() concurrently (e.g. mvn -T4), they corrupt each other's parse state. This causes options like --input-exclude to be silently skipped, resulting in files being incorrectly reported as having unapproved licenses. Making the method synchronized serializes only the configuration parsing phase; the actual file scanning and license checking still runs in parallel. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…gration test Restore RAT-268 to its original single-threaded form as requested by maintainer (integration tests named RAT-XXX are tied to specific Jira tickets and should not be modified for unrelated purposes). Add a new RAT-573 integration test that exercises the rat plugin under parallel Maven builds (-T4) with a multi-module project containing excluded files (src.apt). This directly tests the thread-safety fixes in DefaultLog, DeprecationReporter, and OptionCollection.parseCommands. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Thank you for the thorough review @Claudenw! I've addressed both points:
The Happy to update the RAT-573 Jira description if you'd like it reframed as "make RAT safe for parallel builds" rather than just the DefaultLog issue. |
|
@gnodet , again thank you for your work on parallelizing RAT. I think that the SPDX Matcher needs attention to make this functionally complete. Issue: Analysis: The SPDXMatcherFactory creates instances of the SPDXMatcher class. Each of those instances point contain a reference to the factory. When they are triggered they insert their ID into the When the matcher is executed it calls the When the matcher is queried to see if it has found a match it checks to see if its ID is in the There is a MATCHER_MAP static var in the SPDXMatcherFactory that could probably become a local variable. The data that the Matcher is updating is contained in the SPDXMatcherFactory (probably a bad name at this point). The SPDXBuilder is called when the configuration specifies an SPDX matcher is needed. The Builder calls to the SPDXMatcherFactory to create the matcher. Potential Solution: You have made the creation of the configuration synchronized. During the creation of the configuration we use the factory instance to create SPDX matchers. If the SPDXMatcherFactory were made thread local to the SPDXBuilder, and the static variables in the SPDXMatcherFactory we mitigated I think this would solve the problem. Question: |
SPDXMatcherFactory.INSTANCE is a shared singleton whose lastMatch/checked state and static MATCHER_MAP are corrupted when two threads scan different documents concurrently: both threads see both sets of SPDX IDs. Changes: - MATCHER_MAP: moved from static to instance field so each factory instance maintains its own matcher registry - Constructor: changed from private to package-private, added newInstance() factory method for multi-threaded use - SpdxBuilder: uses a ThreadLocal<SPDXMatcherFactory> so each Maven reactor thread gets its own factory with isolated match state - INSTANCE kept for backward compatibility (single-threaded / tests) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@Claudenw Done — I've addressed the SPDXMatcherFactory thread-safety issue. Here's what changed:
This prevents the scenario you described: two threads scanning different files no longer share All existing tests pass (including |
DefaultLog.removeInstance(), DeprecationReporter.removeLogReporter(), and SpdxBuilder.removeFactory() were never wired into any cleanup path. The ThreadLocal values are naturally overwritten on each mojo execution via setInstance()/setLogReporter()/FACTORY.get(), so explicit removal is unnecessary. Remove the dead methods to avoid suggesting a cleanup contract that isn't enforced. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@Claudenw I did a thorough audit of the codebase for other static mutable state that could cause issues under parallel builds. Here are the results: Already fixed in this PR
Remaining:
|
|
@gnodet thanks, could you point your claude session to the classes mentioned in:
|
|
@ottlinger Thanks for pointing to RAT-553. I looked at Root cause: shared mutable builder instances in
|
MatcherBuilderTracker: use ConcurrentHashMap instead of HashMap for the matcher builder registry, preventing corruption when multiple threads register or look up builders concurrently. StandardCollection: store a Supplier<AbstractFileProcessorBuilder> instead of a shared builder instance. Enum constants are singletons, so the previous shared GitIgnoreBuilder/HgIgnoreBuilder/etc. instances had their mutable state (levelBuilders TreeMap, HgIgnoreBuilder.state) corrupted when two threads called build() concurrently. The supplier creates a fresh builder per invocation, eliminating the race. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Latest commit:
|
- Deprecate INSTANCE field (will be removed in 1.0.0) - Make constructor private - Use computeIfAbsent() in create() method - Update changes.xml to "Make RAT safe for parallel builds" Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@ottlinger I ran This was on JDK 21. The heap space issue you saw with JDK 25 + Maven 3.9.16 in apache-rat-tasks might be environment-specific rather than related to this PR's changes (since master works fine for you and this PR doesn't touch apache-rat-tasks at all). |
|
@ottlinger I investigated the site build issue more thoroughly, now testing with JDK 25.0.3 + Maven 3.9.16 (same versions you reported): Results
The 1GB failure is identical on both This PR makes no changes to If you're seeing a heap space issue specifically in
Could you share the exact error output and your |
|
The build problems remain with JDK 25.0.2 and 25.0.4: Build fails with: If Maven is run with -X -e the following stacktrace is shown: MAVEN_OPTS is set to '-Xmx384M -Xdebug' - once I set it to |
|
@gnodet thanks for your contribution! |
Summary
Make the RAT Maven plugin safe for parallel builds (
mvn -T4) by fixing four concurrency issues:DefaultLog(ThreadLocal) — Replace the shared staticLog instancewith aThreadLocal<Log>so each Maven reactor thread gets its own logger. Without this, thread A'sDefaultLog.setInstance(makeLog())overwrites thread B's logger, causing log output to be mislabeled or lost.DeprecationReporter(ThreadLocal) — Same pattern: replace the shared staticConsumer<Option> consumerwith aThreadLocal. The generatedBaseRatMojoconstructor callssetDeprecationReporter()per-module, creating the same overwrite race asDefaultLog.OptionCollection.parseCommands()(synchronized) — Addsynchronizedbecause this method uses shared mutable state that is not thread-safe:Argenum'sOptionGroup.selectedfield is mutated byDefaultParser.parse()Converters.FILE_CONVERTER.workingDirectoryis overwritten during argument processingWhen two threads parse concurrently, thread A's
--input-excludesetting gets overwritten by thread B's parse, causing exclusions to be silently skipped. Thesynchronizedserializes only the config parsing phase — actual file scanning and license checking still runs in parallel.SPDXMatcherFactory(ThreadLocal per-thread factory) — The singletonINSTANCEshareslastMatch/checkedstate and a staticMATCHER_MAPacross threads. When two threads scan different documents concurrently, both see both sets of SPDX IDs.MATCHER_MAPmoved fromstaticto instance field — each factory has its own matcher registrySpdxBuildernow uses aThreadLocal<SPDXMatcherFactory>so each Maven reactor thread gets its own factory with isolated match stateINSTANCEkept for backward compatibility (single-threaded use / tests)Testing
RAT-573integration test: multi-module project (3 modules + parent) run with-T4, each module containing asrc.aptfile that must be excluded via**/src.aptpattern-T4(exclusion silently skipped → RAT reports unlicensed files)Known remaining statics (out of scope)
MatcherBuilderTracker— tracks custom matchers via static stateThese are more deeply embedded and would require more invasive changes.
Test plan
mvn clean installpasses locally-T4🤖 Generated with Claude Code