Repository navigation
Extend type-profile pollution to the remaining map/set benchmarks (perf toolbox - benchmark) - #12298
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 49 commits intoOct 13, 2026
Conversation
This comment has been minimized.
This comment has been minimized.
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Mirrors StringIndex#contains for the static-arrays path, so callers of the embedded/parallel-array form don't need to spell out indexOf(...) >= 0 themselves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…hmark Reusing the same interned/cached-hash key instances for both building a structure and measuring hit lookups against it understates real hit cost -- String#equals's == fast path and String#hashCode's cached result pay nothing extra for a key already touched during setUp. The new hitFresh scenario looks up separate, never-touched String instances instead, alongside the existing hit (interned literal) and miss (already representative) scenarios. Also pulls in BenchmarkUtils#polluteHashDispatch so the shared hashCode/equals call sites are already megamorphic before the StringSet arms measure their own lookups, matching production where those call sites are hit by every hash-based structure in the JVM. Replaced the javadoc's stale, partial-run tables/findings with a full hit/hitFresh/miss re-run across all six structures together (Fork(5), Threads(8)) -- confirms hitFresh is the slowest of the three scenarios for every hash-based structure, not merely slower than hit as previously guessed from partial data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Recompute the slot index each iteration as (h + probes) & mask instead of carrying a separately-incremented cursor, so the loop has one canonical induction variable for the JIT to reason about.
…ains-only helpers populateTypeProfile() drives only contains(), so it works against both mutable and immutable collections (including the collection instance under test itself); populateTypeProfileMutable() keeps the old add()+contains() behavior for callers that need add() dispatch polluted too.
The instance wrapper reliably beats HashSet on hit (its actual design case) but not on miss/hitFresh, where both are bimodal across forks and the mean already sits at or below HashSet's steady figure. Point miss/fresh-key-heavy callers at EmbeddingSupport directly instead of assuming the wrapper is strictly better than a plain Set.
Object.hashCode()/equals() are JVM-wide shared call sites hit by every hash-based structure in the process; leaving them monomorphic for a single-key-type run understates real dispatch cost, same reasoning as ImmutableSetBenchmark. Rerun and update the javadoc results table with pollution in effect -- StringIndex's get win over HashMap/TagMap/MapN holds up unconditionally here, unlike the access-pattern-dependent Set case.
Apply the reviewer-suggested rewordings for BenchmarkUtils, ImmutableSetBenchmark, ImmutableMapBenchmark, and StringIndex: trim narration and restate a couple of claims (fresh-string hash caching, contains() contract) more precisely. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
polluteHashDispatch() only warmed the shared HashSet/HashMap call site; ConcurrentHashMap has its own, unrelated hashCode()/equals() dispatch sites and needs a dedicated scratch instance. Also add populateTypeProfileMap/populateTypeProfileMutableMap as the Map counterparts to the existing Collection-based helpers, for benchmarks that need finer-grained control than polluteHashDispatch() gives.
…nchmarks Applies the type-profile-pollution fix already used in ImmutableSetBenchmark/ImmutableMapBenchmark to the rest of the HashMap/HashSet/ConcurrentHashMap-comparing benchmarks in this module: SingleThreadedMapBenchmark, SingleThreadedSetBenchmark, ThreadSafeMapBenchmark, HashtableD1Benchmark, HashtableD2Benchmark, CaseInsensitiveMapBenchmark, and TagMapAccessBenchmark. Without this, each file's isolated single-key-type usage left the JDK collections' shared internal hashCode()/equals() dispatch call sites artificially monomorphic, understating their real per-call cost. FlatHashtableIteratorBenchmark is intentionally untouched -- it only exercises the project's own FlatHashtable/HashStrategy, not java.util.HashMap/HashSet/ConcurrentHashMap.
D1.Entry.hash/matches are call sites private to Hashtable.java, distinct from java.util.HashMap/HashSet's own dispatch sites, so pollution literally cannot reach them (not just unlikely to). Since JDK and machine were held constant in this rerun, attribute the observed drop to same-session run-to-run noise, not a JDK effect (that explanation applies elsewhere, where the JDK actually changed).
Same fix as HashtableD1Benchmark: D2.Entry.hash/matches are call sites private to Hashtable.java, structurally distinct from java.util.HashMap dispatch, so pollution cannot reach them regardless of key-type overlap; the observed drop with JDK/machine held constant is same-session noise, not a JDK effect.
The original table is Zulu 21; this rerun is JDK 8, whose C2 backend for Apple Silicon is far less mature than JDK 17+'s. That JDK gap explains a broad-based slowdown across every entry on its own, including flatHashtable/treeMap which don't touch java.util.HashMap dispatch — a more precise explanation than generic "machine variance."
Same fix as CaseInsensitiveMapBenchmark: the original table is Java 17, this rerun is JDK 8, and JDK 8's weaker Apple Silicon C2 codegen explains the across-the-board drop on its own. Distinguish this from HashtableD1Benchmark/HashtableD2Benchmark, where the JDK was held constant and the drop is same-session noise instead.
Controlled rerun (same machine, JDK swapped to Zulu 17.0.7) shows update_hashtable still wins decisively (~4.2x, down from ~14x on JDK 8) and iterate_hashtable flips from a wash to a ~4x win, while add_hashMap edges ahead slightly. Java 17's much better allocator/GC narrows but doesn't erase Hashtable's win on the allocation-heavy paths -- net takeaway: Hashtable is a strong HashMap substitute for simple counter/tally cases with a primitive value. Absolute numbers aren't comparable to the JDK 8 table: JMH auto-selects the cheap "compiler" Blackhole mode on 17 but not on 8, so within-run ratios are the only trustworthy comparison here.
Same controlled rerun as HashtableD1Benchmark. update_hashtable wins ~11.6x (down from ~26x on JDK 8); unlike JDK 8, Hashtable now also wins clearly on add (~1.8x) and iterate (~3.4x, up from a wash) -- Java 17's allocator/GC narrows the update margin but doesn't flip any operation in HashMap's favor for D2. Same Blackhole-mode caveat on absolute numbers as D1; within-run ratios are what's trustworthy here.
Several benchmark javadocs attributed cost to "megamorphic hashCode()/equals() dispatch" that could not have occurred. A megamorphic profile does not always imply virtual dispatch: an exact static type is a proof that outranks the profile, and these keys are statically exact, so C2 devirtualizes without consulting the polluted profile at all. - Add polluteComparatorDispatch so Comparator-based TreeSet/TreeMap/ ConcurrentSkipListMap sites are exercised alongside the natural-ordering ones, which do not share their call sites. - Document the devirtualization model once, in ImmutableMapBenchmark, instead of re-deriving it per file. - Drop the unsupported megamorphic-dispatch attributions in HashtableD1, HashtableD2 and TagMapAccess. String is final and Key2 is a final class built at the call site, so both sharpen to an exact type. - Rewrite TagMapAccess's rerun section against same-JDK Zulu 17 data rather than a cross-JDK comparison, removing that confound entirely. - Replace Objects.hash with HashingUtils.hash in Key2. Objects.hash is varargs and allocated an Object[] per lookup, handicapping the HashMap baseline by 24 B/op. - Fold measured -prof gc results into HashtableD1 and HashtableD2: 24 B/op for D1's boxed Long against zero for the hashtable, and 48 B/op for D2 (Long plus Key2), confirming escape analysis does not eliminate the composite key. HashtableD2's throughput tables still need refreshing from a multi-fork run, since the Key2 change alters that baseline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An identity match yields the same lookup result but never invokes equals(), so the Object decoy contributes a hashCode() receiver sample and no equals() one. The previous wording called the two indistinguishable, which is backwards for the profiling purpose this class exists to serve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both ThreadSafeMap benchmarks claimed that under pollution, megamorphic hashCode()/equals() dispatch dominated the synchronized HashMap cost "far more than lock contention" -- which also asserted that each benchmark failed to measure the thing it was built to measure. The arithmetic rules that out: get_synchronizedHashMap runs at 9.2 ops/us (D1) and 8.8 (D2), roughly 110 ns per op aggregate across eight threads. A megamorphic dispatch costs single-digit nanoseconds; a contended monitor with eight threads parking and unparking does not. Separately, the dispatch is not megamorphic in compiled code anyway: D1 keys are String, a final class, and D2's Key2 is a final class built at the call site, so C2 sharpens to an exact type and devirtualizes without consulting the profile. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Full Zulu 17 run across the four affected classes with -prof gc, so every table now carries allocation per op alongside throughput. - HashtableD2: the Objects.hash fix roughly doubled update_hashMap (196.7 to 463.0), so the update margin is ~5.4x rather than the ~11.6x claimed, and add is now a tie (1018.9 vs 1009.6) rather than a ~1.8x hashtable win. Both prior figures were artifacts of the varargs allocation in the old Key2 constructor. - ThreadSafeMapD1/D2: refreshed tables moved substantially, so the synchronized HashMap contention ratios are ~280-290x, not the ~150x computed from the older numbers a few commits ago. - ThreadSafeMapD2: document that Key2 is scalar-replaced in every map arm (0 B/op), confirmed by LogCompilation showing eliminate_allocation once the never-taken computeIfAbsent branch is pruned as unstable_if. Flag that this depends on a one-sided branch profile the benchmark manufactures by populating in @setup; a production miss rate would restore the allocation, so those numbers are an upper bound. - HashtableD1: the collection count in prose was from the single-fork spot check and disagreed with its own table; now both read 852. Allocation figures decompose exactly, which is a useful property to keep: 24 B/op is one boxed Long, 32 is a HashMap.Node, 48 is Long plus Key2, and 56 is Key2 plus Node with no box because (long) i for i < 128 hits the Long.valueOf cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A one-sided branch profile lets C2 prune the untaken branch as unstable_if, and when that branch held the only store of an object, escape analysis then scalar-replaces an allocation production would keep. Measured here as get-then-guarded-computeIfAbsent, where pre-installing every key in @setup makes a composite key free; a real cache records population-phase misses in the same profile and allocates on every lookup. Also notes what this utility does not cover: branch profiles, and the ordering dependency from JMH running arms sequentially. Flags warmUpArms as the planned follow-on rather than adding unadopted API here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Corrects a claim added in the previous commit. I had written that arms run sequentially and the first shapes profiles the rest inherit; that is wrong. JMH forks a fresh JVM per benchmark method -- verified in our own run log, 29 benchmark methods and 29 VM invokers -- so no profile carries between arms. The real problem is isolation, not ordering, and it is the root cause of all three issues this class deals with. A fresh JVM per arm means C2 sees only that arm's key types, only that arm's branch outcomes, and only the classes that arm loads. So receiver profiles collapse, branch profiles go one-sided, and CHA gets an artificially thin hierarchy. JMH forks to eliminate cross-benchmark profile pollution, but profile pollution is the production condition. Restructured the class javadoc to lead with that and enumerate the three consequences, demoting Blackhole to the orthogonal note it is. Also records that a future warmUpArms should shuffle arm order with a fixed seed, since compilation can trigger part-way through a fixed round-robin and a regular pattern is unrealistically easy on the hardware branch predictor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
From /techdebt and /perf-review over the branch. The J12 benchmark-validity check added in this branch had two problems. It instructed reviewers to call BenchmarkUtils.polluteHashDispatch(), which this same branch made private -- the public entry point is warmUpHashDispatch( Blackhole). And it cited "synchronized HashMap collapsed ~70% once its dispatch was made megamorphic" as its evidence, which is the attribution this PR retracted: those keys are String, a final class, so C2 devirtualizes on an exact type and never consults the profile. The gap there is lock contention. Rewrote J12 around the actual discriminator -- whether the receiver type is statically deducible at the benchmark's own call site -- and noted explicitly that one key class per benchmark is normal, since a production Map<String, V> call site is monotyped too. Evidence is now FlatHashtableIteratorBenchmark's four-implementor strategy dispatch (37.9 to 18.1 M ops/s when poisoned) against the String-keyed maps that were unaffected. Downgraded to flag-as-measure, and corrected the fix to Level.Trial only, since the helper runs 50,000 allocation-heavy passes. Also removed two orphan comments in ImmutableSet/ImmutableMapBenchmark that described a single pollution call deleted earlier in the branch, and widened the boxed-constructor suppression to include "removal", which is the actual warning category for @deprecated(forRemoval = true). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine of the eleven benchmarks wire warmUpHashDispatch into a @State(Scope.Thread) setup, which JMH runs once per thread -- so at @threads(8) the pollution work was being done eight times. Receiver profiles live in the per-method MDO and are JVM-global, so one pass is sufficient however many threads ask for it; the other seven were pure waste, and the allocation they generate lands before measurement starts. Guarded with a CAS rather than restructuring the call sites, so it holds regardless of which state scope a caller uses and stays correct if someone later wires it into Level.Iteration. Note this changes setup behaviour for the nine Scope.Thread benchmarks, so their tables were produced under the old 8x pollution. TagMapAccessBenchmark and ThreadSafeMapCounterBenchmark were already Scope.Benchmark and are unaffected -- which also rules out 8x pollution as the explanation for TagMapAccess's insert_via_ledger drop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n churn Pollution now runs once per JVM rather than once per thread, so the nine Scope.Thread benchmarks needed re-measuring. Zulu 17, -prof gc, same machine and fork config as the previous run. The interesting result is a negative one. Three files had been carrying "warmUpHashDispatch allocates heavily before measurement, which can shift GC state for the whole trial" as an explicitly unmeasured candidate for otherwise unexplained drops. The guard produced an 8x reduction in that allocation with everything else held constant, and nothing moved: update_hashMap 686.0 to 689.6, update_hashtable 2770.7 to 2796.3, iterate_hashMap 19.83 to 19.84, get_concurrentHashMap 1830.7 to 1837.1, get_support 2730.0 to 2731.5. All inside their intervals. That candidate is now tested and ruled out rather than left hanging, and HashtableD1 carries the comparison with D2 pointing at it. HashtableD2's update margin lands at ~5.5x and its add arms remain a tie, now with the hashtable ahead on the means but overlapping intervals. One arm moved outside its interval -- D2 update_hashtable, 2492.7 to 2346.8 -- on the zero-allocation path, so a GC explanation runs the wrong way; recorded as run-to-run drift. Refreshed throughput only for the four classes whose config is comparable between the two runs. For CaseInsensitiveMap, ImmutableMap and SingleThreadedMap the existing tables were taken at a different fork count, so only the allocation figures are added -- bytes/op is deterministic and config-insensitive: - CaseInsensitiveMap: lookup_hashMap is the only allocating lookup arm at 25.5 B/op for the folded String, against zero elsewhere. The variance is real signal, since toLowerCase() returns this for already-lower-case input. - ImmutableMap: the forEach-beats-iterator finding now has a mechanism -- iterate_tagMap 40 B/op versus zero for iterate_tagMap_forEach. - SingleThreadedMap: clone_tagMap 208 B/op against clone_hashMap 488. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bric3
reviewed
Sep 28, 2026
Describe warmUpHashDispatch as broadening shared profiles rather than making them megamorphic (receiver counts still drive C2's inlining), drop the segment-based explanation for ConcurrentHashMap, and remove the false claim that ConcurrentSkipListMap lacks computeIfAbsent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The once-per-thread vs once-per-JVM comparison shows the amount of pollution work doesn't matter; it can't show whether a single pass affects the trial, since both runs include one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unseeded per-fork key draws gave each arm a different case and hit/miss mix. With a fixed sequence, HashMap's apparent ~28% lookup lead over FlatHashtable disappears (384.7 +/- 81.0 vs 402.6 +/- 11.9), and the folded-String allocation becomes deterministic. Default to @fork(5) to match the documented configuration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ls-map-set-pollution
…he JDK Replace the Java 8-to-17 allocator story with a controlled same-JDK comparison (Zulu 17, forced FULL_DONTINLINE Blackhole, -prof gc): disabling EliminateAutoBox changes nothing, while ParallelGC cuts update_hashMap ~4x, matching JDK 8's default-collector number. The mechanism is left unattributed since reported gc.time is similar. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous seeded run's hashMap/treeMap arms showed intra-fork swings consistent with external interference. A clean rerun on the merged code (with the CHA_DEFEAT decoys) restores HashMap's ~20-30% lookup lead over FlatHashtable; FlatHashtable's wide interval is one slow-compiling fork. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bric3
reviewed
Oct 13, 2026
The J12 rubric text said this PR measured FlatHashtableIteratorBenchmark, but that benchmark is not part of the PR; the 37.9 -> 18.1 M ops/s numbers came from an uncommitted experiment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BenchmarkUtils' class javadoc now keeps the fork, profile and escape-analysis constraints while limiting production claims to what the benchmarks show. HashtableD2Benchmark no longer says the remaining setup pass was ruled out: eight passes vs one does not test its effect. Co-authored-by: bric3 <bric3@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 13, 2026
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
dougqh/benchmarkutils-map-set-pollution
branch
October 13, 2026 15:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Makes this module's map/set JMH benchmarks more realistic about JIT dispatch, and records what that does and doesn't change.
BenchmarkUtils.warmUpHashDispatch(Blackhole): one public entry point, called from@Setup(Level.Trial), once per forked JVM. It drives several key classes through the shared JDK collection call sites:hashCode()/equals()inHashMap,HashSetandConcurrentHashMapcompareToinTreeMap,TreeSetandConcurrentSkipListMapMap.of'sMapNlookups on Java 10+Lookups use distinct-but-equal copies of the keys (
distinctEqualCopy), so an identity match can't skipequals(). The class Javadoc explains why this is needed (JMH forks give each arm an unrealistically clean profile), which of the three effects it addresses (receiver-type profiles and, partly, class-hierarchy analysis), and which it doesn't (one-sided branch profiles).Wired into
SingleThreadedMap,SingleThreadedSet,ThreadSafeMapD1/D2/Counter,HashtableD1/D2,CaseInsensitiveMap,TagMapAccess, and the earlierImmutableSet/ImmutableMapbenchmarks.Refreshed results in each benchmark's Javadoc from a Zulu 17
-prof gcrun (CaseInsensitiveMapuses 5 forks and seeded lookup keys, so every arm sees the same key mix).Perf-review rubric check J12 (
checks.md/guide.md): pollution is flag-as-measure, and only when the dispatch under test is actually profile-dependent.Motivation
#12294 made
ImmutableSet/ImmutableMapmore realistic by polluting dispatch. This extends that to the rest of the module, and in doing so pins down where pollution actually matters.The main finding is that pollution only matters when the receiver type isn't statically deducible at the benchmark's own call site. For a final key class like
String, or an exact allocated type, C2 devirtualizes on that proof and never consults the profile, so pollution is a no-op. One key type per benchmark is normal: a productionMap<String, V>call site is monotyped too. It matters for dispatch through an interface with several loaded implementors, such as hashing strategies.Several claims made while this PR was in progress were measured again and retracted:
HashtableD2's ~1.8x/~11.6x wins overHashMapwere artifacts ofObjects.hashvarargs allocation in the composite-key constructor.HashMapgap is lock contention, not megamorphic dispatch (its keys areString).ConcurrentHashMap"segment translation" explanation was wrong: CHM hasn't used segments since Java 8. The ~44% figure stays as a measurement with no attributed cause.HashtableD1's update ratio is attributed to the collector, not the JDK.Additional Notes
warmUpArmshelper for that is planned separately (Add BenchmarkUtils.warmUp to restore cross-arm branch profiles in benchmarks (perf toolkit - benchmarking tools) #12651), as agreed in review.FlatHashtableIteratorBenchmarkevidence (37.9 → 18.1 M ops/s when poisoned) is a one-off measurement, not a committed benchmark, and the rubric now says so.Test plan
./gradlew :internal-api:compileJmhJavaandspotlessCheck-prof gcon Zulu 17 for the changed benchmarks; results are in each file's JavadocString-keyed dispatch resolves statically🤖 Generated with Claude Code