Conversation
… and adding a new cache strategy based on expected read count (apache#24444) (cherry picked from commit cf3d7d1)
…val queue and job (apache#24363) (cherry picked from commit 8ce67b9)
(cherry picked from commit 7ab68b1)
dao-jun
force-pushed
the
dev/pick_batchread_and_cache
branch
2 times, most recently
from
September 20, 2026 17:10
74481c0 to
7b885b2
Compare
…pache#24623) (cherry picked from commit 490ba0c)
…er (apache#25988) (cherry picked from commit fbcfacf)
…tion (apache#26511) (cherry picked from commit 72632e5)
…ck since it's already covered by putIfAbsent (apache#24699) (cherry picked from commit 669ab61)
…ng the parsed instance in the broker cache (apache#24682) (cherry picked from commit 53bbd4a)
…nd improve caching (apache#24836) (cherry picked from commit 9737d03)
…ntryWrapper (apache#25987) (cherry picked from commit 98bac65)
…ad of under the wrapper write lock (apache#26463) (cherry picked from commit ce60798)
…he cache keeps (apache#26495) (cherry picked from commit 9e0f694)
…pache#26526) (cherry picked from commit 7bcae18)
…ction (apache#26529) (cherry picked from commit a3de5dd)
(cherry picked from commit 78688dd)
…che#26523) (cherry picked from commit 30b975a)
…4.2.17 to enable the BK batch read API Dependency enablement for apache#25280 (BK batch read): switch from the com.ascentstream fork artifacts (bookkeeper 4.17.4.0) to Apache org.apache.bookkeeper/org.apache.distributedlog 4.18.0, align Netty to 4.2.17 with io_uring promoted out of the incubator coordinates, raise pulsar.client.compiler.release to 17 (BK 4.18.0 jars are Java 17 bytecode in the client dependency chain), and bump rocksdb to 9.9.3. Carries the API-migration ripple: PulsarFlowControlHandler (netty 4.2), EventLoopUtil, Recycler-based test adaptations, pulsar-metadata BK 4.18 ZK/underreplication API changes, offload LedgerMetadataFormat migration, BKStateStoreProviderImpl, BookKeeperClusterTestCase loopback advertisement, and the module pom/shading updates. Clear the client endpoint identification algorithm in SecurityUtility's client SslContext builders: Netty 4.2's SslContextBuilder.forClient() defaults it to "HTTPS", which silently re-enables hostname verification when tlsHostnameVerificationEnable=false (the per-engine HTTPS overlay in configureSSLHandler still applies it when the flag is on). The algorithm is final at context-build time on the OpenSSL backend, so the default must be countered at the builder (mirrors upstream's TlsContexts.buildNettyClientContext in PIP-478). Extracted from the batch-read WIP (8d38f112a8 + f2b2d43759), excluding the batch-read feature changes that arrive with apache#25280.
…ator type and metrics (apache#26595) (cherry picked from commit e1a1581)
apache#26593) (cherry picked from commit e7cbb33)
…pache#26597) (cherry picked from commit ca58c20)
Co-authored-by: Lari Hotari <lhotari@apache.org> Co-authored-by: Matteo Merli <mmerli@apache.org> (cherry picked from commit f369915)
…apache#26603) (cherry picked from commit 0677be2)
apache#26604) (cherry picked from commit 14e0ede)
Co-authored-by: Matteo Merli <mmerli@apache.org> (cherry picked from commit 1759f73)
…ion settings at startup (apache#26627) (cherry picked from commit bed06a0)
…pache#26628) (cherry picked from commit 3efae19)
…pache#26514) Port of the upstream netty 4.2.17.Final -> 4.2.18.Final upgrade: - root pom netty.version bump; netty-tcnative follows via the netty-bom (2.0.81.Final -> 2.0.84.Final) - LICENSE.bin.txt (server + shell) netty/tcnative version lines realigned with the bundled jars - ProxyProtocolTest.testSniProxyProtocol: use unresolvable-broker-address in the broker service URL so it matches the certificate SAN, as required by the now default-on hostname verification (same change as upstream PIP-478 core migration apache#26282) Fixes the TLS handshake failures ("Connection already closed") seen with netty 4.2.17 in ProxyServiceTlsStarterTest / ProxyProtocolTest. (cherry picked from commit 7ddeac8e2339f3bd2b9e7c1d3694ec7c1a820bdc)
BookKeeper 4.18 jars are Java 17 bytecode (verified: class file major 61, vs major 52 for the previous ascentstream 4.17.4.0 fork) and they are part of the client dependency chain, so the shaded client artifacts cannot run on Java 8/11 anymore. apache/pulsar made the same change when adopting PIP-421 "Require Java 17 as the minimum for Pulsar Java client SDK" (apache#24475): the SHADE_RUN matrix keeps only JDK 17+ entries.
dao-jun
force-pushed
the
dev/pick_batchread_and_cache
branch
from
September 20, 2026 19:01
7b885b2 to
eea0351
Compare
…perties stub to the 6-arg internalAsyncMarkDelete The apache#25796 backport (39f0def) landed both the production fix and its test, but the test stubs the 5-arg internalAsyncMarkDelete overload: that is the arity master's asyncMarkDelete chain reaches, while on this branch the chain routes through the 6-arg overload (adding propagatePersistFailure/durableFallback) that NonDurableCursorImpl overrides. The stub therefore never intercepted the advance-path call and the entry latch deterministically timed out, leaving the test red since the backport landed. Stub the 6-arg overload instead; the null-properties contract asserted by apache#25796 is unchanged.
…pId com.ascentstream
The fork renamed the maven groupId from org.apache.pulsar to
com.ascentstream.pulsar, but the CI infrastructure still referenced the
upstream path ~/.m2/repository/org/apache/pulsar:
- the actions/cache exclusion lists kept caching com.ascentstream
SNAPSHOT reactor artifacts under a key derived only from
hashFiles('**/pom.xml'), so java-only fixes never changed the key and
unit jobs kept testing whatever jars were cached when the key was
first saved (observed: the netty-4.2 TLS builder fix and the
PulsarMockBookKeeper properties fix were amended into the branch
without touching any pom; CI restored a pre-fix cache entry and the
already-fixed AuthenticationTlsHostnameVerificationTest[false],
SchemaTest.testConcurrentCreateSchemaNoOrphanLedger and
ProxyServiceTlsStarterTest.testProducer kept failing with the exact
pre-fix signatures while every local configuration was green);
- the pulsar-maven-repository-binaries tar packed (and therefore the
unit jobs' restore delivered) zero fork artifacts, so the fresh build
could never override the stale cache;
- the coverage/deps tooling (snapshot_pulsar_maven_artifacts, classpath
greps, sourcefiles extraction) silently matched nothing.
Point all of these at com/ascentstream so snapshot reactor artifacts are
never served from the shared cache and unit jobs always consume the
fresh build results, matching the upstream design under the fork's
groupId.
Brings in the base's cursor reset/GC series (checkpoint ledger sweep, batch-index-ack dirty flagging, failed-flush GC race hardening) and the persistentUnackedRangesMaxEntrySize startup validation. The only textual conflict was the add/add on ManagedLedgerClientFactoryTest — resolved by keeping both sides' tests (apache#26627's cache-extension settings plumbing tests plus the base's checkpoint maxEntrySize validation tests). ManagedCursorTest 214/214 and the merged factory test 6/6 green on the merged tree.
…nces start with BookKeeper 4.18 BookKeeper 4.18's allocator classes log through slog, whose Log4j2 backend probes for disruptor with its own classloader and, when disruptor is visible, evaluates log4j's AsyncLogger (implements EventTranslatorVararg on log4j 2.26.x). In the function instance JVM the root classloader holds only java-instance.jar (log4j bundled, disruptor not) while the child classloader (/pulsar/lib) has disruptor via bookkeeper-server, so the probe enabled the async fast path and the parent-delegated AsyncLogger definition failed with NoClassDefFoundError: com/lmax/disruptor/EventTranslatorVararg. Every process-runtime function/source instance died at startup, failing the System jobs: PulsarMetadataStateStoreTest.testJavaWordCountFunction, PulsarFunctionsJavaProcessTest.testAutoSchemaFunctionTest, PulsarDebeziumSourcesTest.testDebeziumMongoDbSource and PulsarDebeziumOracleSourceTest.testDebeziumOracleDbSource. Bundling slog (zero transitive dependencies) into java-instance.jar - the same jar content upstream master reaches via PIP-467 (apache#25508, slog on pulsar-functions-api) - makes the probe run in the root classloader where disruptor is absent, keeping the fast path disabled. JavaInstanceDepsTest whitelists io/github/merlimat/slog, matching upstream. Verified A/B locally against the built distribution: with the previous jar the instance dies with the exact CI signature; with the bundled-slog jar the wordcount instance reaches running=true and processes messages (numReceived/numSuccessfullyProcessed > 0).
…r the fork groupId The previous org/apache/pulsar -> com/ascentstream path fix missed one path segment here: the fork's groupId com.ascentstream.pulsar resolves artifacts under ~/.m2/repository/com/ascentstream/pulsar/<artifactId>/..., so the perl expression captured the constant segment 'pulsar' instead of the artifactId. Every jar then greps multiple lines of the project mapping and the coverage --sourcefiles arguments are dropped. Anchor the pattern one segment deeper, matching the tar exclusion fixed in the same commit.
nodece
reviewed
Sep 23, 2026
Comment on lines
+189
to
+192
| <bookkeeper.groupId>org.apache.bookkeeper</bookkeeper.groupId> | ||
| <distributedlog.groupId>org.apache.distributedlog</distributedlog.groupId> | ||
| <bookkeeper.version>4.18.1</bookkeeper.version> | ||
| <distributedlog.version>4.18.0</distributedlog.version> |
Collaborator
There was a problem hiding this comment.
DO NOT change this package. If you need bk@4.18.0, we need to publish our own bk.
nodece
reviewed
Sep 23, 2026
| bytecode (and netty-transport-classes-io_uring is Java 9) and they are part of the client | ||
| dependency chain, so a Java 8 client bytecode guarantee is no longer possible | ||
| (aligned with Apache Pulsar master). --> | ||
| <pulsar.client.compiler.release>17</pulsar.client.compiler.release> |
Collaborator
There was a problem hiding this comment.
This change breaks client SDK compatibility.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #xyz
Main Issue: #xyz
PIP: #xyz
Motivation
Modifications
Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes