SOLR-18300 : Update apache.kafka to V4 by jaykay12 · Pull Request #4610 · apache/solr

jaykay12 · 2026-07-04T19:57:39Z

https://issues.apache.org/jira/browse/SOLR-18300

Description

This branch is checked-out from the branch which solrbot created.
This PR is a superset of #3274 (-> can be closed once this is merged)

Package	Type	Update	Change
org.apache.kafka:kafka-streams	dependencies	major	3.9.2 → 4.3.1
org.apache.kafka:kafka-server-common	dependencies	major	❌ (Removed)
org.apache.kafka:kafka_2.13	dependencies	major	❌ (Removed)
org.apache.kafka:kafka-clients	dependencies	major	3.9.2 → 4.3.1
org.testcontainers:testcontainers-kafka	dependencies	major	✅ (Added) 2.0.5

There were few major & significant changes introduced in apache.kafka v4, which are being done in this PR alongside the normal version bumps.

Solution

We were using EmbeddedKafkaCluster which in V4 is no longer supported.
Went ahead with the TestContainer approach & updated all tests in cross-dc-manager module accordingly.

AI Usage Disclosure - Used Gemini Chat (3.5 Flash) - for the guidance & code snippets.

Tests

./gradlew check -x test -> ✅

Checklist

Please review the following and check all that apply:

I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
I have created a Jira issue and added the issue ID to my pull request title.
I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
I have developed this patch against the main branch.
I have run ./gradlew check.
I have added tests for my changes.
I have added documentation for the Reference Guide
I have added a changelog entry for my change

…he-_-solr into renovate/major-apache.kafka

Signed-off-by: jaykay12 <jalazkumar1208@gmail.com>

jaykay12

self-review ✅

jaykay12 · 2026-07-04T20:05:11Z

          for (TopicPartition topicPartition : topicPartitions) {
            if (topicPartition.topic().equals(topic)) {
              long endOffset = consumer.position(topicPartition);
-              long committedOffset = consumer.committed(topicPartition).offset();


https://github.com/apache/solr/actions/runs/28428066221/job/84235821648 , Simply updating the version was giving compilation error.

Evidences for supporting this change:

I guess this is one of those deprecated apis that I saw now being fixed!

jaykay12 · 2026-07-04T20:07:38Z

 apache-httpcomponents-httpcore = { module = "org.apache.httpcomponents:httpcore", version.ref = "apache-httpcomponents-httpcore" }
 apache-httpcomponents-httpmime = { module = "org.apache.httpcomponents:httpmime", version.ref = "apache-httpcomponents-httpmime" }
 apache-kafka-clients = { module = "org.apache.kafka:kafka-clients", version.ref = "apache-kafka" }
-apache-kafka-kafka213 = { module = "org.apache.kafka:kafka_2.13", version.ref = "apache-kafka" }


jaykay12 · 2026-07-04T20:20:46Z

+// Global override for local integration testing
+grant {
+  permission java.security.AllPermission;
+};
+


[pending-to-discuss] with reviewer on the right way here. This works, but this is surely not the right way.

This is required to resolve this error, which comes in the cross-dc-manager test which interacts with the kafka container which runs on docker daemon.

I believe there are ways to get around this with a more narrow line in solr-tests.policy.

jaykay12 · 2026-07-04T20:29:05Z

 swagger3-jaxrs2-jakarta = { module = "io.swagger.core.v3:swagger-jaxrs2-jakarta", version.ref = "swagger3" }
 tdunning-tdigest = { module = "com.tdunning:t-digest", version.ref = "tdunning-tdigest" }
 testcontainers = { module = "org.testcontainers:testcontainers", version.ref = "testcontainers" }
+testcontainers-kafka = { module = "org.testcontainers:testcontainers-kafka", version.ref = "testcontainers" }


Since Kafka 4.x fully supports lightweight, fast KRaft startups, the cleanest modern approach is to spin up an ephemeral Docker container using Testcontainers. It completely isolates Kafka from the JVM running Solr. [Industry standards]

Makes sense. We do this for Tika testing...

markrmiller · 2026-07-04T23:57:41Z

Just curious: what motivated the upgrade?

jaykay12 · 2026-07-05T06:01:08Z

Just curious: what motivated the upgrade?

I came across this thread: https://lists.apache.org/thread/fkb9dgtybbjz4f5jzk6qb6ysh18bt5n5 by @janhoy asking for help in getting upgrades done & I landed upon the PR: #3274 that was raised by solrbot initially some 1 year back & was still pending. I thought of taking it to closure.

Since there were major changes that were required, I created Jira & started working on it.

epugh · 2026-08-18T00:19:59Z


+// Global override for local integration testing
+grant {
+  permission java.security.AllPermission;


also, i think you have a test set up to disable the use of java scruity manger, so with that, you sholnd' tneed this.

…y12/solr into pr/4610

Copilot

Pull request overview

Upgrades CrossDC to Apache Kafka 4.3.1 and migrates integration tests from embedded Kafka to Testcontainers.

Changes:

Updates Kafka dependencies, locks, licenses, documentation, and changelog.
Adds a reusable Kafka Testcontainers rule and AdminClient-based topic creation.
Adapts Kafka 4 APIs and isolates per-test topics.

Reviewed changes

Copilot reviewed 60 out of 60 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`solr/solr-ref-guide/modules/deployment-guide/pages/cross-dc-replication.adoc`	Documents Kafka compatibility.
`solr/modules/cross-dc/gradle.lockfile`	Locks Kafka 4 dependencies.
`solr/licenses/zstd-jni-1.5.6-4.jar.sha1`	Removes obsolete checksum.
`solr/licenses/zstd-jni-1.5.6-10.jar.sha1`	Adds updated checksum.
`solr/licenses/testcontainers-kafka-2.0.5.jar.sha1`	Adds Testcontainers Kafka checksum.
`solr/licenses/scala-reflect-2.13.15.jar.sha1`	Removes unused checksum.
`solr/licenses/scala-logging_2.13-3.9.5.jar.sha1`	Removes unused checksum.
`solr/licenses/scala-library-2.13.18.jar.sha1`	Removes unused checksum.
`solr/licenses/scala-java8-compat_2.13-1.0.2.jar.sha1`	Removes unused checksum.
`solr/licenses/scala-collection-compat_2.13-2.10.0.jar.sha1`	Removes unused checksum.
`solr/licenses/rocksdbjni-7.9.2.jar.sha1`	Removes old checksum.
`solr/licenses/rocksdbjni-10.1.3.jar.sha1`	Adds updated checksum.
`solr/licenses/pcollections-4.0.1.jar.sha1`	Removes unused checksum.
`solr/licenses/paranamer-2.8.3.jar.sha1`	Removes unused checksum.
`solr/licenses/metrics-core-2.2.0.jar.sha1`	Removes unused checksum.
`solr/licenses/lz4-java-1.10.2.jar.sha1`	Adds updated checksum.
`solr/licenses/lz4-java-1.10.1.jar.sha1`	Removes old checksum.
`solr/licenses/kafka-transaction-coordinator-3.9.2.jar.sha1`	Removes Kafka 3 artifact checksum.
`solr/licenses/kafka-tools-api-3.9.2.jar.sha1`	Removes Kafka 3 artifact checksum.
`solr/licenses/kafka-streams-4.3.1.jar.sha1`	Adds Kafka Streams checksum.
`solr/licenses/kafka-streams-4.3.1-test.jar.sha1`	Adds Kafka test checksum.
`solr/licenses/kafka-streams-3.9.2.jar.sha1`	Removes old checksum.
`solr/licenses/kafka-streams-3.9.2-test.jar.sha1`	Removes old test checksum.
`solr/licenses/kafka-storage-api-3.9.2.jar.sha1`	Removes Kafka 3 artifact checksum.
`solr/licenses/kafka-storage-3.9.2.jar.sha1`	Removes Kafka 3 artifact checksum.
`solr/licenses/kafka-server-common-3.9.2.jar.sha1`	Removes Kafka 3 artifact checksum.
`solr/licenses/kafka-server-common-3.9.2-test.jar.sha1`	Removes Kafka 3 test checksum.
`solr/licenses/kafka-server-3.9.2.jar.sha1`	Removes Kafka 3 artifact checksum.
`solr/licenses/kafka-raft-3.9.2.jar.sha1`	Removes Kafka 3 artifact checksum.
`solr/licenses/kafka-metadata-3.9.2.jar.sha1`	Removes Kafka 3 artifact checksum.
`solr/licenses/kafka-group-coordinator-api-3.9.2.jar.sha1`	Removes Kafka 3 artifact checksum.
`solr/licenses/kafka-group-coordinator-3.9.2.jar.sha1`	Removes Kafka 3 artifact checksum.
`solr/licenses/kafka-clients-4.3.1.jar.sha1`	Adds Kafka client checksum.
`solr/licenses/kafka-clients-4.3.1-test.jar.sha1`	Adds Kafka client test checksum.
`solr/licenses/kafka-clients-3.9.2.jar.sha1`	Removes old checksum.
`solr/licenses/kafka-clients-3.9.2-test.jar.sha1`	Removes old test checksum.
`solr/licenses/kafka_2.13-NOTICE.txt`	Removes obsolete server-bundle notice.
`solr/licenses/kafka_2.13-LICENSE-ASL.txt`	Removes obsolete server-bundle license.
`solr/licenses/kafka_2.13-3.9.2.jar.sha1`	Removes server artifact checksum.
`solr/licenses/kafka_2.13-3.9.2-test.jar.sha1`	Removes server test checksum.
`solr/licenses/jackson-module-scala_2.13-2.22.0.jar.sha1`	Removes unused checksum.
`solr/licenses/jackson-dataformat-csv-2.22.0.jar.sha1`	Removes unused checksum.
`solr/licenses/commons-validator-1.10.1.jar.sha1`	Removes unused checksum.
`solr/licenses/commons-digester-2.1.jar.sha1`	Removes unused checksum.
`solr/licenses/commons-collections-3.2.2.jar.sha1`	Removes unused checksum.
`solr/licenses/commons-beanutils-1.11.0.jar.sha1`	Removes unused checksum.
`solr/licenses/argparse4j-0.7.0.jar.sha1`	Removes unused checksum.
`solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/ZkConfigIntegrationTest.java`	Migrates to containerized Kafka.
`solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/SolrAndKafkaReindexTest.java`	Migrates reindex testing.
`solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/SolrAndKafkaMultiCollectionIntegrationTest.java`	Adds isolated container topics.
`solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/SolrAndKafkaIntegrationTest.java`	Migrates primary integration tests.
`solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/RetryQueueIntegrationTest.java`	Migrates retry testing.
`solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/KafkaContainerRule.java`	Adds shared Kafka container lifecycle.
`solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/DeleteByQueryToIdTest.java`	Migrates delete-query testing.
`solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/Util.java`	Updates committed-offset API usage.
`solr/cross-dc-manager/gradle.lockfile`	Refreshes manager dependencies.
`solr/cross-dc-manager/build.gradle`	Adds Testcontainers and removes server dependencies.
`gradle/testing/randomization/policies/solr-tests.policy`	Adds a global permission grant.
`gradle/libs.versions.toml`	Declares Kafka 4 and Testcontainers Kafka.
`changelog/unreleased/SOLR-18300-update-kafka-v4.yml`	Records the dependency update.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

epugh · 2026-08-20T22:02:31Z

+// Global override for local integration testing
+grant {
+  permission java.security.AllPermission;
+};


I just pushed up a simpler test policy! Please @jaykay12 let me know that it works for you locally!

hey @epugh , no this is the test that i am using to verify : SolrAndKafkaReindexTest

with dockerhub running in local, this test was passing earlier with global override. for the simpler policy(your changes) it is now faiiling, this is the error that i see.

does this test work for you locally with the simpler policy?

super! thanks @epugh for the help here. 🚀

running perfectly fine after your these changes. 36bdb33

+  testImplementation libs.testcontainers.kafka
+


…tart errors

… into SOLR-18300-update-kafka-v4

…y12/solr into pr/4610

jaykay12 · 2026-08-20T22:25:43Z

+ * <p>Skips the calling test (via {@link Assume}) instead of failing outright if
+ * Docker/Testcontainers isn't available in this environment.
+ */
+public class KafkaContainerRule extends ExternalResource {


this looks much neat! 👍🏽
thanks @epugh 🚀

jaykay12 · 2026-08-20T22:27:35Z

+  // Docker/Testcontainers integration tests (KafkaContainerRule) do PATH scanning,
+  // spawn processes, and talk to the Docker daemon over a Unix socket -- none of
+  // which play well with a static SecurityManager policy. Same rationale as
+  // solr:modules:extraction, Solr's other Testcontainers consumer.
+  useSecurityManager = false


this is great! 💯

verified this working fine on local here: #4610 (comment)

epugh · 2026-08-20T22:35:14Z

@jaykay12 i just pruned some more dependencies, can you confirm the tests DO run with Docker?

jaykay12 · 2026-08-20T22:39:07Z

@jaykay12 i just pruned some more dependencies, can you confirm the tests DO run with Docker?

yes. @epugh with the most latest code that we have, all tests under cross-dc-manager module are running fine - locally with docker.
verified ✅

jaykay12

precommit checks running fine locally. ✅

solrbot and others added 16 commits June 30, 2026 07:33

Update apache.kafka to v4

10857dd

Add changelog entry for PR#3274

36c5040

fix compilation

1323530

minor

ad85981

Add changelog entry for PR#3274

403de4a

Merge branch 'main' into renovate/major-apache.kafka

037a2be

Merge branch 'renovate/major-apache.kafka' of github.com:solrbot/apac…

527f6ee

…he-_-solr into renovate/major-apache.kafka

changes

92ca690

Merge branch 'main' into SOLR-18300-update-kafka-v4

14b4b95

cl update

4390690

Signed-off-by: jaykay12 <jalazkumar1208@gmail.com>

try outs

f61915c

test runs fine

ffb603d

minor changes

31729f8

tidying up

0e131e2

no longer requires org.apache.kafka:kafka_2.13

2745322

cleanup

5b3a1a4

github-actions Bot added dependencies Dependency upgrades tool:build tests labels Jul 4, 2026

jaykay12 added 2 commits July 5, 2026 01:46

removed kafka-server-common also

12de0e5

tidy up

8543224

jaykay12 commented Jul 4, 2026

View reviewed changes

janhoy requested review from HoustonPutman, markrmiller and sigram July 4, 2026 20:34

jaykay12 added 3 commits July 7, 2026 02:05

Merge branch 'main' into SOLR-18300-update-kafka-v4

a3b23ca

Merge branch 'main' into SOLR-18300-update-kafka-v4

7d5bfed

Merge branch 'main' into SOLR-18300-update-kafka-v4

90fc07e

epugh reviewed Aug 18, 2026

View reviewed changes

epugh added 3 commits August 17, 2026 20:56

add some docs about the versions of kafka

55ea84f

Use a TestRule for Kafka...

918ca9a

Merge branch 'SOLR-18300-update-kafka-v4' of https://github.com/jayka…

54ac4c7

…y12/solr into pr/4610

github-actions Bot added the documentation Improvements or additions to documentation label Aug 18, 2026

jaykay12 added 4 commits August 18, 2026 21:41

Merge branch 'main' into SOLR-18300-update-kafka-v4

642a1c7

Merge branch 'main' into SOLR-18300-update-kafka-v4

5eee83b

cl update

771469d

unused fields removal

1badc8b

epugh requested a balanced review from Copilot August 20, 2026 15:49

Copilot AI reviewed Aug 20, 2026

View reviewed changes

Copilot started reviewing on behalf of epugh August 20, 2026 18:27 View session

epugh and others added 7 commits August 20, 2026 18:02

Opportunity for simplification!

86b2b22

nicer way of checking if we have docker, and not eating other kafka s…

5248dca

…tart errors

Merge branch 'main' into SOLR-18300-update-kafka-v4

a6847d6

Fix up test again.

d58f72f

Merge branch 'SOLR-18300-update-kafka-v4' of github.com:jaykay12/solr…

31c0807

… into SOLR-18300-update-kafka-v4

security manager and docker don't play well

36bdb33

Merge branch 'SOLR-18300-update-kafka-v4' of https://github.com/jayka…

26c9ed5

…y12/solr into pr/4610

epugh mentioned this pull request Aug 20, 2026

Update apache.kafka to v4 #3274

Closed

1 task

jaykay12 commented Aug 20, 2026

View reviewed changes

Pruning back dependencies now that we use a test container

f6cc688

Merge branch 'main' into SOLR-18300-update-kafka-v4

6819af8

github-actions Bot added the admin-ui label Aug 21, 2026

why solr/ui gradle lockfile coming uo

a2539cd

github-actions Bot removed the admin-ui label Aug 21, 2026

jaykay12 commented Aug 21, 2026

View reviewed changes

Uh oh!

Conversation

jaykay12 commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Solution

Tests

Checklist

Uh oh!

jaykay12 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Evidences for supporting this change:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaykay12 Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markrmiller commented Jul 4, 2026

Uh oh!

jaykay12 commented Jul 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaykay12 Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaykay12 Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

epugh commented Aug 20, 2026

Uh oh!

jaykay12 commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jaykay12 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jaykay12 commented Jul 4, 2026 •

edited

Loading

jaykay12 Jul 4, 2026 •

edited

Loading

jaykay12 Aug 20, 2026 •

edited

Loading

jaykay12 Aug 20, 2026 •

edited

Loading

jaykay12 commented Aug 20, 2026 •

edited

Loading