[yandrey321]

What changes were proposed in this pull request?

Problem. All Raft groups that two servers share replicate over one gRPC connection.

• GrpcServicesImpl keeps one GrpcServerProtocolClient per peer, shared by every division on the server.
• GrpcServerProtocolClient.appendEntries() always uses the stub on its main channel. Heartbeats move to a separate
  channel only when raft.grpc.server.heartbeat.channel is set.

Netty pins a connection to one event-loop thread. So with TLS, a single thread at each end encrypts, frames and
decrypts the entire log stream between the two servers, state machine data included. That caps replication between
two servers at about one core of TLS work.

Apache Ozone hits this limit. A datanode is one RaftServer hosting 20–40 RATIS/THREE groups (pipelines), and each
WriteChunk entry carries up to 4 MB of data. The measurements below come from a large 3-way write on a 12-datanode
cluster with gRPC TLS on (TestDFSIO 1000 × 1 GB):

• Each datanode replicated 3–3.8 GB/s, about one data connection per peer.
• The busiest connections carried 450–770 MB/s one way.
• Of 80 event-loop threads per datanode, only 10–19 were busy. The hottest averaged 43–72% of a core, with 5 s peaks
  of 78–96%.
• 79–87% of that thread's CPU samples were TLS record encryption and decryption.
• Turning TLS off raised the write from 7.6 to 8.6–8.8 GB/s.

Change.

• New key raft.grpc.server.log-appender.connections (K):
  • default 1, which keeps today's behaviour;
  • allowed range 1–1024;
  • getter and setter in GrpcConfigKeys.Server.
• GrpcServerProtocolClient:
  • keeps K appendEntries channels. The first is the existing main channel, so K = 1 opens no new connection.
  • Each group always uses the same channel: Math.floorMod(groupId.hashCode(), K).
  • close() and resetConnectBackoff() cover all K channels.
• GrpcLogAppender: passes its group id when it opens the appendEntries stream.
• GrpcServicesImpl: reads the key and passes it to each per-peer client.

Key decisions.

• A group is pinned to one connection. Raft ordering is per appender stream, so pinning keeps each group's
  entries in order on one connection while different groups spread over up to K event-loop threads at each end. No
  ordering or retry logic changes.
• Dedicated channels instead of the RATIS-2325 stub pool. The pool hands out stubs per call, which suits short
  readIndex calls. AppendEntries streams are long-lived and ordered, so they need a fixed channel per group.
• Heartbeats are unchanged. With heartbeat.channel on, they keep their own channel; otherwise they share the
  group's appendEntries stream as before.
• The main channel is reused as channel 0. K = 1 behaves exactly as before, and K > 1 adds K − 1 connections per
  peer.

Side effects.

• None with the default K = 1.
• With K > 1, each peer gets K − 1 extra connections. Each extra connection costs an HTTP/2 flow-control window, a TLS
  session and a handshake when it connects.
• K > 1 costs event-loop CPU. For the same throughput, event-loop CPU per MB rose about 15% at K = 2 and 26% at
  K = 4 in the microbenchmark below. The profile puts the increase mostly in waiting for socket events: the same bytes
  arrive over more connections and threads, so each wake-up carries less data. So K should be raised only where one
  connection's event-loop thread is the limit.
• Hashing many groups into a small K can leave the connections unevenly loaded.

Generated-by: Claude Code (Claude Opus 5.5)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-2787

How was this patch tested?

New unit tests. TestGrpcServerProtocolClientChannels:

• testOneConnectionUsesMainChannel: K = 1 has one appendEntries channel, which is the main channel, and every group
  maps to index 0.
• testGroupsSpreadOverConnections: with K = 4:
  • there are 4 distinct channels, the first being the main channel;
  • 100 random groups each map to an index in range, which is stable for equal group ids;
  • all 4 channels get used;
  • close() shuts down every channel.

Benchmark on a Linux server. The benchmark is a standalone harness, not part of this change.

• Topology: 3 Ratis gRPC servers, each in its own JVM, plus a client JVM, on one host over loopback.
• Host: 2 × Xeon Silver 4416+ (40 cores, 80 threads), JDK 17, with the JVM flags Ozone's launcher sets for Netty.
• Pinning: with taskset, all on NUMA node 0. The leader has 10 physical cores, and each follower and the client
  have 5.
• Workload:
  • mutual TLS, Ozone's flow-control window of 5 MB, the Raft log on tmpfs with fsync awaited;
  • 12 groups, all led by the same server, so all replication goes over its connections to the two others;
  • 1 MB entries carried as state machine data, like Ozone's WriteChunk, and not cached in the log;
  • 8 writes in flight per group, 100 timed entries per group after 50 for warm-up.
• Method: each arm in fresh JVMs, the order of K rotated per round. After every run, each follower's state machine
  data was checked against the leader's by checksum, entry by entry, and all matched.
• CPU: per server and per thread, from /proc, during the timed writes.

K	MB/s (3 rounds)	mean MB/s	leader cores	follower cores	hottest threads (%CPU)
1	953 / 979 / 890	941	9.2 of 20	3.5–4.2	all 4 threads of the 2 leader–follower connections at 82–89
2	1,133 / 1,086 / 1,181	1,133 (+20%)	16.2	5.9–6.1	one follower's receiving thread at 86–97
4	1,176 / 1,096 / 1,282	1,185 (+26%)	17.7	6.5–6.7	all at 79 or below

• K = 1 is limited by its connections, not by CPU. The leader uses less than half of its CPUs, while the threads
  of the two leader–follower connections, one sending on the leader and one receiving on each follower, run at
  82–89%.
• K = 2 removes most of that limit. Its slowest round is faster than the fastest K = 1 round. K = 4 adds about 5%
  more, and no thread is above 79%.
• The gain depends on spare CPU. With the leader on 5 physical cores instead, it ran out of CPU at about
  9.4 cores. Then K = 2 and K = 4 gained only 17–18% over K = 1, at 700, 820 and 827 MB/s.

leader cores	flow-control window	K = 1 (MB/s)	K = 2 (MB/s)	K = 4 (MB/s)
5	1 MB (Ratis default)	669	740	782
5	5 MB (Ozone)	700	820	827
10	1 MB	815		1,057
10	5 MB	941	1,133	1,185

The flow-control window also matters: a bigger window raises K = 1 as well, as shown above. At Ratis' 1 MB default,
K = 4 was +17% to +30% over K = 1, depending on the leader's cores.

[amaliujia]

Overall LGTM. Just want to lower the complexity of the assignment algorithm to have a balance between complexity and performance improvement then we good to merge.

[amaliujia]

LGTM

[amaliujia]

This is a testing only call, so the readlock has no performance impact for production.

[szetszwo]

@yandrey321 , thanks for working on this!

The idea of PR is good but the implementation is quite messy. We need to polish it. Filed RATIS-2804.

[szetszwo]

• Why appendEntriesIndex is in GrpcLogAppender but not together with the lists in GrpcServerProtocolClient?
• Also, it is really a bad name. It confuses with the log index in appendEntries.

[szetszwo]

It is better to call it getAppendEntriesIndexForTesting instead of using VisibleForTesting.

[amaliujia]

Nice naming actually.

[szetszwo]

We already has raft.grpc.server.log.appender.listener.factory. This new conf should use the same raft.grpc.server.log.appender prefix.

[szetszwo]

@yandrey321 , have you used any AI tool in this PR? If yes, please stay it in the beginning of the description and the commit message. This is ASF policy.

[yandrey321]

@yandrey321 , have you used any AI tool in this PR? If yes, please stay it in the beginning of the description and the commit message. This is ASF policy.

@szetszwo there is
"Generated-by: Claude Code (Claude Opus 5.5)"
in both commit message and PR description.

[szetszwo]

Generated-by: Claude Code (Claude Opus 5.5)

@yandrey321 , Just saw this line. Please put it in the beginning. It is hidden in the middle of a very long description. Thanks.

BTW, please shorten the description first. It saves everyone's time. I understand that the contributor can type a few words using AI to generate a long description and long PR with many comments/javadocs. It saves a lot of the contributor's time. However, the review is very hard and very time consuming.