perf(rdma): isolate SG writes and restore batch depth

[ketor]

Summary

• isolate multi-SGE device-direct traffic in a dedicated RDMA connection lane
• keep each SG QP at depth 1 while restoring configured batch fanout across independent QPs
• make HiCache rail affinity use the physical PCP rank instead of tp_rank, which is always 0 with DP attention
• add connector regressions for depth propagation and physical-rank rail selection

Problem

Multi-SGE CUDA writes shared the normal data connection pool. With batch depth greater than one, mlx5 could overlap writes from distinct registered CUDA ranges on one QP and report protection/completion errors. The HiCache connector worked around this by effectively running at depth one, which serialized batch GETs and left SSD/RDMA throughput underutilized.

The previous rail_affinity implementation also used tp_rank. Under DP attention every process has tp_rank=0, collapsing all workers onto the same rail.

Design

Lane::kSgData owns a separate idle pool. Its QPs negotiate depth one, but CacheFromMulti still fans a batch across independent connections according to the configured transport depth. This preserves the one-CUDA-range-per-QP invariant without serializing the whole batch.

When explicitly enabled, connector rail affinity now maps pcp_rank % rail_count and disables additional NUMA rail selection for that process.

Validation

Tested on one 8x B200 development node with eight active RDMA rails and two NVMe data devices. The inference container used unlimited memlock so the device pools could be registered once.

• targeted connector tests: PASS
• candidate server/client build: PASS
• 228k-token GLM-5.2 cache retrieval: 227,968 cached tokens
• full hot retrieval: 26.60 s at depth 1 -> 5.24 s at depth 4 (5.08x)
• full cold retrieval: 7.24 s at depth 4
• cold NVMe peaks: 2,184.73 and 2,204.58 MB/s, 85.8% and 87.8% utilization
• RDMA completion error delta after the fix: 0

Runtime prerequisite: deployments using explicit large memory-pool registration must set an adequate memlock limit (for example, container --ulimit memlock=-1:-1).