Add Python benchmark engine (async valkey-glide, redis-py, valkey-py)

[jamesx-improving]

Implements the Python engine (#11), at parity with the Java (reference), Ruby, and C# engines.

What

• New async engine under python/ (pip install -e ., python -m resp_bench), driving three async drivers, one client per connection on a single asyncio event loop:
  • valkey-glide-python — GLIDE async (import glide)
  • redis-py — redis.asyncio
  • valkey-py — valkey.asyncio (Valkey fork of redis-py)
  • recording — in-memory driver for server-free tests
• Consumes the shared driver/workload JSON and emits the exact NDJSON schema (metadata/phase/totals/metrics, unit:"us", uppercased command keys, HDR compressed base64).

Concurrency model

connections = N → N clients (one per connection — the client == connection invariant) driven by N worker coroutines via asyncio.gather, each awaiting one command at a time. This is the faithful async analogue of the Java/Ruby "one in-flight request per connection" model, keeping results comparable across engines. pipeline_depth > 1 is not yet implemented; a >1 request logs a warning and runs at depth 1.

Cross-engine parity

• JavaRandom LCG port (incl. int32-overflow rejection) — verified byte-identical to Java's java.util.Random (seed-0 anchor) and to the Ruby engine (seed 12345).
• sequential_int uses a per-phase shared counter (matches Java's forkForThread); uniform_rand seeded per worker seed+idx.
• Key formatting %0Nd honoring key_size_bytes; leaky-bucket rate limiter; HdrHistogram (1, 600_000_000, 3).

Harness

• Makefile python-build/test/run/clean/info targets.
• Registered the Python driver_ids in scripts/run_benchmark_matrix.py (DRIVER_ENGINE_MAP) and scripts/generate_graphs.py (DRIVER_LANGUAGE_MAP).
• Driver configs under configs/drivers/{default,high-throughput}/ + example-*-standalone.json.
• CI benchmark-python job (matrix over the three drivers) added to generate-graphs needs; drivers.json python list.
• Doc fix: corrected the HdrHistogram range in docs/ADDING_LANGUAGE.md from 3_600_000_000 (1 hour) to 600_000_000 (600s) — every engine uses 600s.

Tests

34 unit + integration tests (cd python && python -m pytest): parity anchors (JavaRandom, key generator), rate limiter, config loader, HDR encode/decode round-trip, NDJSON schema, and end-to-end runs via the recording driver (no server needed).

Notes

• valkey-glide driver_id is valkey-glide-python (not the bare valkey-glide, which is Java's) — matches the valkey-glide-ruby / valkey-glide-csharp convention.
• Live-server runs (glide/redis-py/valkey-py against a real Valkey) were not run in CI-less local dev; the recording-driver e2e covers the engine paths. GLIDE advanced/custom-CA TLS should be validated in a live run.

[jeremyprime]

Note that an earlier attempt (https://github.com/jduo/resp-bench/tree/python-engine) also added Python sync if we want to see how that was done and consider here.

[jamesx-improving]

Note that an earlier attempt (https://github.com/jduo/resp-bench/tree/python-engine) also added Python sync if we want to see how that was done and consider here.

Thanks — I went through that branch before opening this PR. Borrowed from it: the valkey-py driver, which is included here (hence three drivers in the title).

Skipped its sync mode deliberately: #11 calls for the async clients, and that path is ThreadPoolExecutor thread-per-connection (GIL-bound, same shape as Ruby), so it'd add a second non-comparable execution model. Happy to add it as a follow-up if you want sync-client numbers — it'd be additive, async staying the default.

[jeremyprime]

Note that an earlier attempt (https://github.com/jduo/resp-bench/tree/python-engine) also added Python sync if we want to see how that was done and consider here.

Thanks — I went through that branch before opening this PR. Borrowed from it: the valkey-py driver, which is included here (hence three drivers in the title).

Skipped its sync mode deliberately: #11 calls for the async clients, and that path is ThreadPoolExecutor thread-per-connection (GIL-bound, same shape as Ruby), so it'd add a second non-comparable execution model. Happy to add it as a follow-up if you want sync-client numbers — it'd be additive, async staying the default.

That's fine, just wanted to make sure we referenced the other implementation. We can add sync in the future as needed.

[Aryex]

Overall looks good. My main concerns that need to be resolves are that:

1. Need to ensure GLIDE client is shared across threads/connections. This is the recommended way to use GLIDE for the best performance.
2. In a similar vein, we need to ensure peer clients are configured correctly. Since there are many, I would suggest picking only 1, maybe 2, for this PR to start.

[jamesx-improving]

Overall looks good. My main concerns that need to be resolves are that:

1. Need to ensure GLIDE client is shared across threads/connections. This is the recommended way to use GLIDE for the best performance.
2. In a similar vein, we need to ensure peer clients are configured correctly. Since there are many, I would suggest picking only 1, maybe 2, for this PR to start.

Thanks — this was a genuinely useful review; the measured reproductions made every item easy to confirm. I reproduced all of them locally before changing anything, and all 12 are addressed in 4096bac.

On sharing the GLIDE client: I have kept one client per connection for now, because sharing a multiplexing client across workers is what ikolomi#11 explicitly declined — the reasoning there was that one-client-per-transport-connection is the cross-engine comparison baseline, and that transport sharing should be reached via pipelining instead. I do not think I should reverse that inside a language-engine PR. That said your point stands on its own merits: N separate GlideClients each with their own Tokio runtime is not how GLIDE is meant to be used, and it under-represents it. Your other comment also corrected my premise — only lettuce and redis-rb are actually 1:1 among the incumbents — which weakens the comparability argument. Could we settle it as a methodology question with @ikolomi on the original issue? An opt-in shared_client flag (default off, preserving the 1:1 baseline) would give both numbers, but it needs his agreement rather than just ours. The current choice and its rationale are documented in the meantime.

On peer drivers: agreed, and your own findings are the argument for it — the two peers were not configured equivalently. Dropped valkey-py from this PR; it is now GLIDE + redis-py only, with redis-py's protocol, retry policy, parser and command timeout all pinned explicitly rather than inherited, and those settings recorded in the metrics metadata. valkey-py can return as a follow-up once there is a pattern for holding peer configuration equivalent. @jeremyprime — this is the driver borrowed from jduo's branch, so flagging that it is deferred rather than dropped on the merits.

The four correctness bugs (worker cancellation, loop starvation, warmup, config validation) have tests; the suite is now 59 tests. Two of your findings led somewhere broader than the original report: the starvation hole also existed on the success path (any non-suspending driver let one connection monopolise a duration phase — [460010, 0, 0, 0] on the recording driver, now even), and the new ERROR status did not actually stop the matrix runner scoring a bad cell as ok, so the CLI now exits non-zero on any ERROR phase.

Given the severity of the starvation and validation issues I would rather you re-reviewed than merged on the earlier approval.

[jeremyprime]

Looks like we are following the conclusion of ikolomi#11 (i.e. client==connection, and use pipelining for shared transport). My only remaining concern I have is that Java/Ruby/C# all support pipelining and can set pipeline_depth>1, so I think we need the same pipeline support in the Python client.

[Aryex]

@jamesx-improving +1 on the pipeline_depth support

[yipin-chen]

I suggest to support both sync and async benchmark.

[jamesx-improving]

My only remaining concern I have is that Java/Ruby/C# all support pipelining and can set pipeline_depth>1, so I think we need the same pipeline support in the Python client.

Implemented in 6b3c8ab71e96b4480e9ff6051415e7cf73b9fa9c.

For connections = N, pipeline_depth = D: N clients, N×D worker coroutines, so each connection holds up to D requests in flight. A connection owns one key generator and command selector shared by its D slots, so raising depth changes how many requests are outstanding, not which keys a connection touches — same ownership as the C# engine.

One caveat worth knowing, because it isn't symmetric across drivers:

Driver	Mechanism	Sockets per client
valkey-glide-python	multiplexed over one socket	1
redis-py	one pooled connection per in-flight command	up to pipeline_depth

redis-py has no single-socket path to real depth — single_connection_client=True serialises behind a lock, and pipeline() batches D commands into one round trip with one shared latency, which measures something else. So its pool is capped at pipeline_depth, each driver records its mechanism as pipelining in the metrics metadata, and phase rows now carry pipeline_depth, sockets_per_client and total_sockets. That last part matters for the matrix, which sweeps connections as the x-axis: without it a 2-socket GLIDE point and an 8-socket redis-py point sit at the same x with nothing in the data saying so. Verified against a server — at connections=2, depth=8 redis-py holds 16 real connections and reports 16; GLIDE holds 2 and reports 2.

Two bugs this shook out, both relevant beyond Python:

• Lazy pool sockets landed inside the measured window. redis-py opens a pooled socket only on first use, so at depth D the other D−1 connected during the benchmark, charging TCP connect + HELLO to the first requests (p50 6759µs vs 1188µs at depth 8 on loopback). There's now an explicit prime() step after connect, independent of warmup_requests. Worth checking whether C# has the same gap — StackExchange.Redis multiplexes so probably not, but its warmup is serial too.
• Failed phases dragged down reported throughput. A failed phase still emits a row so it stays visible, but with requests=0, duration_ms=0 — and generate_graphs.py filtered only on phase id, averaging it in as 0 RPS (two good 50k runs + one failed cell reported 33k). Fixed in load_results. The same pattern exists at three sites in generate_interactive_graphs.py (:252, :473, :1064); I left those alone since this PR doesn't otherwise touch that file, but they want the same guard.

Also flagging that the premise isn't quite right, since it affects what "the same support" means: only C# actually pipelines today.

• Java: runPipelinedLoop exists, but every client returns an already-completed future — AsyncHelper.timed() runs the operation inline on the calling thread (AsyncHelper.java:51-60) and each driver lambda ends in .get() (e.g. ValkeyGlideBenchmarkClient.java:161-172). The fill loop does D sequential blocking round trips and anyOf().join() returns with everything already done, so there is never more than one request in flight. It also loses command attribution on failure (BenchmarkEngine.java:437).
• Ruby: pipeline_depth is parsed, stored on ClientSlot and logged, then never read by any request loop (benchmark_engine.rb:599-605 vs :302, :443, :472).
• C# is internally inconsistent: its GLIDE driver awaits (StringSetAsync) but StackExchangeRedisBenchmarkClient blocks via AsyncHelper.Timed(() => _db.StringSet(...)), the same inline trick as Java. So C# depth>1 is real for GLIDE and a no-op for StackExchange.Redis.

I matched C#-GLIDE, not Java. Happy to file those separately — an engine that can't pipeline should reject pipeline_depth > 1 rather than silently report depth-1 numbers labelled depth-N.

[jamesx-improving]

+1 on the pipeline_depth support

Thanks @Aryex — landed, details in the comment above.