Skip to content

TCP T1/T2: native-client benchmarks, the transport comparison, and a working comparison job - #597

Draft
alex-clickhouse wants to merge 10 commits into
tcp/epic-t3-examplesfrom
tcp/epic-t1-t2-benchmarks
Draft

TCP T1/T2: native-client benchmarks, the transport comparison, and a working comparison job#597
alex-clickhouse wants to merge 10 commits into
tcp/epic-t3-examplesfrom
tcp/epic-t1-t2-benchmarks

Conversation

@alex-clickhouse

@alex-clickhouse alex-clickhouse commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Epic T1/T2: benchmarks for the native client, the cross-transport comparison, and the PR comparison job that runs them.

Ratios below are time relative to HTTP, so lower is faster — the same convention as BenchmarkDotNet's own Ratio column.

What this adds

Native-protocol benchmarks (tcp-regression, 41 methods). Each class prices the tiers a caller chooses between, not one number per operation:

Class Compares
TcpSelectColumn 16 expressions, block tier, mirroring SelectColumn
TcpReadTiers block spans / typed indexer / boxed GetValue / object[] / POCO
TcpProjectedRead stored type / ReadAs<T> indexer / ReadAs<T>.Values
TcpCompositeRead composite interface / indexer / Values / row tier, over 5 composites
TcpStringRead IStringColumn.Offsets / GetBytes / decoded strings
TcpInsertShapes columnar / object[] / POCO
TcpArrayWriteShape dense offsets against one array per row
TcpConnectionCost warm ping / warm scalar / client-per-operation
TcpPoolConcurrency 64 queries serially against widths 1, 8, 32
TcpCompression none / lz4 / zstd, read and insert

Cross-transport benchmarks (cross, 14 methods): TransportRead, TransportReadPoco, TransportInsert, TransportLatency. Each pairs an HTTP arm with its native counterpart on identical data in one process, with the codec and the server's insert buffering equalised so that a ratio reports the transport and not a setting that lands on one side only.

A working PR comparison job. benchmark-compare.yml has been failing to compile since the Common assembly split: the outer dotnet run left ClickHouseDriverVersion unpinned, so it restored the default release from nuget.org and could not compile against any API added since. It now pins the version and source, publishes 9000, and defaults baseline_ref to the PR's own base branch — a tcp/** PR sits on a stack whose base is not main, so comparing against main cannot compile.

Category selection from the diff. A change under ClickHouse.Driver.Tcp/ runs the native set, ClickHouse.Driver/ the HTTP set, either Compression/ directory the codec sweep, and the harness itself runs everything. Both transports moving adds the cross table. A push to main runs the regression sets; the nightly build runs all of them.

TCP against HTTP

One ClickHouse 26.6.1.1193 container serving both 8123 and 9000, so both transports hit the same data and CPU budget. 3 warmup + 30 iterations × 2 launches, on 4 logical and 2 physical cores. Loopback, so this measures client and protocol work and none of the transfer saving that compression exists for. The read sections use 2M rows; the insert section uses 5M, for the reason given there.

Every time below is the fastest of 120 iterations: 30 × 2 launches, run twice. This host interferes in bursts, so an iteration can land at 4.8 s where its neighbours sit at 0.3 s. Means and medians do not survive that: the two runs put TcpPoco's insert median at 277 ms and 424 ms. Best-case iterations do survive, agreeing between runs to within 0.02% to 8% on every arm. Interference is additive, so the fastest iteration is the arm's own cost and the tail is the box. Allocation does not depend on this and is quoted as BenchmarkDotNet reports it.

Every arm below runs uncompressed on both sides. The transports do not default to the same
codec (HTTP gzip or ZSTD, native LZ4), so at their defaults the codec difference reports itself as a
transport difference, and over loopback a codec costs CPU while saving nothing. An earlier revision
of this PR ran the read arms at both clients' defaults; that inflated the native client's advantage,
and the corrected figures are below. What compression is actually worth is measured on a real
network further down.

Row reads — TransportRead

Http and TcpRows both consume every value; TcpBlocks is the columnar ceiling.

Shape HTTP TCP rows TCP blocks HTTP alloc TCP rows alloc TCP blocks alloc
1 × UInt64 52.3 ms 58.2 ms (1.11) 8.1 ms (0.16) 46,890 KB 109,397 KB (2.33) 22.1 KB
UInt64+String+Float64 179.8 ms 179.3 ms (1.00) 122.7 ms (0.68) 170,330 KB 264,111 KB (1.55) 76,606 KB (0.45)
3 × String 232.3 ms 326.2 ms (1.40) 416.1 ms (1.79) 232,048 KB 325,835 KB (1.40) 232,107 KB (1.00)

Equalising the codec removed the native row tier's read advantage. At the two clients' defaults an earlier revision of this PR measured TcpRows at 0.66 / 0.72 / 1.02; uncompressed it reads 1.11 / 1.00 / 1.40. HTTP had been paying ZSTD on the response body while the native side paid LZ4, and that gap was most of the earlier win. On this hardware the native row tier is not faster than the ADO reader.

The block tier's advantage is real and is where the value is: 0.16 on fixed-width data, at 22.1 KB against HTTP's 46,890 KB, because nothing is boxed and nothing is copied.

The native row tier allocates more than HTTP's reader — 2.33× on the narrow shape, 56 B/row against 24 B/row. HTTP's Read() boxes each value into one reused object[]; QueryAsync yields a fresh object[] per row, because a consumer is free to keep it. That is the row API's shape rather than the protocol's, and it is not fixable without changing the contract.

On string-heavy data both TCP tiers are slower than HTTP. The block arm calls .Values on the string columns, materialising 6M strings, and pays far worse GC for it (Gen1 34,000 and Gen2 4,000 per 1,000 ops against HTTP's zero) because three 2M-element arrays live for the whole block and get promoted, while HTTP's per-row boxes die in Gen0.

That is what IStringColumn is for. TcpStringRead reads the same strings for 20.6 ms and 12.09 KB against 37.8 ms and 19,543 KB: 0.54 on time, and 1/1,600 of the allocation. So TCP's read advantage on strings depends entirely on whether the caller decodes them.

POCO reads — TransportReadPoco

Arm Fastest Ratio Allocated Alloc ratio
HTTP QueryAsync<T> 170.5 ms 1.00 151.08 MB 1.00
TCP QueryAsync<T> 165.2 ms 0.97 151.11 MB 1.00
TCP blocks → POCO by hand 166.2 ms 0.97 151.11 MB 1.00

All three allocate within 1% of each other, and all three take the same time. 151.08 MB over 2M rows is 79 B/row, which is the POCO (40 B) plus one fresh string per row (~32 B) and essentially nothing else. The objects are the allocation, no transport can avoid them, and that shared floor leaves nothing for a transport to win. Hand-building POCOs from spans buys nothing over QueryAsync<T> for the same reason.

Hand-building from blocks is the worst of the three on GC pressure — Gen1 23,000 per 1,000 ops against QueryAsync<T>'s 1,000 — because the intermediate column arrays survive the block.

An earlier revision reported 0.70 here. That was the codec difference, not the transport. A caller who wants materially better than parity has to stop building objects — which is what the 0.16 on narrow blocks shows.

Latency — TransportLatency

500 sequential SELECT 1 on warm connections: HTTP 824.6 ms (1.65 ms/query, 6.98 MB) against TCP 590.6 ms (1.18 ms/query, 4.61 MB), ratio 0.72, allocation 0.66. TCP saves about 0.47 ms of fixed per-request cost.

Inserts — TransportInsert

5,000,000 rows, because a smaller insert measures the fixed cost of an insert as much as the serialization the arms compare. Fitting both sizes gives each arm 6 to 8 ms of fixed cost, which is a fifth of a 500k-row measurement, so the ratios there compress toward 1 (TcpColumnar reads 0.62 at 500k against 0.51 at 5M, TcpRows 1.33 against 1.35, TcpPoco 1.00 against 0.94). 5M costs about 1 GB of source rows held for the run, which is the reason not to go further.

The native arms send this insert as 100 wire blocks of 50,000 rows, which is MaxRowsPerBlock at its default, and convert rows to columns one block at a time.

Two things are turned off, because each lands on the transports unequally and would report itself as a protocol difference. async_insert = 0: a server with async inserts on holds an insert's response until its buffer flushes, waiting from async_insert_busy_timeout_min_ms (50 ms) upward, and the native arms never paid it. No request compression: the transports do not default to the same codec (HTTP ZSTD, native LZ4), and the codec costs more than the serialization the arms exist to compare. Over loopback there is no bandwidth to save and TcpCompression owns the codec axis, so nothing of value is dropped.

Arm Fastest Ratio ns/row Allocated
HttpRows 257.2 ms 1.00 50.2 29.7 KB
TcpRows 347.5 ms 1.35 68.0 519.2 KB
HttpPoco 259.7 ms 1.01 50.7 30.9 KB
TcpPoco 241.2 ms 0.94 46.6 518.9 KB
TcpColumnar 131.1 ms 0.51 24.9 519.0 KB
HttpRowsDefaultBatching 386.4 ms 1.50 479.6 KB

The ns/row column is the slope between the two sizes, so it excludes the fixed cost.

The native protocol's insert advantage needs the columnar shape to appear. TcpColumnar is the fastest thing here at 0.51, but hand the same client rows and it is behind HTTP at 1.35, even though the server reads its blocks for a third of the CPU that parsing RowBinary costs it. The whole difference is the row-to-column transpose, and the two sizes price it twice over: as totals, 347.5 − 131.1 = 216.4 ms for 15M values, or 14.4 ns per value; as slopes, 68.0 − 24.9 = 43.1 ns per row, which is the same 14.4 ns per value. HTTP's RowBinary is row-major, so it matches the input layout and encodes straight into the request body while sending; the native block format is column-major, so every column must be fully materialised before a byte of it goes out, and the transpose runs inside the schema => … callback, wedged between two round trips with nothing to overlap.

The POCO tier beats the object[] tier on the native side, 241.2 against 347.5 ms — 7.2 ns per value against 14.4 — because the compiled typed accessors read fields in place instead of unboxing scattered values. On HTTP the same swap is a wash on the clock (259.7 against 257.2 ms) but not in the client: a probe tagging each insert with a query_id puts the POCO path at 239 ms of client CPU against the object[] path's 297 ms. The saving is real and it lands in the overlap. Both HTTP paths serialise into the request body as it is being sent, so the client's encoding runs against the server's parse of what has already arrived, and on HTTP the server is the slower stage: it spends 150 to 180 ms of CPU on the identical 109 MiB of RowBinary either way and waits only 20 to 40 ms for the client. The native arms are the mirror image, with the server idle for 160 to 255 ms waiting, which is why the same swap moves their wall time by 97 ms.

The object[] penalty is also smaller on HTTP to begin with, 59 ms of CPU against the native side's 156 ms, and that is where the row-major layout helps: the client walks each row once in order, where the column-major transpose visits every row once per column and dereferences a separate box each time.

Allocation is per unit of framing, and the transports frame differently. A native insert allocates about 13 KB fixed plus 5.1 KB per wire block, which the two sizes pin down: 10 blocks cost 63 KB and 100 cost 519 KB. HTTP allocates about 9 KB per request, so one request costs 30 KB and the fifty of HttpRowsDefaultBatching cost 480 KB. Neither figure includes row handling, because the source objects are built once in GlobalSetup, and no arm triggers a single GC collection. Neither HTTP path boxes during the operation: the POCO path uses the typed Action<T, ExtendedBinaryWriter> writers, and InsertOptions.Format defaults to RowBinary, so the boxed getters serve only RowBinaryWithDefaults and failure diagnostics. (HttpPoco is the one arm whose allocation does not reproduce: 30.9 KB in one run against 358.9 KB in the other. Unexplained, and on the HTTP side.)

InsertOptions.BatchSize: the default 100,000 sends fifty requests for 5M rows, costing 386.4 against 257.2 ms — about 2.6 ms per extra request, which is the round trip and little else — plus 16× the allocation in per-request buffers. An earlier revision of this PR reported a 4× penalty here; that was fifty exposed async_insert waits and the claim is withdrawn.

Compression, and the columnar layout, on a real network

Loopback cannot answer whether a codec pays for itself, so this was measured against a ClickHouse Cloud service (26.4.1.2212): 500k rows, median of 3 after a warm-up, rows taken from ClickBench hits so the payload is not a compressor-friendly sequence. This is characterisation from a one-off probe, not a committed benchmark. It is also the one section not re-measured for this revision, because it needs that service; its arms run for 0.4 s to 30 s each on a path where transfer dominates client work by an order of magnitude, so nothing the client does moves them. The arms run sequentially over a WAN, so read differences under ~15% are noise.

Reads:

Shape HTTP HTTP compressed Native Native LZ4 Native ZSTD
10 × hits, mixed 30,203 ms 3,103 ms (0.10) 26,776 ms 5,863 ms (0.22) 5,294 ms (0.20)
10 × hits, numeric 2,332 ms 245 ms (0.10) 2,499 ms 397 ms (0.16) 506 ms (0.20)
3 × synthetic 2,004 ms 692 ms (0.35) 2,792 ms 1,303 ms (0.47) 1,398 ms (0.50)

Inserts:

Shape HTTP HTTP gzip HTTP ZSTD Native Native LZ4 Native ZSTD
10 × hits, mixed 19,279 ms 3,772 ms (0.20) 3,059 ms (0.16) 16,847 ms 4,314 ms (0.26) 3,348 ms (0.20)
10 × hits, numeric 1,865 ms 830 ms (0.45) 463 ms (0.25) 2,784 ms 687 ms (0.25) 415 ms (0.15)
3 × synthetic 2,458 ms 802 ms (0.33) 791 ms (0.32) 1,953 ms 1,208 ms (0.62) 448 ms (0.23)

Compression is the dominant factor and everything else is secondary. It takes 0.10 to 0.26 of the uncompressed time. This is the number the loopback suite structurally cannot produce, and it settles the direction of T1a: compression on by default is right.

ZSTD, not LZ4, is the better codec on this path — it wins 4 of 6 arms, including both large payloads and every insert. LZ4 holds only on the two small, highly compressible reads. One WAN probe is not enough to change a shipped default, but it is evidence T1a did not have.

HTTP's compressed reads beat the native client's on every shape, by 1.6× to 2.1×. Uncompressed the two are level or favour the native client (0.89 on the mixed hits shape), so the gap is specific to the compressed read path, and it is not explained by bytes: the layout table below has the native format sending fewer bytes on this data. Compression improves HTTP's mixed-shape read by 9.7× against a 6.49× byte reduction, while it improves the native client's by only 5.1× against its own 7.05×. The native compressed read is the one arm whose speedup falls short of its byte reduction, so something in that path is not converting the saving. Per-block framing is the leading suspect — a codec context, a CityHash checksum and a frame header per block rather than once per response — but this is unverified, and it is worth a look before the client ships.

Inserts are near parity on real data: HTTP is 21% ahead on the mixed shape, the native client 10% ahead on the numeric one. The native client's large insert win shows up only on the synthetic shape.

The columnar layout does not compress better than RowBinary on realistic data

An earlier revision of this PR claimed it did, on the strength of a synthetic shape, and that claim is withdrawn. Comparing FORMAT RowBinary against FORMAT Native for the same 1M rows under ZSTD-3, compressed in 1 MB frames so neither layout gets cross-block context the native protocol would never have:

Shape RowBinary Native Columnar sends
1 × WatchID 1.00× 1.00× 1.00× (incompressible control)
10 × hits, mixed 6.49× 7.05× 1.09× fewer bytes
3 × hits String 7.70× 8.04× 1.04× fewer bytes
10 × hits, numeric 30.09× 24.39× 1.23× more bytes
3 × synthetic 4.91× 10.73× 2.19× fewer bytes

These are server-side format sizes, so no client change can move them.

The synthetic row is the outlier, and it is the one the withdrawn claim rested on. Its columns are number, number % 100 and number / 7: three short-period sequences that compress far better once grouped by column. Real hits columns are high-cardinality — WatchID and UserID are near-random, URL and Title are varied text — so grouping them by column unlocks little the compressor was not already finding.

On ten narrow numeric columns row-major wins outright, for a reason that is a property of real traffic rather than of either format: many hits repeat an identical (CounterID, RegionID, OS, UserAgent, Resolution*, Flash*, Net*) tuple. Row-major keeps each duplicate row contiguous, so the compressor matches one long run per repeat; column-major scatters that tuple across ten distant regions and can only find per-column runs.

The frame size barely mattered — whole-stream compression reads 6.98× against the chunked 6.49× on the mixed shape — so the distortion in the withdrawn claim was the synthetic data, not the compression window.

Per-type decode — SelectColumn against TcpSelectColumn

Same 16 expressions. These are not equal work. HTTP's Read() decodes each row into a reused object[], boxing every value (ClickHouse.Driver/ADO/Readers/ClickHouseDataReader.cs:565); the block reader decodes into typed storage and boxes nothing, and for String never builds the strings at all. Read the times as directional and ignore TCP's flat ~12 KB as an allocation win on the variable-length types — that is the drain loop not materialising, not the protocol. TransportRead is the honest allocation comparison.

Type HTTP TCP Ratio
DateTime 23.5 ms 3.8 ms 0.16
Date 18.3 ms 3.2 ms 0.17
Date32 20.6 ms 3.4 ms 0.17
Float32 18.3 ms 3.7 ms 0.20
Int32 18.2 ms 3.8 ms 0.21
UInt32 18.4 ms 3.8 ms 0.21
Nullable(Int32) 19.5 ms 4.3 ms 0.22
Float64 26.9 ms 8.1 ms 0.30
UInt64 27.6 ms 8.2 ms 0.30
Int64 27.3 ms 8.4 ms 0.31
Decimal256 51.4 ms 16.4 ms 0.32
Array 54.3 ms 17.5 ms 0.32
Decimal64 36.7 ms 12.1 ms 0.33
String 38.2 ms 19.4 ms 0.51
Tuple 51.6 ms 26.9 ms 0.52
Decimal128 42.5 ms 93.0 ms 2.19

TCP scales with width — Int32 3.8 ms to Int64 8.4 ms is 2.2× — which HTTP does not.

What these turned up

Decimal128 reads allocate ~357 B/row. 93.0 ms and 174,459 KB, against its own Decimal64 at 12.1 ms / 20 KB and Decimal256 at 16.4 ms / 37 KB. Gen0 is 28,000 per 1,000 ops where every other TCP type is zero. It is the one type in the sweep where TCP loses to HTTP. ClickHouseTcpDecimal(Int128 mantissa, int scale) (Numerics/ClickHouseTcpDecimal.cs:39) is documented as sign-extending to 256 bits but reaches Int256 through Int256.FromBigInteger(mantissa), and the implicit Int128BigInteger conversion heap-allocates once per row. The Int256(ulong, ulong, ulong, ulong) constructor next to it costs nothing. Not fixed here — this PR only measures.

Not TCP, and not a defect: the HTTP insert cost is async_insert. A server with async_insert = 1 and wait_for_async_insert = 1 holds an insert's response until its buffer flushes, waiting from async_insert_busy_timeout_min_ms (50 ms) upward. On 26.6.1.1193 a one-row HTTP insert costs 52 ms against 2.0 ms with the setting off, where an empty SELECT 1 round trip is 2.2 ms. It reproduces with curl, on Null/MergeTree/Memory alike and for VALUES/FORMAT CSV alike, and a buffered ExecuteNonQueryAsync literal pays it too — so it is the server's policy, not the driver's path. INSERT ... SELECT costs 2.4 ms, which is the line it falls on: async inserts apply only when the data comes from the client. Worth documenting rather than fixing, since BatchSize multiplies it and async_insert = 1 is the ClickHouse Cloud default.

RowBinary costs the server about three times what the native block format does. For the same 109 MiB, the same 5M rows and the same ENGINE Null target, the server burns 150 to 180 ms of CPU parsing RowBinary against 55 to 68 ms reading native blocks, and on the native path it spends most of its time waiting for the client rather than parsing. So HTTP has a server-side floor the client cannot get under, and the native protocol's insert advantage is partly the server's work, not only the client's. Measured by tagging every insert with a query_id and reading query_duration_ms, ProfileEvents['UserTimeMicroseconds'] and ProfileEvents['NetworkReceiveElapsedMicroseconds'] out of system.query_log.

Where the row tier's cost is, and where it is not. The native row tier costs 14.4 ns per value above the columnar arm, 7.2 ns when the values come off POCO fields, and neither figure has a confirmed cause. Two candidates are now ruled out:

  • Type.IsInstanceOfType, called once per value in the gather that UntypedRowColumns.CreateBuilder compiles, is not it: removing it moved TcpRows by less than its standard deviation.
  • Cache locality in the transpose is not it either, and that one was settled by a change rather than an experiment. TCP N9: POCO row insert — InsertRowsAsync<T> over a compiled per-column gather #559 replaced the whole-insert gather with one that fills a 50,000-row window per wire block, which is what the locality argument asks for. Measured back to back on this box, the tip before that change and the tip with it agree on every arm: TcpRows 356.4 against 347.5 ms, TcpPoco 263.5 against 241.2, TcpColumnar 129.4 against 131.1, and the same ratios to within 0.06. All of those are inside the current tip's own run-to-run spread. At three columns the row array is walked three times either way, and a 50,000-row window is still 200,000 scattered heap objects, about 5 MB of them. Whether locality pays on wide rows is a question three columns cannot answer.

That change is visible in this suite only as allocation: 5.1 KB per wire block is unchanged by it, but the default block size moved, so the same 5M-row insert allocates 519 KB as 100 blocks where it allocated 40 KB as 5. Its real subject, the peak memory of the conversion buffers, is not something a timing table shows.

Caveats

  • Loopback for everything committed. Compression is priced at its cost with none of its benefit, so TcpCompression cannot rank codecs; its own docs say so. The Cloud figures above come from a one-off probe that is not part of the suite, and T1a still owns making that repeatable.
  • This host is not a measurement instrument. Two physical cores run the client and the server together, and the host interferes in bursts. That is why the figures are best-case iterations, and why an arm's mean sits 15% to 40% above its best case. Ratios between arms reproduce; absolute times are this box, not yours.
  • The comparison job reports, it does not gate. Nothing decides which rows matter: on a 2-core runner the same code on both sides reads as −21% to +11% at 3 iterations. Filed as T5 — a row should be flagged only when the ratio's interval clears RatioSD, and the job should say how many rows it suppressed.
  • tcp-investigation is defined and unused, kept as the slot symmetric with http-investigation.
  • BenchmarkDotNet's MinIterationTime advisory fires on the sub-100 ms arms. Under RunStrategy.Monitoring there is no invocation amortisation to increase; the measurements are still five orders of magnitude above timer resolution.

Verification

  • Builds in local mode and in comparison mode; both Program guards pass in both.
  • Category counts, from --list flat --anyCategories: http-regression 40, tcp-regression 41, cross 14, http-investigation 8, tcp-investigation 0, compression 8, 111 total.
  • Every figure above was measured twice on the branch as it stands, and the runs agree to within 0.02% to 8% per arm on best-case iterations. Where they disagree the disagreement is reported, not averaged: HttpPoco's allocation, and TcpColumnar at 500k, whose first run carried 11 ms of interference that the fixed-cost fit exposed.
  • The insert arms were also measured on the tip before TCP N9: POCO row insert — InsertRowsAsync<T> over a compiled per-column gather #559's ranged gather, back to back on the same box, to separate that change from the host. They agree, and the comparison is in the row-tier section above.
  • The async_insert finding was re-checked with curl against the same server: 52 ms per one-row insert at the server default, 2.0 ms with the setting off.
  • The path-to-category mapping has 12 cases covering each project, the Compression/ subdirectory rule, the harness escape hatch, and the docs/tests/examples fallbacks.
  • Every new class ran end to end against a real server; three arms were reworked after their first results showed they measured something other than what their docs claimed.

Stacked on #595. Nothing here depends on the examples, so the diff to review is the commits on top of tcp/epic-t3-examples.

🤖 Generated with Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 635c2ff. Configure here.

// method. Without this a class carrying [Benchmark(Baseline = true)] makes that one method
// the baseline for the whole table, and every other row's ratio mixes the transport change
// with the method difference — unreadable as a regression signal.
AddLogicalGroupRules(BenchmarkLogicalGroupRule.ByMethod);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comparison groups ignore benchmark params

Medium Severity

AddLogicalGroupRules is called with only BenchmarkLogicalGroupRule.ByMethod, so every parameter combination of a method shares one logical group. Most new classes are parametrized (Shape, Codec, Degree, ElementsPerRow), so the comparison job can treat one param’s baseline job as the ratio baseline for the others and mix the parameter effect into the PR-versus-baseline signal.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 635c2ff. Configure here.

@alex-clickhouse
alex-clickhouse marked this pull request as draft August 30, 2026 11:15
@alex-clickhouse
alex-clickhouse requested a balanced review from Copilot August 30, 2026 11:15
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch from 837cd92 to d917ea3 Compare August 30, 2026 11:17
@alex-clickhouse
alex-clickhouse changed the base branch from tcp/epic-u1-block-projection to tcp/epic-t3-examples August 30, 2026 11:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds native TCP and cross-transport benchmarks, category-based execution, and updated comparison workflows.

Changes:

  • Adds TCP read, insert, compression, latency, and concurrency benchmarks.
  • Categorizes existing HTTP benchmarks and supports package-to-package comparisons.
  • Updates benchmark documentation and GitHub Actions automation.

Reviewed changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
CONTRIBUTING.md Documents benchmark categories and execution.
ClickHouse.Driver.Benchmark/TransportReadPoco.cs Compares POCO reads across transports.
ClickHouse.Driver.Benchmark/TransportRead.cs Compares transport read tiers.
ClickHouse.Driver.Benchmark/TransportLatency.cs Measures warm request latency.
ClickHouse.Driver.Benchmark/TransportInsert.cs Compares insert shapes across transports.
ClickHouse.Driver.Benchmark/TcpStringRead.cs Benchmarks native string access tiers.
ClickHouse.Driver.Benchmark/TcpSelectColumn.cs Adds native per-type read benchmarks.
ClickHouse.Driver.Benchmark/TcpReadTiers.cs Compares native materialization tiers.
ClickHouse.Driver.Benchmark/TcpProjectedRead.cs Measures projected column reads.
ClickHouse.Driver.Benchmark/TcpPoolConcurrency.cs Measures pool concurrency scaling.
ClickHouse.Driver.Benchmark/TcpInsertShapes.cs Compares native insert shapes.
ClickHouse.Driver.Benchmark/TcpConnectionCost.cs Measures connection lifecycle costs.
ClickHouse.Driver.Benchmark/TcpCompression.cs Benchmarks native compression codecs.
ClickHouse.Driver.Benchmark/TcpCompositeRead.cs Measures composite access paths.
ClickHouse.Driver.Benchmark/TcpArrayWriteShape.cs Compares dense and jagged arrays.
ClickHouse.Driver.Benchmark/SelectColumn.cs Categorizes HTTP column benchmarks.
ClickHouse.Driver.Benchmark/ResponseDecompressionBenchmark.cs Categorizes decompression benchmarks.
ClickHouse.Driver.Benchmark/ReadValueBenchmark.cs Categorizes reads and adjusts baselines.
ClickHouse.Driver.Benchmark/Program.cs Validates categories and comparison baselines.
ClickHouse.Driver.Benchmark/PocoReadBenchmark.cs Categorizes POCO reads.
ClickHouse.Driver.Benchmark/PocoInsertColumn.cs Categorizes POCO column inserts.
ClickHouse.Driver.Benchmark/PocoInsertBenchmark.cs Categorizes POCO inserts.
ClickHouse.Driver.Benchmark/MultidimArrayInsert.cs Categorizes array investigation benchmarks.
ClickHouse.Driver.Benchmark/InsertCompressionBreakdownBenchmark.cs Categorizes compression breakdowns.
ClickHouse.Driver.Benchmark/DynamicReadBenchmark.cs Categorizes dynamic-read investigation.
ClickHouse.Driver.Benchmark/ComparisonConfig.cs Configures job-based comparison ratios.
ClickHouse.Driver.Benchmark/ClickHouse.Driver.Benchmark.csproj Adds TCP references and comparison symbols.
ClickHouse.Driver.Benchmark/BulkInsertColumn.cs Categorizes bulk inserts.
ClickHouse.Driver.Benchmark/BinaryInsertStreamingBenchmark.cs Categorizes streaming investigation.
ClickHouse.Driver.Benchmark/BinaryInsertObjectArrayBenchmark.cs Categorizes object-array investigation.
ClickHouse.Driver.Benchmark/BinaryInsertCompressionBenchmark.cs Categorizes insert compression.
ClickHouse.Driver.Benchmark/BenchmarkServer.cs Centralizes HTTP/TCP endpoints.
ClickHouse.Driver.Benchmark/BenchmarkModes.cs Switches baseline behavior by build mode.
ClickHouse.Driver.Benchmark/BenchmarkCategories.cs Defines benchmark categories.
ClickHouse.Driver.Benchmark/BatchQueryLineBenchmark.cs Categorizes batch investigation.
.github/workflows/nightly.yml Adds the nightly full benchmark suite.
.github/workflows/benchmark.yml Adds TCP and category selection.
.github/workflows/benchmark-compare.yml Selects baselines/categories and runs comparisons.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +33 to +35
/// Every arm consumes every value, because a transport that only decodes is not comparable with one
/// that also materializes. The per-row <c>switch</c> on <see cref="Shape"/> is identical in all three
/// arms, so it cannot bias the comparison.
[GlobalCleanup]
public void Cleanup() => client?.Dispose();

/// <summary>Borrowed columnar spans: no per-row call, no allocation per row.</summary>
Comment on lines +164 to +166
dotnet build baseline/ClickHouse.Driver/ClickHouse.Driver.csproj \
--configuration Release \
/p:Version=0.0.0-main
cp main/ClickHouse.Driver/bin/Release/*.nupkg pr/.nupkg/
/p:Version=0.0.0-baseline
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch from 1bc008d to 8ffa63f Compare August 30, 2026 19:03
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch from 8ffa63f to a55f032 Compare August 31, 2026 09:31
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch from a55f032 to 45eecaa Compare August 31, 2026 15:34
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch from 45eecaa to b2389e0 Compare August 31, 2026 16:40
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch from b2389e0 to b2e873c Compare August 31, 2026 17:23
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch from b2e873c to 9a6b277 Compare September 1, 2026 08:23
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch 2 times, most recently from a11d5a4 to 2d30cf1 Compare September 1, 2026 18:20
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch 2 times, most recently from db0e523 to f0dece7 Compare September 2, 2026 11:07
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch from f0dece7 to d01f5cb Compare September 3, 2026 09:25
Comment thread .github/workflows/benchmark-compare.yml Fixed
Comment thread .github/workflows/benchmark-compare.yml Fixed
Comment thread .github/workflows/benchmark-compare.yml Fixed
@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch 2 times, most recently from bfe6007 to 35ae198 Compare September 3, 2026 09:46
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch 2 times, most recently from e08c54a to e7ceb9e Compare September 4, 2026 09:02
alex-clickhouse and others added 10 commits September 4, 2026 11:18
The CI runs select benchmarks with --anyCategories, so a class with no
category matches no filtered run. Program fails the run rather than let one
disappear from CI silently.

BenchmarkDotNet takes either a baseline method or a baseline job, not both:
given both, the method wins and every other row's ratio mixes the package
difference with the method difference. Local runs want the method baseline
(PocoReadBenchmark exists to report POCO against manual GetValue); the PR
comparison wants the job baseline. BenchmarkModes.MethodBaseline selects per
mode, and a second guard fails a comparison-mode run if a literal
Baseline = true appears later.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covers the tiers a caller chooses between rather than one number per
operation: the block tier's borrowed spans against the typed indexer,
Values, and the row tier; ReadAs<T>'s projection cost; IStringColumn's
bytes against decoded strings; the dense Array(T) write shape against one
array per row; and the pool's speedup at three widths.

TcpSelectColumn mirrors SelectColumn's 16 expressions at the same row
count. The two are not equal work and its docs say so: HTTP's Read()
boxes every value into a reused object[], while the block reader decodes
into typed storage.

TcpCompression prices the codecs on the client's own work only. Over
loopback there is no bandwidth to save, so it cannot rank them; T1a owns
that question.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each class pairs an HTTP arm with its native counterpart on identical data,
so the choice between transports comes from numbers rather than from the
protocol's reputation. Both clients run at their own defaults, which is what
a caller gets.

TransportInsert carries one-request arms next to the default ones:
InsertOptions.BatchSize defaults to 100,000, so a 500k-row HTTP insert goes
as five sequential requests where the native arms send one block. Without
the control the table reports that batching as a protocol difference.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comparison job has been failing to compile since the Common assembly
split: the outer dotnet run left ClickHouseDriverVersion unpinned, so it
restored the default release from nuget.org and could not see any API added
since. It now pins the baseline version and source, and publishes 9000 so
the native benchmarks have a server.

baseline_ref defaults to the PR's own base branch. A tcp/** PR sits on a
stack whose base is not main, and its benchmarks reference code main does
not have, so comparing against main cannot compile.

Categories come from the files the PR touched: a change under
ClickHouse.Driver.Tcp/ runs the native set, one under ClickHouse.Driver/ the
HTTP set, one under either Compression/ directory the codec sweep, and a
change to the harness runs everything. Both transports moving adds the cross
table. A push to main runs the regression sets; the nightly build runs all
of them, where an hours-long job costs nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A server with async_insert on buffers any insert carrying a client data block
and, with wait_for_async_insert, holds the response until that buffer
flushes. The adaptive wait starts at async_insert_busy_timeout_min_ms, 50 ms
by default, so on 26.6.1.1193 a one-row HTTP insert costs 55 ms against
1.9 ms with the setting off. It reproduces with curl, on Null, MergeTree and
Memory alike, and for VALUES and FORMAT CSV alike, so it is the server's
policy and not the client's path. INSERT ... SELECT does not pay it, which is
the line the measurements fall on: async inserts apply only when the data
comes from the client.

The native arms did not pay it, so leaving it on charged one transport for a
server setting and reported the difference as a protocol difference. Every
arm now sets async_insert = 0.

HttpRowsDefaultBatching keeps InsertOptions.BatchSize at its default so the
batching cost stays visible on its own. With the buffering out of the way it
is not measurable in time, only in the per-request buffers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two transports do not default to the same request codec — HTTP to ZSTD,
the native client to LZ4 — and on this payload the codec costs more than the
serialization the arms exist to compare, so it reported itself as a protocol
difference. Both sides now run uncompressed. Over loopback there is no
bandwidth to save and TcpCompression owns the codec axis, so nothing of value
is left out.

With the codec and the server's insert buffering both out of the way, the
native row tier is slower than HTTP's, not faster: it transposes rows into
columns while HTTP writes each row straight to the stream. Columnar wins by
skipping the transpose. The POCO tier beats the object[] tier on the native
side, which is the compiled typed writers against unboxing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
At 500,000 every arm ran under 60 ms and the ratios carried a standard
deviation of 0.16 to 0.76, which tells a reader nothing. At ten times the
size they land at 0.04 to 0.11 and hold across the change: the native row
tier reads 1.30 at both sizes, the POCO tier 0.98, and the columnar tier 0.50
against 0.56. It costs about 1 GB of source rows held for the run.

The batching arm also becomes quotable: fifty requests against one costs
about 2.6 ms per extra request, which is the round trip and little else.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The read arms carried the same confound the insert arms did: HTTP defaults to
gzip or ZSTD and the native client to LZ4, so the codec difference reported
itself as a transport difference. Over loopback a codec costs CPU and saves
nothing, so the uncompressed pair is the one that answers which client does
less work. BenchmarkServer.HttpUncompressed and CreateUncompressedTcpClient
put the knob in one place.

It moves the read ratios a long way, and mostly against the native client:
HTTP was paying for an expensive codec while the native side paid for a cheap
one. On a bandwidth-bound path the codec pays for itself either way, measured
against ClickHouse Cloud at 0.22x to 0.56x of the uncompressed time on both
transports.

The layout is not what decides it. On ClickBench hits the native block
format's columnar layout sends 1.04x to 1.09x fewer bytes than RowBinary
under ZSTD, and 1.23x more bytes on ten narrow numeric columns, where real
data repeats whole rows and row-major keeps each duplicate contiguous. A
synthetic three-column shape reports 2.19x for the columnar layout, but only
because its columns are short repeating sequences. TcpCompression and a real
network own that question; this class stays on client work.

Read rows go from 500,000 to 2,000,000, where the arms no longer sit in the
tens of milliseconds with a relative standard deviation near 30%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Its three columns are number, number % 100 and number / 7, which repeat far
more than production columns do. Measured against ClickBench hits, that shape
reports a 2.19x layout advantage for the native format where real data gives
1.04x to 1.09x, so a codec scores better here than a caller should expect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alex-clickhouse
alex-clickhouse force-pushed the tcp/epic-t1-t2-benchmarks branch from e7ceb9e to 9d2bf66 Compare September 4, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants