TCP TT: a test audit of the native client, and the defects it found

[alex-clickhouse]

What this is

A test audit of the native client, and the defects it found.

The suite up to this point grew alongside the epics: each epic proved its own feature. This PR
attacks it from the other side, asking of each area what a test would have to do to fail, and
writing that test. 57 backlog items, one commit each where the item needed one, plus six commits
closing a review pass over the result.

Roughly half the commits change client behavior. Those came first as a failing test against a
real server, so every one of them is a defect an existing test did not catch.

Correctness fixes

A dense Variant could be written as a different Variant of the same arity.
ClickHouse.Driver.Tcp/Types/Codecs/VariantColumnCodec.cs. The dense write gate compared the
number of alternatives, then reused the source column's own discriminator bytes. Reading
Variant(Int64, String) and inserting it into Variant(Bool, Int64) passed CanWrite, sent
discriminators that named the wrong alternative, and then hit an InvalidCastException inside the
body, after the discriminators were already on the wire. The gate now compares the alternative names
in order; a column that does not match is scattered by each value's own type, so a read-back of one
Variant lands in the right alternative of another.

A backticked identifier in a type string broke the header parse.
Types/QuotedText.cs (new), Types/TypeTokenizer.cs, Types/NamedElementParser.cs. The server
quotes an identifier it cannot spell bare — a JSON typed path, a Tuple or Nested field name — so
a real header carries `a(b` or `a,b`. Breaking on the comma or the paren inside it split
the argument list and the read failed on the header, before a row decoded. QuotedText scans a
quoted span once, for its boundaries and its text, so the tokenizer and both parsers agree on where
a span ends.

Enum labels decoded wrong. Types/Codecs/EnumColumnCodec.cs. A label holding a newline arrives
spelled 'a\nb'; dropping the backslash and keeping the next character decoded that to anb, so
the label a caller read was wrong and writing the real label threw. Only \\ and \' came out
right. Labels now go through the same QuotedText scanner as identifiers.

A timezone .NET cannot represent killed the whole column.
Types/Codecs/DateTimeZones.cs, DateTimeColumn.cs, DateTime64Column.cs. ClickHouse accepts
fixed offsets TimeZoneInfo cannot hold and 26.6 applies them: Fixed/UTC+19:00:00 is past .NET's
±14 hours and Fixed/UTC+05:30:15 is not a whole number of minutes. Resolving the zone while
building the codec made each of those a dead read — no column and no row arrived, including for a
caller who only wanted the epoch counts, which the zone does not affect. ResolvedTimeZone now
carries either the zone or the reason there is none, and only the calendar uses (TimeZone,
GetDateTimeOffset, a DateTime/DateTimeOffset projection, writing an Unspecified DateTime)
report it.

DateTime64 reported a bare OverflowException. Types/Codecs/DateTime64ColumnCodec.cs. A
DateTimeOffset past the scale's range fell out of the multiply with no context. Range-checked
before scaling, naming the value and the column, as the DateTime path already did.

A Utc or Local DateTime could not be written to a column whose zone .NET cannot represent.
Same file plus DateTimeColumnCodec.cs. ToUtc consults the zone only for Unspecified, but both
callers evaluated ResolvedTimeZone.Value before it could look at Kind, so the check above did not
finish the job: writing a UTC instant to DateTime('Fixed/UTC+19:00:00') still threw, and so did a
null row, whose placeholder is DateTime.UnixEpoch. ToUtc now takes the resolved zone and asks for
it after the Kind test.

A negative Time64 finer than a tick read back as midnight.
Types/ColumnValueProjections.cs. Time64ToTimeOnly shifted the raw count to 100 ns ticks and then
checked it was a time of day. The shift truncates toward zero, so every count from −1 to −99 at scale
9 reached zero and passed the check — a negative value presented as 00:00, which is the exact
outcome the refusal exists to prevent. Checked on the raw count against [0, 86400 · 10^scale).

Diagnostics

• Array(Boolean) said "ClickHouse type 'Boolean' is not supported". Resolution recurses and a
  child node knows nothing about where it sits, so the message named a type the caller never wrote
  (the server spells it Bool). Types/ColumnCodecRegistry.cs now keeps the child's refusal and
  adds the type it was handed. Also drops "yet": Object('json') was removed from ClickHouse and
  MultiPoint never existed, so promising them later is wrong.
• An AggregateFunction refusal named the serialization version as the function. 26.6 reports
  sumMapState(...) as AggregateFunction(1, sumMap, ...), so the message said "the '1' aggregate
  function" and suggested SELECT 1Merge(column), which is not a function. Now
  sumMapMerge(column), and sumMapFilteredMerge([1, 2])(column) with parameters — both run.
• One parameter message covered two unrelated failures. Parameters/TcpParameterFormatter.cs
  answered "Cannot convert value of type 'System.String' (abc) to ClickHouse type VARCHAR" both for
  a type name this client does not know and for a known type whose arm declined the value. For the
  first the value was never the problem. Split, with the registry answering whether a name is known
  so the two stay in step as codecs are added.
• Two client limits read as copies of a server rule. The QBit element switch and the Variant
  refusal of a Dynamic alternative. Neither is a server rule. Wording only.

HTTP-client parity

A query written against the shipped HTTP driver should move to this client without respelling its
types. Five things stopped that:

• Type-name aliases. {p:VARCHAR}, {p:BIGINT}, {p:DEC(4,2)}, {p:Boolean}, {p:json} all
  threw client-side, before the query was sent. Types/TypeAliases.cs (new) is
  system.data_type_families on 26.6: every row with a non-empty alias_to. The registry consults
  it when an exact lookup misses and resolves under the canonical name, so DEC(4, 2) reports
  itself as Decimal(4, 2). Child nodes come back through the same lookup, which is what makes
  Array(Map(String, Tuple(Int32, BIGINT))) resolve. An integration test compares the table against
  the live server in both directions, which is what caught the entries this would otherwise have
  mistyped.
• Case. The registry now matches any case of a name it knows. It deliberately does not carry a
  copy of which families the server matches case-insensitively: that rule moves between versions,
  and resolution serves reads, so a stale copy would refuse a header the server legitimately sent.
• The same, inside a Variant. Picking the alternative is a separate match from formatting the
  value, and it read the name as the caller wrote it, so {p:Variant(BIGINT, String)} accepted no
  Int64 at all while the server resolves that declaration to Variant(Int64, String).
  ParameterTypeInference.Accepts now canonicalizes first, as the formatter's dispatch already did.
• A bare Enum('A' = 1, 'B' = 2), which the server accepts and the HTTP driver resolves. The
  client now picks the width the server would: Enum8 while every ordinal fits Int8, Enum16
  otherwise. A header always names a width, so this only comes from a caller's hint.
• TimeOnly as a parameter. It reached neither time arm: Time called Convert.ToInt32 on it,
  which a TimeOnly does not support, and Time64 matched TimeSpan only.
• Dynamic, Geometry and SimpleAggregateFunction parameters, all three refused client-side
  while the server takes a text value for each (checked on 26.6). SimpleAggregateFunction writes
  as its inner type; the other two write the value's own text and leave the parse to the server. For
  Dynamic the server's answer turns out to be String whatever the text is — it reports
  dynamicType String for a bare 42 too — so what this buys is that the value arrives, not a
  particular runtime type.
• A value read from a Map column, sent back as a parameter. A Map row surfaces as
  KeyValuePair<K, V>[] so duplicate keys and pair order survive, and the explicit Map(K, V)
  formatter already took that shape — but ParameterTypeInference.Infer had no arm for it, so it
  fell to the Array arm, whose element inference has no reading for a KeyValuePair. A Dynamic
  parameter asks Infer what the value is, so a Map read-back could not be sent as one.

Ergonomics

• Time and Time64 now take a TimeOnly. Date takes a DateOnly, so a caller reasonably
  expects the symmetry; ClickHouseTcpColumn.Create over TimeOnly values had no codec at all.
  Reading back as a TimeOnly is the one narrowing in the read surface — a Time value may be
  negative or past 24 hours and is then not a time of day — so such a row is refused with a message
  pointing at TimeSpan, rather than reduced modulo a day, which would present a different value as
  the stored one.
• Variant(Time, String) accepted no time value at all.
  Parameters/ParameterTypeInference.cs had an arm for the instant types and none for the time ones,
  so a TimeSpan matched a Time64 alternative and never a Time one, and a Variant that matches
  nothing is refused whole.

New option: SendJsonAndDynamicSerializationSettings

ClickHouseTcpClientOptions and the connection string. Public API surface updated.

Every query, insert and execute from the high-level client carried
output_format_native_use_flattened_dynamic_and_json_serialization and
output_format_native_write_json_as_string. Both server defaults are 0, so each operation was a
setting modification, and a user under a readonly profile got Code 164 for all of them — the
high-level client was unusable for such a user while ClickHouseTcpConnection worked. The option is
on by default and turns the injection off when cleared. Sending the settings as "0" is not an
alternative: the server counts that as a modification too. The cost of clearing it is that a JSON
or Dynamic column may arrive in a serialization this client does not read; no other type is
affected.

Coverage added

Most of the diff. New integration fixtures, all against a real server:

Fixture	What it reaches that nothing did
MidStreamFailureIntegrationTests	an Exception packet after data blocks — its body is unframed while the blocks around it are framed, so uncompressed, LZ4 and ZSTD
InsertInterruptionIntegrationTests	a data phase that does not finish: cancelled from OnBlockWritten, and a constraint rejecting every block after the first
ComputedColumnInsertIntegrationTests	the insert schema block naming fewer columns than the table has
MultiBlockStateIntegrationTests	a LowCardinality dictionary, a Dynamic type list and a version word across more than one block — the only multi-block reads were of UInt64
ConnectionPoolIntegrationTests (+250)	the server hanging up on an idle connection (idle_connection_timeout, polled for with a bounded budget rather than slept past); a retired connection closed on the server, by system.query_log client ports and the TCPConnection metric; permit accounting at a pool of one
TlsTransportIntegrationTests + TlsTerminatingProxy	a query, a block and an insert through a real SslStream. The existing TLS tests answer with a canned Hello, and the Cloud fixture is skipped unless a service is configured
TimezoneColumnIntegrationTests	a non-UTC column asserted against the server's own rendering, including the ambiguous DST hour
FloatSpecialValueIntegrationTests	signed zero, which the corpus comparison cannot see because (-0.0).Equals(0.0)
ReadonlyUserIntegrationTests	a user who is not the fixture superuser
ClickHouseTcpCancellationIntegrationTests	the read deadline actually firing

Plus ~330 lines of InsertRoundTripCase values: UInt256/Int256 at 2^255 and both ends;
Decimal at scale 0, at scale == precision, and Decimal(76, 0); a fully populated IPv6 and the
IPv4-mapped form; LowCardinality over 1-, 8- and 16-byte dictionaries; a dictionary leaf under two
levels of offsets; Map with a LowCardinality key and with a Tuple key; QBit(Float32, 768);
DateTime past 2038; both Int64 ends of DateTime64(9); NaN/±∞/−0 on all three float widths in
bare, Nullable and Array form; Nullable(Tuple(...)) and Nullable(Point), which the corpus
note had said could not be written.

Two things worth flagging to a reviewer

Some coverage was written to make a test able to fail, not to add a case. Three examples: the
read-deadline test could not fire because the server's Progress packet every interactive_delay
counts as speaking, so SELECT sleep(3) finished under a one-second deadline — raising
interactive_delay past the sleep buys the silence. The timezone assertions previously compared the
read-back against a value the same conversion produced, so they held even with the direction
inverted. The signed-zero test asserts stored bits through reinterpretAsUInt32, because
[assembly: DefaultFloatingPointTolerance(1e-15)] and Equals both hide the sign.

Three backlog claims did not survive checking, and are closed on evidence rather than a fix. The
BFloat16 1-ULP disagreement (client and server truncate identically, bit for bit), the
Array(Dynamic) zero-row desync (no desync; the codec's stated reason was wrong and is corrected,
the behavior was right), and >255 Dynamic runtime types (the server caps max_types at 254, so
the wide discriminator is unreachable end to end).

What a review pass then found

The branch was reviewed by Codex with the repository's own review criteria, a live 26.6 to run
against, and licence to mutate the code to test a claim. It found nine things; four were real defects
in the branch, three were tests that could not support their names, and two are recorded rather than
fixed. The four defects and the three test repairs are in the commits above and below.

The most useful finding is one my own mutation discipline missed: the mismatched-Variant insert
test passed with the fix reverted. It used Variant(Date, Int64) → Variant(Date32, Int64), where
the alternative keeps its index and both surface as IColumn<DateOnly>, so the old arity-only
shortcut reached the same answer. It is now Variant(Int64, String) → Variant(Bool, Int64), which
moves the alternative from index 0 to index 1; reverting the gate now fails it with the
InvalidCastException the fix's commit message describes.

Two findings are declined, with the reasoning on record:

• Time/Time64 parameters round where the binary write truncates, so TimeOnly.MaxValue
  formats as 24:00:00 and stores 86400 seconds, which no longer reads back as a TimeOnly. Real,
  but the rounding is deliberate parity with the shipped HTTP driver, which formats TimeOnly
  through the identical Math.Round(TotalSeconds) path, and the divergence applies to TimeSpan
  equally, so it predates this branch. Changing one client and not the other would reopen the parity
  gap five commits here exist to close. Filed as its own TODO item; the test that pinned it now says
  why it is pinned, and the case named "truncated to the scale" — which used a value that could not
  tell rounding from truncation — is replaced by midpoint cases that can.
• No docs/ update. docs/ has two files and no native-client page at all; documentation lands
  for the client as a whole. The examples half of that finding was valid and is done: the
  Time/Time64 example now writes a TimeOnly and states the read-back narrowing, and the
  Variant/Dynamic/JSON example documents the new option and the readonly trade-off.

Verification

Full suite, both ends of the supported range:

Server	Result
26.6.1.1193	4065 passed, 6 skipped, 0 failed
25.8.32.4 (the CI floor)	4001 passed, 12 skipped, 0 failed

The 25.8 run is the reason for the version gates in the last commit. It found a pre-existing red
test, two ungated cases, and one server behavior change: 25.8's MergeTree part writer normalizes a
negative zero away, where a Memory table keeps it, so the signed-zero test uses Memory.

Every behavior change was verified by mutation — revert the fix, watch the new test fail. Three
mutations were inert on the first attempt and the tests were redesigned until they bit; the third of
those is the Variant case the review caught rather than I did.

Not in this PR

• Docs. docs/ has no native-client page yet; documentation lands for the client as a whole. The
  two user-visible additions are documented in the examples instead, which is where the client's
  usage documentation currently lives.
• No changelog fragment, per the standing rule for the tcp/** epic stack.
• Roles and grants. Deferred to a post-0.1 item. There is no protocol feature to add — the
  native protocol carries no role field, SET ROLE is SQL — so the work is coverage plus a
  documented pattern: over a pooled client, SET ROLE has to run inside OpenSessionAsync or the
  next statement lands on another connection with the role unset.
• Five backlog items are triaged as will-not-fix, four of them under the standing rule that the
  client does not gate on server type rules.

[Copilot]

Pull request overview

Audits the native TCP client with expanded real-server coverage and fixes defects uncovered across codecs, type parsing, parameters, transport, and diagnostics.

Changes:

• Fixes Variant, timezone, enum, alias, time, and parameter handling.
• Adds readonly-profile serialization controls and public API support.
• Expands integration coverage for TLS, interruptions, framing, state, and edge cases.

Reviewed changes

Copilot reviewed 64 out of 64 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
examples/Tcp/Types/Tcp_004_VariantDynamicJson.cs	Documents serialization settings.
examples/Tcp/Types/Tcp_002_DateTimeAndTimezones.cs	Demonstrates TimeOnly support.
ClickHouse.Driver.Tcp/Types/TypeTokenizer.cs	Handles quoted spans and whitespace.
ClickHouse.Driver.Tcp/Types/TypeAliases.cs	Defines server type aliases.
ClickHouse.Driver.Tcp/Types/QuotedText.cs	Adds shared quoted-text decoding.
ClickHouse.Driver.Tcp/Types/NamedElementParser.cs	Parses quoted field names.
ClickHouse.Driver.Tcp/Types/IDateTimeColumn.cs	Documents unrepresentable timezones.
ClickHouse.Driver.Tcp/Types/DateTimeColumn.cs	Defers timezone failures.
ClickHouse.Driver.Tcp/Types/DateTime64Column.cs	Defers timezone failures.
ClickHouse.Driver.Tcp/Types/ColumnValueProjections.cs	Adds TimeOnly projections.
ClickHouse.Driver.Tcp/Types/ColumnCodecRegistry.cs	Resolves aliases and case variants.
ClickHouse.Driver.Tcp/Types/Codecs/VariantColumnCodec.cs	Corrects dense Variant writes.
ClickHouse.Driver.Tcp/Types/Codecs/TimeColumnCodec.cs	Adds TimeOnly support.
ClickHouse.Driver.Tcp/Types/Codecs/Time64ColumnCodec.cs	Adds scaled TimeOnly support.
ClickHouse.Driver.Tcp/Types/Codecs/QBitColumnCodec.cs	Clarifies client limitations.
ClickHouse.Driver.Tcp/Types/Codecs/EnumColumnCodec.cs	Fixes labels and bare Enum sizing.
ClickHouse.Driver.Tcp/Types/Codecs/DynamicColumnCodec.cs	Clarifies zero-row behavior.
ClickHouse.Driver.Tcp/Types/Codecs/DateTimeZones.cs	Represents unresolved timezones safely.
ClickHouse.Driver.Tcp/Types/Codecs/DateTimeColumnCodec.cs	Delays timezone use during reads/writes.
ClickHouse.Driver.Tcp/Types/Codecs/DateTime64ColumnCodec.cs	Adds range checks and delayed timezone use.
ClickHouse.Driver.Tcp/Types/Codecs/AggregateFunctionColumnCodecs.cs	Corrects aggregate-function diagnostics.
ClickHouse.Driver.Tcp/PublicAPI/PublicAPI.Unshipped.txt	Tracks the new public option.
ClickHouse.Driver.Tcp/Parameters/TcpParameterFormatter.cs	Expands parameter formatting parity.
ClickHouse.Driver.Tcp/Parameters/ParameterTypeInference.cs	Adds time and map inference.
ClickHouse.Driver.Tcp/Format/BlockReader.cs	Clarifies per-block state handling.
ClickHouse.Driver.Tcp/Compression/FramedPackets.cs	Documents exception framing.
ClickHouse.Driver.Tcp/Client/ClickHouseTcpConnectionStringBuilder.cs	Exposes serialization configuration.
ClickHouse.Driver.Tcp/Client/ClickHouseTcpClientOptions.cs	Defines the new serialization option.
ClickHouse.Driver.Tcp/Client/ClickHouseTcpClient.cs	Applies conditional setting injection.
ClickHouse.Driver.Tcp.Tests/Utilities/TlsTerminatingProxy.cs	Adds a real TLS test proxy.
ClickHouse.Driver.Tcp.Tests/Utilities/NestedArrayShape.cs	Extends nested-array depth coverage.
ClickHouse.Driver.Tcp.Tests/Types/VariantColumnCodecTests.cs	Tests mismatched dense Variants.
ClickHouse.Driver.Tcp.Tests/Types/TypeParserTests.cs	Tests quoting and whitespace parsing.
ClickHouse.Driver.Tcp.Tests/Types/TupleColumnCodecTests.cs	Tests quoted tuple fields.
ClickHouse.Driver.Tcp.Tests/Types/NestedColumnCodecTests.cs	Tests quoted nested fields.
ClickHouse.Driver.Tcp.Tests/Types/EnumColumnCodecTests.cs	Tests enum escape decoding.
ClickHouse.Driver.Tcp.Tests/Types/DateTimeZonesTests.cs	Tests deferred timezone failures.
ClickHouse.Driver.Tcp.Tests/Types/DateTimeColumnCodecTests.cs	Covers unrepresentable timezone behavior.
ClickHouse.Driver.Tcp.Tests/Types/DateTime64ColumnCodecTests.cs	Covers ranges and timezone behavior.
ClickHouse.Driver.Tcp.Tests/Types/ColumnReadProjectionTests.cs	Tests TimeOnly projections.
ClickHouse.Driver.Tcp.Tests/Types/ColumnCodecRegistryTests.cs	Tests aliases and diagnostics.
ClickHouse.Driver.Tcp.Tests/Types/ClickHouseTcpTypesTests.cs	Tests alias capability queries.
ClickHouse.Driver.Tcp.Tests/Parameters/TcpParameterFormatterEdgeCaseTests.cs	Covers new parameter edge cases.
ClickHouse.Driver.Tcp.Tests/Integration/TlsTransportIntegrationTests.cs	Exercises TLS end to end.
ClickHouse.Driver.Tcp.Tests/Integration/TimezoneColumnIntegrationTests.cs	Verifies timezone writes against the server.
ClickHouse.Driver.Tcp.Tests/Integration/ReadonlyUserIntegrationTests.cs	Verifies readonly-user operation.
ClickHouse.Driver.Tcp.Tests/Integration/MultiBlockStateIntegrationTests.cs	Tests state across multiple blocks.
ClickHouse.Driver.Tcp.Tests/Integration/MidStreamFailureIntegrationTests.cs	Tests errors after data blocks.
ClickHouse.Driver.Tcp.Tests/Integration/InsertInterruptionIntegrationTests.cs	Tests interrupted inserts and pool recovery.
ClickHouse.Driver.Tcp.Tests/Integration/FloatSpecialValueIntegrationTests.cs	Verifies signed-zero preservation.
ClickHouse.Driver.Tcp.Tests/Integration/ComputedColumnInsertIntegrationTests.cs	Covers computed-column insert schemas.
ClickHouse.Driver.Tcp.Tests/Integration/ColumnReadProjectionIntegrationTests.cs	Tests timezone projection failures.
ClickHouse.Driver.Tcp.Tests/Integration/ColumnarReadSurfaceIntegrationTests.cs	Covers Nullable(Nothing) alignment.
ClickHouse.Driver.Tcp.Tests/Integration/ClickHouseTcpTypesIntegrationTests.cs	Compares aliases with the server.
ClickHouse.Driver.Tcp.Tests/Integration/ClickHouseTcpParameterIntegrationTests.cs	Verifies parameter parity end to end.
ClickHouse.Driver.Tcp.Tests/Integration/ClickHouseTcpExceptionIntegrationTests.cs	Expands protocol-failure coverage.
ClickHouse.Driver.Tcp.Tests/Integration/ClickHouseTcpConnectionMetadataIntegrationTests.cs	Updates metadata coverage documentation.
ClickHouse.Driver.Tcp.Tests/Integration/ClickHouseTcpConnectionInsertIntegrationTests.cs	Tests dense Variant reinsertion.
ClickHouse.Driver.Tcp.Tests/Integration/ClickHouseTcpCancellationIntegrationTests.cs	Verifies read deadlines.
ClickHouse.Driver.Tcp.Tests/Client/ClickHouseTcpConnectionStringBuilderTests.cs	Tests serialization option parsing.
ClickHouse.Driver.Tcp.Tests/Client/ClickHouseTcpClientSettingsTests.cs	Tests conditional setting injection.
ClickHouse.Driver.Tcp.Tests/Client/ClickHouseTcpClientOptionsTests.cs	Tests option copying.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!