Skip to content

TCP T3: usage examples for the native client - #595

Draft
alex-clickhouse wants to merge 16 commits into
tcp/epic-u1-block-projectionfrom
tcp/epic-t3-examples
Draft

TCP T3: usage examples for the native client#595
alex-clickhouse wants to merge 16 commits into
tcp/epic-u1-block-projectionfrom
tcp/epic-t3-examples

Conversation

@alex-clickhouse

@alex-clickhouse alex-clickhouse commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Epic T3. Adds 30 runnable examples for the native (TCP) client and reorganises examples/ so the two
transports live side by side.

Stacked on #594 (tcp/epic-u1-block-projection), which acts on section B of the follow-up list these examples produced. Below that, #592 froze the surface they use.

Replaces #593, closed so the stack reads in number order.

What is here

30 native examples in seven directories, numbered from 001 within each one, as the HTTP examples are:

Directory Examples Subject
Tcp/Core 001-004 first query, connection string and options, DI, migrating from the HTTP client
Tcp/Read 001-004 the three read tiers, blocks and columns, parameters, POCOs
Tcp/Write 001-002 columnar insert, composite write shapes
Tcp/Types 001-005 scalars, date and time, composites, Variant/Dynamic/JSON, QBit
Tcp/Connection 001-004 sessions, pool tuning, TLS, timeouts
Tcp/Advanced 001-006 settings and query id, callbacks, cancellation, errors and retries, compression, server info
Tcp/Observability 001-005 logging, OpenTelemetry, metadata blocks, health checks, Testcontainers

Each is a public static class with a public static Task Run(), registered in Program.cs and listed in
the README, and each prints what it did and what came back rather than asserting.

The 54 HTTP examples move to examples/Http/ (pure renames) and now take their server from one place.

Reorganisation

  • examples/ExampleConfig.cs is the single source of the endpoint. CLICKHOUSE_HOST,
    CLICKHOUSE_HTTP_PORT, CLICKHOUSE_TCP_PORT, CLICKHOUSE_USER, CLICKHOUSE_PASSWORD,
    CLICKHOUSE_DATABASE, or CLICKHOUSE_{HTTP,TCP}_CONNECTION_STRING to replace a whole string. Four
    examples are exempt and say so: two teach configuration, two start their own server.
  • examples/ExamplePreflight.cs reaches the endpoints the selected examples need before any of them
    runs, and prints what to fix. It exits non-zero rather than skipping, because CI runs the suite unfiltered
    and a skip would leave the run green with nothing exercised.
  • dotnet run -- --tcp and --http run one transport; --list takes them too.
  • examples/AGENTS.md (plus a CLAUDE.md pointing at it) writes down the five steps for adding an
    example — the registration in Program.cs is the one that gets forgotten — the naming rule that keeps the
    two transports' class names apart, and the table-naming and endpoint rules.
  • examples/Tcp/README.md covers the port, the CHTCP0001 opt-in, what the native client cannot do
    (no format other than Native, no ADO.NET and so no ORM, no JWT, no custom headers, no per-query role or
    database) and what only it does (blocks, real sessions, live progress, block compression, QBit planes,
    W3C trace context).

.github/workflows/examples.yml publishes 9000 and watches ClickHouse.Driver.Tcp/**.

Review

The branch was reviewed per directory. Everything the review raised about examples/ is fixed in
3f88b6e, the last commit — four P1s in the runner (a whole-string override that did not reach the
builders; --http running the three examples that need a cluster, Cloud or a token; a transport flag
ignored alongside --filter; preflight holding up the two self-contained examples), version gates for
QBit and Geometry, which fail outright on 25.8, and nine prose claims that were wrong.

Two behaviours worth calling out, because they are the kind of thing an example teaches by accident:

  • 42 stream loops ended in break after their first block. Stopping a StreamAsync enumeration early makes
    the client cancel the query and discard the connection, so every one of those loops was demonstrating the
    abandon path. All 42 read a single-block result, so they now drain.
  • Every fixed-name table is dropped before it is created as well as after. A run interrupted before its
    finally used to break the next run at CREATE TABLE, and the runner pauses between examples when
    interactive, so that is a common way to leave one behind.

What the review found about the driver rather than the examples is recorded as follow-ups, not fixed here:
TimeoutException cannot be attributed to the pool, the dial or the read, so a health check cannot honestly
separate "busy" from "down"; and GetServerInfoAsync describes the connection it borrowed, which a version
gate can outlive.

The surface #594 adds

Five examples use it, so the block tier is taught with the projection and composition it has:

  • Types/Tcp_001_ScalarTypes writes an Enum from a label and a String from raw bytes, then reads both
    back through IEnumColumn, IStringColumn, and Block.ReadAs. Its earlier advice was to change the
    schema to FixedString(N) for bytes.
  • Types/Tcp_002_DateTimeAndTimezones reads a whole DateTime column as DateTimeOffset.
  • Types/Tcp_003_CompositeRead walks every column through the non-generic composite views, naming the trap
    that a wrong type argument on a typed view compiles and never matches.
  • Types/Tcp_004_VariantDynamicJson names each row's alternative from IVariantColumn.TypeNames rather than
    printing bare discriminators.
  • Write/Tcp_002_CompositeWrites builds the flat array shape with ClickHouseTcpColumn.CreateArray, and asks
    ClickHouseTcpTypes.CanWrite which CLR types a target column accepts instead of listing them by hand.

Verification

Ran against a live 26.6.1.1193:

  • dotnet run -- --tcp, 30/30, exit 0, re-run after the TCP U1: projection and composition on the block tier #594 surface went in.
  • dotnet run, 79 examples across both transports, exit 0. Every exception in the logs is one an example
    raises on purpose.
  • CLICKHOUSE_TCP_PORT=1 dotnet run -- --filter tcptestcontainers starts its container and handshakes,
    confirming the self-contained examples no longer depend on the configured endpoint.

No changelog fragment: this is an epic branch in the tcp/** stack.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Aug 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

alex-clickhouse and others added 16 commits September 4, 2026 11:17
The HTTP examples move under examples/Http/, making room for a Tcp/ series
alongside them. Declared namespaces are unchanged, since ExampleRunner matches
ClickHouse.Driver.Examples exactly whatever folder a file sits in.

AGENTS.md carries the checklist for adding an example, because RunAllExamples in
Program.cs is hand-maintained: an example left out of it still compiles, still
lists, and still runs under --filter, so nothing reports that CI never runs it.
DateTimeHandling, AsyncInsert and QBitSimilaritySearch were in that state; they
are registered now, and the three that need a cluster, Cloud credentials or a
JWT are listed as deliberate omissions.

QBitSimilaritySearch read L2DistanceTransposed with GetFloat, which throws
because the server returns Float64.

The README's filter section described matching on the file name; the runner
matches the class name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ExampleConfig assembles the connection string from CLICKHOUSE_HOST,
CLICKHOUSE_HTTP_PORT, CLICKHOUSE_TCP_PORT, CLICKHOUSE_USER,
CLICKHOUSE_PASSWORD and CLICKHOUSE_DATABASE, each falling back to what a stock
server container exposes on localhost, so the suite runs with nothing set and
points somewhere else without editing 46 files.
CLICKHOUSE_HTTP_CONNECTION_STRING replaces the whole string for an endpoint the
pieces cannot describe.

HttpBuilder() covers the examples that change one of the five keys the assembled
string already sets; appending covers the rest, since no other key collides.

Core_002_ConnectionStringConfiguration and Core_003_DependencyInjection keep
their literals, because configuration is what they teach, and
Testing_001_Testcontainers starts its own server.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ClickHouse.Driver references ClickHouse.Driver.Tcp with PrivateAssets="all", so
the native client's types do not reach a consumer of the driver project; the
examples reference it directly. CHTCP0001 is suppressed project-wide, and
Tcp/README.md explains the opt-in a consumer has to make for themselves.

An example's transport comes from its class name. Every example shares one
namespace, so a native-protocol example cannot reuse an HTTP example's class
name, and the Tcp prefix that keeps them apart also says which endpoint it
needs. --http and --tcp select on it, and --list takes it too.

ExamplePreflight reaches those endpoints once before anything runs and reports
the endpoint, the reason, and the variables that change it. It exits non-zero
rather than skipping, because CI runs the suite with no filter and a skip would
leave the run green having exercised nothing. Asking for a transport checks that
transport, so --tcp reports whether port 9000 answers even with no example
written yet.

The workflow's server publishes 9000 alongside 8123, and its paths filter covers
the Tcp project.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four under examples/Tcp/Core/: constructing a client and running the four
operations; the connection string's native key set and deriving an options
variant; AddClickHouseTcpDataSource with keyed registrations and who disposes
the pool; and the same task written over both transports, with the call-for-call
mapping and an honest account of what the native client cannot do.

An example that talks to both interfaces needs both endpoints checked, which its
class-name prefix does not say, so ExampleInfo carries RequiredTransports
alongside the transport it is filed under.

Tcp/README.md said every public type of the native client is experimental. Six
are; the options record, the builder, Block, the columns and the exceptions are
not. It also now records that per-query roles and databases have no native
equivalent, and that a timestamp column reaches the row and block tiers as the
integer the wire carried.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
IDateTimeColumn and ITimeColumn reach the timezone and scale, so the row tier is
the only one that hands back the raw count.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four under examples/Tcp/Read/: the three read tiers with what each allocates,
measured over 200,000 rows; the block tier in depth, including the borrowed-span
contract and the temporal column interfaces; parameter binding with the three
traps that cost real time; and POCO reads and writes over one class.

Tcp_001 presented QueryAsync<T> as the only way to get a calendar value out of a
timestamp column, which the block tier's IDateTimeColumn now also does.

Tcp/README.md's list of experimental types omitted the AddClickHouseTcpDataSource
overloads and counted the session twice.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two under examples/Tcp/Write/: the columnar insert tier, matching by name with a
named subset and the server defaulting the rest; and composite writes, covering
both accepted Array(T) shapes and re-inserting a column read straight out of a
block for five composite families with nothing rebuilt.

These close the write-path documentation debt the plan recorded: the two array
shapes, the non-nullable-row rule, and the dense round trip now have a runnable
demonstration each.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five under examples/Tcp/Types/: the scalar CLR map including the 256-bit
integers and when a Decimal needs ClickHouseTcpDecimal; the temporal types, with
the presentation timezone measured under five session_timezone values and the
insert side of the same model; composite reads through the typed views and the
geo aliases as ValueTuple; Variant, Dynamic and JSON, including the
normalization that makes text in differ from text out; and QBit's bit planes,
rebuilding a vector at the precision L2DistanceTransposed would use.

The QBit and strided-QBit cases this server refuses print its own refusal and
the version they need rather than being left out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four under examples/Tcp/Connection/: sessions, showing what one pinned
connection buys and that disposal closes rather than pools it; the pool's seven
knobs with the cap, the PoolTimeout expiry, the sweep and Lifo against Fifo all
measured; TLS configuration and the mistakes the constructor refuses; and the
timeouts, separating ReadTimeout's idle deadline from a total time limit.

Pool state is not reportable, so Tcp_017 reads the pool's own log lines and
polls system.processes from a second client for the concurrency figure: from the
client side a query waiting for a slot and a query running look alike.

Tcp_018 needs a TLS endpoint for its last section and CI has none, so that part
is opt-in on CLICKHOUSE_TCP_TLS_CONNECTION_STRING and says why it skipped. The
rest needs no server: the port derivation, and the refusals.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six under examples/Tcp/Advanced/: settings precedence and query ids; the
progress, profile-info and profile-event callbacks, with the timeline showing
progress arriving while the rows do; cancellation, including what the server
records and why the connection is closed rather than pooled; the exception
hierarchy and which errors are worth retrying; compression measured in bytes on
the wire; and the server info a caller gates behaviour on.

Tcp_024 measures wire size through a counting socket and prints no wall-clock
ranking: over loopback there is no bandwidth to save, so timing there measures
the CPU cost and none of the benefit. The bytes say the client's codec does not
decide what the server sends - a zstd client reads an LZ4 response, because the
query packet carries one flag and the server frames with its own
network_compression_method - while it does decide what an insert writes.

Tcp_020 and Tcp_023 each hold a connection busy to demonstrate a real transient
failure. Both start the holder with AsTask() rather than Task.Run, so the query
packet is sent on the caller's thread: queued behind a busy thread pool, the
holder could start second and be the query that was refused. Tcp_023 also lets
only the retrying side declare the concurrency limit, so which query loses is
fixed by construction rather than by ordering.

Tcp_020 runs its unparseable-setting-value case on a throwaway client. That
error is raised while the server parses the settings list, so it closes the
socket and the pool keeps the connection, and the next operation on it fails.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five under examples/Tcp/Observability/: the three log categories and what each
reports, with two filter sets; OpenTelemetry, collecting the client's spans and
showing the server's own joining the same trace; the three block-shaped
callbacks Tcp_021 does not cover, copying out of a borrowed block; PingAsync as
a health check, measured against SELECT 1; and a throwaway container reached
over the native port.

Tcp_030 waits on a handshake rather than on either wait strategy. Neither probe
tests the native protocol - the HTTP one tests the other listener, and
UntilInternalTcpPortIsAvailable only says the port is bound - so on a loaded
machine the first handshake is refused after both have passed. Forced with a
no-condition strategy that took 43 attempts over 4.4 seconds. Only a transport
exception is retried: a server exception means it answered and rejected us.

Tcp/README.md now records W3C trace context propagation under what only the
native client does. The client sends the current Activity's ids with each query,
so the server's spans join the caller's trace; the HTTP transport sends no
traceparent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Runner and configuration:

- A whole-string CLICKHOUSE_{HTTP,TCP}_CONNECTION_STRING override now reaches
  ExampleConfig.HttpBuilder()/TcpBuilder(), which rebuilt from the component
  variables and so pointed an example at a different server than preflight
  checked. Host/HttpPort/TcpPort are private; HttpEndpoint/TcpEndpoint answer
  for the effective endpoint.
- --http and --tcp no longer include the three examples that need a cluster,
  Cloud credentials or a token. --list still shows them, marked.
- A transport flag now narrows --filter instead of being ignored.
- The two Testcontainers examples start their own server, so preflight no
  longer holds them up on the configured endpoint.
- TcpOpenTelemetry queries both interfaces, so it is registered as
  cross-transport and both endpoints are checked.

Native examples:

- Gate QBit (Tcp_015) and Geometry (Tcp_013) on 25.11, and QBit on the HTTP
  side too: both fail outright on 25.8, the floor of the support matrix.
- Every fixed-name table is dropped before it is created, so a run interrupted
  before its finally does not break the next one.
- Drop the trailing break from 42 single-block loops. Stopping a StreamAsync
  enumeration early makes the client cancel the query and discard the
  connection, which is not the pattern to teach.
- Tcp_012 pins session_timezone=UTC in the comparison that reads a
  Kind=Unspecified DateTime, which otherwise means something different per
  server.
- Tcp_017's concurrency marker and the deliberately-missing table names in
  Tcp_026 and Tcp_027 are unique per run.
- Tcp_020 polls its own table instead of issuing SYSTEM FLUSH ASYNC INSERT
  QUEUE, which flushes every client's pending inserts.
- WaitUntilRunning throws instead of returning after its last poll, and
  Tcp_023 reports retry exhaustion instead of throwing out of the example.
- Tcp_016 creates its role and user inside the try that drops them.
- Tcp_030 retries only a socket failure; the same exception type also carries
  TLS and DNS failures.

Corrected claims: values in an object[] row are boxed only when the column is
a value type; MaxRowsPerBlock is block geometry and MaxSendBufferBytes is the
memory bound; toString of a NULL is NULL, not an empty string; BFloat16, IPv4,
IPv6 and String are CLR surfaces rather than the wire bytes; a parameterized
query below revision 54459 is refused by the client before it is sent; a
TimeoutException does not say whether the pool, the dial or the read timed out;
a callback that copies values can still throw; GetServerInfoAsync describes the
connection it borrowed.

Also fixes three HTTP examples that built settings from Host = "localhost"
instead of ExampleConfig.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five native examples and the two indexes now teach what the block tier
can do:

- Tcp_001_ScalarTypes writes an Enum from a label and a String from raw
  bytes, then reads both back through IEnumColumn, IStringColumn, and
  Block.ReadAs.
- Tcp_002_DateTimeAndTimezones reads a whole DateTime column as
  DateTimeOffset with Block.ReadAs.
- Tcp_003_CompositeRead walks every column through the non-generic
  composite views, with no element type known.
- Tcp_004_VariantDynamicJson names each row's alternative from
  IVariantColumn.TypeNames instead of printing bare discriminators.
- Tcp_002_CompositeWrites builds the flat array shape with
  ClickHouseTcpColumn.CreateArray and asks ClickHouseTcpTypes.CanWrite
  which CLR types a target column accepts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each of these hand-rolled something the client can now be asked for, or asserted
something it can now show.

- ITupleColumn.FieldNames is empty rather than null for an unnamed tuple, so the
  null-coalesce is dead.
- ClickHouseTcpClientOptions.ResolvedPort replaces two hand-written derivations
  of the default port, one of which hard-coded 9000 and 9440.
- ClickHouseTcpInsertOptions.DeduplicationToken replaces the raw setting.
- ClickHouseTcpServerInfo separates the three protocol revisions, so the server
  info example prints which one is in force rather than one unlabelled number.
- A query with no QueryId gets one from the client, which the log lines carry.
- OnBlockWritten shows the block sizing MaxRowsPerBlock produced, and measures
  that the client's codec governs what an insert writes: 160,023 bytes go out as
  80,083 under LZ4 and 33,850 under ZSTD.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant