test(perf): compare Trino worker size and execution capacity

[bill-ph]

The frozen benchmark currently fixes Trino at three 1-CPU/4-GiB workers. Add a manual trino_perf_shape choice to scenario-dev so the same workload can distinguish worker-size effects from additional execution capacity.

Shape	Workers	CPU / memory per worker	Total CPU / memory
baseline	3	1 / 4 GiB	3 / 12 GiB
large	1	3 / 12 GiB	3 / 12 GiB
scaleout	6	1 / 4 GiB	6 / 24 GiB
large-scaleout	2	3 / 12 GiB	6 / 24 GiB

Heap and query-memory limits scale with the selected shape. The coordinator resources, Duckgres resources, Trino queries, cache policy, and iteration counts are unchanged. Scheduled runs retain the full baseline. Experiments upload resource/image/mode provenance with their results and skip baseline historical publishing. Invalid selections fail before cloud mutations; separate deploy/test invocations reject conflicting saved provenance, while teardown remains available.

Choose trino_perf_shape=all to run all four configurations in one workflow dispatch on this branch. Images build once, and a sequential matrix runs each shape in a separate temporary stack with its own credentials and timeout. Each job attempts teardown before the next job starts; failures do not cancel the remaining shapes. This is one measurement round with the existing warmup and four measured iterations, not repeated deployment rounds.

The final comparison job combines per-query Trino medians, baseline-relative workload speedup, and allocated CPU-budget efficiency in the workflow summary and a downloadable Markdown artifact. Shape artifacts have distinct names and include deployment/scenario/teardown outcomes. Missing, failed, inconsistent, or partial results are marked incomplete and fail the comparison job instead of producing a misleading comparison. The entire all run, including its baseline member, is excluded from historical publishing.

The baseline runs the full cross-engine comparison (42 warmups and 168 measured queries). Each nonbaseline shape, including individual dispatches, measures Trino only (7 warmups and 28 measured queries), retaining identical Trino queries and iterations and the same provisioning, data setup, validation, and cleanup. Nonbaseline shapes skip Athena configuration and the runner's Athena Pod Identity association. The harness derives and records perf_mode; unused Athena fields and requirements are removed before environment validation/template resolution. The combined report compares only Trino and accepts both the new mixed workloads and older full-workload artifacts.

The perf README documents one-dispatch and individual-shape usage, comparison interpretation, and cleanup/recovery. Rerun the whole workflow for a complete new comparison; artifacts from different attempts are not combined. The latest update applies to future dispatches; it does not cancel, restart, or change an in-flight benchmark. Execution profiling is not added.

Example:

gh workflow run scenario-dev.yml --ref codex/trino-perf-shape-experiments \
  -f scenario=posthog_frozen_perf \
  -f trino_perf_shape=all

Validation:

• Test-first coverage for all four rendered deployments, replica counts, memory settings, invalid selections, provenance, artifact collection, teardown, and workflow publication isolation.
• just test-scenario and the workflow/publisher script tests pass.
• The complete tests/mw-dev harness passes with the largest experimental shape exported, verifying workflow environment isolation.
• bash -n tests/mw-dev/run.sh and git diff --check pass.
• Planner tests cover single/all/default selections and invalid input; workflow tests enforce sequential jobs, shared builds, distinct identities/artifacts, cleanup, and historical publication isolation.
• Summary/CLI tests cover medians and efficiency calculations, malformed/missing/duplicate/failed artifacts, measurement completeness, and dataset/query/image consistency.
• Trino-only runner tests verify missing Athena credentials are allowed, unrelated modes are rejected, setup/validation/cleanup stay intact, and the real shared catalog executes exactly 7 warmups plus 28 measurements without constructing other engine drivers. Mixed-workload summary tests cover the new mode and legacy artifacts.
• Container entrypoint preflight tests verify Trino-only reaches the runner without Athena credentials, while baseline still requires them and invalid modes fail before execution.
• Actionlint v1.7.7 passes for the updated workflow.
• just lint reports six existing SA4023 diagnostics at three unchanged control-plane call sites on this macOS toolchain. The identical failures reproduce on base commit d7e852d8; lint restricted to changes reports zero issues.

Benchmark results

Four-shape run, September 8, 2026, at 07e3c6d1. All four shapes completed successfully with zero query errors and successful teardown. This run predates the Trino-only optimization in 1fd5cfcc: every shape ran the full cross-engine workload (42 warmups and 168 measured executions). The results below compare only the identical Trino queries, with one warmup and four measured iterations per query. Dataset and Trino image provenance matched across shapes.

Latency and scaling

Workload latency is the sum of the seven measured query medians, not total scenario duration or concurrent-query throughput. Resource totals below cover execution workers; the unchanged coordinator is additional.

Shape	Total worker CPU / memory	Workload latency	Baseline speedup	Allocated CPU-budget efficiency
baseline: 3 small workers	3 CPU / 12 GiB	608.0s	1.00x	100.0%
large: 1 large worker	3 CPU / 12 GiB	604.3s	1.01x	100.6%
scaleout: 6 small workers	6 CPU / 24 GiB	320.0s	1.90x	95.0%
large-scaleout: 2 large workers	6 CPU / 24 GiB	289.7s	2.10x	104.9%

Efficiency is speedup divided by the worker CPU-budget increase; it is not measured CPU utilization. Values slightly above 100% are not evidence of reliably superlinear scaling from this single deployment round.

Median query latency, in seconds:

Query	baseline	large	scaleout	large-scaleout
Events by name	92.504	88.941	50.420	45.015
Events count, one day	0.151	0.156	0.137	0.152
Events by day	87.201	74.996	47.212	39.046
Exact distinct persons in events	370.039	381.290	192.063	177.084
Events total	0.187	0.182	0.168	0.169
Persons by day	57.733	58.572	29.823	28.085
Persons total	0.189	0.189	0.183	0.183

Observations:

• Consolidating the same 3-CPU/12-GiB budget improved workload latency by only 0.6%; distinct became slightly slower. This weakens memory fragmentation or inter-worker exchange as the dominant explanation for the six-minute distinct query.
• Doubling the execution budget approximately halved the expensive queries with either worker size. Simple count queries barely changed.
• Two large workers were fastest, with 9.5% lower workload latency than six small workers. Treat that topology advantage as provisional: unchanged uncached Duckgres controls also ran about 7% faster in that deployment. One deployment per shape cannot isolate placement/time variability.
• Even the fastest Trino shape took 289.7s versus 161.6s for uncached Duckgres in the same deployment: about 1.8x slower despite twice the execution CPU and memory. Cached Duckgres took 20.0s. These are distinct cache policies, not interchangeable baselines.
• Exact distinct still accounts for about 61% of Trino workload latency, making it the highest-impact query to investigate.

Historical CPU, memory, and network inspection

Historical cAdvisor metrics were recovered after teardown and matched to all four measured distinct executions per shape. CPU and network figures use one-minute rates sampled every 15 seconds, restricted to query start +60 seconds through query end -15 seconds to exclude preceding-query overlap. Samples overlap and are not independent trials. Effective CPU quotas were checked against container quota/period metrics.

Shape	Allocated CPU	Mean CPU used	Used / allocated	Approx. aggregate inbound traffic
baseline	3	1.65 cores	55%	81 MiB/s
large	3	1.43 cores	48%	73 MiB/s
scaleout	6	3.35 cores	56%	161 MiB/s
large-scaleout	6	3.01 cores	50%	161 MiB/s

• All worker pods were active, with reasonably balanced CPU usage; this was not an idle-worker-pool pattern.
• CPU quotas were not continuously saturated. Baseline workers averaged throttling in 14-19% of CFS scheduling periods, versus 2.4% for the single large worker, without a distinct-query speedup. This measures periods experiencing throttling, not percentage of query time lost.
• Sampled container working sets remained below limits: baseline workers peaked at at most 2.34 GiB against 4 GiB; the single large worker at 7.22 GiB against 12 GiB. This does not rule out JVM GC or query-memory pressure.
• Coordinator CPU averaged roughly 0.04-0.06 cores during these windows, with sampled one-minute rates below 0.09 cores.
• Inbound traffic approximately doubled with the execution budget. This is pod network throughput, not measured network capacity or S3-specific physical input: it includes exchange and other traffic. Throughput can rise because processing advances faster, so correlation alone does not establish an I/O bottleneck.

CPU-dependent concurrency is a confounding factor

The pinned Trino build defaults task.max-worker-threads to Runtime.availableProcessors() * 2 and task.min-drivers to twice that value. Its enabled thread-per-driver scheduler uses these for execution slots and its global leaf-driver target. Pinned task configuration, scheduler implementation.

Assuming the JVM detected the configured quotas, the expected aggregate defaults are:

Shape	Execution slots	Leaf-driver target
baseline	6	12
large	6	12
scaleout	12	24
large-scaleout	12	24

These are source-derived expectations, not captured runtime settings or counts of simultaneous S3 requests. They provide a plausible explanation for similar throughput at equal CPU budgets and approximately doubled throughput at the larger budget: the experiment changed CPU, memory, and default execution concurrency together.

Conclusion: the evidence does not establish sustained CPU-quota saturation as the bottleneck. Read/scan concurrency and waiting between CPU bursts deserve investigation. Neither a hard network limit nor a specific GC, exchange, or hashing bottleneck has been demonstrated.

Recommended next experiment: keep CPU, memory, worker shape, dataset, and cache policy fixed; explicitly increase execution/leaf-driver concurrency. Capture effective settings, running/blocked drivers, operator CPU and blocked time, physical-read and exchange statistics, GC, and container telemetry. If throughput improves while spare CPU is used, concurrency tuning may recover some of the scaling gain without doubling resources. This experiment has not been run.

The completed run did not retain Trino server query IDs, operator profiles, or GC statistics. Its server_metrics.prom artifact contains harness latency/request/row metrics, not Trino server telemetry. A targeted diagnostic execution is still required to identify the precise bottleneck.

[github-actions]

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area	Added	Changed	Deleted
Test files	6	8	0
E2E/journey files	0	0	0
Workflow files	0	1	0

Signals

• Test cases: +25 / -0
• Assertions: +107 / -0
• Skips or known failures added: 0
• Workflow continue-on-error added: 1
• Workflow path filters added: 0
• Test commands removed from justfile: 0
• E2E/journey retry lines added: 0

Coverage risk: likely reduced

Warnings

• CI/test runner behavior changed (likely reduced)
  • 1 continue-on-error line(s) added

[bill-ph]

Closing this experiment without merging its shape-selection machinery. The completed benchmark results and analysis remain in the description. The follow-up will start from main and explicitly increase Trino execution/leaf-driver concurrency at the existing worker resource budget to test the read-concurrency hypothesis.