feat(observability): OTel instrumentation + HTTP idle-pool options

[bdchatham]

Summary

Implements the observability design specified in sei-protocol/platform#128 (docs/designs/sei-load-observability.md), consumed by autobake and the forthcoming benchmarks namespace on harbor.

Two bundled pieces of scope, captured in one PR because they share the same code path and dependency set:

1. HTTP client idle-pool tuning (the original scope of this PR) — parameterize newHttpClient via a functional-options pattern with raised defaults (MaxIdleConns: 500, MaxIdleConnsPerHost: 50) so high-concurrency callers can tune without churning the constructor.
2. Full OTel instrumentation — new observability package, run-scope Resource, full tracing with exemplar support, W3C context propagation, optional OTLP push exporter, and new application metrics.

OTel scope — what's in

New observability package (observability/{setup.go,setup_test.go,README.md}):

• Setup(ctx, Config) installs MeterProvider + TracerProvider with a Resource built from SEILOAD_RUN_ID, SEILOAD_CHAIN_ID, SEILOAD_COMMIT_ID (+ derived commit_id_short), SEILOAD_WORKLOAD, SEILOAD_SERVICE_VERSION, and SEILOAD_INSTANCE_ID (falls back to os.Hostname() so cluster-of-seiload deployments disambiguate by pod name without explicit wiring).
• Prometheus reader (always) + OTLP gRPC exporter (gated on OTEL_EXPORTER_OTLP_ENDPOINT).
• Composite TraceContext + Baggage propagator installed globally.

Package-level meters work before Setup. sender/metrics.go and stats/metrics.go acquire their meter via var meter = otel.Meter(...) at package-init time, before observability.Setup installs the real MeterProvider. This is safe because OTel Go's global is a delegating provider — meters and instruments acquired against it forward to the real provider once SetMeterProvider runs. Documented inline so the next reader doesn't have to re-derive it.

Transport instrumentation (sender/worker.go):

• newHttpClient wraps the base transport with otelhttp.NewTransport (W3C traceparent injection + auto http.client.* metrics).

Tracing (full scope — one-shot):

• seiload.run top-level span in main.go wrapping the entire invocation.
• sender.send_tx, sender.check_receipt, sender.dial_endpoint child spans in sender/worker.go.
• All histogram emissions happen inside sampled span contexts, so trace-based exemplar filter (OTel default) attaches trace IDs automatically.

New metrics:

Metric	Type	Purpose
seiload.txs.accepted	Counter	Accepted submissions — alert denominator
seiload.txs.rejected	Counter + reason	Rejection taxonomy
seiload.http.errors	Counter + status_code	HTTP failure breakdown
seiload.tps.achieved	Observable gauge	Live TPS (producer pump wired in follow-up)
seiload.run.tps.final	Gauge	Final TPS at run end — 1 series per run via Resource
seiload.run.duration.seconds	Gauge	Run wall-clock duration
seiload.run.txs.accepted.total	Gauge	Run's accepted-txs total

Run-summary gauges are 1-series-per-run-per-metric after the Resource join — the correct cardinality shape for cross-run benchmark dashboards.

Review rounds addressed

Tri-specialist pre-merge sanity check (platform-engineer + kubernetes-specialist + OpenTelemetry-expert):

1. promhttp.Handler() silently drops exemplars. Default opts leave EnableOpenMetrics off → handler never negotiates OpenMetrics regardless of scraper Accept header → SDK-computed exemplars get stripped. Fixed with promhttp.HandlerFor(..., HandlerOpts{EnableOpenMetrics: true}). Highest-impact finding — without it, the entire exemplar path was inert.
2. Shutdown order lost final OTLP batches. Provider shutdown is the only path that flushes the PeriodicReader / BatchSpanProcessor buffers; explicit exporter shutdowns must come AFTER providers (or be omitted entirely). Reordered.
3. Cardinality on send_latency/receipt_latency. Dropped worker_id from labels — per-worker fan-out multiplied series by ~50 at realistic autobake scale, risking the 50k guardrail in platform PR #128's SeiLoadCardinalityHigh alert. Worker-level attribution lives on span attributes (accessible via exemplar pivot).
4. defer cancel() inside a for loop in Worker.watchTransactions accumulated unbounded deferred state. Explicit cancel at end-of-iteration.

@amir-deris review feedback:

• OTEL_SERVICE_VERSION → SEILOAD_SERVICE_VERSION (the env var is sei-load's own version, distinct from seiload.commit_id which names what's under test; OTEL_SERVICE_VERSION isn't an OTel spec env var anyway — the spec path is OTEL_RESOURCE_ATTRIBUTES=service.version=…).
• receipt_latency description distinguished from send_latency.
• OTel delegation pattern documented inline in sender/metrics.go so the pattern is self-explaining.
• ethclient.DialContext(dialCtx, ...) instead of ethclient.Dial so the dial honors the span context.
• go.mod bumped to 1.25.1 to match CI toolchain.
• /metrics server moved to http.Server{} + deferred Shutdown(ctx) so the scrape endpoint stays available through the OTel shutdown window.
• HttpClientOption / WithMaxIdleConns / WithMaxIdleConnsPerHost kept exported intentionally as the opt-in tuning API for a follow-up that plumbs settings.HttpMaxIdleConns[PerHost] through.

Test plan

• go build ./... clean
• go test ./... passes across all packages
• golangci-lint run ./... clean
• observability/setup_test.go covers Resource population from env, hostname fallback for service.instance.id, propagator install, OTLP gating, scheme stripping, short-commit derivation
• Existing sender tests updated for the split between newHttpTransport (inner, inspectable) and newHttpClient (outer, otelhttp-wrapped)

Post-merge validation on harbor (via autobake's first run after both PRs land):

• /metrics scrape response includes # {trace_id=...} exemplar lines (OpenMetrics content-type negotiation)
• target_info{seiload_run_id="...",seiload_commit_id_short="..."} series present in harbor Prometheus
• seiload_run_duration_seconds emitted at run end (AutobakeRunFailed alert clears)

Known follow-ups (tracked separately, NOT in this PR)

• Wire the RecordTPSSample producer from stats.Collector so seiload.tps.achieved emits datapoints (currently registered + inert).
• Platform-side: set SEILOAD_INSTANCE_ID via downward API metadata.name in autobake Job template (works today via hostname fallback, but explicit downward API is cleaner for cluster-of-seiload).
• Tempo deployment on harbor — platform#127. Until it lands, spans are created + propagated + dropped at export.
• Consider otlpmetricgrpc.WithEndpointURL vs the stripScheme helper; bump otelhttp to latest for fresher HTTP semconv.
• Thread HttpClientOption through from settings so the exported tuning API has a caller.

🤖 Generated with Claude Code

[bdchatham]

Addressed in bb74d4d — dropped the Worker.Run call-site override so we rely on the 500/50 defaults. Kept the WithMaxIdleConns/WithMaxIdleConnsPerHost options available for explicit opt-in later, and the tests still validate the options API so a follow-up that plumbs settings.HttpMaxIdleConns[PerHost] through to newHttpClient has test coverage ready.

[bdchatham]

@amir-deris thanks for the careful pass. All addressed in 95fe37c:

#	Comment	Resolution
1	OTEL_SERVICE_VERSION naming	Renamed to SEILOAD_SERVICE_VERSION throughout (observability/setup.go:38, test, PR body). You're right that OTEL_SERVICE_VERSION isn't a spec env var — OTel's path is OTEL_RESOURCE_ATTRIBUTES=service.version=…. SEILOAD_* namespace is the right home.
2	receipt_latency description	Updated to "Latency from transaction submission to receipt confirmation in seconds".
3	Stale sync.OnceValue claim in PR body	PR body rewritten. The actual pattern is package-level var meter = otel.Meter(...) relying on OTel's delegating global provider. Added an inline comment at sender/metrics.go:12 documenting the mechanism so the next reader doesn't have to re-derive it. stats/metrics.go points to that same comment.
4	HttpClientOption / WithMaxIdleConns[PerHost] exported with no external caller	Leaving exported intentionally as the opt-in tuning API for a follow-up that plumbs settings.HttpMaxIdleConns[PerHost] through to newHttpClient. Called out in the PR body's follow-up list. Tests validate the mechanism so the wiring PR lands with coverage ready.
5	ethclient.Dial → DialContext	Fixed in sender/worker.go:156 — ethclient.DialContext(dialCtx, ...) so the dial honors the span context.
6	go.mod 1.25.0 vs CI 1.25.1	go.mod bumped to 1.25.1 to match.
7	Graceful metrics server shutdown	Switched to http.Server{} with a deferred Shutdown(ctx) using a 5s timeout. Also set ReadHeaderTimeout: 5s while there. Ordered so metricsServer.Shutdown runs before obsShutdown returns (first deferred = last executed), keeping the scrape endpoint available during the OTel provider flush.

Lint + tests clean locally.