feat(hub): add explicit telemetry foundation

hub/.env.example

@@ -4,6 +4,10 @@ PORT=3001
 # API logging threshold: trace, debug, info, warn, error, fatal, or silent.
 LOG_LEVEL=info
 
+# Explicit spans only. See docs/telemetry.md before enabling export.
+OTEL_TRACES_EXPORTER=none
+# OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:4318/v1/traces
+
 # Independent worker concurrency. Both default to one (supported range: 1-100).
 ON_DEMAND_JOB_CONCURRENCY=1
 SCHEDULED_JOB_CONCURRENCY=1

hub/AGENTS.md

@@ -6,6 +6,7 @@ Hub is being rebuilt in TypeScript. The app starts locally, but its product feat
 - [Code architecture conventions](docs/backend-architecture.md) define repositories, integrations, utilities, domains, and allowed imports.
 - [Coding conventions](docs/coding-conventions.md) explain frontend layers, formatting, and interface rules.
 - [Database conventions](docs/database.md) define Prisma files, model section order, UUID keys, timestamps, and database naming.
+- [Telemetry](docs/telemetry.md) defines explicit spans, safe attributes, operator configuration, and API and worker lifetime.
 - [Development](docs/development.md) and [testing](docs/testing-and-operations.md) own local and CI commands.
 - [Storybook](docs/storybook.md#ui-changes-and-pull-request-evidence) owns presentation coverage, cleanup, and PR screenshots.
 

hub/docs/README.md

@@ -8,6 +8,7 @@ Hub is being rebuilt. The [Hub overview](../README.md) describes what the baseli
 - [Development](development.md): local API and frontend setup.
 - [Database](database.md): Prisma schema conventions, migrations, and the shared client.
 - [Logging](logging.md): application events, automatic request records, and payload safety.
+- [Telemetry](telemetry.md): explicit OpenTelemetry spans, safe attributes, and process lifetime.
 - [Background jobs](background-jobs.md): PostgreSQL queues, workers, schedules, setup, and test isolation.
 - [WebSocket updates](websockets.md): local broadcasts, subscriptions, access hooks, and connection limits.
 - [Testing and operations](testing-and-operations.md): baseline checks and their limits.

hub/docs/architecture.md

@@ -22,6 +22,8 @@ The [Pino integration](logging.md) writes structured application and HTTP logs t
 
 The [background-job foundation](background-jobs.md) uses BullMQ with PostgreSQL and separate on-demand and scheduled worker processes. Setup migrates BullMQ's own schema independently of Prisma. Product processors and schedules are not registered yet.
 
+The [telemetry integration](telemetry.md) provides explicit server-side OpenTelemetry spans with optional operator-configured OTLP export. API and worker entrypoints own startup and flush telemetry after draining active work. Automatic instrumentation and product analytics are not implemented.
+
 The [WebSocket foundation](websockets.md) attaches to the API's HTTP server and broadcasts to local channel subscribers without persistence. The application rejects upgrades until access checks are supplied. Product channels, client recovery, and communication between processes are not implemented.
 
 ## Frontend

hub/docs/development.md

@@ -42,6 +42,8 @@ Vite also forwards WebSocket upgrades under `/api`. The `/api/realtime` endpoint
 
 The API reads an optional `PORT` environment variable, defaulting to 3001. If you change it, update the local Vite proxy target to match. The editor forwards ports 3001, 5173, and 6006 from the devcontainer. API startup reads `.env` when present; Compose's environment takes precedence.
 
+On `SIGINT` or `SIGTERM`, the API stops accepting requests and allows active requests up to 30 seconds to finish, then flushes [optional telemetry](telemetry.md) within five seconds. No automatic request tracing is installed.
+
 ## Background workers
 
 The API and frontend do not start background consumers. After starting PostgreSQL, run `npm run setup` once, then start `npm run worker:on-demand` and `npm run worker:scheduled` in separate terminals. Both use `DATABASE_URL`; their concurrency settings are in `.env.example`.

hub/docs/telemetry.md

@@ -0,0 +1,99 @@
+# Telemetry
+
+Hub operators can send explicitly recorded server-side traces to an OpenTelemetry Protocol (OTLP) recipient of their choice. Tracing is disabled by default. Configuration belongs to the operator's process environment, not to an API endpoint or browser setting.
+
+The foundation provides two functions under `server/integrations/opentelemetry/`: `startTelemetry()` and `withTelemetrySpan()`. It does not automatically trace requests, database queries, outbound HTTP, or jobs. The health endpoint does not produce spans. Browser tracking and durable product analytics are not implemented.
+
+## Configure export
+
+Set these variables in the API and worker environments, or in a private local `.env`:
+
+```sh
+OTEL_TRACES_EXPORTER=otlp
+OTEL_SERVICE_NAME=replica-hub
+OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:4318/v1/traces
+OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
+OTEL_TRACES_SAMPLER=parentbased_traceidratio
+OTEL_TRACES_SAMPLER_ARG=1
+```
+
+The recipient can be a provider that accepts OTLP over HTTP/protobuf or an operator-managed OpenTelemetry Collector that forwards to other platforms. Use HTTPS for remote recipients. This integration does not include provider-specific SDKs, configure a Collector, or prove that a provider displays accepted traces.
+
+| Variable | Contract |
+| --- | --- |
+| `OTEL_TRACES_EXPORTER` | `none` is the default; `otlp` enables export. Other values fail startup. |
+| `OTEL_SDK_DISABLED` | `true`, ignoring case, disables telemetry regardless of other settings. With export enabled, other nonempty values must be `false`. |
+| `OTEL_SERVICE_NAME` | Defaults to `replica-hub`. Use a static service label of at most 100 printable characters, not secrets or personal data. |
+| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Explicit absolute HTTP(S) URL with a request path. Credentials, query strings, and fragments are rejected. |
+| `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` | Only `http/protobuf` is supported. |
+| `OTEL_EXPORTER_OTLP_TRACES_HEADERS` | Optional comma-separated URL-encoded `key=value` authentication headers. For example, `Authorization=Bearer%20example`. Keep real values private. |
+| `OTEL_TRACES_SAMPLER` | `parentbased_traceidratio` is the default. `always_on` and `always_off` are also supported. Other samplers fail startup. |
+| `OTEL_TRACES_SAMPLER_ARG` | A finite fraction from `0` through `1`, defaulting to `1`, controls the root sampling ratio. It is validated but has no effect for `always_on` and `always_off`. |
+
+Common `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_PROTOCOL`, and `OTEL_EXPORTER_OTLP_HEADERS` settings are also accepted. Trace-specific settings take precedence, even when empty. The common endpoint is a base URL; Hub appends `/v1/traces` to its path. Enabled export requires an explicit endpoint. Disabled telemetry does not validate unused export settings or create an exporter.
+
+Hub sets its own resource, sampler, batching, and context policies rather than loading arbitrary SDK environment configuration. `OTEL_RESOURCE_ATTRIBUTES`, other sampler names, automatic instrumentation, and custom propagation are not part of this contract. Do not infer support for every standard OpenTelemetry variable.
+
+## Process lifetime
+
+`startTelemetry(process.env, logger)` validates enabled configuration and registers one tracer provider and asynchronous context manager per process. The logger is supplied by composition code; this integration does not import the Pino integration. A second enabled initialization before shutdown is rejected.
+
+The API and worker entrypoints already call it before accepting requests or starting consumers. Static imports are sufficient because this foundation does not patch dependencies. `createApp()` does not initialize global telemetry or install signal handlers.
+
+The returned `{ shutdown() }` belongs to the entrypoint, not to domains or request handlers. The entrypoint waits for API requests and WebSockets, or active worker jobs, to drain, then awaits shutdown in `finally`. API and worker drain limits are 30 seconds; telemetry shutdown adds at most five seconds. Operators must allow both windows before terminating a process. Startup failures and shutdown timeouts exit unsuccessfully.
+
+Shutdown flushes queued spans and closes the SDK. It is idempotent, including when telemetry is disabled. Normal batching holds at most 256 spans, exports at most 64 per batch, and attempts export at one-second intervals with a two-second export timeout. Queue saturation or failed delivery can lose spans. Export failures produce safe static logs, not errors in the feature being traced. Shutdown failure is reported separately and rejects the shutdown call.
+
+## Trace a ported feature
+
+Call `withTelemetrySpan()` around one meaningful operation in a route or domain. From a function directory under `server/domains/build/`, wrapping an existing `importBuild()` operation would look like:
+
+```ts
+import { withTelemetrySpan } from "../../../integrations/opentelemetry/with-telemetry-span/index.ts"
+
+return withTelemetrySpan(
+  "build.import",
+  { "replica.build.id": buildId },
+  () => importBuild(buildId),
+)
+```
+
+The callback runs once. The helper awaits it, returns its original result, and finishes the span on success or failure. A failure marks the span as failed and rethrows the exact original value. It does not attach exception messages, stacks, or thrown objects. A failed operation is not converted into a successful result.
+
+Nested calls become parent and child spans across `await`; concurrently running operations keep their contexts separate. Disabled or unsampled telemetry still runs the callback normally. Completed spans receive success or failure status without additional diagnostic payloads.
+
+Use static operation names such as `build.import`, not record IDs, branch names, URLs, or user input. Names contain at most 100 lowercase ASCII letters, digits, and separating `.`, `_`, or `-` characters, beginning with a letter. Format validation cannot determine whether a name is genuinely static; reviewers must check the call site.
+
+Attributes accept only these optional UUID strings:
+
+- `replica.repository.id`
+- `replica.prototype.id`
+- `replica.build.id`
+- `replica.thread.id`
+- `replica.comment.id`
+
+The integration validates names and attributes before invoking the callback, even when tracing is disabled. Unknown keys and malformed IDs reject with a safe error instead of silently recording unexpected data. An invalid instrumentation call is a programming error, not an exporter outage. To add an attribute, update the integration's type, runtime schema, this guide, and privacy coverage together.
+
+Do not pass request or response bodies, headers, cookies, credentials, SQL, job arguments, review text, profile fields, or raw exceptions. IDs are pseudonymous, not anonymous. Exported data needs operator-controlled access and retention. Pino redaction does not sanitize traces. Use this wrapper rather than directly modifying SDK spans or importing the OpenTelemetry API in feature code.
+
+## Architecture and future ports
+
+Keep configuration and telemetry-specific helpers inside `integrations/opentelemetry/`. Use one function directory with colocated types and tests; do not add a barrel export, telemetry domain, or telemetry utilities outside the integration.
+
+Follow the [backend import boundaries](backend-architecture.md#import-boundaries). Domains and routes may call the integration. Repositories and other integrations must receive a tracing callback from composition code rather than import OpenTelemetry integration functions or types. Business decisions about what to trace stay with the feature.
+
+There is no request-header extraction, outbound context injection, baggage handling, or job-context transport yet. Each independently started operation is a root unless it already has an active in-process parent. Adding cross-process propagation requires a real feature boundary and an explicit policy for trusted incoming context; never assume client-supplied trace headers are trusted. Automatic instrumentation may need earlier initialization or ESM loader support and must be verified for the selected libraries before changing startup.
+
+Sampled, best-effort traces explain execution. They are not durable usage counts, security audits, or delivery guarantees. Port product analytics separately when its domains, access checks, transactional events, and delivery requirements exist.
+
+## Verify a port
+
+Add a colocated test that proves the operation's span, safe attributes, result, and failure behavior. Keep the actual SDK and asynchronous context handling real; replace only export transport when network delivery is not under test. Release the runtime and any test-owned global providers or context managers after each case.
+
+From `hub/`, run the foundation checks with:
+
+```sh
+npm test -- server/integrations/opentelemetry server/run-api
+```
+
+For an actual recipient check, configure a local OTLP receiver and invoke an instrumented operation. Visiting the health route alone cannot verify export. A receiver accepting a batch does not establish that an analytics or APM report contains it.

hub/docs/websockets.md

@@ -4,6 +4,8 @@ Hub can send live updates to the client app and Copilot plugin through the `ws`
 
 The application attaches `/api/realtime` to the same HTTP server as Express and passes its broadcast function to `createApp()`. Vite forwards WebSocket upgrades under `/api` during development. The endpoint rejects connections until application access checks are supplied. There are no comment channels or product events yet.
 
+`runApi()` owns the shared server and signal handlers. It waits for HTTP requests and WebSockets to close before the entrypoint flushes [telemetry](telemetry.md). A failure closing either transport is reported after both cleanup operations settle, unless the overall API shutdown deadline expires.
+
 ## Server operations
 
 `attachRealtimeServer({ server, path, logger, limits?, authenticate?, authorizeSubscription? })` receives an existing Node HTTP server and an injected Pino logger. It does not start listening or open another port. It returns an instance with two operations: