Skip to content

fix(sdk): report export outages once and keep errors off agent TUIs - #86

Merged
ephraimduncan merged 3 commits into
mainfrom
fix/sdk/pi-telemetry-errors
Oct 27, 2026
Merged

ephraimduncan merged 3 commits into
mainfrom
fix/sdk/pi-telemetry-errors

Conversation

@ephraimduncan

Copy link
Copy Markdown
Contributor

During a telemetry.dev ingest outage, pi showed error blocks over its TUI. One prompt with two tool calls wrote 19 reports to stderr, and pi did not keep them. After this change, no telemetry.dev text reaches the pi terminal, and the outage is one entry in ~/.pi/agent/telemetry-dev.log. omp and opencode 1.x had the same problem and get the same correction.

[telemetry.dev] [TypeError: fetch failed] {
  [cause]: Error: connect ECONNREFUSED 127.0.0.1:4999
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:2083:16) {
    errno: -61,

One report for an outage

Each failed batch went to onError and to the console. flush() reported the same failure a second time, because the span processor rejects its flush when an export fails. The exporters of one client and the client flush now share one failure state, from createExportFailures in @telemetry-dev/otel.

stateDiagram-v2
    [*] --> Healthy
    Healthy --> Reported: export or flush fails / onError(error)
    Reported --> Reported: export or flush fails / no report
    Reported --> Healthy: a request reaches the ingest
Loading

An export without data sends no request, so it does not end the series. TelemetrySpanProcessor shares the state between its trace and metric exporters. The change applies to all SDK users. It keeps the onError contract that the SDK can merge repeated failures into one report.

Errors go to the log of the host

The pi, omp, and opencode integrations now default to logLevel: "silent". Their default onError writes to the log that each host keeps:

Host Default onError
pi Appends to telemetry-dev.log in the pi agent directory, with mode 0600. When the file is more than 5 MB, it moves to telemetry-dev.log.1, as pi does with mcp.log. pi has no logger for extensions.
omp pi.logger.error("telemetry.dev error", ...). omp keeps this file logger off the console because console output corrupts its TUI.
opencode client.app.log with the service telemetry-dev. opencode 1.18.9 draws plugin console output over its TUI. opencode 2.x runs plugins in opencode serve --service, where stdout is /dev/null.

An explicit onError or logLevel replaces the default. Outside these integrations, the SDK default stays warn on the console. The cursor integration does not change. Its daemon starts with stdio: "ignore", so its output never reaches a terminal.

sdk-logs 0.218 and the next release

The TypeError: Cannot read properties of null (reading 'exportCompleted') in pi comes from @opentelemetry/sdk-logs 0.218. After it, the pi process sends no more logs. #80 moved the catalog to ^0.223.0, but npm still has @telemetry-dev/otel 0.1.6 and @telemetry-dev/sdk 0.1.9 with ^0.218.0. The open release PR does not change the otel version, because #80 changed no file in packages/otel.

This PR changes packages/otel, so the next release publishes otel too. With otel at 0.1.7, pnpm pack writes these ranges:

@telemetry-dev/otel 0.1.7   sdk-logs ^0.223.0, otlp-transformer ^0.223.0
@telemetry-dev/sdk          sdk-logs ^0.223.0, api-logs ^0.223.0, @telemetry-dev/otel ^0.1.7

^0.1.7 excludes otel 0.1.6, and ^0.223.0 excludes 0.218.x. The sdk also imports createExportFailures, which otel 0.1.6 does not export. A new test in logs.test.ts keeps a scheduled log export open during a flush. With the 0.218 read, the test fails.

Tests

Command Packages Result
pnpm run check (vp check and the OpenAI v6 type check) all pass
vp test run --changed otel, sdk, pi, omp, opencode 77, 135, 22, 20, and 24 passed
pnpm run test all TypeScript packages pass

New and changed tests: transport.test.ts "does not retry non-retryable statuses and reports a series of failures once", otel-byo.test.ts "an outage that stops traces and metrics reaches onError once", client.test.ts "an ingest outage reaches onError once until an export gets through again", and logs.test.ts "logs keep reaching /v1/logs after a flush starts during a scheduled export". pi.test.ts, omp.test.ts, and opencode.test.ts each have one "errors go to ..." test. Each new assertion failed against a deliberate error in the owner code, for example a report for each batch, the 0.218 _flushAll read, or the warn log level.

pi session check

pi 1.4.0 loaded the worktree build with pi -e sdk/packages/pi/dist/register.mjs. Its imports resolved to the worktree packages. A local fake ingest answered each request after 700 ms. For the outage, TELEMETRY_DEV_BASE_URL pointed to a port with no server. "Before" is SDK main without this PR.

Check Before After
Outage, one prompt with two tool calls: [telemetry.dev] reports on stderr 19 0
Outage: screen captures with [telemetry.dev] text 4 of 7 0 of 7
Outage: entries in ~/.pi/agent/telemetry-dev.log no file 1
Outage through 3 prompts, Escape, /new, /reload, and exit: entries not run 1
Fake ingest: /v1/logs requests after the 5-tool prompt, /new, and /reload 7, 2, 2 7, 2, 2
Fake ingest: stderr bytes for the full run 0 0

Related: #85

During an ingest outage, each failed batch went to onError and to the
console. The client also reported the flush that failed because of a
span export. In pi, one prompt with two tool calls wrote 19 reports.

The exporters of one client and the client flush now share one failure
state. The first failure goes to onError. Later failures stay quiet
until an export reaches the ingest again. An export that sends no
request does not end the series. TelemetrySpanProcessor shares the
state between its trace and metric exporters.
@opentelemetry/sdk-logs 0.218 reads _currentExport after the export
completion sets it to null. The flush throws a TypeError, and the
processor sends no more logs. Version 0.223 keeps the in-flight export
in a local variable.

The test keeps a scheduled log export open and starts a flush. Then it
makes sure that the next log record reaches /v1/logs. With the 0.218
read, the test fails.
The pi, omp, and opencode integrations used the SDK default log level,
warn. SDK errors went to stderr. In pi, omp, and opencode 1.x, stderr is
the terminal that shows the TUI. Thus the errors showed over the TUI,
and the host did not keep them.

The integrations now default to logLevel "silent" and send errors to
the log of the host. pi has no logger for extensions, so the pi
integration appends to telemetry-dev.log in the pi agent directory. The
file has mode 0600. When the file is more than 5 MB, it moves to
telemetry-dev.log.1, as pi does with mcp.log. omp gets the errors
through pi.logger, and opencode through client.app.log. An onError or
logLevel option replaces the default.
@ephraimduncan
ephraimduncan merged commit d52effd into main Oct 27, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant