Repository navigation
fix(sdk): report export outages once and keep errors off agent TUIs - #86
Merged
Merged
Conversation
During an ingest outage, each failed batch went to onError and to the console. The client also reported the flush that failed because of a span export. In pi, one prompt with two tool calls wrote 19 reports. The exporters of one client and the client flush now share one failure state. The first failure goes to onError. Later failures stay quiet until an export reaches the ingest again. An export that sends no request does not end the series. TelemetrySpanProcessor shares the state between its trace and metric exporters.
@opentelemetry/sdk-logs 0.218 reads _currentExport after the export completion sets it to null. The flush throws a TypeError, and the processor sends no more logs. Version 0.223 keeps the in-flight export in a local variable. The test keeps a scheduled log export open and starts a flush. Then it makes sure that the next log record reaches /v1/logs. With the 0.218 read, the test fails.
The pi, omp, and opencode integrations used the SDK default log level, warn. SDK errors went to stderr. In pi, omp, and opencode 1.x, stderr is the terminal that shows the TUI. Thus the errors showed over the TUI, and the host did not keep them. The integrations now default to logLevel "silent" and send errors to the log of the host. pi has no logger for extensions, so the pi integration appends to telemetry-dev.log in the pi agent directory. The file has mode 0600. When the file is more than 5 MB, it moves to telemetry-dev.log.1, as pi does with mcp.log. omp gets the errors through pi.logger, and opencode through client.app.log. An onError or logLevel option replaces the default.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During a telemetry.dev ingest outage, pi showed error blocks over its TUI. One prompt with two tool calls wrote 19 reports to stderr, and pi did not keep them. After this change, no telemetry.dev text reaches the pi terminal, and the outage is one entry in
~/.pi/agent/telemetry-dev.log. omp and opencode 1.x had the same problem and get the same correction.One report for an outage
Each failed batch went to
onErrorand to the console.flush()reported the same failure a second time, because the span processor rejects its flush when an export fails. The exporters of one client and the client flush now share one failure state, fromcreateExportFailuresin@telemetry-dev/otel.stateDiagram-v2 [*] --> Healthy Healthy --> Reported: export or flush fails / onError(error) Reported --> Reported: export or flush fails / no report Reported --> Healthy: a request reaches the ingestAn export without data sends no request, so it does not end the series.
TelemetrySpanProcessorshares the state between its trace and metric exporters. The change applies to all SDK users. It keeps theonErrorcontract that the SDK can merge repeated failures into one report.Errors go to the log of the host
The pi, omp, and opencode integrations now default to
logLevel: "silent". Their defaultonErrorwrites to the log that each host keeps:onErrortelemetry-dev.login the pi agent directory, with mode0600. When the file is more than 5 MB, it moves totelemetry-dev.log.1, as pi does withmcp.log. pi has no logger for extensions.pi.logger.error("telemetry.dev error", ...). omp keeps this file logger off the console because console output corrupts its TUI.client.app.logwith the servicetelemetry-dev. opencode 1.18.9 draws plugin console output over its TUI. opencode 2.x runs plugins inopencode serve --service, where stdout is/dev/null.An explicit
onErrororlogLevelreplaces the default. Outside these integrations, the SDK default stayswarnon the console. The cursor integration does not change. Its daemon starts withstdio: "ignore", so its output never reaches a terminal.sdk-logs 0.218 and the next release
The
TypeError: Cannot read properties of null (reading 'exportCompleted')in pi comes from@opentelemetry/sdk-logs0.218. After it, the pi process sends no more logs. #80 moved the catalog to^0.223.0, but npm still has@telemetry-dev/otel0.1.6 and@telemetry-dev/sdk0.1.9 with^0.218.0. The open release PR does not change the otel version, because #80 changed no file inpackages/otel.This PR changes
packages/otel, so the next release publishes otel too. With otel at 0.1.7,pnpm packwrites these ranges:^0.1.7excludes otel 0.1.6, and^0.223.0excludes 0.218.x. The sdk also importscreateExportFailures, which otel 0.1.6 does not export. A new test inlogs.test.tskeeps a scheduled log export open during a flush. With the 0.218 read, the test fails.Tests
pnpm run check(vp checkand the OpenAI v6 type check)vp test run --changedpnpm run testNew and changed tests:
transport.test.ts"does not retry non-retryable statuses and reports a series of failures once",otel-byo.test.ts"an outage that stops traces and metrics reaches onError once",client.test.ts"an ingest outage reaches onError once until an export gets through again", andlogs.test.ts"logs keep reaching /v1/logs after a flush starts during a scheduled export".pi.test.ts,omp.test.ts, andopencode.test.tseach have one "errors go to ..." test. Each new assertion failed against a deliberate error in the owner code, for example a report for each batch, the 0.218_flushAllread, or thewarnlog level.pi session check
pi 1.4.0 loaded the worktree build with
pi -e sdk/packages/pi/dist/register.mjs. Its imports resolved to the worktree packages. A local fake ingest answered each request after 700 ms. For the outage,TELEMETRY_DEV_BASE_URLpointed to a port with no server. "Before" is SDKmainwithout this PR.[telemetry.dev]reports on stderr[telemetry.dev]text~/.pi/agent/telemetry-dev.log/new,/reload, and exit: entries/v1/logsrequests after the 5-tool prompt,/new, and/reloadRelated: #85