Skip to content

fix(sdk-python): report a series of failed exports once - #87

Merged
ephraimduncan merged 1 commit into
mainfrom
fix/sdk-python/export-failure-series
Oct 27, 2026
Merged

ephraimduncan merged 1 commit into
mainfrom
fix/sdk-python/export-failure-series

Conversation

@ephraimduncan

@ephraimduncan ephraimduncan commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

During an ingest outage, the Python SDK sent one on_error call and one telemetry_dev log line for each failed batch. With three flushes in each of two outages, a local run got 18 reports. With this change, the same run gets one report for each outage, as the TypeScript SDK does.

The trace, log, and metric exporters of one client use one ExportFailures state. The first failed export goes to on_error and the logger. The SDK sends no report for subsequent failures until an export reaches the ingest again. A metric collection with no data points sends no request, thus it does not stop the series. The exporters run on background threads, thus the state holds a lock. report_error runs outside the lock. As a result, a slow on_error does not stop other exporters.

TelemetrySpanProcessor uses one state for its span and metric exporters. Before this change, its span exporter sent no export failures to on_error, and its metric exporter sent them only to the logger. With this change, the two exporters send the first failure of a series to on_error and the logger, as the TypeScript processor does. A span_exporter that you supply does not report through this state.

Client.flush(), Client.shutdown(), and the force_flush() and shutdown() methods of the processor continue to call report_error. In OpenTelemetry Python, the batch processors and the periodic metric reader catch export failures, thus these calls do not raise for a failed export. The local run shows no flush or shutdown report before or after the change.

The OTLP exporters of OpenTelemetry continue to log each failed batch on the opentelemetry.exporter.otlp.proto.http loggers. This PR does not change those loggers.

Fixes #85

Related: #86

Local run

A temporary script called init() with a fake key and a base_url on a free local port with no server. It made spans, logs, and metrics and called flush() three times. Then a local HTTP server answered 200 for two flushes. Then the server stopped for three more flushes. The baseline is main (d52effd), and the candidate is this branch.

Phase on_error calls, main on_error calls, this branch telemetry_dev log lines, main telemetry_dev log lines, this branch
First outage 9 1 9 1
Recovery 0 0 0 0
Second outage 9 1 9 1

Tests

pnpm run py:check and pnpm run py:test passed for the seven Python packages. .github/scripts/python_dependency_bounds.sh sdks/python passed with OpenTelemetry 1.39.0 on Python 3.10 and 1.45.1 on Python 3.13. These tests in sdks/python/tests/test_export_on_error.py are new or changed:

test_an_ingest_outage_is_one_report_until_an_export_reaches_the_ingest_again
test_a_failure_during_a_slow_report_on_another_thread_is_quiet_and_does_not_wait
test_an_outage_that_stops_byo_traces_and_metrics_is_one_report
test_metric_export_skips_empty_collections_without_reporting_or_ending_a_failure_series

The other export tests in that file and test_quiet_interval_emits_zero_metric_posts in tests/test_metrics.py use the new constructor arguments.

During an ingest outage, each failed batch sent one more report to
on_error and to the telemetry_dev logger.

The trace, log, and metric exporters of a client use one failure state.
The first failed export goes to on_error and the logger. The SDK sends
no report for subsequent failures until an export reaches the ingest
again. A metric export with no data sends no request, thus it does not
stop the series. The state holds a lock, because the exporters run on
background threads.

TelemetrySpanProcessor uses one state for its span and metric exporters.
Its span exporter also sends export failures to on_error, as the
TypeScript processor does.

Flush and shutdown errors go to report_error as before. OpenTelemetry
catches export failures in force_flush and shutdown, thus a flush or
shutdown error is different from an export failure.
@ephraimduncan
ephraimduncan merged commit 06b75e9 into main Oct 27, 2026
10 checks passed
@aikins01 aikins01 mentioned this pull request Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python SDK: report a series of failed exports once

1 participant