Repository navigation
fix(sdk-python): report a series of failed exports once - #87
Merged
Merged
Conversation
During an ingest outage, each failed batch sent one more report to on_error and to the telemetry_dev logger. The trace, log, and metric exporters of a client use one failure state. The first failed export goes to on_error and the logger. The SDK sends no report for subsequent failures until an export reaches the ingest again. A metric export with no data sends no request, thus it does not stop the series. The state holds a lock, because the exporters run on background threads. TelemetrySpanProcessor uses one state for its span and metric exporters. Its span exporter also sends export failures to on_error, as the TypeScript processor does. Flush and shutdown errors go to report_error as before. OpenTelemetry catches export failures in force_flush and shutdown, thus a flush or shutdown error is different from an export failure.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During an ingest outage, the Python SDK sent one
on_errorcall and onetelemetry_devlog line for each failed batch. With three flushes in each of two outages, a local run got 18 reports. With this change, the same run gets one report for each outage, as the TypeScript SDK does.The trace, log, and metric exporters of one client use one
ExportFailuresstate. The first failed export goes toon_errorand the logger. The SDK sends no report for subsequent failures until an export reaches the ingest again. A metric collection with no data points sends no request, thus it does not stop the series. The exporters run on background threads, thus the state holds a lock.report_errorruns outside the lock. As a result, a slowon_errordoes not stop other exporters.TelemetrySpanProcessoruses one state for its span and metric exporters. Before this change, its span exporter sent no export failures toon_error, and its metric exporter sent them only to the logger. With this change, the two exporters send the first failure of a series toon_errorand the logger, as the TypeScript processor does. Aspan_exporterthat you supply does not report through this state.Client.flush(),Client.shutdown(), and theforce_flush()andshutdown()methods of the processor continue to callreport_error. In OpenTelemetry Python, the batch processors and the periodic metric reader catch export failures, thus these calls do not raise for a failed export. The local run shows no flush or shutdown report before or after the change.The OTLP exporters of OpenTelemetry continue to log each failed batch on the
opentelemetry.exporter.otlp.proto.httploggers. This PR does not change those loggers.Fixes #85
Related: #86
Local run
A temporary script called
init()with a fake key and abase_urlon a free local port with no server. It made spans, logs, and metrics and calledflush()three times. Then a local HTTP server answered 200 for two flushes. Then the server stopped for three more flushes. The baseline ismain(d52effd), and the candidate is this branch.on_errorcalls,mainon_errorcalls, this branchtelemetry_devlog lines,maintelemetry_devlog lines, this branchTests
pnpm run py:checkandpnpm run py:testpassed for the seven Python packages..github/scripts/python_dependency_bounds.sh sdks/pythonpassed with OpenTelemetry 1.39.0 on Python 3.10 and 1.45.1 on Python 3.13. These tests insdks/python/tests/test_export_on_error.pyare new or changed:The other export tests in that file and
test_quiet_interval_emits_zero_metric_postsintests/test_metrics.pyuse the new constructor arguments.