feat: PPL OpenTelemetry tracing integration - #5708
Draft
penghuo wants to merge 1 commit into
Draft
Conversation
Contributor
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 3efa12e.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
penghuo
force-pushed
the
feat/ppl-otel-tracing
branch
2 times, most recently
from
August 20, 2026 15:53
be86ef0 to
3bdabc0
Compare
penghuo
force-pushed
the
feat/ppl-otel-tracing
branch
from
August 20, 2026 16:44
3bdabc0 to
85ae0c4
Compare
Add distributed tracing spans across the PPL Calcite query execution
pipeline, reusing the existing query-profiling boundaries so a phase
means the same thing whether you read it from `profile` output or from a
trace. When the telemetry feature flag is off, NoopTracer is used with
near-zero overhead.
Spans emitted per PPL query:
opensearch.query (CLIENT, root)
-> opensearch.query.prepare (INTERNAL, trace-only)
-> opensearch.query.analyze (INTERNAL)
-> opensearch.query.optimize (INTERNAL)
-> opensearch.query.execute (INTERNAL)
Root-span attributes follow OTel DB semantic conventions:
db.system.name, db.query.type=ppl, db.query.id (UUID),
db.operation.name (EXECUTE/EXPLAIN), and db.query.text (anonymized via
PPLQueryDataAnonymizer). PPLService hands the anonymized text to a
Consumer supplied by the transport action, which owns the Span.
Module boundaries: core and ppl must not depend on the opensearch
module, so neither references org.opensearch.telemetry. Instead core
gains a PhaseListener interface and a ProfileScope helper that times one
boundary and feeds both a ProfileMetric and — when a listener is
installed — a span. The opensearch module supplies TracingPhaseListener,
backed by Tracer, which TransportPPLQueryAction installs at
construction. Tracer reaches the transport action via the node injector
and is bound into the plugin's child injector by OpenSearchPluginModule.
Profile metric definitions are preserved from main -- the golden rule
is: do not be misled by internal function names, follow the profile's
existing semantics. Each phase's MetricName marker is placed where main
already had its manual System.nanoTime() timing, just wrapped in a
ProfileScope so the trace span shares the same boundary.
ANALYZE - QueryService.executeWithCalcite: analyze() + convert +
CalciteToolsHelper.optimize() (Calcite rule pass).
OPTIMIZE - CalciteToolsHelper.OpenSearchRelRunners.run: shuttle pass
(LogicalTableScan -> BindableTableScan) + prepareStatement.
The name looks like it should mean "compile"; per main's
profile it is called OPTIMIZE. Left as-is.
EXECUTE - OpenSearchExecutionEngine.execute: executeQuery() +
buildResultSet(). Materialize is inside EXECUTE, matching
main; there is no separate MATERIALIZE phase.
FORMAT - unchanged, populated by QueryService.analyze onResponse.
The phase span name is derived once from MetricName.name().toLowerCase
(ROOT) -- the same rule QueryProfile uses -- so a span name and its
profile phase key cannot drift apart. To keep the trace tree flat under
root, OpenSearchRelRunners.run (OPTIMIZE) is called BEFORE the EXECUTE
ProfileScope opens in OpenSearchExecutionEngine.execute, so OPTIMIZE
closes as a sibling of EXECUTE rather than nesting inside it.
Async lifecycle: the root Span is created on the transport thread and
ended on the worker thread by core's TraceableActionListener; the
SpanScope closes synchronously on the transport thread once
execute()/explain() returns.
The EXECUTE scope closes (records its metric, ends its span) BEFORE
listener.onResponse fires: onResponse drives the downstream pipeline
that snapshots the profile via QueryProfiling.finish(), so recording
after that snapshot would drop the metric while still emitting the span
-- the exact profile/trace divergence this design prevents.
Two hierarchy fixes verified against the observability stack:
* The analyze endpoint (also profile:true) re-runs the compile pass on
the caller thread to capture the physical plan text via a Calcite
hook. That produced a second `compile` span per query while the
profile stayed accurate because QueryProfiling.noop() was set. Added
ProfileScope.withSuppression, a thread-local mute for BOTH metric
and span; wrapped the re-run in it. One compile span per query now.
* The explain path opened its ANALYZE ProfileScope around the entire
operation, including the call into executionEngine.explain which
opens its own compile span. That made `compile` a grandchild of the
root instead of a sibling. Narrowed the analyze scope to just the
parse-to-plan work; explain traces now show the flat root ->
{analyze, compile} shape.
Also added a trace-only `opensearch.query.prepare` span in PPLService,
covering parse + AST build + anonymize on the transport thread before
the sql-worker hop. Without it, cold-start ANTLR grammar init (several
hundred ms) showed up as an unlabeled gap between the root span start
and analyze. It is trace-only because QueryProfiling isn't active on
the transport thread yet -- the profile output is unaffected. Verified
against the observability stack: post-prepare gap dropped from tens to
hundreds of ms cold and 2-8ms warm to <1ms in both cases; the prepare
span itself now shows the actual parse cost (~350ms cold, 1-3ms warm).
Adds a `-DenableTelemetry` toggle to the plugin `run` gradle task that
installs the telemetry-otel plugin from a local OpenSearch source tree
(path required via `-DtelemetryOtelSrc=<path>`), enables the tracer,
sets sampling to 100%, and wires OtlpGrpcSpanExporter to
http://localhost:4317 for the observability-stack docker-compose.
Docs: local-run runbook under docs/dev/.
Tested end-to-end against opensearch-project/observability-stack. Spans
land in Trace Analytics nested under OpenSearch's own REST and transport
spans, with the execute span parenting the downstream shard-search
spans, which confirms context propagation survives the fork-thread
handoff. Per-phase span durations match the profile output within
System.nanoTime() noise, and failures record status.code=2 with the
exception message on the root span.
Signed-off-by: Peng Huo <penghuo@gmail.com>
penghuo
force-pushed
the
feat/ppl-otel-tracing
branch
from
August 20, 2026 16:51
85ae0c4 to
3efa12e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add distributed OpenTelemetry tracing across the PPL Calcite query execution pipeline, reusing the existing query-profiling boundaries so a phase means the same thing whether you read it from
profileoutput or from a trace. When the telemetry feature flag is off,NoopTraceris used with near-zero overhead.Spans emitted per PPL query
Root-span attributes follow OTel DB semantic conventions:
db.system.name=opensearchdb.query.type=ppldb.query.id(UUID)db.operation.name(EXECUTE/EXPLAIN)db.query.text— anonymized PPL queryRelated Issues
Resolves #5300
Check List
--signoffor-s.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.