Feat: Lineage telemetry plugin — two facts-only spans per exchange

[JoshSag]

What it does

Adds a lineage-telemetry plugin that emits two facts-only OTel spans per
HTTP exchange crossing the sidecar:

• a request span the moment the sidecar sees the request,
• a response span at stream end, carrying the outcome,
• joined by lineage.exchange.id (the request span's own id).

Span names are {self_id} {protocol} {operation}, with the response span
appending response. The facts are lineage.role, lineage.direction,
lineage.self.id, lineage.peer.host, lineage.protocol,
lineage.principal.{sub,client}, lineage.outcome, lineage.denied_by,
lineage.parent.source, plus url.scheme and url.path. With
capture_io: true the parsed message content rides along as input.value /
output.value, so a trace viewer shows the actual A2A message, MCP tool
arguments or LLM prompt inline.

The producer records facts, not meaning. No hop classification, no trust
vocabulary, no identity guessing. Anything interpretive — what kind of hop this
is, which entity it belongs to — lives in whatever consumes the spans. That
separation is the design, and it is why the plugin stays small and the
vocabulary can change without touching Go.

Configuration

Six keys, decoded with DisallowUnknownFields so a typo is a boot error
rather than a silent default:

- name: lineage-telemetry
  config:
    otel_endpoint: "otel-collector.rossoctl-system.svc.cluster.local:4317"
    capture_io: true
    self_id: "weather-service"

capture_io is off by default — payloads may contain user messages and
model output. self_id falls back to self_id_file, defaulting to
/shared/client-id.txt, the operator-mounted credential. bypass_paths and
bypass_hosts keep agent-card discovery, health probes and telemetry backends
out of the graph by default.

Cross-pod parenting rides one tracestate member

Each sidecar parents an exchange from the dg-parent tracestate member when
present (else the wire parent), and re-stamps that member with its own request
span id. The forwarded traceparent is never modified — an app with its own
tracing keeps its chain intact toward its own backend. No mechanism guesses a
parent: missing data degrades to an explicit unknown or fails loudly.

The wire format is specified at v1.5.3 in a document we maintain alongside
the consumer, with a consumer test suite pinned to it. Every attribute name,
its conditional emission, and the parenting rule are contract.

Why lane 1 matters

The plugin writes its tracestate stamp into pctx.Headers. In extproc and
forwardproxy as they stand today, that write never reaches the wire —
only Authorization is forwarded. The stamp dies in the pipeline context, the
next hop sees no dg-parent, and the reconstructed graph degrades into
phantom-root forests: an exchange that should derive as 2 interactions under 1
root came out as 3 interactions under 2 roots when measured.

So: lane 1 is a prerequisite for this plugin to be useful, not for it to
build. The diffs never collide — only review order matters. If lane 1 is not
wanted, this plugin still works correctly in reverseproxy mode, which already
has the header sync.

Opt-out is a build tag you control

The plugin registers through your one-tag-file-per-plugin convention
(cmd/authbridge-{envoy,proxy}/plugins_lineage.go, 5 lines each). A build with
exclude_plugin_* tags links none of the plugin and none of its OTel
dependency subtree.

Verified rather than asserted: the lite variant (authbridge-proxy built with
the seven exclude_plugin_* tags CI uses) builds and passes go test -race
on this branch.

Dependencies

Four direct, three of which are promotions of modules already in your graph
as indirect dependencies:

module	before	after
go.opentelemetry.io/otel	indirect	direct
go.opentelemetry.io/otel/sdk	indirect	direct
go.opentelemetry.io/otel/trace	indirect	direct
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc	—	direct (new)

Plus five new indirect: otlptrace, proto/otlp, cenkalti/backoff/v5,
grpc-ecosystem/grpc-gateway/v2, genproto/googleapis/api.

Licences, checked at the module proxy: Apache-2.0 for every OTel module and
genproto, MIT for backoff/v5, BSD-3-Clause for grpc-gateway/v2.
All permissive; none on your dependency-review deny list (GPL / AGPL-3.0).

No go.sum change is needed anywhere — your existing sums already cover
these modules, which is why the diff contains none. A reviewer expecting one
might otherwise read its absence as an omission. go mod tidy is byte-clean on
all three modules.

Verification

Under golang:1.26, mirroring .github/workflows/ci.yaml:

gate	result
go vet · build · test -race -cover (authlib)	PASS — 47 packages ok, 0 failed
same on cmd/authbridge-envoy and cmd/authbridge-proxy (GOWORK=off)	PASS
lite variant, 7 exclude_plugin_* tags — build + test -race	PASS
go mod tidy byte-clean × 3 modules	PASS
gofmt -l	16 dirty — the same 16 as main itself; the new package is gofmt-clean

All of the above was run on this branch alone, without the listener fix applied — which is the
direct evidence for the claim above that this compiles and tests green independently of it.

The plugin's own suite is 858 lines.

Reproducible evidence that it does what it claims is the demo submitted
separately (authbridge/demos/lineage/): enable the plugin, point
otel_endpoint at any OTLP sink, and one A2A request yields the pair. On a
stock install that sink is the platform's own collector, whose default pipeline
exports to debug — so the spans are readable straight from its log, with no
extra service to deploy. (Phoenix is not installed by default;
components.phoenix.enabled is false, so it is one helm value away rather
than already there.) Run against a live cluster, that is literally:

weather-lineage a2a message/send            role=request   direction=inbound  protocol=a2a
weather-lineage a2a message/send response   role=response  direction=inbound  protocol=a2a  outcome=ok

both carrying the same lineage.exchange.id. Nothing beyond this repo and a
cluster is required to reproduce it.

Limits, stated plainly

1. An HTTPS hop produces no span at all — not a span with the body missing.
   The outbound listener has two filter chains. A connection matching
   transport_protocol: tls goes to envoy.filters.network.tcp_proxy and is
   forwarded to its original destination as bytes; a connection matching
   raw_buffer goes to the HTTP connection manager, which is the only chain
   carrying the ext_proc filter. So for TLS traffic the plugin is never
   invoked: there is no method, no path, no host, no status — nothing to attach
   a payload to. The only thing observable is the SNI name at handshake, which
   is why an SNI observer is the named follow-up rather than "parse the body".
   Our probe asserts both sides: the same external endpoint called over plaintext
   HTTP derives exactly one hop, and called over HTTPS derives zero rows, while
   both calls return 200 to the app.
2. No producer-side payload size cap. With capture_io: true, a large
   message is attached whole. There is no truncation in the plugin (checked).
3. A denial that happens before the plugin runs emits no spans at all. The
   pipeline YAML places this plugin after the gate plugins (ordering is by
   position in the list — it is not soft-declared under this capabilities
   model), and the pipeline short-circuits on a request-phase reject — so an
   exchange refused by a gate is invisible to lineage. Denials after
   OnRequest are captured (lineage.outcome=denied + lineage.denied_by).
   Moving lineage ahead of the gates is a named follow-up, not current
   behaviour. Documented in the package doc; it matters to anyone who would
   reach for these spans as an audit trail.
4. The originating caller is usually unattributed, by design.
   lineage.principal.sub and lineage.principal.client are emitted only on
   inbound request spans and only from a validated JWT — the plugin reads
   pctx.Identity, which is nil unless a gate plugin verified a token
   (plugin.go:530-539). An entry call that arrives without one therefore
   carries no principal fact at all. That is deliberate: the alternative is
   inferring a caller from a network address, which is a guess, and this
   producer does not guess. The consequence is that the first hop of a trace is
   typically anonymous.
5. A read-only variant is marked, not built. plugin.go:268 carries an
   explicit >>> OPTION-4 DELETION POINT <<<: deleting the selectParent and
   restampTracestate calls (and the parent.source fact) yields a sidecar
   that parents on the wire context alone and writes no header at all. We have
   not built that variant; the marker is there so the choice stays visible.

---

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5574fddf-eb16-4972-97ea-84473f4f3dc0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}