Skip to content

Add Azure Durable Functions tracing support - #12618

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 21 commits into
masterfrom
storms/azure-durable-fx-support
Oct 12, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 21 commits into
masterfrom
storms/azure-durable-fx-support

Conversation

@jcstorms1

@jcstorms1 jcstorms1 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Adds tracing support for Azure Durable Functions running on the Java worker:

  • Instruments durable orchestrator, activity, and entity invocations through FunctionExecutionMiddleware.
  • Continues the W3C trace context propagated by the Azure Functions host.
  • Suppresses successful orchestrator replays while retaining initial executions and replays caused by activity or sub-orchestration failures.
  • Marks application failures as errors without treating Durable SDK replay-control exceptions as failures.
  • Adds Durable function name, trigger type, serverless span type, and component metadata.
  • Adds Java/JUnit 5 coverage for both span attribute schema versions and the latest supported dependency set.

Motivation

The existing Azure Functions instrumentation covers HTTP entry points, but Java Durable orchestrators and activities execute through worker middleware and were not represented in the distributed trace. This left the asynchronous Durable workflow invisible after the starter function scheduled an instance.

This instrumentation connects the Java Durable workflow to the originating trace and avoids emitting a duplicate orchestrator span for every successful replay.

Validation

  • ./gradlew :dd-java-agent:instrumentation:azure-functions-worker-2.7:spotlessCheck
  • ./gradlew :dd-java-agent:instrumentation:azure-functions-worker-2.7:test
  • ./gradlew :dd-java-agent:instrumentation:azure-functions-worker-2.7:forkedTest
  • ./gradlew :dd-java-agent:instrumentation:azure-functions-worker-2.7:latestDepTest
  • The Java/JUnit 5 suite runs 18 cases in each test task, covering trigger spans, replay suppression and retention, sampling propagation, replay-control exceptions, and application failures.
  • ./gradlew :dd-trace-core:test --tests datadog.trace.core.propagation.ExtractedContextTest
  • ./gradlew :dd-java-agent:shadowJar :dd-java-agent:verifyAgentJarContents :dd-java-agent:verifyAgentJarIntegrations
  • Deployed the rebuilt agent to an Azure Java Durable Functions POC using Durable tracing V2 and the Azure Storage provider.
  • Verified a successful orchestration contains one orchestrator span and both activity spans.
  • Verified a failed activity is marked as an error and retains the failure replay of the orchestrator.

No error

No error trace

With error

Trace with activity error

Additional Notes

  • The instrumentation targets Azure Functions Java worker 2.7+ and was validated with durabletask-azure-functions 1.0.1.
  • The instrumentation tests are implemented in Java/JUnit 5 using the repository's Groovy-to-Java migration conventions.
  • No new user-facing configuration flags are introduced.

Contributor Checklist

  • Use a user-facing infinitive title.
  • Assign type: feature and inst: azure-functions labels.
  • Add the new instrumentation directory to CODEOWNERS.
  • Add focused Java/JUnit 5 instrumentation and propagation tests.
  • No public configuration documentation update is required because this adds no configuration flags.

@jcstorms1 jcstorms1 added type: feature Enhancements and improvements inst: azure-functions Azure Functions instrumentation labels Sep 23, 2026
@datadog-datadog-us1-prod

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.98 s 13.91 s [-0.1%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent 12.90 s 12.93 s [-0.9%; +0.5%] (no difference)
startup:petclinic:appsec:Agent 17.08 s 16.96 s [-0.2%; +1.6%] (no difference)
startup:petclinic:iast:Agent 16.84 s 17.00 s [-1.7%; -0.2%] (maybe better)
startup:petclinic:profiling:Agent 16.55 s 16.58 s [-1.1%; +0.8%] (no difference)
startup:petclinic:sca:Agent 17.06 s 16.97 s [-0.4%; +1.5%] (no difference)
startup:petclinic:tracing:Agent 16.02 s 16.20 s [-2.2%; -0.0%] (maybe better)

Commit: 81868593 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@jcstorms1

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-23T20:07:08.148907Z 12dec94 Draft marked ready
🔒 Security Review ✅ Completed 2026-09-23T20:07:30.900452Z 12dec94 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 376c24d851

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 376c24d851

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jcstorms1
jcstorms1 marked this pull request as ready for review September 23, 2026 20:00
@jcstorms1
jcstorms1 requested review from a team as code owners September 23, 2026 20:00
@jcstorms1
jcstorms1 requested review from AlexeyKuznetsov-DD, ValentinZakharov and mcculls and removed request for a team September 23, 2026 20:00

@datadog-datadog-us1-prod datadog-datadog-us1-prod Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

An exception with a cyclic cause chain can keep a worker thread in an endless loop. Error spans created after a suppressed replay also omit the time spent in the invocation.

Open Bits AI session

🤖 Bits Code Review · Commit 12dec94 · @DataDog review to ask questions

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left several comments. My main concern that core classes changed, we need to double check on that.

Comment thread dd-java-agent/instrumentation/azure-functions-worker-2.7/build.gradle Outdated

@mcculls mcculls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The overall handling of spans is unusual - especially the error spans created after the fact. Some of this logic is what's leading to needing several @Advice.Local slots.

Left various comments with suggested improvements/cleanup, but no obvious blockers from apm-sdk-capabilities. (After the refactoring, none of the files here are actually owned by us so this PR is not gated by our review.)

@jcstorms1
jcstorms1 requested a review from dougqh October 2, 2026 16:50

@lucaspimentel lucaspimentel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved from the apm-serverless pov:

  • Span naming and metadata: Verified against the .NET Azure Functions instrumentation for cross-runtime consistency: resource name, trigger values, and tags match .NET; operation name, span type (serverless), component, and span kind match the sibling
    azure-functions-1.2.2 module, with the correct v1 default for Azure Functions. Service name is left to the default function-app naming, which is right for serverless.
  • Sampling behavior: honors the W3C sampled flag, which differs from .NET's durable propagation (which force-keeps on a positive Datadog decision and clears Azure's implicit rejection). This divergence is deliberate and agreed: the core team's position is that the sampled flag must be respected, and the Java host's telemetryMode: OpenTelemetry requirement covers the drop scenario. The PR description now matches the implementation.
  • Replay suppression and error retention: Successful replays produce no span while user work during a replay stays attached to the trace; failed activities/sub-orchestrations retain the replay with proper error tags; Durable SDK control-flow exceptions are correctly not treated as failures.

I did not look at JVM-specific performance and allocation characteristics of the new code paths or Muzzle wiring and internal-class coupling details beyond what's already discussed in review threads.

@ValentinZakharov ValentinZakharov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jcstorms1
jcstorms1 enabled auto-merge October 12, 2026 13:14
@jcstorms1
jcstorms1 added this pull request to the merge queue Oct 12, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-12 14:07:20 UTC ℹ️ Start processing command /merge


2026-10-12 14:07:24 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-10-12 15:01:56 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 83b7bd4:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

  • Investigate the failures and when ready, re-add your pull request to the queue!
  • If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
  • Any question, go check the FAQ.
Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 12, 2026
@jcstorms1
jcstorms1 added this pull request to the merge queue Oct 12, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-12 21:20:27 UTC ℹ️ Start processing command /merge


2026-10-12 21:20:31 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-10-12 22:35:23 UTC ℹ️ MergeQueue: This merge request was merged

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 12, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit deb81bb into master Oct 12, 2026
611 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the storms/azure-durable-fx-support branch October 12, 2026 22:35
@github-actions github-actions Bot added this to the 1.68.0 milestone Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: azure-functions Azure Functions instrumentation type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants