Skip to content

refactor: move components shared between aap and iast to internal.security - #21162

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 14 commits into
mainfrom
emmett.butler/security-internal
Oct 28, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 14 commits into
mainfrom
emmett.butler/security-internal

Conversation

@emmettbutler

Copy link
Copy Markdown
Collaborator

Description

Removes the 10 dependency direction violations from IAST (product:iast) into AAP (product:aap). Every one of them pointed at code both products use, so that code now lives in a new ddtrace/internal/security/ package, which both may import. Being part of ddtrace.internal, the package imports no product code.

Extracted from AAP modules:

  • STACK_TRACE from ddtrace/appsec/_constants.py → ddtrace/internal/security/_constants.py
  • get_entry_span and the _ASM_CONTEXT core-context key from ddtrace/appsec/_asm_request_context.py → ddtrace/internal/security/_context.py. report_stack needs the entry span; it now looks it up through the shared key instead of importing _asm_request_context.
  • _asm_manual_keep from ddtrace/appsec/_trace_utils.py → ddtrace/internal/security/_trace_utils.py

The original AAP modules re-export the extracted names, so existing AAP call sites are unchanged. IAST now imports everything from ddtrace.internal.security.

Risks

  • The C extension's import path changed from ddtrace.appsec._shared._stacktrace to ddtrace.internal.security._stacktrace. Packaging that removes the extension by its old path (as the AWS Lambda layer does to keep it out of serverless deployments) needs to remove it at the new path; otherwise the unused extension is shipped. Nothing imports it at serverless startup, so this affects package size only. There's a PR updating the exclusion rule in the Lambda layer here: chore: exclude stacktrace C extension's new location datadog-lambda-python#875

@emmettbutler
emmettbutler requested review from a team as code owners October 21, 2026 18:06
@emmettbutler
emmettbutler requested review from avara1986, brettlangdon, kathiehuang and lym953 and removed request for a team October 21, 2026 18:06
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T16:31:11.371421Z dd3b972 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@emmettbutler emmettbutler added the changelog/no-changelog A changelog entry is not required for this PR. label Oct 21, 2026
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Pipelines  Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: dd3b972 | Docs | View more details | Give us feedback!

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

.github/CODEOWNERS                                                      @DataDog/python-guild @DataDog/apm-core-python
benchmarks/suitespec.yml                                                @DataDog/apm-core-python
ddtrace/appsec/_asm_request_context.py                                  @DataDog/asm-python
ddtrace/appsec/_constants.py                                            @DataDog/asm-python
ddtrace/appsec/_contrib/filesystem/patch.py                             @DataDog/asm-python
ddtrace/appsec/_contrib/stripe/patch.py                                 @DataDog/asm-python
ddtrace/appsec/_iast/__init__.py                                        @DataDog/asm-python
ddtrace/appsec/_iast/_metrics.py                                        @DataDog/asm-python
ddtrace/appsec/_iast/_patch_modules.py                                  @DataDog/asm-python
ddtrace/appsec/_iast/reporter.py                                        @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/_base.py                               @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/insecure_cookie.py                     @DataDog/asm-python
ddtrace/appsec/_iast/taint_sinks/weak_hash.py                           @DataDog/asm-python
ddtrace/appsec/_metrics.py                                              @DataDog/asm-python
ddtrace/appsec/_processor.py                                            @DataDog/asm-python
ddtrace/appsec/_trace_utils.py                                          @DataDog/asm-python
ddtrace/appsec/sca/_instrumenter.py                                     @DataDog/asm-python
ddtrace/internal/security/__init__.py                                   @DataDog/apm-core-python
ddtrace/internal/security/_constants.py                                 @DataDog/apm-core-python
ddtrace/internal/security/_context.py                                   @DataDog/apm-core-python
ddtrace/internal/security/_deduplications.py                            @DataDog/apm-core-python
ddtrace/internal/security/_patch_utils.py                               @DataDog/apm-core-python
ddtrace/internal/security/_stacktrace.c                                 @DataDog/apm-core-python
ddtrace/internal/security/_stacktrace.pyi                               @DataDog/apm-core-python
ddtrace/internal/security/_trace_utils.py                               @DataDog/apm-core-python
ddtrace/internal/security/stack_traces.py                               @DataDog/apm-core-python
setup.py                                                                @DataDog/python-guild
tests/appsec/appsec/test_common_modules.py                              @DataDog/asm-python
tests/appsec/appsec/test_exploit_prevention.py                          @DataDog/asm-python
tests/appsec/appsec/test_telemetry.py                                   @DataDog/asm-python
tests/appsec/architectures/mini.py                                      @DataDog/asm-python
tests/appsec/architectures/test_appsec_loading_modules.py               @DataDog/asm-python
tests/appsec/iast/taint_sinks/test_weak_hash.py                         @DataDog/asm-python
tests/appsec/iast/test_stacktrace.py                                    @DataDog/asm-python
tests/appsec/iast_memcheck/fixtures/stacktrace.py                       @DataDog/asm-python
tests/appsec/iast_memcheck/test_iast_mem_check.py                       @DataDog/asm-python
tests/appsec/suitespec.yml                                              @DataDog/asm-python
tests/internal/test_serverless.py                                       @DataDog/apm-serverless @DataDog/asm-python @DataDog/serverless-aws @DataDog/serverless-azure-and-gcp

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 106 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 106 highest severity)
ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=121)
ddtrace.llmobs._utils -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.google_adk -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.litellm -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.typesafe -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

✅ Dependency direction violations removed

10 violation(s) have been removed by this PR.

Show removed violations (showing 5 of 10 highest severity)
ddtrace.appsec._iast.reporter -×-> ddtrace.appsec._constants  (product:iast -> product:aap, score=26)
ddtrace.appsec._iast._patch_modules -×-> ddtrace.appsec._patch_utils  (product:iast -> product:aap, score=8)
ddtrace.appsec._iast.taint_sinks.insecure_cookie -×-> ddtrace.appsec._patch_utils  (product:iast -> product:aap, score=8)
ddtrace.appsec._iast.taint_sinks._base -×-> ddtrace.appsec._patch_utils  (product:iast -> product:aap, score=8)
ddtrace.appsec._iast.taint_sinks.weak_hash -×-> ddtrace.appsec._patch_utils  (product:iast -> product:aap, score=8)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The architecture fixture excludes the new security namespace, making the relocated stacktrace extension’s loading assertion impossible to satisfy when IAST is enabled on a supported non-Lambda platform.

Open Bits AI session

🤖 Bits Code Review · Commit 08ffb01

@pr-commenter

pr-commenter Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-10-28 16:53:51

Comparing candidate commit dd3b972 in PR branch emmett.butler/security-internal with baseline commit 55bb739 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 609 metrics, 10 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

  • unstable execution_time [-799.088ns; +653.638ns] or [-7.406%; +6.058%]

scenario:coreapiscenario-core_dispatch_1_listener

  • unstable execution_time [-37.706ns; +33.290ns] or [-5.458%; +4.818%]

scenario:coreapiscenario-core_dispatch_50_listeners

  • unstable execution_time [-2025.485ns; +1801.065ns] or [-10.111%; +8.991%]

scenario:coreapiscenario-core_dispatch_exception_listeners

  • unstable execution_time [-1849.911ns; +1777.463ns] or [-9.707%; +9.327%]

scenario:coreapiscenario-core_dispatch_listeners

  • unstable execution_time [-402.400ns; +350.128ns] or [-9.357%; +8.142%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

  • unstable execution_time [-227.497ns; +222.948ns] or [-8.387%; +8.219%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

  • unstable execution_time [-110.940ns; +69.003ns] or [-7.680%; +4.777%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

  • unstable execution_time [-5834.826ns; +3911.715ns] or [-11.390%; +7.636%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

  • unstable execution_time [-1194.830ns; +800.867ns] or [-11.063%; +7.415%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

  • unstable execution_time [-50.132µs; +62.637µs] or [-8.626%; +10.778%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

  • scenario:packagespackageforrootmodulemapping-cache_off

@florentinl florentinl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's awesome. Thank you for the refactor. Two small things:

  • Can you add ddtrace/internal/security/* to the security test and benchmark mappings. Changes limited to these files currently select only internal and detect_global_locks, skipping AAP/IAST/SCA coverage.
  • Would you mind adding us (@DataDog/asm-python) as the codeowners of the ddtrace/internal/security folder so that we keep the ownership of this code ?

@emmettbutler
emmettbutler requested a review from a team as a code owner October 26, 2026 14:17

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Entry-span lookup preserves its existing context and span fallbacks, and the relocated native extension’s imports match its build target. The specialist returned no actionable findings.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit dd943ea

@gh-worker-ownership-write-b05516
gh-worker-ownership-write-b05516 Bot removed the request for review from a team October 26, 2026 14:25

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a0205f01f5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread benchmarks/suitespec.yml Resolved
@emmettbutler

Copy link
Copy Markdown
Collaborator Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-28 16:27:43 UTC ℹ️ Start processing command /merge


2026-10-28 16:27:50 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

  • Run /code blockers to see what is blocking it.
  • Run /remove to cancel it.

2026-10-28 17:10:10 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 1h (p90).


2026-10-28 17:55:11 UTC ❌ MergeQueue: The build pipeline failed for this merge request

Build pipeline has failing jobs for eac9adb:

What to do next?

  • Investigate the failures and when ready, re-add your pull request to the queue!
  • If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
  • Any question, go check the FAQ.

This build will be kept for 3h more. During that period, you can retry the pipeline or the failing jobs and then re-run the same command!
If you want to reset everything, you can cancel your previous command by rerunning it with the -c flag.

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 6000afb into main Oct 28, 2026
857 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the emmett.butler/security-internal branch October 28, 2026 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/no-changelog A changelog entry is not required for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants