Skip to content

feat(rum): add rum_permanent_retention_filters resource - #718

Open
michael-richey wants to merge 5 commits into
michael.richey/add-rum-retention-filters-orderfrom
michael.richey/add-rum-permanent-retention-filters
Open

michael-richey wants to merge 5 commits into
michael.richey/add-rum-retention-filters-orderfrom
michael.richey/add-rum-permanent-retention-filters

Conversation

@michael-richey

Copy link
Copy Markdown
Collaborator

Summary

Permanent RUM retention filters — system-defined with fixed ids (rum_apm_flat_sampling, synthetics_sessions, forced_replay_sessions) identical across orgs, so the filter id needs no remapping; only the parent application id does. PATCH-only (no POST/DELETE): create_resource delegates to update_resource, delete_resource is a no-op (mirrors logs_archives_order).

Stacked on #717 (rum_retention_filters_order).

Design

  • Parent-scoped like rum_retention_filters: synthetic _application_id injected during enumeration, remapped via resource_connections, kept out of excluded_attributes (must survive prep_resource), excluded from diffs via deep_diff_config.exclude_regex_paths.
  • Only attributes.cross_product_sampling is updatable, so attributes.name/description/editability are in excluded_attributes (removed by prep, not needed for update).

Changes

  • datadog_sync/model/rum_permanent_retention_filters.py (new)
  • datadog_sync/models/__init__.py — register
  • tests/unit/test_rum_permanent_retention_filters.py (new) — 6 unit tests
  • README.md — add rum_permanent_retention_filters (depends on rum_applications)

Testing

  • pytest tests/unit/test_rum_permanent_retention_filters.py → 6 passed
  • Full unit suite: no new failures.

Follow-up

Integration tests + VCR cassettes deferred (require sandbox-org API access).

@michael-richey michael-richey left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this model + coverage. I did find one blocking correctness issue:

Blocking: key collision across applications for permanent filters

import_resource() returns resource["id"] as the state key. For permanent filters, IDs are fixed/system-defined (synthetics_sessions, etc.) and repeat under every app. That means imports from multiple apps collide in state.source[self.resource_type][_id], so only the last app survives.

Concretely, with two apps both containing synthetics_sessions, state ends up with a single key:

  • imported synthetics_sessions from app A
  • imported synthetics_sessions from app B
  • final state key set: ['synthetics_sessions'] (app A overwritten)

This breaks multi-app sync for rum_permanent_retention_filters.

Suggested fix

Use a composite key for this resource type (e.g. "{application_id}:{filter_id}") for import/state indexing, while keeping payload data.id as the server filter id.

That preserves per-app cardinality and keeps create/update/delete URL construction unambiguous.

Happy to re-review after this is addressed.

@michael-richey
michael-richey force-pushed the michael.richey/add-rum-permanent-retention-filters branch from d79679a to 7561097 Compare September 25, 2026 20:53
@michael-richey

Copy link
Copy Markdown
Collaborator Author

Thanks for catching this — fixed in 75610977. The state key is now a composite "{application_id}:{filter_id}" to avoid collisions when multiple apps have the same fixed filter id. The payload data.id remains the server filter id. update_resource now uses the resource's id and _application_id directly (the id is fixed across orgs, and _application_id was remapped by connect_resources) instead of looking up state, which also simplifies the first-create path. Tests updated to verify composite keys and multi-app cardinality.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Excluding the parent application ID from diffs can skip configuration after a destination application is recreated.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Adds synchronization support for application-scoped permanent RUM retention filters.

Changes:

  • Adds PATCH-only resource handling with composite state keys.
  • Registers and documents the resource and dependency.
  • Adds unit coverage for retrieval, updates, remapping, and deletion behavior.
File Description
datadog_sync/​model/​rum_permanent_retention_filters.py Implements the resource model.
datadog_sync/​models/​__init__.py Registers the model.
tests/​unit/​test_rum_permanent_retention_filters.py Adds unit tests.
README.md Documents support and dependencies.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread datadog_sync/model/rum_permanent_retention_filters.py
@michael-richey
michael-richey force-pushed the michael.richey/add-rum-retention-filters-order branch from e298847 to c2e4117 Compare September 28, 2026 14:10
@michael-richey
michael-richey force-pushed the michael.richey/add-rum-permanent-retention-filters branch from 7561097 to 9192e66 Compare September 28, 2026 14:10
@michael-richey

Copy link
Copy Markdown
Collaborator Author

Fixed in 9192e664. _application_id is no longer excluded from the diff (exclude_regex_paths removed from deep_diff_config). Now a changed parent mapping (e.g. destination app deleted and recreated with a new id) forces a PATCH rather than being silently skipped. Added regression test verifying _application_id is not in exclude_regex_paths.

@michael-richey

Copy link
Copy Markdown
Collaborator Author

Integration Test Findings (EU1 → US5 sync)

Ran a full integration test: imported all RUM resources from the EU1 internal testing org, then synced to the US5 internal testing org. Found 1 failure in rum_permanent_retention_filters:

rum_apm_flat_sampling — 400 Bad Request

trace fields are not editable for the rum_apm_flat_sampling filter

The sync is trying to PATCH the cross_product_sampling.trace_sample_rate and trace_enabled fields on this permanent retention filter, but the API rejects it because trace fields are not editable for this specific filter.

The source state from EU1 includes:

{
  "id": "rum_apm_flat_sampling",
  "attributes": {
    "cross_product_sampling": {
      "trace_sample_rate": 100,
      "trace_enabled": true
    },
    "editability": {
      "trace_editable": false
    },
    "name": "RUM APM Flat Sampling"
  }
}

Note the "trace_editable": false in the editability field — this signals that trace fields cannot be modified for this filter.

Suggested fix

Respect the editability.trace_editable flag in the model. When trace_editable is false, omit the cross_product_sampling.trace_sample_rate and trace_enabled fields from the PATCH body. Only send fields that the editability metadata allows.

What worked

The other 2 permanent retention filters (forced_replay_sessions and synthetics_sessions) synced successfully — their PATCH calls completed without error.

@michael-richey
michael-richey force-pushed the michael.richey/add-rum-retention-filters-order branch from c2e4117 to 56d4b34 Compare October 1, 2026 15:54
@michael-richey
michael-richey force-pushed the michael.richey/add-rum-permanent-retention-filters branch from 9192e66 to 864e6dd Compare October 1, 2026 15:54
@michael-richey

Copy link
Copy Markdown
Collaborator Author

Fixed in e214c9ed. update_resource now checks the editability.trace_editable flag and strips trace_sample_rate and trace_enabled from cross_product_sampling when the flag is false, so only non-trace attributes are sent in the PATCH body. Two new tests: one verifying trace fields are stripped when trace_editable=false, one verifying they are kept when trace_editable=true.

@michael-richey

Copy link
Copy Markdown
Collaborator Author

Re-test Results (after fix 864e6dd)

Re-ran the full integration test with the updated branch. The fix is not working — rum_apm_flat_sampling still fails with the same error:

400 Bad Request - trace fields are not editable for the rum_apm_flat_sampling filter

Root cause

attributes.editability is in excluded_attributes:

excluded_attributes=[
    "attributes.name",
    "attributes.description",
    "attributes.editability",  # <-- this is the problem
]

The prep_resource phase strips editability from the resource before update_resource runs. So when update_resource checks:

editability = resource.get("attributes", {}).get("editability", {})
if not editability.get("trace_editable", True):

editability is {} (already stripped), editability.get("trace_editable", True) returns True (the default), and the trace fields are not stripped from the PATCH body.

Suggested fix

Remove "attributes.editability" from excluded_attributes so it survives prep_resource and is available in update_resource. The editability field is needed at update time to decide which fields to strip, so it must not be excluded. Alternatively, read editability from the source state before prep_resource strips it, or store it in a separate field.

@michael-richey
michael-richey force-pushed the michael.richey/add-rum-retention-filters-order branch from 56d4b34 to 221d602 Compare October 1, 2026 18:13
@michael-richey
michael-richey force-pushed the michael.richey/add-rum-permanent-retention-filters branch from 864e6dd to b66155e Compare October 1, 2026 18:13
@michael-richey

Copy link
Copy Markdown
Collaborator Author

Fixed in b66155e7. attributes.editability was in excluded_attributes, so prep_resource stripped it before update_resource could read editability.trace_editable. The trace fields were never stripped, causing 400 on rum_apm_flat_sampling.

Fix: removed attributes.editability from excluded_attributes so it survives prep_resource. Added editability to deep_diff_config.exclude_regex_paths so it doesn't cause a perpetual diff (it's read-only). Two new regression tests verify editability is not in excluded_attributes but is in exclude_regex_paths.

@michael-richey
michael-richey force-pushed the michael.richey/add-rum-retention-filters-order branch from 221d602 to 390b682 Compare October 1, 2026 20:37
@michael-richey
michael-richey force-pushed the michael.richey/add-rum-permanent-retention-filters branch from b66155e to 3585b3a Compare October 1, 2026 20:37
@michael-richey
michael-richey force-pushed the michael.richey/add-rum-retention-filters-order branch from 390b682 to 48de449 Compare October 1, 2026 21:17
@michael-richey
michael-richey force-pushed the michael.richey/add-rum-permanent-retention-filters branch from 3585b3a to a6191f3 Compare October 1, 2026 21:17
Permanent RUM retention filters are system-defined with fixed ids
(rum_apm_flat_sampling, synthetics_sessions, forced_replay_sessions)
identical across orgs, so the filter id needs no remapping -- only the parent
application id does. The endpoint set is PATCH-only (no POST/DELETE), so
create_resource delegates to update_resource and delete_resource is a no-op,
mirroring logs_archances_order.

Parent-scoped like rum_retention_filters: synthetic _application_id injected
during enumeration, remapped via resource_connections, kept out of
excluded_attributes (must survive prep_resource) and excluded from diffs via
deep_diff_config.exclude_regex_paths. Only attributes.cross_product_sampling
is updatable, so attributes.name/description/editability are excluded.

- datadog_sync/model/rum_permanent_retention_filters.py (new)
- datadog_sync/models/__init__.py -- register
- tests/unit/test_rum_permanent_retention_filters.py (new) -- 6 unit tests
- README.md -- add rum_permanent_retention_filters (depends on rum_applications)

Integration tests + VCR cassettes deferred (require sandbox-org API access).
BLOCKING fix per review: permanent filter IDs are fixed/system-defined
(synthetics_sessions, rum_apm_flat_sampling, forced_replay_sessions) and
repeat under every application. Using resource['id'] as the state key
caused collisions when multiple apps were synced -- only the last app
survived.

Fix: use a composite key '{application_id}:{filter_id}' for state indexing.
The payload data.id remains the server filter id. update_resource now uses
the resource's id and _application_id directly (the id is fixed across orgs,
and _application_id was remapped by connect_resources) instead of looking
up state, simplifying the first-create path.
Per review: excluding _application_id from the diff hid a real parent remap.
If a destination RUM application is deleted and recreated, the parent ID
changes but persisted state still references the old ID; the exclusion made
the sync skip the PATCH, leaving the new application's permanent filters
at their defaults. Now _application_id participates in the diff so a
changed parent mapping forces the PATCH. Added regression test verifying
_application_id is not in exclude_regex_paths.
Per integration test findings (EU1 -> US5 sync):

rum_apm_flat_sampling (400 'trace fields are not editable for the
rum_apm_flat_sampling filter'): the API rejects PATCHes that include
cross_product_sampling.trace_sample_rate or trace_enabled when
editability.trace_editable is false.

update_resource now checks the editability.trace_editable flag and strips
trace_sample_rate and trace_enabled from cross_product_sampling when the
flag is false, so only non-trace attributes are sent in the PATCH body.

Two new tests: one verifying trace fields are stripped when
trace_editable=false, one verifying they are kept when trace_editable=true.
Per integration test re-test: attributes.editability was in
excluded_attributes, so prep_resource stripped it before update_resource
could read editability.trace_editable. The trace fields were never stripped,
causing 400 on rum_apm_flat_sampling (trace_editable=false).

Fix: removed attributes.editability from excluded_attributes so it survives
prep_resource. Added editability to deep_diff_config.exclude_regex_paths so
it doesn't cause a perpetual diff (it's read-only). Two new regression tests
verify editability is not excluded but is excluded from diffs.
@michael-richey
michael-richey force-pushed the michael.richey/add-rum-retention-filters-order branch from 48de449 to d42eac2 Compare October 2, 2026 01:45
@michael-richey
michael-richey force-pushed the michael.richey/add-rum-permanent-retention-filters branch from a6191f3 to 46857db Compare October 2, 2026 01:45
@michael-richey

Copy link
Copy Markdown
Collaborator Author

Nice work on this one.

Highlights:

  • Composite keying with <application_id>:<filter_id> avoids the multi-app collision class.
  • import_resource() normalization from <app_id>:<id> back to API id is clean.
  • pre_resource_action_hook + PATCH-only behavior match the endpoint semantics well.

No blocking issues from my side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants