Skip to content

test(observability): preserve #345 UA wire-capture checks as in-tree diagnostics + runbook #893

Description

@scottschreckengaust

Summary

PR #345 (#319, solution attribution via native AWS_SDK_UA_APP_ID) shipped unit tests for the UA helpers (agent/tests/test_ua.py, cdk/test/handlers/shared/ua.test.ts, cli/test/ua.test.ts) and the SolutionUaAspect. During that work, two wire-capture verification scripts were written but never committed. They should be preserved in the repo as diagnostics, because they verify something the unit tests structurally cannot.

Why they matter (the gap unit tests can't cover)

The outbound User-Agent carries two solution segments:

app/uksb-wt64nei4u6#{AWS_SDK_UA_APP_ID}   <- injected by the SDK/botocore itself, from the env var
md/uksb-wt64nei4u6#{component}            <- from our helper (abcaUserAgent() / ua.py)

Unit tests assert the helper returns the right md/… string. But the app/… segment is SDK-native — our code never produces it — so a helper unit test can't observe it. The only ways to confirm the assembled header on the wire are CloudTrail or a wire capture, and CloudTrail is unavailable here (this account blocks DynamoDB data events). That is exactly the recurring hazard AGENTS.md calls out: "Dropping solution UA on a new AWS client (#319)." A repeatable wire-check is the regression guard for it.

The two scripts (currently uncommitted, in the feat-319-ua-appid worktree)

  • agent/ua_wire_check.py — imports the real agent/src/ua.py, builds boto3 clients like aws_session.platform_client, registers a botocore before-send hook, and prints the assembled UA across STS/DynamoDB/S3/SecretsManager.
  • cdk/ua-wire-check.ts — the SDK v3 twin: imports the real src/handlers/shared/ua.ts, attaches a finalizeRequest middleware, prints the UA across Lambda/DynamoDB/S3/SecretsManager, varying ABCA_COMPONENT.

Both import the real helpers (no mirror → no drift) and capture the UA at request-build time, so even a perms failure still prints the header.

Known issues to fix during relocation

  • cdk/ua-wire-check.ts has a duplicate import line (@aws-sdk/client-lambda imported twice) — won't compile as-is.
  • Both import paths assume the repo root; they need adjusting for their new locations.
  • Neither has a documented home or run instructions in-tree.

Proposed placement

  • agent/ua_wire_check.pyagent/scripts/diagnostics/ua_wire_check.py (alongside test_sdk_smoke.py).
  • cdk/ua-wire-check.tscdk/scripts/ua-wire-check.ts (add a row to cdk/scripts/README.md; run via npx tsx).
  • Add a runbook: docs/verification/ua-wire-check-runbook.md — how to run both tiers and interpret output (incl. AWS_SDK_UA_APP_ID='' opt-out check).

These paths are deliberately outside tsc/eslint (cdk scopes src/**+test/**), vulture (agent scopes src), and knip analysis — matching the existing cdk/scripts/generate-*.ts and agent/scripts/diagnostics/*.py helper scripts. They are manual, credentialed diagnostics, not CI tests.

Acceptance

  • Both scripts committed at the proposed paths, with corrected imports and a fixed (compiling) cdk script.
  • cdk/scripts/README.md lists the new script.
  • docs/verification/ua-wire-check-runbook.md documents how to run each tier and what a pass looks like.
  • No new failures in mise //cdk:eslint, mise //cdk:build, or agent quality (scripts are outside those scopes by design).

Notes

Follow-up to merged #345 (#319). Small chore/test change; no runtime/behavioral impact. Needs maintainer approved + assignment before implementation (ADR-003).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

approvedWhen an issue has been approved and readyv1Version 1

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions