Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

chore(testing): mark provenance_enforcement storyboard as known-failing pending training-agent implementation

Adds `creative_sales_agent/provenance_enforcement` to `KNOWN_FAILING_STORYBOARDS` in `server/tests/manual/run-storyboards.ts`. The storyboard exercises the `PROVENANCE_*` rejection paths and the round-trip of `creative_policy.{provenance_required, provenance_requirements, accepted_verifiers}` through `get_products` / `sync_creatives`, but the training agent has no provenance enforcement yet — the spec is landing in this PR ahead of the reference implementation.
Adds `media_buy_seller/provenance_enforcement` to `KNOWN_FAILING_STORYBOARDS` in `server/tests/manual/run-storyboards.ts`. The storyboard exercises the `PROVENANCE_*` rejection paths and the round-trip of `creative_policy.{provenance_required, provenance_requirements, accepted_verifiers}` through `get_products` / `sync_creatives`, but the training agent has no provenance enforcement yet — the spec is landing in this PR ahead of the reference implementation.

Tracked in #3777; the entry is removed once the training agent enforces provenance per the spec (`get_products` surfacing the seeded `creative_policy` fields and `sync_creatives` emitting the six `PROVENANCE_*` codes for structural rejections).

Expand Down
23 changes: 23 additions & 0 deletions .changeset/training-agent-provenance-enforcement.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
---

feat(training-agent): implement provenance enforcement (closes #3777)

Bring the reference training agent up to the spec landed in #3468:

- `handleGetProducts` now overlays `comply_test_controller`-seeded products onto the response so storyboard-seeded `creative_policy.{provenance_required, provenance_requirements, accepted_verifiers}` fields round-trip through `get_products`. Previously only `handleCreateMediaBuy` saw seeded fixtures.
- `backfillProductDefaults` fills in spec-required Product fields (`name`, `description`, `publisher_properties`, `format_ids`, `pricing_options`, `reporting_capabilities` and its required sub-fields) for fixture-seeded products that historically only carried fields `create_media_buy` validation needed. Closes the response-schema gap that surfaced once seeded products began round-tripping through `get_products`.
- `handleSyncCreatives` enforces `creative_policy` from session-seeded products with the structural-rejection family on `error-code.json`:
- `PROVENANCE_REQUIRED`
- `PROVENANCE_DIGITAL_SOURCE_TYPE_MISSING`
- `PROVENANCE_DISCLOSURE_MISSING`
- `PROVENANCE_EMBEDDED_MISSING`
- `PROVENANCE_VERIFIER_NOT_ACCEPTED` — buyer-supplied `verify_agent.agent_url` cross-checked against the seller's `accepted_verifiers` allowlist (canonicalized) before any outbound call. Off-list URLs reject without contacting them.
- Per-creative failures emit `action: 'failed'` + per-creative `errors[]` with `field`/`recovery`. The `SyncCreativeResult` interface gains the `failed` action variant and an optional `errors[]` field.
- Truth-of-claim (`PROVENANCE_CLAIM_CONTRADICTED`, requires calling `get_creative_features` against an on-list verifier) is out of scope for this initial implementation — the structural codes are sufficient to make the conformance scenario pass and exercise the wire contract.

Removes `media_buy_seller/provenance_enforcement` from `KNOWN_FAILING_STORYBOARDS` in `server/tests/manual/run-storyboards.ts` (it now passes 5/5 steps in both legacy and framework modes). Bumps `min_clean_storyboards` (53→65) and `min_passing_steps` (388→444 legacy, 401→462 framework) in `.github/workflows/training-agent-storyboards.yml` to reflect the new clean baselines.

Updates the storyboard fixture (`media_buy_seller/provenance_enforcement`) with a unique product name/description so brief-mode scoring places it at `products[0]`, and switches per-creative error assertions to `field_value` paths since the spec sync_creatives shape carries failures per-item, not at the top-level errors envelope.

Refs: #3468, #3777.
21 changes: 17 additions & 4 deletions .github/workflows/training-agent-storyboards.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,20 @@ jobs:
# (#3081) now passes 4 steps instead of grading not_applicable.
# 384→388 after bumping @adcp/client to 5.18.0 (#3191) and
# implementing force_task_completion (#3194).
# The 53/388 floors had drifted below the actual baseline
# (true main was 64 storyboards / 439 steps); #3777 brings
# the count to 65 / 446 by adding the
# media_buy_seller/provenance_enforcement scenario (six
# phases exercising PROVENANCE_REQUIRED,
# PROVENANCE_DIGITAL_SOURCE_TYPE_MISSING,
# PROVENANCE_VERIFIER_NOT_ACCEPTED,
# PROVENANCE_DISCLOSURE_MISSING, and an accept path).
# Passing-step asymmetry with framework mode is intentional:
# the two modes declare different capabilities, so the
# storyboard runner skips a different subset of steps in each.
# Both modes pass every step they run.
min_clean_storyboards: 53
min_passing_steps: 388
min_clean_storyboards: 65
min_passing_steps: 446
- mode: framework
flag_value: '1'
# 390→393 after bumping @adcp/client to 5.15 + seller catalog
Expand All @@ -67,8 +75,13 @@ jobs:
# injection — the upstream fix for #940. Framework parity with
# legacy fully restored; passing-step asymmetry remains because
# framework declares more capabilities (more steps run, all pass).
min_clean_storyboards: 53
min_passing_steps: 401
# The 53/401 floors had drifted below the actual baseline
# (true main was 64 storyboards / 457 steps); #3777 brings the
# count to 65 / 464 — same six-phase provenance_enforcement
# scenario as legacy; framework picks up the extra steps from
# its broader capability declaration.
min_clean_storyboards: 65
min_passing_steps: 464
steps:
- uses: actions/checkout@v6

Expand Down
7 changes: 4 additions & 3 deletions server/src/shared/formats.ts
Original file line number Diff line number Diff line change
Expand Up @@ -540,9 +540,10 @@ export function buildFormats(agentUrl: string): TrainingFormat[] {
requirements: { mime_types: ['image/png', 'image/svg+xml'], max_file_size_bytes: 5_000_000 } },
],
},
// Storyboard-hardcoded format ids (creative_lifecycle, creative_sales_agent).
// Aliased to close-enough shapes; storyboards pass a format_id string
// and don't care about subtle shape differences.
// Storyboard-hardcoded format ids (creative_lifecycle,
// media_buy_seller/creative_reception). Aliased to close-enough shapes;
// storyboards pass a format_id string and don't care about subtle
// shape differences.
{
format_id: { agent_url: agentUrl, id: 'video_30s' },
name: '30-second video',
Expand Down
Loading
Loading