feat: first-class decision & effect auditing (the audit seam)

[terylt]

Summary

CPEX could not audit its own enforcement. An observation-only plugin — the
reference audit-logger, or an OCSF emitter — only ever sees allowed
post-hook traffic; a blocked call, an approval rejection, a delegation failure,
or an injection-stop produced no audit record at all. And irreversible
external actions a plugin causes (a token mint, an approval grant) were not
recorded crash-safely — a process that died between "about to mint" and "minted"
left no trace.

This PR makes auditing first-class in the executor. The core owns a decision
record and emits it at every verdict; any audit sink consumes it. Irreversible
effects are recorded write-ahead, crash-safe, and reconcilable. Each decision
carries the provenance needed to reconstruct a causal graph (span + taint +
content hash). The OAuth delegator is wired as the first real consumer.

Everything is opt-in — no behavior changes unless the operator configures a
sink, an effect WAL, or content provenance.

What's included

Decision auditing.

• A new AuditHook family, auto-attached by the PluginManager, fired at the
  executor's verdict return points (not a pipeline phase) — so allow, deny,
  and modify all produce a record. "Which phase, before or after which deny"
  stops being a question.
• A DecisionLog — executor-owned, handed only to audit sinks, never placed
  on PluginContext (the thing that records must not be able to change what it
  records). Carries the ordered plugin steps and the terminal verdict.

Effect auditing (irreversible external actions).

• A capability-gated, two-phase write-ahead protocol: a plugin holding
  emit_effect calls ext.begin_effect to durably record intent before the
  act (fail-closed — no durable record, no act) and ext.complete_effect
  to record the outcome (confirmed / rejected / unknown).
• FileEffectLog — a durable WAL (append + fsync, serialized against
  concurrent writers, self-compacting past a configurable threshold).
• Crash recovery: PluginManager::recover_effects compacts completed effects
  and reconciles crash-orphaned ones against the issuing participant via an
  EffectReconciler seam. The default (LogUnknownsReconciler) logs and leaves
  them unknown — correct for any participant with no lookup-by-key.
• Extensions::perform_effect brackets the whole protocol so a caller cannot
  skip, reorder, or forget it.

Provenance on the decision node (for downstream causal-graph reconstruction).

• Span / causal parent — DecisionLog::span() carries a W3C
  trace_id/span_id/parent_span_id (child-span model: a fresh span whose
  parent is the request's span), set by the executor at pipeline entry.
• Taint — the labels the request arrived with, captured at entry; the sink
  diffs them against the final labels to show the taint this node added.
• Content hash (opt-in) — PluginPayload::audit_bytes() (per-type opt-in
  via impl_plugin_payload!(_, audit_serialize)) feeds a sha256: content ref.
  The executor hashes the payload at entry behind capture_content_provenance;
  the sink hashes the output lazily. Only digests are kept, never content —
  provenance without re-spilling the data a PII scanner exists to redact.

First real consumer.

• cpex-plugin-delegator-oauth brackets both mint legs — the workload
  client_assertion base-token mint and the RFC 8693 exchange — with
  begin_effect/complete_effect, mapping a successful exchange to confirmed,
  a definitive IdP rejection to rejected, and a timeout/unreachable IdP to
  unknown (reconciled later, never assumed minted).

Reference sink.

• audit-logger now renders the verdict, ordered steps, span, taint, and
  (when enabled) content hashes.

Notable design decisions

• Fire on the verdict, not a phase. The two deny sites straddle the AUDIT
  phase; emitting at the executor's return points is complete by construction.
• Isolation as a type contract. The DecisionLog reaching an audit handler
  but never PluginContext is a real security property; an AuditHook family
  (not a manager special-case) makes "sees verdicts, cannot influence them"
  type-level.
• Err → unknown, not rejected. A failed mint may still have landed at the
  participant, so recovery reconciles it rather than assuming it didn't happen.
  The delegator, which knows a clean 4xx from a timeout, maps precisely.
• The reconciler is generic, not plugin-specific. It reads the
  self-describing EffectRecord and does a keyed ledger lookup — participant-
  specific at most, and today no participant offers one, so the default suffices.

Opt-in / compatibility

• No new source-level breaking changes. PluginPayload::audit_bytes() has a
  default (None); AuditHook/effect emitter/WAL/hashing all engage only when
  configured (effect_log_path, capture_content_provenance, emit_effect).
• New config knobs on plugin_settings: effect_log_path,
  effect_log_compaction_threshold, capture_content_provenance — all default
  to off.

Testing

• Unit + integration across cpex-core, audit-logger, and delegator-oauth:
  verdict-point emit, capability gating, WAL durability / concurrent-append
  integrity / recovery+compaction / reconciliation, perform_effect state
  mapping, span child-model, taint delta, content-hash gating, and the OAuth
  delegator emitting prepared → confirmed/rejected against a mock IdP.
• cargo fmt clean, cargo clippy --workspace --all-targets clean, full
  workspace test green.

Out of scope (follow-ups)

• Host-side trace-context propagation downstream so spans chain across hops.
• ocsf-audit mapping span()/on_effect into OCSF trace/span and the
  Authentication class.
• A real ledger reconciler (an issuance ledger with keyed lookup) — see
  docs/effect-ledger-integration-note.md; would make unknown truly resolvable
  and, if it sits in the mint path, deliver v2 structural enforcement too.

[Levaj2000]

Reviewed from the perspective of the OCSF audit plugin that will consume this seam.

Overall: this is the right architecture, and I would like to put three of its decisions on the record before suggesting anything — verdict-point emission (completeness by construction — observation-only sinks finally see denials), DecisionLog kept out of PluginContext (the observation-only contract becomes type-enforced rather than promised), and Err → unknown on effects (the honest crash semantics; a timed-out action may have landed). Those three are what make this an enforcement record rather than an enforcement log.

A few suggestions, all non-blocking:

1. Consider a monotonic sequence number on emitted decision records. Downstream evidence chains (ours included) can prove order of what they received, but not completeness — a sink can't distinguish "no denials this hour" from "denials emitted but never persisted." A per-executor (or per-stream) monotonic counter stamped at emission lets any downstream verifier prove completeness of an exported stream without trusting the exporter, and gives cross-restart continuity a handle. This is the same shape as the audit.sequence.stream_id chain-scoping idea now in the OTel Audit Logging draft, so there's convergent prior art if you want it.

2. A test pinning on_effect delivery on the begin_effect leg. Per Slack, both legs firing is the intention — suggest a test asserting a sink observes the prepared record before the effect body runs, not just the completion. Evidence-of-intent is the WAL's whole value; a regression that silently reduced sinks to completions-only wouldn't fail any current assertion (or if one exists and I missed it, ignore me).

3. Document canonicalization expectations for audit_bytes(). If two runs serialize identical content to different bytes, the resulting hashes lose cross-run comparability — an auditor can no longer say "same input" by digest. Doesn't need to be solved in this PR; a doc comment stating whether byte-stability is guaranteed (or explicitly not) would keep consumers from assuming it.

One design fact worth pinning in the docs (from our Slack thread): AuditHandler::handle is awaited at the verdict return point, not fire-and-forget. That's the right default for an evidence seam — a crash can't lose a verdict that was emitted, and sinks don't need drop-detection machinery for the steady state.

Two consequences worth a doc note: (a) it's now a contract consumers will design around — our chain relies on it, so a future "optimization" to fire-and-forget would be a silent semantics break, and it'd be good if a comment on the trait said so; (b) sink latency sits on the request path, so handler implementations need to stay cheap — ours is serialize + hash + append, and anything slower (network sinks, say) probably belongs behind an internal queue on the handler's side of the boundary, which might also be worth a sentence in the trait docs.

Happy to be the guinea-pig consumer: our plugin is a near-twin of audit-logger, so I'll port it against this seam as soon as the PR settles and report anything that doesn't match intent?

[jkershawrh]

Reviewing from the downstream persistence side and building the append-only ledger @terylt that we talked about. That would be the durable sink for these decision and effect records.

+1 on the monotonic sequence number and audit_bytes() canonicalization asks. We need both: the sequence number lets us prove completeness of an exported stream (no gaps), and byte-stable serialization lets content hashes mean "same input" across runs rather than "same serialization attempt."

The Err → unknown crash semantics on effects are exactly right for our use case. An immutable ledger can serve as the
reconciliation backend for unknown states. If there's no confirmed record in the chain, it didn't land. That's the
resolution path the EffectReconciler seam is shaped for.

Happy to collaborate on wiring up the integration once this settles. The ledger already has an OCSF adapter;
extending it to consume DecisionLog and EffectRecord as chained entries is a natural fit.

[Levaj2000]

Glad the sequence + canonicalization asks line up with what the ledger needs — two independent consumers wanting the same semantics is a good sign for the seam.

Would be great to compare notes on the OCSF shape: our plugin emits the 6003/ai_operation form with a DSSE-signed fingerprint chain, and if your adapter and our emitter agree on the event shape, evidence becomes portable across both sinks by construction.

Happy to share the reference bundle.

[jkershawrh]

Here's our OCSF adapter. It's thin by design: adapters/ocsf/

The mapping is straightforward: raw OCSF JSON goes in as content with content_type: "application/ocsf+json", keyed by
class_uid → entry_type, with correlation_id pulled from trace context. The ledger preserves the bytes unmodified and hash-chains them.

We don't have 6003/ai_operation yet. If your emitter's event shape lands as OCSF JSONL, adding it is a one-line class
map entry and the fingerprint chain comes through as content, byte-for-byte, which is where the canonicalization ask on the PR matters for us too.

Happy to share the full sample events and field mapping if useful, or just compare a sample 6003 event against what
the adapter expects. @terylt @Levaj2000 Maybe we should get a call together to discuss more, but either way this is great.

[terylt]

Thanks @Levaj2000 and @jkershawrh — Thank you for the feedback! I made the suggested changes. Let me know if you like how the sequencing is done or would prefer a different mechanism. Not sure if I over designed it.

Status on the four items:

2 — prepared-leg delivery test. Done. There's now a test that asserts a sink observes the prepared record before the effect body runs (not just at completion), so a regression that silently reduced sinks to completions-only fails a test. Evidence-of-intent is pinned.

3 — audit_bytes() canonicalization. Documented on the trait. The guarantee: identical content serializes to identical bytes across runs/processes (object keys sorted via a serde_json::Value/BTreeMap round-trip), so equal digests mean "same content" within a deployment — with the explicit caveat that it's sorted-key JSON, not full RFC 8785 (JCS): number formatting follows serde_json, stable within a version but not spec-guaranteed across toolchains. So: treat digest equality as same-content within a build; don't assume cross-toolchain canonicalization. Let me know if this works or we want something different.

4 — handle awaited contract. Pinned as a contract on the trait doc (a future switch to fire-and-forget would be a silent semantics break; consumers may rely on it), with the consequence spelled out: sink latency is on the request path (bounded per sink by the plugin timeout, panic-contained, sequential), so keep handlers cheap and put anything slow — a network sink — behind an internal queue on the handler's side. Same note is in the operator guide's "writing a custom sink" section.

1 — sequence numbers. Built, and this is the one I'd like your input on.

The design is two per-type streams plus a global order counter. Every decision and effect record now carries:

• stream_id + stream_seq — a gap-free counter within its own type (dec-… / eff-…, fresh per executor lifetime). This is the completeness handle, scoped the way the OTel audit.sequence.stream_id shape scopes it.
• emission_seq — a single global counter across both decision and effect emissions.

The reason we didn't use one shared counter for completeness: a consumer that receives only one record type would see phantom gaps. Concretely — if the OCSF side routes decisions to 6003/ai_operation and effects (a mint) to an Authentication event, a downstream that only holds the decision stream from a shared counter sees …41, 43, 44… and can't tell "42 was an effect I never receive" from "42 was dropped." Per-type stream_seq keeps each type's completeness provable independently.

emission_seq is then what lets a consumer that merges both — the ledger — reconstruct the interleave (a mint's prepared/confirmed emit during handle, so they carry lower emission_seq than the decision emitted at the verdict). Worth noting: because handle/on_effect are awaited in emission order (item 4), a merging sink already gets the interleave from its own append order — emission_seq makes that ordering a property of the record, provable and portable across sinks/re-exports rather than an artifact of one sink's write order.

So the question for you:

• @jkershawrh — you chain both into one ledger, so emission_seq is your interleave key and stream_seq is per-type gap-detection. Does (stream_id, stream_seq) + emission_seq give the ledger what it needs, or would you rather a single monotonic counter (you receive the full stream, so no phantom-gap problem on your side)?
• @Levaj2000 — if your emitter splits decisions and effects across OCSF classes/destinations, the per-type stream_seq is what avoids phantom gaps for a single-class consumer. Does the per-type + global split match the audit.sequence.stream_id model you had in mind, or would you scope it differently (e.g., one stream per class rather than per record-type)?

Happy to change the shape — it's cheap either way (a couple of atomics + fields). We leaned two-streams-plus-global because it degrades gracefully for both "merge everything" and "consume one type," but if you'd both rather a single counter (or a different stream scoping), say so and we'll match it before this lands.

And yes to porting against it — the seam is stable enough now that a guinea-pig consumer would flush out anything that doesn't match intent faster than we can guess at it.

Note, I also added some auditing documentation to the PR today.

[Levaj2000]

sample6003bundle.zip

@jkershawrh, Great — and the call's on the calendar for Friday, thanks @terylt for setting it up.

Taking you up on the sample exchange ahead of that. Attached zipped are two real 6003/ai_operation events from our emitter, an Invoke Tool and a Completion chained to it, plus a field-mapping doc against your adapter's conventions. They're the merged ocsf-schema#1661 shape on OCSF 1.9.0, emitted as JSONL, so your one-line class map entry should be exactly that.

Two things worth calling out from the mapping doc:

agent_id should come from ai_agent.uid, not metadata.uid. In ai_operation events, metadata.uid is the record id — it's what the next record's prev_event.uid points at — so the OpenShell-style metadata.uid → agent_id mapping would give you one "agent" per event. correlation_id maps cleanly from metadata.correlation_uid.

Byte-for-byte content preservation means our chain is verifiable from your stored entries alone. Strip fingerprint/signatures and the two unmapped.signature_* extras, JCS-canonicalize (RFC 8785), SHA-256, compare — no knowledge of our crate required. The mapping doc has the full recipe, including how unmapped.signature_b64/signature_key_id can populate your V3 writer_signature/signer_key_reference so both integrity layers cover the same bytes.

Which is also why I want to underline that there are now two independent consumers needing the same thing from audit_bytes(): our fingerprints commit to canonical bytes of the event, and your ledger's envelope commits to whatever bytes arrive. One canonical-serialization guarantee at the emitter keeps hashes comparable for every downstream consumer — happy to help spec that if useful.

On Teryl's sequencing question — from where we sit, streams mapping to separate entry_type chains (your parallel-chains scaling model) with emission_seq as the cross-chain interleave key seems like the natural fit, but that one's yours to call.

Thanks, sir.

[Levaj2000]

@terylt, very nicely done!

This matches what I had in mind, and the dual-stream split is the right call in my opinion — the phantom-gap rationale is exactly why. From the serializer side, (stream_id, stream_seq) maps directly onto our chain scoping: each stream becomes its own fingerprint chain (and, downstream, its own ledger entry_type), so within-stream gap detection falls out of the dense counter, and emission_seq gives us the cross-stream interleave for reconstructing total order — decision-before-effect causality without merging the chains.

Two things I think are worth pinning down in the docs so verifiers don't misuse one counter for the other's job:

Separate the claims. stream_seq is a completeness claim (dense within its stream — a gap means a missing record); emission_seq is an ordering claim only (a single-stream consumer will legitimately see it sparse). Stating that explicitly prevents someone from "detecting loss" off emission_seq gaps.

Restart semantics. For completeness verification to survive a crash, the counters need to be either durable across restarts or epoch-scoped with the epoch visible in the record (so (epoch, stream_seq) is monotonic and a verifier can distinguish "counter reset" from "records lost"). Given the Err→unknown crash semantics elsewhere in this PR, an explicit epoch/boot id feels consistent — works either way as long as it's stated.

And assuming both counters land inside audit_bytes(), they're covered by the content hashes — which is where this connects back to the canonicalization thread: sequence integrity and byte stability together are what make the downstream evidence chain verifiable end-to-end.

good stuff- Jeff

[jkershawrh]

Thanks both. The dual-counter design is what the ledger wants. stream_seq per type maps to our per-entry_type chains for gap detection on ingest; emission_seq goes into the entry body as metadata for cross-chain causal ordering. No changes needed from the ledger side.

@Levaj2000 - Jeff's two counter-semantics points are worth pinning:

1. Separating the claims - agree, make it explicit. stream_seq = completeness (dense, gaps mean loss); emission_seq =
   ordering only (legitimately sparse for single-stream consumers). The adapter will validate accordingly.
2. Restart semantics - an epoch/boot-id scoping stream_seq would be clean. The ledger needs to distinguish "counter reset" from "records lost."
   @terylt - does the current stream_id already capture this (e.g., dec-{boot_id}), or does it need an explicit epoch field?

On the sample data. Thanks Jeff. Will fix agent_id to pull from ai_agent.uid instead of metadata.uid, and map correlation_uid → source_id. The signature_b64 / signature_key_id → V3 writer_signature path is noted and will prototype in the adapter.

The byte-preservation point ties it together: if audit_bytes() is what both Jeff's fingerprint chain and the ledger's entry hash commit to, we get end-to-end verification from two independent integrity layers covering the same canonical bytes. Happy to help spec that boundary if useful.

Ready to port against this shape. See you Friday.

[terylt]

Thanks @Levaj2000 - I updated the PR per your comments plus one clarification on where the counters live.

Separate the claims (docs). Updated:

• stream_seq is a completeness claim — dense within (epoch, stream_id); a gap means a dropped record.
• emission_seq is an ordering claim only — monotonic across both streams within an epoch; a single-stream consumer sees it sparse by design, and those gaps are the other stream's records, never a loss signal.

That's on the set_stream doc, the EffectRecord fields, and a new "completeness vs. order" section in the operator guide.

Restart / epoch. Added an explicit epoch on every record. I went with the executor's boot time (Unix nanoseconds) rather than a uuid or a persisted counter, for two reasons: it needs no durable state (a fresh process just reads its clock), and because it's ordered, (epoch, emission_seq) is now a total order across restarts you can compute from the record alone — a new, larger epoch marks a restart, so a counter reset is distinguishable from a loss. Note, it's wall-clock, so a backward NTP step could in principle misorder two boots; if we ever want strict monotonicity we'd swap in a persisted counter (or a ULID), but for reset-vs-loss + cross-restart ordering the boot timestamp does the job with zero coordination.

One clarification on audit_bytes(). The sequence counters are not inside audit_bytes() — that hash is scoped to the payload content (provenance without the plaintext). The epoch/stream_seq/emission_seq fields live on the emitted record, so they're covered by a sink's full-event fingerprint — precisely what your emitter already computes over the whole 6003 event. So sequence integrity is end-to-end; it just rides on the event fingerprint, not the content hash. (Which also keeps the two hashes cleanly separated: audit_bytes answers "same input?", the event fingerprint answers "same record, unaltered?".) The canonicalization guarantee we documented is about audit_bytes — happy to help spec a matching one for the full-event bytes if that's useful for the two-consumer story.

And noted on the mapping-doc catches (agent_id ← ai_agent.uid, not metadata.uid; correlation_id ← metadata.correlation_uid) — those are on the emitter/adapter side rather than this seam, but good to have them pinned before Friday. Looking forward to the call.

[terylt]

@jkershawrh Yes, the last change I made just added in a epoch value.

I think there might be some confusion on the audit_bytes(). It canonicalizes the payload in a hash, not the entire audit record. It's a single field content.input_hash inside the audit record. I think the whole event record canonicalization would be handled in that auditor plugin?

[araujof]

A few things I noticed that haven't come up in the thread yet, mostly around the crash-safety story this PR is building:

• recover_effects doesn't seem to be called from any production startup path yet, only from tests. Worth wiring it into the FFI and Python bindings' init so a crash mid-mint actually gets reconciled on restart.
• Concurrent-phase plugins don't get the emit_effect capability the way serial-phase ones do. If a mint-performing plugin runs with mode: concurrent, begin_effect/complete_effect will no-op, which quietly defeats the fail-closed guarantee for that config.
• A TRANSFORM-phase plugin trying to deny gets ignored, since TRANSFORM can't block. Worth deciding whether that's intended, since the log ends up recording it as an allow.
• When a hook resolves to zero plugins, which is a normal case, the executor returns early and skips audit emission. Might be worth emitting a record there too for completeness.
• Serial-phase plugins don't have the panic containment that concurrent ones do. A panic between begin_effect and complete_effect would leave that WAL entry orphaned until recovery runs.
• Config hot-reload builds a fresh FileEffectLog while in-flight requests are still using the old one. There's a narrow race where the old instance's write and the new instance's compaction could overlap on the same path.
• audit-logger's empty hooks: case used to fail startup; now it's treated as intentional audit-only mode with an info-level log. Might be worth a slightly louder signal so a config typo doesn't go unnoticed.
• WAL recovery resolves by key across the whole file rather than per attempt. Not an issue today since the OAuth delegator always uses a fresh UUID, but worth keeping in mind if a future plugin reuses a stable idempotency key on retry.
• Aborted concurrent branches get logged as Error("aborted"), which reads the same as a real crash even though a comment elsewhere in the file treats this case as intentional and not an error.
• The OAuth delegator's completion write swallows its error silently, while the core primitive it mirrors logs on the same failure. Might be worth aligning the two so a failed write there isn't completely silent.

[jkershawrh]

Noted. Good catches across the board, @araujof.

Three of these touch the ledger integration directly:

recover_effects not wired into startup (#1). The ledger is shaped to be the reconciliation backend for unknown effect states. If there's no confirmed record in the chain, it didn't land. But that only works if recover_effects actually runs on restart. Without it, orphaned WAL entries with prepared status sit indefinitely, and the serial-phase panic case (#5) compounds this. A panic between begin_effect and complete_effect leaves an orphan that nothing reconciles. Worth wiring before the delegator goes live, even with the default LogUnknownsReconciler.

Zero-plugin hooks skipping audit emission (#4). Our adapter's gap detector will flag this as a stream_seq gap, which is the correct signal, but the root cause is "no record emitted" rather than "record lost in transit." If zero-plugin is a normal case, either emit a no-op decision record to keep the stream dense, or document it as an expected gap so consumers don't chase a phantom loss.

Concurrent-phase emit_effect (#2). If a mint-performing plugin runs with mode: concurrent, the ledger never sees the effect records. That's a silent hole in the audit trail. The decision chain shows an allow, but the effect chain has no corresponding prepared/confirmed. Worth deciding whether concurrent-phase effects should be gated at config validation (reject the combination) rather than silently no-op'd at runtime.

The rest are internal to the CPEX runtime and I'll defer to Teryl on those.

[Levaj2000]

@terylt Thanks — the doc updates and the epoch addition close out both callouts flagged. Confirming from the OCSF/evidence side:

Completeness vs. ordering split — the split reads exactly right now. Dense stream_seq within (epoch, stream_id) is precisely what our gap detector keys on, and documenting emission_seq as ordering-only (sparse by design for single-stream consumers) removes the ambiguity that was worrying me: nobody will misread an emission_seq gap as a dropped record.

Epoch semantics — boot-time-in-Unix-ns works for the fingerprint-chain model. We only need epochs to be distinct and increasing across restarts so (epoch, emission_seq) gives a total order, and "new, larger epoch marks a restart" maps cleanly onto how we scope chains. One non-blocking doc suggestion: this implicitly assumes the host clock doesn't step backwards across a restart (NTP correction, VM restore). Maybe worth a single sentence stating that assumption, since a non-increasing epoch would silently merge two boots into one chain scope.

audit_bytes() scope — payload-only hashing with sequence fields covered by the sink's full-event fingerprint is consistent with how we've mapped it: audit_bytes() feeds the payload digest, and the ledger's fingerprint covers the full record including epoch/stream_seq/emission_seq. No changes needed to the 6003 field mapping I posted earlier.

---

One addition to @araujof list, from the evidence-mapping angle — item 3 (TRANSFORM-phase denials ignored but logged as allows) is more than an enforcement-semantics bug for us. If the audit record says allow while the plugin's intent was deny, the mapped OCSF ai_operation event inherits that verdict as its disposition, and the exported evidence now contradicts what the plugin actually decided. Whichever way this is resolved (honor the deny, or reject deny-capable plugins at TRANSFORM registration), the invariant that matters for the audit seam is: the emitted record must reflect the actual enforcement outcome, never the discarded intent. If TRANSFORM denies are intentionally ignored, the record for that hook should say so explicitly rather than recording a plain allow.

Thanks, Jeff