Approval outcomes as signed receipt evidence

Scope (rescoped 2026-10-22)

Keep approval outcomes and their receipt evidence for the loan-review governance PoC.
This issue no longer covers an RL training loop or a separate feedback database.

Context

Main already contains related work from #206 and #207.
pkg/approval handles approve, reject, and edit decisions stamped by the admission webhook.
pkg/receipts records human-layer decisions with a pseudonymous approver digest.
This rescope does not mark that work complete for the PoC.

Acceptance

Approve, reject, and edit outcomes from the loan-review console each produce a signed human-layer receipt.
Each receipt binds the outcome to the pending action and the approver digest.
Offline verification passes for an untouched chain and fails after tampering.
A recorded approval decision is not lost if the operator restarts before its receipt is written. Carried over from RL feedback persistence — store feedback history in etcd/PVC for crash recovery #88.
Tests cover each outcome and the restart case.

Non-goals

RL training, calibration, or threshold tuning.
A Postgres feedback_history table or a Kafka topic.

Original scope before the 2026-10-22 rescope

RL feedback controller — capture approve/reject signals from OPA gates

Build the RL feedback controller that listens to OPA approval/rejection events and persists them as training signal.

**Acceptance:**
- [ ] OPA approval webhook fires ApprovalEvent → Postgres feedback_history
- [ ] Event includes: agent_id, action_class, proposed_confidence, human_decision, decision_latency_ms
- [ ] Audit trail for compliance

Deep research synthesis (2026-04-26)

Full research note (~2.5k words, 12 primary sources): Obsidian/NineVigil/01-Research/RL Feedback Loop - Deep Research. Posting the load-bearing conclusions here so the design lives on the issue.

TL;DR design choice for v0.2.0

Ship as per-bucket online recalibration + threshold optimisation with ADWIN drift detection and shrinkage-based cold-start, marketed as the "RL feedback loop". Not full RL, not a neural policy network, not contextual bandits — the data regime (~10²/week feedback) does not support those, and the literature shows they fail badly at this scale.

Why not full RL (PPO / RLHF style)

InstructGPT (Ouyang et al. 2022) used ~33k preference comparisons just to bootstrap a usable reward model. We will see that in two years, not in a milestone.
Self-Rewarding LMs (Yuan et al. 2024) explicitly documents the reward-hacking failure mode that hits exactly this kind of feedback loop. Mitigation in our design: agent never sees the threshold.

Why not contextual bandits (yet)

Even Thompson sampling needs O(dozens) of impressions per (persona, action_type) cell. With ~50 cells and ~10²/week, per-cell convergence takes ~12+ months.
Defer to v0.3 once persistence (RL feedback persistence — store feedback history in etcd/PVC for crash recovery #88) gives us a year of history.

Why calibration + threshold tuning IS the right frame

Tian et al. (EMNLP 2023) shows verbalized LLM confidence (which we already collect) reduces calibration error ~50% vs logit-based — but RLHF models are systematically miscalibrated, so we must recalibrate against ground truth.
Xiong et al. (ICLR 2024) shows LLMs systematically overconfident; consistency-across-samples helps. Cheap upgrade for v0.3.
Sparrow (Glaese et al. 2022) shows per-rule rejection signals carry far more information than thumbs up/down. → CLI command (agentctl approve / agentctl reject — CLI commands for human-in-the-loop feedback #84) must let the human say which OPA rule they think failed, not just --reject.

Drift detection: ADWIN per bucket

river.drift.ADWIN on the per-bucket reward stream (1=approved, 0=rejected). When drift fires:

Clamp threshold back to global prior.
Set Status.Conditions[type=CalibrationDrift]=True on AgentPersona.
Emit Prometheus alert.

This catches model-upgrade drift, persona-behaviour drift, and tool-set changes. Without it we silently auto-approve dangerous actions for weeks.

Cold-start (first 50 decisions per bucket)

Empirical-Bayes shrinkage to a higher-than-steady-state global prior. Fall back through (bucket → persona-marginal → action-type-marginal → global). Better to over-ask the human in week 1 than to under-ask.

Concrete data model (informs #88)

FeedbackEvent {
  workloadName, actionUID, persona, actionType, severity,
  reportedConfidence, consistencyRate?,
  decision: enum{auto_approved, human_approved, human_rejected, expired},
  rejectedRule?: string,    # which OPA rule the human said failed (Sparrow pattern)
  ts, tenantID
}

Persisted in Postgres (same DB as LangGraph checkpoints), not etcd.

Open design questions (need decisions before #82 implementation lands)

Cost asymmetry α/β for agent does wrong thing vs extra human ask. Default proposal: α=100, β=1, configurable per AgentPersona.
Cross-tenant federation of the global prior — opt-in per tenant via feedbackContributesToGlobalPrior: bool?
Expose calibrated Status.CurrentThreshold per persona in CRD status — yes/no. (My recommendation: yes; hiding it makes ops blind.)
Benchmark target for Benchmark: measure escalation rate reduction over 100 feedback cycles #87 — proposed: 40% reduction in human-approval requests over 100 cycles, ≤ 2% post-execution-rejection rate. Doable; documented; if missed, defines the v0.3 backlog.

Cascade impact on sibling issues

Bucket calibration + threshold optimiser (was: Policy network training loop) #83 rename suggested: "Bucket calibration + threshold optimiser" (no neural net in v0.2)
agentctl approve / agentctl reject — CLI commands for human-in-the-loop feedback #84 must emit --rejected-rule flag, not just bool
Confidence drift tracking — detect when agent behavior regresses after updates #85 = ADWIN per bucket, exactly this design
Agent evolution dashboard — Grafana panels for escalation rate, confidence over time, feedback volume #86 dashboard panels: per-bucket calibration curves, drift events, threshold-over-time
Benchmark: measure escalation rate reduction over 100 feedback cycles #87 benchmark = the 40%/2% goal above
RL feedback persistence — store feedback history in etcd/PVC for crash recovery #88 schema = FeedbackEvent above

Upgrade path (no rewrites required)

When	Add	Source
v0.3	Conformal coverage guarantee on auto-approve rate	Angelopoulos & Bates 2021
v0.4	LLM-as-judge augmentation (RLAIF) when human feedback is sparse	Lee et al. ICML 2024
v1.x	Contextual bandits per bucket once data density supports it	Vowpal Wabbit / river

Each swaps one component without touching the controller.

Sources (full bibliography in research note)

12 primary sources: arxiv 2207.05221, 2212.08073, 2203.02155, 2209.14375, 2309.00267, 2401.10020, 2305.14975, 2306.13063, 2107.07511, river docs, Langfuse user-feedback docs, LangSmith.

cc #83 #84 #85 #86 #87 #88 — this synthesis is the input for those design discussions.

feat(cli): add agentctl reject command with --rule and --reason flags (…

Rescoped to approval outcomes and receipt evidence for the loan-review governance PoC.
The RL training loop and the separate feedback database are outside this phase.
The restart durability requirement from #88 is preserved here.
#88 closes as not planned.
Related code exists on main, but this issue is not marked completed.
Labels now reflect the current scope, and the old RL milestone was removed.
The original scope is kept in the body.

Navigation Menu

Repository navigation

Approval outcomes as signed receipt evidence #82

Scope (rescoped 2026-10-22)

Context

Acceptance

Non-goals

Deep research synthesis (2026-04-26)

TL;DR design choice for v0.2.0

Why not full RL (PPO / RLHF style)

Why not contextual bandits (yet)

Why calibration + threshold tuning IS the right frame

Drift detection: ADWIN per bucket

Cold-start (first 50 decisions per bucket)

Concrete data model (informs #88)

Open design questions (need decisions before #82 implementation lands)

Cascade impact on sibling issues

Upgrade path (no rewrites required)

Sources (full bibliography in research note)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Approval outcomes as signed receipt evidence #82

Description

Scope (rescoped 2026-10-22)

Context

Acceptance

Non-goals

Activity

shreyanshjain7174 commented on Apr 26, 2026

Deep research synthesis (2026-04-26)

TL;DR design choice for v0.2.0

Why not full RL (PPO / RLHF style)

Why not contextual bandits (yet)

Why calibration + threshold tuning IS the right frame

Drift detection: ADWIN per bucket

Cold-start (first 50 decisions per bucket)

Concrete data model (informs #88)

Open design questions (need decisions before #82 implementation lands)

Cascade impact on sibling issues

Upgrade path (no rewrites required)

Sources (full bibliography in research note)

shreyanshjain7174 commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions