feat: agent remote configuration (overlay, apply modes, safeguards)

[ccf-lisa]

Summary

The agent side of agent remote configuration:

• the agent polls an API-stored overlay (RFC 7396 merge over its file config);
• it applies the overlay according to its local remote_config.mode (off, report, apply_safe, apply_all), behind the shared Classify safeguards. With credentials and no mode set, the mode is report: applying is opt-in;
• it reports what it runs back to the API.

An overlay can change plugins.<p>.config, policy_data, labels, schedule, enabled, source, policies (OCI or local sources), policy_behavior, agent_evidence and verbosity. Inline policy bundles are out of scope: an overlay that sets policy_bundles is rejected with unknown-field.

Dependency: this pins compliance-framework/api#465 (head b53be8f), then gets re-pinned to the API release tag. Ship order: api#465, then this PR, then compliance-framework/ui#318.

What's in it

Config, state and reloads

• Declared config: api/pkg/agentconfig.Config is the declared form, and the existing structs remain the runtime form. The config hash is byte-identical, pinned by golden tests.
• Per-config state dir and instance ID: --state-dir/CCF_STATE_DIR and --instance-id/CCF_INSTANCE_ID.
• Prepare-then-cancel reloads: for file and remote triggers. An invalid file or a failed download no longer exits; the agent keeps running the last good configuration, with a 5-minute drain.
• Startup ladder: fetched overlay, then last applied, then the file alone.

Remote config (cmd/reconciler.go, cmd/report.go)

• Cache: 0600, checksummed and bound to the API URL and client ID. The opaque ETag is sent back verbatim.
• prepare:
  1. ValidateOverlay;
  2. the Classify/WillApply mode gate (forbidden keys are rejected in every mode);
  3. Merge;
  4. Validate;
  5. ${env:} resolution in plugins.*.config;
  6. prefetch.
• Rejections and failures: a rejected revision is remembered per (ETag, base). It is re-reported after a restart, so the API keeps showing rejected, with the reason and unsafe list. Failed applies back off from 1 to 10 minutes.
• Reports:
  • contents: the redacted, unresolved base and effective config, revisions, status, reason, unsafe list and warnings; and plugins[] with lib-version, read from each plugin binary's Go build info (diagnostics only, nothing is gated on it);
  • redaction and the digest come from the API's pkg/agentconfig (Redact, Digest): secret-like keys and secret-looking values (URL passwords, PEM private keys, password= assignments, known token formats), plus literal text mixed with ${env:} under secret-like keys. The agent does not re-implement it;
  • sent at startup, on change and every 24h, with backoff on 401/403/404/409/413; oversized reports drop base.
• Heartbeat: carries config_revision / config_digest.
• Evidence props: agent-config-revision.

Evidence (runner)

• Evaluation-time artifact uploads are unchanged from main (runner/policy_artifacts.go is byte-identical to main). There is no report-time policy bundle inventory and no configuration-time artifact upload.
• _policy_source for old plugins: plugins that send no policy evaluations still get _policy_source/_policy_digest, keyed from the evidence's _policy_path label.

Plugin environment

• Plugins get the host environment minus CCF_API_AUTH_*.

policy-manager (used by plugins built on this lib)

• Unchanged from main. Evidence seeding is pinned by the new evidence_seed_test.go.

Docs

• docs/configuration.md: remote config (default mode report), the classification table, redaction, plugin library versions, and the state directory.
• docs/adr/0003-remote-config-overlay.md.
• docs/policy_artifacts.md, docs/running_as_a_service.md, AGENTS.md, README.

Notable decisions

• Inline policy bundles are out of scope (decision of 2026-10-02). Keeping vendor evidence streams, checking the policy contract and sandboxing the checks made them the largest and riskiest part of the change. The agent therefore has no policy errors, no path shadowing and no plugin-lib gate.
• Applying is opt-in (security review, finding 1): an unset remote_config.mode with credentials is report, not apply_safe.
• No policy bundle inventory or report-time uploads (security review, findings 4 and 5): the report carries no policy-bundles[] and the API dropped the artifact file routes.
• File validation shared with the API stays non-breaking: a bad schedule in the file skips that plugin, and a negative verbosity or a literal ${env:} outside plugins.*.config is only a warning (R34). An unset variable that the file references is passed through unchanged (R60).

Testing

• With GOWORK=off against api b53be8f, all of these are clean: go build, go vet, make test (race and coverage), gofmt, make check-opa-version, make build, a GOOS=windows build, and golangci-lint v2.11.4 on the diff from main.
• Reconciler tests:
  • invalid edits, races, the startup ladder, 304 handling;
  • rejected memory, including the restart re-report;
  • backoff and truncation.
• Report tests: redaction (env-sourced values, a dsn mixing literal text and placeholders, a password in a URL), truncation, and plugin lib-versions.
• Mode tests: credentials without a mode (with and without a remote_config block) run in report: they report not-applicable, never fetch, and don't apply a cached applied overlay.
• The local-dev end-to-end runs were done on the earlier, larger scope and have not been repeated on this one.

Release notes

• Remote config is opt-in. Agents with api.auth credentials and no remote_config.mode now run in report mode: they report their configuration but no longer fetch or apply remote configuration (including a cached overlay) unless remote_config.mode (or CCF_REMOTE_CONFIG_MODE) is set to apply_safe or apply_all.
• Reports redact by value too. Effective digests change for configs with newly masked values; agent and API must use the same pkg/agentconfig for the reported digest to match.
• The report's policy-bundles field is gone (the API ignores it if an older agent build sends it).

Known limitations

• Old plugins upload no evaluation artifacts. Plugins built on old agent libs send no PolicyEvaluation, so their evidence has no input or policy artifacts and can't be played back.
• Dev stacks need a state wipe. State layouts from earlier in-branch iterations are not migrated (never released). A remote config cache that remembers a rejection with policy errors fails its checksum and is reported once as cache-corrupt.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 4879d585-73e9-4261-9e5a-f59bac38dac2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gusfcarvalho]

Findings are inline, reviewed at b9c1cad. build, vet, test -race and gofmt are clean locally, and CI is green. There are 6 blocking threads:

• A denied builtin reachable from authored Rego still executes during the _test.rego run. This is security-relevant and was probe-confirmed.
• File configs that load on main are now fatal (R34/R51).
• A no-op revision is never recorded as applied.
• A remembered-rejected overlay is re-prepared on every poll.
• SIGTERM is lost during the 5-minute reload drain (R33).
• The agent-run evidence for startup download failures is gone.

The rest are non-blocking suggestions and one owner question. The overall shape (prepare-then-cancel, the agent-side Classify gate, opaque ETags, env filtering and redaction) looks solid.

[gusfcarvalho]

Re-review of b9c1cad..0c1a8c7: all 15 threads from my previous review are addressed, and I verified each fix. This supersedes my REQUEST_CHANGES.

• Security: Check now returns before the tests when a denied builtin is reachable. The tests also run under SandboxCapabilities, with every denied builtin rewritten to a stub that errors. I re-ran my probe: 0 requests reached a local server across authored→vendor helper, with f as http.send, dynamic data[p][q] refs, walk(data), a vendor test calling http.send, and, with the walk bypassed entirely, direct/http["send"]/with calls and opa.runtime.
• R34/R51/R60: a file with verbosity: -1, a literal ${env:} in labels/policy_data, or an unset ${env:X} in config (whole or embedded) loads, and the plugin gets exactly the literal values main passes. All are reported as warnings and no plugin is skipped.
• Reconciler: a no-op revision is adopted in place: one Prefetch, then applied-revision/report/heartbeat = 2, same runtime, clean under -race. The remembered-rejected early return, the SIGTERM handling during the drain, the restored startup-failure evidence, the empty-ETag key, the notify-after-bind ordering, the file-only backoff, the bounded network phase and GC after each swap all look correct.

build, vet, test -race and gofmt are clean locally, and CI is green. One non-blocking note for the symlink follow-up is inline.

[ccf-lisa]

PR approved. Marking ready for e2e.

[ccf-lisa]

Round 2 (design §13: R62, R63, R65, R66, R67)

API pin bumped to api#465 round 2 (6c801a3). Commits 66af7bf..f88bde9.

R62: one channel for policy sources

• New internal/policytree: ReadTree + TarFiles (sorted, fixed headers). It is the only archiver, so a tree uploaded at config time and at evaluation time produces the same bytes and the same artifact. A symlinked root now works; inner symlinks are skipped, as OPA does.
• runner.ArtifactUploader is exported and keyed per API. One instance per process is shared by the reconciler (remoteAPI.UploadArtifact) and every plugin run (runner.WithArtifactUploader).
• At report time (mode ≠ off, only for the candidate that runs, within the remote request timeout), the reconciler uploads every reported tree once, memoized by tree digest per API:
  • inline bundles
  • the tree each one extends
  • OCI and local sources
• It fills artifact-digest on policy-bundles[] and on policy-bundles[].extends. The field survives truncation.
• Failures never reject or fail a revision; the field is left empty:
  • 404/405 (old API): backoff
  • 400/413: not retried
  • 429/5xx/transport errors/timeouts: retried on the next report

R63: policy contract (agent is authoritative)

• After compile and tests, inlinepolicy.Check runs CheckContract only on vendor-only packages (as warnings). It never re-runs it on authored modules.
• Then a sandboxed dry run on {} plus policy_data through policyeval.Execute and policy-manager GetRiskTemplates.
• Results are located PolicyErrors with contract codes:
  • error for packages with an authored non-test module
  • warning for vendor-only packages, for conflicts on {}, and for input-dependent titles
• A broken package is excluded and the dry run retried, so it doesn't hide the others.

R65: a compile error in a vendor file whose package an override also defines gets the hint vendor test references rules removed by the override of `<file>`; keep the rule or add `delete: [<test>]` . It stays an error. The exact e2e repro is a test.

R66: if a plugin uses an inline bundle and the same package appears in two of its policy paths, the agent emits a duplicate-policy-package warning that names both paths.

R67: stable evidence identity

• Plugins receive <state>/inline/<name>/current/bundle.
• current is a symlink, swapped by an atomic rename to the content-addressed <name>/<digest>/bundle.
  • Why not <name>/current: OPA's dir loader does not descend into a symlinked root directory and would load nothing.
• Swap timing: the run loop swaps only between configuration runs (after the reload drain) and on fallback. Never at prepare time.
• The API helper resolves the path when the run starts, so evidence artifacts are the tree that was actually evaluated.
• GC and activation share a lock. GC never deletes the tree current points to.
• Tested: two inline revisions with an unchanged package produce the same policy_file and evidence UUID.

UI wire changes

• artifact-digest on PolicyBundleReport and PolicyExtendsReport (sha256:<64 hex>).
• New policy error codes:
  • policyeval.Issue* (static contract and dry run)
  • eval-error, eval-conflict, dry-run-timeout
  • duplicate-policy-package (R66)
• R65 hints are appended to the compile error message (no code).

Behaviour change: an authored package with no title now rejects the revision, because it would record no evidence. Fixtures were updated.

Checks: go build, go vet, gofmt, go test -race ./... and make check-opa-version are green, and GOOS=windows builds. Three self-review passes were run; deferred lows are in the lisa-selfreview note. Docs updated: configuration.md, ADR 0003, policy_artifacts.md.

🤖 Generated with Claude Code

[gusfcarvalho]

Round-2 (§13) re-review of 0c1a8c7..f88bde9: approve. build, vet, test -race and gofmt are clean locally, and CI is green. What I verified:

• R67: I confirmed independently that OPA v1.14.1 loads 0 modules from a symlinked root, 1 from the real dir and 1 from current/bundle, so the <name>/current/bundle deviation is justified. TestActivate_EvidenceIdentityIsStable_R67 goes through policy-manager Execute/GenerateResults and fails on empty results, so it proves that modules load through the stable path and that an unchanged package keeps its UUID. The swap happens only in start() after the previous run returned (and on fallback), under inlineMu. GC runs on the same lock, keeps active, pending, starting and fallback plus the current target, and runs on the materializing goroutine.
• R62: both upload paths use policytree.ReadTree and TarFiles, so the tars are identical. Uploads run at report time and never feed status. The memo resets when the remote client is rebuilt.
• R63: the dry run uses sandboxed copies with stubs and SandboxCapabilities, only after the reachability check and tests. 0 requests reached a probe server across authored→vendor helper, with … as http.send, a vendor test, a vendor violation rule and risk_templates calling http.send. Severity is an error only for authored non-test packages; timeouts are bounded warnings. The wording-based classification can only fail open (a warning), never reject wrongly, and tests pin the codes at this API pin.
• R65/R66: the e2e repro test and the duplicate-package test are both in place.

One non-blocking owner question is inline (bundle data.json in artifacts vs Redact). The ConfigMap follow-up thread stays open, as agreed.

[gusfcarvalho]

Re-reviewed bce3ad3: M1 (view Ensure errors now only fail that plugin's runs; blast-radius + restart test), M2 (_inline), R88 removal, shared policyeval/regocheck helpers, and the per-(bundle, plugin) policy-stream-forked warning for unshadowed extends bundles all look right. CI green. Note for release: pluginlib.MinPolicyID = v0.9.0 must match the release that ships this PR. Dev stacks need a state wipe (R67 migration removed).

[gusfcarvalho]

One logic flaw found while bringing the RC stack up on bce3ad3 (pre-existing, not from this round): a remembered rejection isn't re-reported after restart, so the instance shows 'pending'. Small fix inline.

[gusfcarvalho]

Re-reviewed e8afe72: remembered rejections are re-reported at startup and kept through 304/fallback re-prepares (unsafe + policy errors persisted, bounded); restart test covers both paths. CI green.

[ccf-lisa]

Closing in favour of a stack of smaller PRs (stack #112). The top of the stack is byte-identical to this branch merged with main; each PR builds, vets and passes go test -race ./... on its own.

1. test: pin agent configuration hashes and evidence seeds #101 test: pin agent configuration hashes and evidence seeds
2. feat(runner): policy source from the _policy_path label; extra evidence props #102 feat(runner): policy source from the _policy_path label; extra evidence props
3. refactor(config): adopt api/pkg/agentconfig as the declared config #103 refactor(config): adopt api/pkg/agentconfig as the declared config
4. feat(agent): stable instance ID and per-instance state directory #104 feat(agent): stable instance ID and per-instance state directory
5. feat(agent): AgentRunner primitives for prepare-then-cancel reloads #105 feat(agent): AgentRunner primitives for prepare-then-cancel reloads
6. feat(agent): prepare-then-cancel config file reloads #106 feat(agent): prepare-then-cancel config file reloads
7. feat(pluginlib): read the agent library version a plugin binary was built with #107 feat(pluginlib): read the agent library version a plugin binary was built with
8. feat(agent): report the configuration to the API (remote_config off/report) #108 feat(agent): report the configuration to the API (remote_config off/report)
9. feat(agentstate): persisted cache for the remote configuration overlay #109 feat(agentstate): persisted cache for the remote configuration overlay
10. feat(agent): pull and apply remote configuration overlays (apply_safe/apply_all) #110 feat(agent): pull and apply remote configuration overlays (apply_safe/apply_all) — size-exception (≈780 of its 1417 lines are tests)
11. feat(config): ${env:NAME} placeholders in plugin config; ADR 0003 #111 feat(config): ${env:NAME} placeholders in plugin config; ADR 0003

Review threads on this PR stay here for reference. The branch lisa/in-flight-agent-config is kept.

🤖 Generated with Claude Code