Skip to content

Expanded redaction rules (core v0.85.0), saved upstream keys, rule text from core - #129

Merged
fylorn merged 3 commits into
devfrom
feat/redact-rules-expand
Oct 29, 2026
Merged

fylorn merged 3 commits into
devfrom
feat/redact-rules-expand

Conversation

@fylorn

@fylorn fylorn commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

What this changes

thinkwatch-core moves to v0.85.0, which grows the shared redaction catalog from 30 built-in rules to 120 in 20 groups (more prefixed credentials, credentials by position, cryptocurrency keys, and personal information). The gateway now recognises the API keys saved for its providers by value, and the console shows rule names and descriptions as core writes them, in English and Chinese.

Why

Many providers' keys have no prefix (32 hex digits, UUIDs), so no pattern rule can find them, yet the gateway knows each one exactly. Keeping a second copy of rule names in the console meant every new core rule showed up here as a bare id.

Details

Saved upstream keys

  • Collected from each active provider's credential headers (Authorization with the scheme left out, x-api-key, api-key, x-goog-api-key, and custom headers whose name says key/token/secret/auth/password) and a Bedrock provider's secret access key; values shorter than 12 characters are left out. Configuration headers such as anthropic-version are not collected.
  • Built when the providers load (startup, provider CRUD, the cross-instance router reload); the guards and the at-rest redactor are recompiled only when the list changes. One compiled set (tw_guard Saved) is shared by outbound redaction, the at-rest redactor, dangerous commands ("credential sent to an unknown host" allows each key to its provider's host) and the console trial (trial::run_with).
  • Audit: the thinkwatch-saved-secret event carries sources: [{"kind": "upstream-key", "name": <provider>}] from core's Finding::source, and never a value, masked or not. Per-rule aggregation is unchanged.

Console

  • Rule names, reasons, descriptions (name_zh, why_zh, about / about_zh) and group names (kinds) come from the rule view. The console's own tables (contentSecurity.kind, check, redactRule, contentRule, contentWhy, toolRule, cardNetwork) and the per-matcher redaction descriptions are removed; check-i18n.mjs is updated accordingly.
  • Security-log rows and trial hits look the rule name up by id. A saved-key hit shows the provider whose key it was. The rule search also matches prefixes, field names and descriptions.

Other

  • The one-time conversion of the old PII pattern list switches on or off only the four personal-information rules that list could express; new personal rules keep their factory setting.
  • CHANGELOG: entry under Unreleased, including the "Read before upgrading" notes (new credential rules ship on; some keys now report under their own rule id and the metric's kind label follows the new groups).

How it was verified

  • cargo fmt --check, cargo clippy --workspace --all-targets: clean.
  • Unit tests: 900 passed. New tests cover collection (credential vs. configuration headers, scheme stripping, short values, duplicates, Bedrock secret, Debug without values), matching a saved key as written and JSON-escaped with its source named, and per-rule audit aggregation with sources and no masked values.
  • Integration suite against fresh Postgres, Redis and ClickHouse containers: 482 passed. The new provider_credentials_reach_redaction_when_the_providers_load covers collection with the provider's host, guard recompilation, the key replaced on the wire and restored for the caller, the trial reporting thinkwatch-saved-secret, no recompilation when nothing changed, and removal when the provider is deactivated.
  • Web: check-i18n, eslint, vitest (202 passed), tsc -b, vite build.

🤖 Generated with Claude Code

fylorn and others added 3 commits October 29, 2026 21:15
…xpanded rule catalog

Prepares the console and gateway for thinkwatch-core's expanded redaction
catalog (about 115 built-in rules in 20 kinds).

- Saved secrets: the API keys in the providers' credential headers
  (the auth scheme stripped) and Bedrock secret access keys are collected
  whenever the providers load (startup, provider CRUD, the cross-instance
  reload), under the provider's name with its base URL host. The guards
  and the at-rest redactor are recompiled when they change. They reach the
  rule set in one place, `guard_policy::saved::with_saved`, which hands
  them to core once `RuleSet::with_saved` is released.
- Audit: a saved-secret event names the provider (`sources`), never any
  part of the value, masked or not; aggregation per rule is unchanged.
- Legacy PII list conversion: switch on or off only the personal rules
  the old list could express; new personal rules ship as they are.
- Console: group headers for the new kinds in core's order, saved keys
  first; zh/en names for every new rule id; descriptions for the new
  matcher shapes (prefixes, saved, assignment, labeled) and for rules
  checked in code; the security log shows whose key a saved-secret hit
  was; the rule search also matches prefixes and field names.

Temporary: the core crates are pinned to the branch
`feat/redact-rules-expand` until the expanded catalog is released;
re-pin to the release tag before merging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Saved secrets: the providers' keys are compiled once with core's
  `Saved` and handed to every rule set (`RuleSet::with_saved_set`, in
  `guard_policy::saved::with_saved`), to dangerous commands
  (`Rules::with_saved`, so a provider key sent to its own host is not
  flagged) and to the console's trial (`trial::run_with`).
- Audit: a saved-secret event's `sources` now come from core's
  `Finding::source` instead of searching the request again.
- Console: rule names, reasons and descriptions (`name_zh`, `why_zh`,
  `about` / `about_zh`) and group names (`kinds`) come from the rule
  view in both languages. The console's own tables of rule names, group
  names, check texts and card networks are removed, and so are the
  per-matcher descriptions the server's `about` replaces. Security log
  rows and trial hits look the name up by rule id.

Temporary: the core crates are pinned to the branch `feat/redact-catalog`
until the expanded catalog is released; re-pin to the release tag before
merging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The core crates move from the temporary branch pin to the v0.85.0 tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fylorn
fylorn merged commit d90345e into dev Oct 29, 2026
7 checks passed
@fylorn
fylorn deleted the feat/redact-rules-expand branch October 29, 2026 21:44
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant