Redaction exception values and scan cache (core v0.86.0)

CHANGELOG.md

@@ -15,11 +15,14 @@ Secrets and privacy recognises far more: the built-in redaction catalog
 grows from 30 rules to 120 in 20 groups, and the API keys saved
 for the providers are recognised by value, whatever their format. The
 console shows the rules' names and descriptions as thinkwatch-core
-writes them, in both languages. The thinkwatch-core crates move from
-v0.83.1 to v0.85.0.
+writes them, in both languages. A value recognised by mistake can be made
+an exception without switching its rule off, and long conversations are
+no longer scanned from the start every turn. The thinkwatch-core crates
+move from v0.83.1 to v0.86.0.
 
-No new setting, column or route; the redaction policy's shape is
-unchanged.
+No new setting or column. The redaction policy gains one optional field
+(`exceptions`), with endpoints of its own under
+`/api/admin/security/redact/exceptions`.
 
 ### Read before upgrading
 
@@ -40,7 +43,7 @@ unchanged.
 
 ### Added
 
-- **Expanded redaction rules** (thinkwatch-core v0.85.0): about 50 more
+- **Expanded redaction rules** (thinkwatch-core v0.85.0 and later): about 50 more
   prefixed credentials (AI services, code hosting, cloud, payments and
   collaboration), credentials by position (request headers, command
   lines, URL parameters, bot webhooks, configuration assignments),
@@ -67,8 +70,32 @@ unchanged.
   and no masked values; the security log shows it as the provider's API
   key.
 
+- **Exception values** (`security.redact.exceptions`, the Exceptions
+  section under the Secrets and privacy rules). A value listed there is
+  neither replaced nor recorded, whichever rule finds it; the rules keep
+  working for every other value. A value is added by pasting it: the
+  server hashes it on arrival and keeps only its SHA-256, its masked form,
+  the rule that recognised it and when it was added, never the value. At
+  most 500; at the cap, adding is unavailable. Saved provider keys cannot
+  be exceptions. Listing them takes `settings:read`; adding and removing
+  take `settings:write` and `pii_redactor:write`, and each is written to
+  the audit log (`security.redact_exception_added`,
+  `security.redact_exception_removed`) with the rule and the masked value
+  only. The trial marks an excepted value as left as is.
+- **Unchanged conversation history is not scanned again.** Each gateway
+  instance keeps, in memory, where the redaction rules matched in the
+  parts of recent requests it has scanned (positions and rule ids, never
+  values); the next turn of the same conversation takes those parts from
+  it and scans only what is new. What is found is the same as without it.
+  The cache is rebuilt whenever the rules, the providers' keys or the
+  exceptions change.
+
 ### Changed
 
+- **A redaction policy written through `PATCH /api/admin/settings` keeps
+  the stored exceptions**, whatever it carries: they are changed only
+  through their own endpoints, and the console writes the rest of the
+  policy back from the rule view, which does not carry them.
 - **Rule names and descriptions come from thinkwatch-core.** The security
   page shows each built-in rule's name, reason and description and each
   group's name as the rule view gives them, in English or Chinese; the

Cargo.toml

@@ -70,10 +70,10 @@ opt-level = 3
 # never re-exported through a local shim. And the reverse: something only
 # this side uses (the at-rest crypto, IMDSv2 credentials, the gateway error)
 # lives here, not in core.
-tw-bedrock = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.85.0" }
-tw-breaker = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.85.0" }
-tw-dialect = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.85.0" }
-tw-guard = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.85.0" }
+tw-bedrock = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.86.0" }
+tw-breaker = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.86.0" }
+tw-dialect = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.86.0" }
+tw-guard = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.86.0" }
 
 # Web framework
 axum = { version = "0.8", features = ["macros", "ws"] }

crates/common/src/guard_policy/exceptions.rs

@@ -0,0 +1,102 @@
+//! Exception values of outbound redaction (`security.redact.exceptions`):
+//! values that are never replaced or recorded, whichever rule finds them.
+//!
+//! They live in the redaction policy, as thinkwatch-core defines it
+//! (`tw_guard::policy::RedactException`): **the SHA-256 of each value and
+//! its masked form, never the value**. The value an admin pastes is hashed
+//! as soon as it arrives and goes no further.
+//!
+//! **They are edited only through their own endpoints**
+//! (`/api/admin/security/redact/exceptions`). The console writes the rest
+//! of the policy back whole, rebuilt from the rule view, and the view does
+//! not carry them: a policy written through the settings endpoint keeps the
+//! stored exceptions ([`keep_stored`]) rather than losing them.
+
+use serde_json::Value;
+use tw_guard::policy::RedactException;
+
+/// The exceptions a stored redaction policy holds. A policy that is
+/// missing, or whose exceptions cannot be read, holds none.
+pub fn stored(policy: Option<&Value>) -> Vec<RedactException> {
+    policy
+        .and_then(|p| p.get("exceptions"))
+        .and_then(|e| serde_json::from_value(e.clone()).ok())
+        .unwrap_or_default()
+}
+
+/// `policy` (a stored redaction policy, or none) with `list` as its
+/// exceptions; the field is left out when the list is empty, as the
+/// factory policy leaves it out. Everything else stays as stored.
+pub fn with_list(policy: Option<Value>, list: &[RedactException]) -> Value {
+    let mut policy = match policy {
+        Some(Value::Object(map)) => Value::Object(map),
+        _ => Value::Object(Default::default()),
+    };
+    let map = policy.as_object_mut().expect("an object, just made one");
+    if list.is_empty() {
+        map.remove("exceptions");
+    } else {
+        map.insert(
+            "exceptions".into(),
+            serde_json::to_value(list).expect("exceptions serialise"),
+        );
+    }
+    policy
+}
+
+/// A redaction policy about to be saved through the settings endpoint,
+/// with the exceptions `stored` holds instead of whatever it carries.
+pub fn keep_stored(incoming: Value, stored_policy: Option<&Value>) -> Value {
+    match incoming {
+        Value::Object(_) => with_list(Some(incoming), &stored(stored_policy)),
+        // Not an object: validation refuses it as it is.
+        other => other,
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use serde_json::json;
+
+    fn one(sha: char) -> RedactException {
+        RedactException {
+            sha256: sha.to_string().repeat(64),
+            masked: "sk-an…7f9c".into(),
+            rule: "anthropic-api-key".into(),
+            added_at: 1,
+        }
+    }
+
+    #[test]
+    fn a_settings_write_keeps_the_stored_exceptions() {
+        let stored_policy = with_list(Some(json!({"mode": "observe"})), &[one('a')]);
+        assert_eq!(stored(Some(&stored_policy)), [one('a')]);
+        // The console writes the policy without them, or with others.
+        for incoming in [
+            json!({"mode": "enforce"}),
+            json!({"mode": "enforce", "exceptions": [one('b')]}),
+            json!({"mode": "enforce", "exceptions": []}),
+        ] {
+            let kept = keep_stored(incoming, Some(&stored_policy));
+            assert_eq!(kept["mode"], "enforce");
+            assert_eq!(stored(Some(&kept)), [one('a')]);
+        }
+        // Nothing stored: none come in either.
+        let kept = keep_stored(json!({"exceptions": [one('b')]}), None);
+        assert!(kept.get("exceptions").is_none(), "{kept}");
+        // Not an object: left for validation to refuse.
+        assert_eq!(keep_stored(json!([]), Some(&stored_policy)), json!([]));
+    }
+
+    #[test]
+    fn an_empty_list_leaves_the_field_out_and_the_rest_as_stored() {
+        let p = with_list(
+            Some(json!({"mode": "enforce", "disable": ["jwt"], "exceptions": [one('a')]})),
+            &[],
+        );
+        assert_eq!(p, json!({"mode": "enforce", "disable": ["jwt"]}));
+        assert_eq!(with_list(None, &[]), json!({}));
+        assert!(stored(Some(&json!({"exceptions": "not a list"}))).is_empty());
+    }
+}

crates/common/src/guard_policy/mod.rs

@@ -29,6 +29,7 @@ use tw_guard::redact::rules::RuleSet;
 
 use crate::dynamic_config::DynamicConfig;
 
+pub mod exceptions;
 pub mod legacy;
 pub mod saved;
 

crates/gateway/src/redaction.rs

@@ -36,19 +36,31 @@ use std::sync::Arc;
 use think_watch_common::audit::AuditLogger;
 use think_watch_common::guard_policy::saved::{self, SavedSecrets};
 use tw_guard::policy::{Mode, RedactPolicy};
-use tw_guard::redact::replace::Ledger;
+use tw_guard::redact::flow::ScanCache;
+use tw_guard::redact::replace::{Ledger, Scheme};
 use tw_guard::redact::rules::{Finding, Kind, Rule, RuleSet, SavedSource};
 
 pub use tw_guard::redact::flow::Shape;
 
 use crate::guards::Caller;
 
 /// Outbound redaction as configured: a mode, and the rules — with the
-/// providers' credentials, recognised by value.
+/// providers' credentials, recognised by value, and the exception values
+/// that are not.
+///
+/// **One scan cache per compiled rule set** (thinkwatch-core's
+/// [`ScanCache`]): a client sends the whole conversation every turn, and
+/// the parts of it already scanned last turn are taken from the cache
+/// instead of being scanned again. What is found is exactly what a scan
+/// without it finds. The cache keeps hit positions and rule ids, never a
+/// value, in this process's memory only; it is built with the rules and
+/// dropped with them when a policy, the providers' keys or the exceptions
+/// change.
 #[derive(Debug, Clone)]
 pub struct Redaction {
     pub mode: Mode,
     pub rules: Arc<RuleSet>,
+    cache: Arc<ScanCache>,
 }
 
 impl Redaction {
@@ -59,14 +71,23 @@ impl Redaction {
                 think_watch_common::guard_policy::redact_rules(policy),
                 saved,
             )),
+            cache: Arc::new(ScanCache::new()),
         }
     }
 
     /// Look at what the caller sent, a request of `shape`: what was found
     /// in its written text (masked), and in enforce mode the ledger every
     /// hop is replaced with. Nothing when off.
     pub fn look(&self, body: &[u8], shape: Shape) -> (Vec<Finding>, Ledger) {
-        tw_guard::redact::flow::look(self.mode, &self.rules, body, shape)
+        let l = tw_guard::redact::flow::look_hits_with(
+            self.mode,
+            &self.rules,
+            body,
+            shape,
+            Ledger::new(Scheme::SECRET),
+            Some(&self.cache),
+        );
+        (l.found, l.ledger)
     }
 
     /// The bytes one hop sends, a request of `shape` — the format it goes
@@ -77,12 +98,13 @@ impl Redaction {
     /// Byte for byte what came in, and the same ledger, unless in enforce
     /// mode and something is found.
     pub fn replace(&self, body: Vec<u8>, ledger: &Ledger, shape: Shape) -> (Vec<u8>, Ledger) {
-        let (out, ledger) = tw_guard::redact::flow::replace(
+        let (out, ledger) = tw_guard::redact::flow::replace_with(
             self.mode,
             &self.rules,
             bytes::Bytes::from(body),
             ledger,
             shape,
+            Some(&self.cache),
         );
         (Vec::from(out), ledger)
     }
@@ -552,6 +574,67 @@ mod tests {
         assert!(!writes_masked(&Rule::Custom("ssn".into())));
     }
 
+    #[test]
+    fn an_exception_value_is_neither_replaced_nor_recorded() {
+        let other = "sk-ant-api03-BBBBBBBBBBBBBBBBBBBBBBBBBBBB";
+        let r = Redaction::new(
+            &RedactPolicy {
+                mode: Mode::Enforce,
+                exceptions: vec![tw_guard::policy::RedactException {
+                    sha256: tw_guard::redact::exceptions::sha256_hex(KEY),
+                    masked: "sk-an…AAAA".into(),
+                    rule: "anthropic-api-key".into(),
+                    added_at: 1,
+                }],
+                ..Default::default()
+            },
+            &Default::default(),
+        );
+        let body = serde_json::json!({"messages": [
+            {"role": "user", "content": format!("{KEY} and {other}")}
+        ]})
+        .to_string();
+        let (found, ledger) = r.look(body.as_bytes(), CHAT);
+        assert_eq!(found.len(), 1, "{found:?}");
+        let (sent, _) = r.replace(body.into_bytes(), &ledger, CHAT);
+        let sent = String::from_utf8(sent).unwrap();
+        assert!(sent.contains(KEY) && !sent.contains(other), "{sent}");
+    }
+
+    /// The scan cache takes the unchanged history from the turn before; the
+    /// findings, the numbering and what is sent are the same as without it.
+    #[test]
+    fn a_growing_conversation_redacts_the_same_turn_after_turn() {
+        let r = redaction(Mode::Enforce);
+        let filler = "lorem ipsum dolor sit amet ".repeat(200);
+        let mut messages = Vec::new();
+        for turn in 0..30 {
+            let content = if turn % 7 == 0 {
+                format!(
+                    "{filler} turn {turn} key sk-ant-api03-{}",
+                    format!("{turn:02}").repeat(14)
+                )
+            } else {
+                format!("{filler} turn {turn}")
+            };
+            messages.push(serde_json::json!({"role": "user", "content": content}));
+            let body = serde_json::json!({"messages": messages}).to_string();
+            let cached = r.look(body.as_bytes(), CHAT);
+            let plain =
+                tw_guard::redact::flow::look(Mode::Enforce, &r.rules, body.as_bytes(), CHAT);
+            assert_eq!(cached.0, plain.0, "turn {turn}");
+            let (a, _) = r.replace(body.clone().into_bytes(), &cached.1, CHAT);
+            let (b, _) = tw_guard::redact::flow::replace(
+                Mode::Enforce,
+                &r.rules,
+                bytes::Bytes::from(body),
+                &plain.1,
+                CHAT,
+            );
+            assert_eq!(a, b.to_vec(), "turn {turn}");
+        }
+    }
+
     #[test]
     fn findings_are_one_entry_per_rule_and_a_custom_rule_keeps_no_values() {
         let r = Redaction::new(