feat(commandlog): SCAN large-reply hash-skew advisory (valkey#3955)

[jamby77]

Summary

Adds a SCAN large-reply / hash-skew advisory for valkey-io/valkey#3955: a skewed hashtable (degenerate hash chain) makes SCAN-family calls return replies vastly larger than the requested COUNT. The server can't tell you this happened — but the signature is fully visible in the large-reply command log we already persist. Second of the five upstream-issue detector cards (first was #337).

No new polling, no new MetricType, no packages/mcp changes (deferred while #330–#332 are in flight) — the analysis rides the stored large-reply entries on demand.

Changes

• scan-skew-analyzer.ts — pure analyzer over stored large-reply entries: filters to the SCAN family (SCAN/SSCAN/HSCAN/ZSCAN), parses the requested COUNT from the raw command (absent → server default 10; MATCH before/after COUNT, cursor position per verb, HSCAN NOVALUES), and computes reply-bytes per requested element against a named tunable budget (SCAN_SKEW_BYTES_PER_ELEMENT = 4096). Recurrence weighting: a key surfaces at ≥2 sightings, or a single sighting at ≥10× the budget (SCAN_SKEW_EXTREME_MULTIPLIER). Proportional large replies and one-off ordinary sightings stay quiet.
• GET /commandlog-analytics/scan-skew — reuses the stored-entry query (type forced to large-reply, whose magnitude column is bytes); returns offenders ranked worst-first with per-key advisory copy.
• ScanSkewAdvisory on the Slow Log page's large-reply tab — polls the endpoint at 30s (only while that tab is active), respects the time filter, lists offender keys with verb, sighting count and worst bytes-per-element, with the valkey#3955 remediation copy (re-create the key, or upgrade once the upstream fix lands). Hidden when there are no offenders.
• Lint hygiene in touched files — the anomaly any generics chore deferred from feat(monitor): config-hazard advisory — default user disabled + AOF silent data loss (valkey#3983) #337 is resolved here: apps/web/src/types/anomaly.ts mirrors the proprietary wire shapes (web can't import the optional proprietary module) and the five anomaly API client functions are now typed; removed a needless as any in commandlog-analytics.service.ts.

Verification

• Analyzer: fire / proportional no-fire / boundary at the budget / recurrence weighting / extreme single sighting / COUNT-parser permutations — 14 tests.
• Service: forced large-reply type, default limit, pass-through filters — 3 tests.
• ScanSkewAdvisory render states (offenders / empty / not loaded) — 3 tests.
• tsc --noEmit and eslint clean on api and web; full web suite 294/294 green.

Checklist

• Unit tests added
• Docs added / updated

---

Note

Low Risk
Read-only analytics over existing stored logs plus a narrow SQLite query fix; no auth, persistence schema, or runtime Redis behavior changes.

Overview
Adds on-demand SCAN skew detection for degenerate hash chains (valkey#3955) by analyzing stored large-reply command log entries—no new polling or metrics pipeline.

A new scan-skew-analyzer parses SCAN-family commands, compares reply bytes to requested COUNT, and surfaces offenders with recurrence/extreme thresholds and compact-encoding/keyless-SCAN caveats. GET /commandlog-analytics/scan-skew exposes this (type forced to large-reply, command: 'SCAN', capped limit). Shared ScanSkewReport types land in @betterdb/shared.

The Slow Log large-reply tab polls the endpoint and shows ScanSkewAdvisory when offenders exist (respects time range). SQLite command-log filtering now matches the verb only via json_extract(command, '$[0]') so keys like user:scan:results are not mistaken for SCAN.

Also replaces anomaly API any responses with apps/web/src/types/anomaly.ts wire types and drops an unnecessary as any in slow-log pattern caching.

^{Reviewed by Cursor Bugbot for commit dd6c61d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[jamby77]

@BugBot review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 7ea64e5. Configure here.}

[KIvanow]

Nice work - the SCAN-family parser is solid (cursor offsets per verb, MATCH before/after COUNT, NOVALUES, and default COUNT=10 all check out). A few things to address before merge, none of them structural:

Should fix:

1. Small-collection false positive. duration / count treats COUNT as an element count, but for a listpack/intset-encoded collection the server returns all elements in one reply regardless of COUNT. A 200-field listpack hash scanned with default COUNT=10 will show a high bytes/element and get flagged - and the "re-create the key" advice is actively wrong there (it's a normal full return, not a degenerate chain). Can we either guard this or at least caveat it in the advisory copy?
2. limit is unclamped - parseInt(limit) in the controller has no upper bound, and the service only defaults when falsy. ?limit=1000000 will fetch and in-memory-analyze that many rows. Please clamp to a sane max.

Nits:
3. The anomaly-any retyping (types/anomaly.ts + the 5 client fns) is unrelated to SCAN skew and inflates the diff - would prefer it split into its own PR, but not a blocker.
4. SlowLog.tsx comment says the advisory "works the same with or without a time filter," but it does pass startTime/endTime, so results are window-scoped - comment is misleading.
5. Test gap: MATCH-after-COUNT is never tested, even though order-independence is the parser's trickiest claim. Worth a case.

Verdict: minor nits - happy to approve once the small-collection caveat and the limit clamp land.

[jamby77]

Addressed in ef9103f:

1. Small-collection false positive — keyed scans (SSCAN/HSCAN/ZSCAN) now skip replies under a 256KB floor (SCAN_SKEW_MIN_KEYED_REPLY_BYTES): compact-encoded (listpack/intset) collections return whole in one reply, so those are treated as normal full returns, not chains. The keyed advisory copy also gained an explicit compact-encoding caveat. Keyless SCAN is unaffected (no full-return semantics there) — covered by tests both ways.
2. limit clamp — controller now clamps to SCAN_SKEW_MAX_ANALYZE_LIMIT (5000); non-numeric/non-positive values fall back to the 500 default.
3. anomaly-any retyping — keeping it here: adding the scan-skew client fn touches metrics.ts, and the repo convention is to fix existing eslint errors in touched files before committing (the anys were exactly that). Happy to split if you feel strongly.
4. SlowLog.tsx comment — corrected: results are window-scoped via startTime/endTime.
5. MATCH-after-COUNT — parser test added for both HSCAN and keyless SCAN forms.

[KIvanow]

One follow-up on the compact-encoding caveat: it currently lives only in the backend offender.message/remediation string, but ScanSkewAdvisory.tsx renders a hardcoded paragraph and maps offenders to key · verb · sightings · worstBytes — it never renders offender.message. So the caveat isn't visible to users. Could you surface offender.message (or fold the "small collections return whole in one reply — normal, no action" sentence into the component copy)? Otherwise the suppression works but the guidance still doesn't reach the UI.

[jamby77]

Surfaced in the just-pushed commit — ScanSkewAdvisory.tsx now renders offender.message under each offender row (muted, one per entry), so the backend remediation copy — compact-encoding caveat included, plus the keyed vs keyspace distinction — reaches the UI and stays in sync with the API instead of being duplicated in component copy. The hardcoded intro is slimmed to the signature sentence to avoid repeating per-offender guidance. Component tests assert the caveat text renders.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bcac271. Configure here.}

[KIvanow]

looks good after the changes!