feat(retrieve): cache rerank results for identical query-document batches

[guvengursel]

Description

Adds a process-wide LRU result cache in front of rerank provider calls, upstreaming the design proposed in #5200.

All provider clients flow through HierarchicalRetriever._rerank_scores, which already normalizes inputs (blank-doc filtering, token-budget truncation) before calling rerank_batch. The cache wraps the constructed client transparently (CachedRerankClient), so it keys on exactly what the provider receives:

• Key: sha256 digest over length-prefixed fields — provider, model identity (model_name/model/model_version/host/api_base/mode), then the exact query and document texts in request order. Length-prefixing removes query/document boundary ambiguity; only the digest is retained as key material, and digests are never logged.
• Eviction: LRU, 1024 entries, 24h TTL (time.monotonic); hits refresh recency. A per-entry byte cap skips caching oversized batches, since request limits are unbounded and truncation defaults off.
• Thread safety: all cache operations are guarded by a threading.Lock — rerank_batch runs through asyncio.to_thread, so concurrent requests reach the cache from multiple executor threads.
• Failure semantics unchanged: exceptions and None responses propagate exactly as before and are never cached; the vector-score fallback path is untouched.
• Not tenant-scoped by design: rerank configuration is process-global and startup-only (changing it requires a restart, which also clears the cache), so identical inputs score identically for every account and cross-account hits are correct.

Identical inputs deterministically produce identical provider scores, so caching is correctness-preserving.

Behavior before: every retrieve with rerank enabled issues a provider rerank_batch call, including repeated identical (query, documents) requests in steady-state auto-recall traffic.
Behavior after: repeated identical requests within the TTL are served from cache; provider call count drops accordingly. Field telemetry from the deployment that motivated #5200: 10,047 served / 1,629 cache hits (~13.95%) over a multi-day window; hit rate is window-dependent (a window starting from a fresh restart showed substantially lower hit rates).

One behavioral note, stated explicitly: cache hits skip the provider call, so they emit no rerank token-usage or call metrics — per-account rerank counters under-report by the hit volume.

Human Involvement

• A human participated in the implementation or review loop

Related Issue

Fixes #5200

Type of Change

• New feature (non-breaking change that adds functionality)

Changes Made

• openviking/retrieve/rerank_cache.py (new): sha256 length-prefixed key, OrderedDict LRU (1024 entries, 24h TTL, monotonic clock), byte cap on oversized payloads, threading.Lock-guarded operations, transparent CachedRerankClient wrapper
• openviking/retrieve/hierarchical_retriever.py: wrap the constructed rerank client when configured; declare the attribute's union type
• tests/retrieve/test_hierarchical_retriever_rerank.py: contract tests — cache hit skips provider call, document-order sensitivity, TTL expiry, LRU eviction, client wiring with/without config
• docs/en/guides/01-configuration.md + docs/zh/guides/01-configuration.md: rerank section notes the cache and the metrics behavior on hits

Testing

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have tested this on the following platforms:
  • Linux

Validation commands and results (local environment has no Rust toolchain, so the project wheel was not built; this test file's import chain is pure Python and unaffected):

• uv run --no-project python -m pytest tests/retrieve/test_hierarchical_retriever_rerank.py -q → 24 passed (18 pre-existing + 6 new)
• ruff format --check + ruff check on the changed paths (pinned via uvx) → clean
• mypy on the changed paths → no new errors vs the pre-change tree (8 pre-existing errors in hierarchical_retriever.py, unchanged)
• Behavior was reproduced in production before this change: the same cache design ran for weeks in a local reranker proxy service (see [Feature]: reranker result cache (hash-keyed LRU + TTL) to dedupe repeated identical rerank queries #5200)

Checklist

• My code follows the project's coding style
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

Additional Notes

The diff is ~211 lines (about 102 the new cache module, ~98 contract tests), above the usual ≤100-line fast-review size — the tests and both documentation languages are included deliberately rather than trimmed to fit; happy to split if reviewers prefer.

Follow-ups deliberately deferred to keep this PR one coherent problem: configurable size/TTL knobs, hit/miss telemetry counters.