Repository navigation
Cache implementation - #131
Gaudy Blanco (gaudyb) wants to merge 10 commits into
Conversation
- Add new caching mechanisms in `cache.py` and related modules. - Update CLI and documentation to reflect new features. - Enhance tests for chunk assertions and caching functionality.
Andres Morales (andresmor-ms)
left a comment
There was a problem hiding this comment.
Regression review through b0278bc, including the SQLite initialization retry and strengthened trial-independence tests. Eight actionable findings are attached inline. The concurrency and result-integrity issues should be fixed before merging.
Performance: A local Windows storage-only microbenchmark of 500 warm chunk-cache lookups, median of three runs, measured approximately 0.04 ms on the base versus 3.21 seconds with SQLite. The base reads an already-loaded dictionary; the new path makes individual threaded database reads. This is not end-to-end LLM timing, but warrants batched reads or a carefully designed read-through cache for large, mostly cached evaluations.
Compatibility: Existing cache_dir configurations are explicitly rejected, several Python APIs change to cache_config, and standard scoring gains persistent caching by default. These appear intentional but need migration communication. Score IDs no longer force fresh judgments, and the existing pairwise output-CSV shortcut remains independent of judgment-cache invalidation.
Evidence and limits: At this head, eight deterministic regression probes failed while seven controls/updated tests passed. An earlier broader selection passed 186 existing tests. Local execution used upstream GraphRAG 3.2.0 source with the existing Python environment because locked dependency restoration encountered TLS failures; live-provider and Azure Blob behavior were not exercised. Current-head CI independently reports 11 failed, 248 passed, and one skipped: https://github.com/microsoft/benchmark-qed/actions/runs/36925433961. The findings below distinguish reproducible defects from the performance and compatibility observations above.
This PR standardizes BenchmarkQED caching on the shared graphrag-cache and graphrag-storage infrastructure and expands persistent cache support across evaluation workflows.
Caching now supports interruption recovery, repeated-run reuse, deterministic configuration invalidation, and cross-process work coordination. Generated configurations use persistent SQLite caches by default, while JSON, memory, and disabled caching remain available.