Skip to content

Cache implementation - #131

Open
Gaudy Blanco (gaudyb) wants to merge 10 commits into
mainfrom
cache-implementation
Open

Gaudy Blanco (gaudyb) wants to merge 10 commits into
mainfrom
cache-implementation

Conversation

@gaudyb

Copy link
Copy Markdown
Contributor

This PR standardizes BenchmarkQED caching on the shared graphrag-cache and graphrag-storage infrastructure and expands persistent cache support across evaluation workflows.

Caching now supports interruption recovery, repeated-run reuse, deterministic configuration invalidation, and cross-process work coordination. Generated configurations use persistent SQLite caches by default, while JSON, memory, and disabled caching remain available.

- Add new caching mechanisms in `cache.py` and related modules.
- Update CLI and documentation to reflect new features.
- Enhance tests for chunk assertions and caching functionality.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regression review through b0278bc, including the SQLite initialization retry and strengthened trial-independence tests. Eight actionable findings are attached inline. The concurrency and result-integrity issues should be fixed before merging.

Performance: A local Windows storage-only microbenchmark of 500 warm chunk-cache lookups, median of three runs, measured approximately 0.04 ms on the base versus 3.21 seconds with SQLite. The base reads an already-loaded dictionary; the new path makes individual threaded database reads. This is not end-to-end LLM timing, but warrants batched reads or a carefully designed read-through cache for large, mostly cached evaluations.

Compatibility: Existing cache_dir configurations are explicitly rejected, several Python APIs change to cache_config, and standard scoring gains persistent caching by default. These appear intentional but need migration communication. Score IDs no longer force fresh judgments, and the existing pairwise output-CSV shortcut remains independent of judgment-cache invalidation.

Evidence and limits: At this head, eight deterministic regression probes failed while seven controls/updated tests passed. An earlier broader selection passed 186 existing tests. Local execution used upstream GraphRAG 3.2.0 source with the existing Python environment because locked dependency restoration encountered TLS failures; live-provider and Azure Blob behavior were not exercised. Current-head CI independently reports 11 failed, 248 passed, and one skipped: https://github.com/microsoft/benchmark-qed/actions/runs/36925433961. The findings below distinguish reproducible defects from the performance and compatibility observations above.

Comment thread benchmark_qed/cache.py Outdated
Comment thread benchmark_qed/cache.py Outdated
Comment thread benchmark_qed/autoe/retrieval_metrics/relevance_assessment/base.py Outdated
Comment thread benchmark_qed/autoe/pairwise/differential.py
Comment thread benchmark_qed/autoe/pairwise/scores.py
Comment thread benchmark_qed/cache.py
Comment thread tests/test_load_config.py Outdated
Comment thread docs/notebooks/cache_parallelism_benchmark.ipynb Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants