perf: cache the OpenRouter model catalog with a 30-minute TTL - #90
Draft
aryansk wants to merge 1 commit into
Draft
perf: cache the OpenRouter model catalog with a 30-minute TTL#90aryansk wants to merge 1 commit into
aryansk wants to merge 1 commit into
Conversation
Repeated list_free_models calls (startup + /models) inside the TTL now reuse the last successful response instead of a network round-trip. ttl_seconds=0 forces a fresh fetch; failed fetches are never cached. Closes shauryagangrade#56.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
list_free_modelshitshttps://openrouter.ai/api/v1/modelson every call./modelsand startup both fetch the full catalog each time — a wastednetwork round-trip and a slow menu on a flaky connection. Closes #56.
Change
Adds an in-process cache to
list_free_modelsingcode/models.py:repeated calls inside the TTL make one HTTP request.
ttl_seconds=0is the escape hatch for callers that need fresh data.best-effort contract (empty list + actionable error) is unchanged.
clear_model_catalog_cache()is exposed for tests/consumers.Why this approach
In-process only, no new dependencies, and the default matches the issue's
suggested 10–60 min window. Only successful responses are cached so a
transient outage is never masked by a stale catalog, and the escape hatch
keeps
/modelshonest when the user explicitly asks for a fresh list.Testing
Acceptance criteria covered: two consecutive calls inside the TTL make one
HTTP request (asserted via
mock_get.call_count == 1), and cacheinvalidation on expiry is tested with the real clock.
Documentation and release impact
Review notes