Repository navigation
perf(worker): cache bot list and model pricing reads - #112
Merged
Merged
Conversation
- GET /api/bots/: per-user infinite in-memory list cache with write-through invalidation on POST/PUT/PATCH/DELETE, OAuth provisioning, and onboarding bootstrap; single batched aimodel lookup replaces N per-bot queries - usage.ts: bots_aimodel pricing loaded once per isolate (runtime- static table); shared by rawCostForToday and baselinePricingSnapshot - tests: reset both caches in beforeEach (fresh DB per case) Verified: tsc --noEmit clean, vitest 51/51 pass.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 5 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
- botListCache: per-isolate epoch; in-flight GETs whose reads started before a bot write no longer store pre-write rows (P2) - usage: never cache an empty bots_aimodel result, or costs pin at 0 and the daily cap stops enforcing (P2) - extract bot-list cache to worker/src/botListCache.ts so onboarding and oauth no longer import the route module (P3) - tests/cache.test.ts: epoch-guard, eviction, and empty-pricing regression tests (55/55 pass)
Owner
Author
|
@cubic-dev-ai incremental review |
@tpaulshippy I’ve started reviewing the changes since the last completed review. I’ll post the results here when it’s done. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts the hottest D1 readers to near zero.
Bot list (
GET /api/bots/,worker/src/routes/bots.ts)SELECT model_id FROM bots_aimodelN+1 replaced with one batchedWHERE id IN (...)Model pricing (
worker/src/usage.ts)bots_aimodelis runtime-static (no worker route writes it), so rows load once per isolate and are shared byrawCostForToday+baselinePricingSnapshot(previously 2 full scans per quota check, on every chat turn)Tests (
worker/tests/api.test.ts)beforeEach— fresh DB per case would otherwise leak rows through module stateVerified
tsc --noEmitclean,vitest51/51 pass (local seeded DB; no prod reads)Staleness notes
bots_botwriter must callinvalidateBotListCache.Summary by cubic
Reduces the hottest D1 reads to near zero by caching the per-user bot list and the
bots_aimodelpricing table in worker memory. The bot list's per-bot N+1 lookup becomes a single batched query, and pricing rows load once per isolate instead of twice per quota check on every chat turn.Cache notes
bots_aimodel; any future runtime writer must callinvalidateModelsCache(). An empty table is never cached, so costs can't pin at zero and silently stop enforcing the daily cap.beforeEach; a newcache.test.tscovers the epoch guard, eviction cap, and empty-pricing behavior.Written for commit 44489b9. Summary will update on new commits.