perf(covers): stop enumerating MediaCover/Books for every book without a cover folder - #237
Open
jordanfelle wants to merge 2 commits into
Open
jordanfelle wants to merge 2 commits into
jordanfelle wants to merge 2 commits into
Conversation
…t a cover folder GET /api/v1/book?authorId=N maps each book's covers, and MediaCoverService looked up cover-metadata.json with IDiskProvider.FileExists. When the file is missing, FileExists falls back to a case-insensitive path resolution that lists and normalises every entry of MediaCover/Books (one folder per book, ~195k here) for every book that has no cover folder. The request was therefore quadratic in catalogue size: ~3ms/book at 1k books, but no response in 100s+ for an author with 2.5k books (2,286 of them without a cover folder) and several minutes for 12k. Check FolderExists (a plain stat) first and cache the negative result.
jordanfelle
added a commit
to jordanfelle/chaptarr
that referenced
this pull request
Sep 26, 2026
Author
|
Live timings with this change deployed (library of 346k books, 195k folders in MediaCover/Books), one request each:
|
jordanfelle
added a commit
to jordanfelle/chaptarr
that referenced
this pull request
Sep 26, 2026
…ache entry TryReadBookCoverMetadata checks the disk and then writes its result with the indexer. A cover download that stores its metadata (and its cache entry) between the disk check and the cache write was overwritten by the stale negative result, so the book looked coverless until the entry was next invalidated. Use TryAdd for the read path's cache writes; writers and invalidation keep the indexer and TryRemove. Regression test drives the interleaving deterministically.
jordanfelle
added a commit
to jordanfelle/chaptarr
that referenced
this pull request
Sep 27, 2026
jordanfelle
added a commit
to jordanfelle/chaptarr
that referenced
this pull request
Sep 27, 2026
jordanfelle
added a commit
to jordanfelle/chaptarr
that referenced
this pull request
Sep 27, 2026
jordanfelle
added a commit
to jordanfelle/chaptarr
that referenced
this pull request
Sep 27, 2026
jordanfelle
added a commit
to jordanfelle/chaptarr
that referenced
this pull request
Sep 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GET /api/v1/book?authorId=N(what the book and author pages load) is quadratic in the size of the catalogue. On a live instance the time per book is flat at about 3 ms for small authors, then falls off a cliff:The database is not the bottleneck.
pg_stat_statementsshowed no activity for 12 s or more mid-request, and the controller's own mapping is linear (a benchmark of the realGetBooksandMapToResourcewith stubbed services maps 5,000 books in about 110 ms).Cause
Stack snapshots taken while a request was running put every request thread in
MediaCoverService.TryReadBookCoverMetadata->DiskProviderBase.FileExists->ResolveExistingFilePath->ResolvePathSegment.TryReadBookCoverMetadatalooks upcover-metadata.jsonper book. When the file does not exist,FileExistsfalls back to a case-insensitive path resolution that lists and normalises every entry of the parent directory. Here that isMediaCover/Bookswith 194,613 folders (one per book), and most books have no cover folder: 2,286 of the 2,522 books of the author above. That is about 445 million comparisons for one request.Fix
Check
FolderExists(a plainDirectory.Exists) for the book's cover folder first, and cache the negative result the same way the missing-file case already was. Books that do have a cover folder behave as before.Verification
converting_a_book_without_a_cover_folder_should_not_probe_for_its_metadata_file: fails without the change, passes with it.MediaCovertests pass.Measured live after deploying:
GET /api/v1/book?authorId=552(2,522 books) went from more than 100 s (timeout) to 1.36 s, andauthorId=2698(12,000 books) from never returning to 5.9 s.Review update: the read path now writes the negative cache entry with
TryAdd, so a cover download that stores its metadata between the disk check and the cache write is no longer overwritten by a stale 'missing' result (test:a_concurrent_cover_write_should_not_be_overwritten_by_the_missing_folder_negative_cache_entry, deterministic via anOnFolderExistshook on the test disk stub) (3,022 core tests pass).