fix(cache): unique HTTP cache URL and race-safe upsert to stop duplicate rows - #252
Open
jordanfelle wants to merge 2 commits into
Open
jordanfelle wants to merge 2 commits into
jordanfelle wants to merge 2 commits into
Conversation
…fe upsert The HttpResponse URL index was not unique and CachedHttpResponseService did find-then-insert, so two concurrent lookups of one URL could each insert a row (measured: 7,846 rows for 6,261 URLs, 782 URLs duplicated). Migration 108 keeps the newest row per URL (latest refresh, then highest id) and makes IX_HttpResponse_Url unique. The service now writes through UpsertByUrl, which on a unique violation (SQLite constraint 19 / Postgres 23505) updates the winning row instead.
jordanfelle
added a commit
to jordanfelle/chaptarr
that referenced
this pull request
Sep 27, 2026
Open PR Chaptarr#177 already adds migration 108 (add_ignored_genres_to_metadata_profile); two migrations with the same version fail at startup when both are applied, so use 109.
jordanfelle
added a commit
to jordanfelle/chaptarr
that referenced
this pull request
Sep 27, 2026
jordanfelle
added a commit
to jordanfelle/chaptarr
that referenced
this pull request
Sep 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #223.
Problem
The
HttpResponsecache table has a non-uniqueIX_HttpResponse_Urlindex andCachedHttpResponseService.Getdoes find-then-insert, so two concurrent lookups of the same URL can each insert a row. Measured on a live cache: 7,846 rows for 6,261 distinct URLs, 782 URLs with duplicates (one with 47 rows). Every later lookup of an affected URL used to throwSequence contains more than one element(see #206, which makes reads tolerate duplicates).Fix
LastRefresh, then highestId), then recreatesIX_HttpResponse_Urlas a unique index. Plain SQL that works on SQLite and PostgreSQL; safe on a cache of this size.CachedHttpResponseRepository.UpsertByUrl: the service now writes through it. It inserts, and on a unique violation (SQLite constraint 19 / PostgreSQL 23505) updates the winning row instead of creating a duplicate.Complementary to #206 (tolerant reads); either merge order works.
Verification
HttpResponseUniqueUrlMigrationFixture: runs migration 109 against a SQLite cache with duplicates and ties, checks the newest row survives per URL and the index is unique afterwards.CachedHttpResponseRepositoryUniqueUrlFixture: a plain second insert is rejected by the index,UpsertByUrlupdates the existing row, and 16 concurrent upserts for one URL leave a single row (both upsert tests fail if the unique-violation handling is removed).Untested on PostgreSQL (no instance available in CI here): the migration SQL and the
23505handling follow the same pattern as the existingEditionServiceunique-violation handling, but the Postgres path was not exercised.Note: uses migration number 109 because open PR #177 already adds migration 108; two migrations with one version fail at startup when both are applied.
Postgres verification: the migration was applied on a live Postgres cache database (about 7.8k rows before) and left 6,253 rows for 6,253 distinct URLs with
VersionInfoat 109, and chaptarr started and ran normally afterwards. Failure mode to be aware of: a migration error on the cache database stops startup ("Error creating main database"); see the linked issue.