Skip to content

fix(cache): unique HTTP cache URL and race-safe upsert to stop duplicate rows - #252

Open
jordanfelle wants to merge 2 commits into
Chaptarr:developfrom
jordanfelle:fix-http-cache-unique-url
Open

jordanfelle wants to merge 2 commits into
Chaptarr:developfrom
jordanfelle:fix-http-cache-unique-url

Conversation

@jordanfelle

@jordanfelle jordanfelle commented Sep 27, 2026 •

Copy link
Copy Markdown

Fixes #223.

Problem

The HttpResponse cache table has a non-unique IX_HttpResponse_Url index and CachedHttpResponseService.Get does find-then-insert, so two concurrent lookups of the same URL can each insert a row. Measured on a live cache: 7,846 rows for 6,261 distinct URLs, 782 URLs with duplicates (one with 47 rows). Every later lookup of an affected URL used to throw Sequence contains more than one element (see #206, which makes reads tolerate duplicates).

Fix

  • Migration 109 (cache database): deletes duplicate rows keeping the newest per URL (latest LastRefresh, then highest Id), then recreates IX_HttpResponse_Url as a unique index. Plain SQL that works on SQLite and PostgreSQL; safe on a cache of this size.
  • CachedHttpResponseRepository.UpsertByUrl: the service now writes through it. It inserts, and on a unique violation (SQLite constraint 19 / PostgreSQL 23505) updates the winning row instead of creating a duplicate.

Complementary to #206 (tolerant reads); either merge order works.

Verification

  • HttpResponseUniqueUrlMigrationFixture: runs migration 109 against a SQLite cache with duplicates and ties, checks the newest row survives per URL and the index is unique afterwards.
  • CachedHttpResponseRepositoryUniqueUrlFixture: a plain second insert is rejected by the index, UpsertByUrl updates the existing row, and 16 concurrent upserts for one URL leave a single row (both upsert tests fail if the unique-violation handling is removed).
  • 3,024 core tests pass.

Untested on PostgreSQL (no instance available in CI here): the migration SQL and the 23505 handling follow the same pattern as the existing EditionService unique-violation handling, but the Postgres path was not exercised.

Note: uses migration number 109 because open PR #177 already adds migration 108; two migrations with one version fail at startup when both are applied.

Postgres verification: the migration was applied on a live Postgres cache database (about 7.8k rows before) and left 6,253 rows for 6,253 distinct URLs with VersionInfo at 109, and chaptarr started and ran normally afterwards. Failure mode to be aware of: a migration error on the cache database stops startup ("Error creating main database"); see the linked issue.

…fe upsert

The HttpResponse URL index was not unique and CachedHttpResponseService did find-then-insert, so two concurrent lookups of one URL could each insert a row (measured: 7,846 rows for 6,261 URLs, 782 URLs duplicated). Migration 108 keeps the newest row per URL (latest refresh, then highest id) and makes IX_HttpResponse_Url unique. The service now writes through UpsertByUrl, which on a unique violation (SQLite constraint 19 / Postgres 23505) updates the winning row instead.
jordanfelle added a commit to jordanfelle/chaptarr that referenced this pull request Sep 27, 2026
Open PR Chaptarr#177 already adds migration 108 (add_ignored_genres_to_metadata_profile); two migrations with the same version fail at startup when both are applied, so use 109.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] HTTP response cache: non-unique Url index and find-then-insert race create duplicate rows

1 participant