Cache catalogue figures and index the search columns - #38
Merged
Merged
Conversation
Catalogue pages took 1.4-3.4s against 31,000 listings. Three causes, only two of which caching can touch. - Cache stats/0 and top_tags/1 for a day in a small ETS table, dropped on any write and after an official sync, so the TTL is a ceiling rather than a staleness guarantee - Count tags with GROUP BY instead of shipping ninety thousand rows to Elixir to tally, and fold five full-table aggregates into one pass - Add trigram indexes so search stops sequentially scanning the table Search keeps ILIKE rather than moving to full-text, so matching is unchanged and the index is only a faster path to the same rows. Verified across eight terms: zero differing rows. Indexing the array columns needed an IMMUTABLE mcp_array_to_text/1, because Postgres will not index array_to_string/2, and one unindexed OR branch would have sent the whole predicate back to a scan. Measured on 31,000 rows: selective search 47.4ms to 0.4ms, top_tags 19.8ms to 9.7ms uncached and nothing when cached, stats 6.9ms to 4.2ms uncached. --- Pages affected: - [MCP Registry](https://ai.mcpharbor.dev/) — landing page, which renders the counters and tag cloud. - [Browse MCP servers](https://ai.mcpharbor.dev/servers) — the catalogue and its search. - [JSON API](https://ai.mcpharbor.dev/api/v0/servers) — machine-readable listings, served by the same queries. - [Submit a server](https://ai.mcpharbor.dev/submit) — publishing drops the cached figures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Migrations run from the service unit's ExecStartPre, so a migration that raises means the release never starts. pg_trgm is a trusted extension from PostgreSQL 13 and the database owner can install it, but on an older server or a restricted role it needs superuser. - Create mcp_array_to_text/1 unconditionally, since filter_q/2 calls it and a missing function would make every search raise - Wrap the extension and the indexes, logging what to run by hand instead Losing an index is a slow catalogue. Losing the boot is an outage. --- Pages affected: - [MCP Registry](https://ai.mcpharbor.dev/) — landing page served by the same release. - [Browse MCP servers](https://ai.mcpharbor.dev/servers) — the catalogue and its search. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #37.
Catalogue pages took 1.4–3.4s in production against ~31,000 listings. Three causes; caching only addresses two of them.
stats/0— five full-table aggregates/and/serverstop_tags/1— shipped ~90k rows to Elixir to tally/and/serversGROUP BYin SQL, cached a dayILIKE '%…%'× 5, no usable index?q=Measured on 31,000 rows
top_tagsuncachedstatsuncachedstats/top_tagscachedResults do not change
Search keeps
ILIKE; the index is only a faster path to the same rows, and Postgres rechecks the real condition on every candidate. Full-text search would have been faster still and would have changed results — stemming, stop words, no substring matching — so it is deliberately not used.Verified by diffing result sets across eight terms (
github,database,tool_3,developer-tools,bench,Search,zz,io.github): zero differing rows in either direction.Two things worth knowing
ORbranches had to become indexable. Postgres combinesORbranches through a BitmapOr only when every branch has an index — one unindexed branch sends the whole predicate back to a sequential scan. The array columns needed anIMMUTABLEmcp_array_to_text/1wrapper, because Postgres refuses to index an expression containingarray_to_string/2. Iffilter_q/2and that function ever drift apart, search silently falls back to a scan.Cache behaviour
A day is a ceiling, not a staleness window: the cache is dropped on every listing write and once at the end of each official-registry sync, so a submission shows up on the next request. It holds a handful of tuples in ETS — deliberately not a dependency and deliberately tiny, given #36.
Indexes are built
CONCURRENTLY, so the migration does not take a write lock on the live catalogue. 85 tests pass; migration verified from a clean database.Pages affected:
🤖 Generated with Claude Code