Repository navigation
Facets all values - #184
Merged
Merged
Conversation
added 3 commits
October 18, 2026 16:20
The SPARQL facet queries that /facets replaced returned every value; /facets capped each facet at the requested size (default 50, max 500). SynBioHub puts dc:creator on the submission collection only, not on its parts, so a private collection's creator has a count of 1 and never made the top 50: the owner could not find their own name under Creator. Locally 1,652 of 5,309 creators occur once. Raise MAX_FACET_SIZE to 10,000 (five facets x 10,000 stays below Elasticsearch's search.max_buckets of 65,536); the frontend asks for it. Warm, locally: 0.65-1.3 s and ~1.1 MB with every value, versus 0.1-0.4 s for the top 50.
The facet fields were filled by streaming (subject, value) pairs sorted by
subject with keyset paging (FILTER > last + ORDER BY). Virtuoso filtered
and re-sorted the remaining rows for every page, so the cost grew roughly
quadratically: on production (~1M top-level objects) the facet pass took
hours and dominated the rebuild.
query_value_pages() now reads unsorted OFFSET pages (like
query_parts_paged), and index_facet_fields() merges one facet at a time
in memory into {subject: values} before writing one partial update per
part, as before. Memory is one facet at a time (~127 MB peak for 130k
parts locally).
Measured locally, same data and run back to back: old 263 s (Virtuoso
123 s), new 174 s (Virtuoso 43 s); all 132,922 documents end up with
identical facet values. An intermediate variant that appended per page
with a Painless script was correct but slower (373 s) and is not used.
When the index did not exist (e.g. a fresh Elasticsearch volume), startup() ran the full rebuild at import time, so Flask did not listen until it finished (an hour or more on production). The container healthcheck on /info failed after a few minutes and autoheal restarted Explorer, which started the rebuild over, indefinitely. When a restart happened after the index had been created, the next start found the half-built index and skipped the rebuild. startup() now starts the rebuild in a daemon thread, so /info answers immediately (searches still return 503 until the index exists). A lock lets only one full rebuild run at a time: /update returns 409 while one is running, and the auto-update thread skips its turn, instead of a second rebuild deleting the index the first is writing. Checked locally against an empty index: /info answered after 2 s, search returned 503, and /update during the build returned 409.
cjmyers
approved these changes
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I got Explorer running and finished indexing all the data in Elasticsearch, but a few issues remain: