Repository navigation
Upgrade es - #183
Merged
Merged
Upgrade es#183
Conversation
added 5 commits
October 9, 2026 16:30
Facet counts (role, sboltype, type, creator, collections) and the total are computed in one ES request using the same text query as the result list, so counts agree with the results. Each facet ignores its own selection (self-exclusion); filters are OR within a facet and AND across facets. Without graph params only public documents are counted, and collections are listed only if the collection itself is visible. Facet values are stored in new keyword-array fields (facet_*), filled after the main index build from Virtuoso, keyset-paged on (subject, value) because Virtuoso rejects ORDER BY + large OFFSET. Parts keep every role/collection instead of only the last row's value. Incremental updates fill the fields per part. The multi_match used by the string searches is extracted to search.text_query() and shared with /facets; search behavior is unchanged.
ES 8+ caps an ids query at max_result_window (10,000) ids; ES 6.3 had no limit. A text search combined with a broad filter (e.g. "promoter" in the iGEM collection, 75,333 allowed parts) failed with "Too many ids specified". search_es_allowed_subjects and search_es_allowed_subjects_empty_string now restrict with a terms query on the keyword subject field (equal to _id) via subjects_query(). Its limit, index.max_terms_count, is raised to 1,000,000 when the index is created (default 65,536). Both queries give each match the same constant score, so ranking is unchanged (verified identical hits, order and scores on a 3,000-id restriction). ES9_UPGRADE.md documents the difference and how to raise the setting on an existing index.
SynBioHub's PrepareSubmissionJob sends /incrementalupdate while it prepares a submission, before submit.js uploads it to Virtuoso. The facet fields of new parts were looked up in Virtuoso at that moment, so they came back empty (and overwrote nothing useful): new parts were searchable but missing from the facet counts until the next full /update, which production only runs when an admin triggers it. - facet_role / facet_sboltype / facet_type are now set from the role, sboltype and type SynBioHub already sends in the payload. - add_facet_fields() adds Virtuoso's values instead of replacing them, so payload values survive when Virtuoso does not have the part yet. - Collections and creator, which only Virtuoso has, are fetched again in the background (FACET_REFRESH_DELAYS: 30 s, then 2 min, then 10 min), writing whatever Virtuoso has by then. Parts that have a collection are done; the rest go to the next attempt. At most three small queries per submission. Verified end to end in the real order (update first, Virtuoso upload after): the first refresh filled collections, creator and a second role.
Step-by-step instructions for the people who run the SynBioHub VM and maintain synbiohub-docker: pre-checks (vm.max_map_count, live SBOLExplorer config values, current image digest for rollback), the new docker-compose.explorer.yml (ES 9.5.3, new esdata9 volume, restart policies, ES port bound to loopback), switch-over commands for docker-compose v1 with --no-deps, the startup index build, verification and rollback. Notes that merging upgrade_ES moves the snapshot-synbiohub tag, so the switch must follow the merge. Background and code details stay in ES9_UPGRADE.md.
cjmyers
approved these changes
Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To test this PR, pull 3 PRs from SynBioHub, SynBioHub3, and SBOLExplorer together.