Repository navigation
Conversation
added 9 commits
October 20, 2026 12:33
First step towards vector (kNN) search. embedding.py holds the model and field settings: all-MiniLM-L6-v2 through fastembed (384 dimensions, cosine), and the mapping, which uses Better Binary Quantization (bbq_hnsw) so ~1M parts need little Elasticsearch memory; Elasticsearch rescores the best candidates with the full vectors. create_parts_index adds the field to new indexes. Nothing fills it yet. Checked on Elasticsearch 9.5.3: the mapping is accepted and a kNN query with a graph filter returns the expected parts.
The tokenizer pads every text in a model call to the longest one. Part texts are short (median 22 tokens, 99th percentile 56, max ~210), so a single long text in a batch made the model process mostly padding. Truncation would not help: only 0.04% of texts exceed 128 tokens. embed() now sorts each round by length before calling the model and returns the vectors in the original order; model calls take 32 texts instead of 64, and each round reads 2,048 parts instead of 256 so the batches are more even. The vectors are identical (padding is masked out; max difference 0.0). Locally, first build of 78,262 parts from an empty cache: 1,198 s -> 584 s, peak process memory 859 MB -> 532 MB.
With vector_search on, parts that SynBioHub sends to /incrementalupdate, and parts refreshed with /update?subject=, now get their embedding before they are indexed, so a new (also private) part is found by vector search right away instead of after the next full /update. Same text, model and cache as a full build; Sequences are skipped. A failure is logged and the part is still indexed without a vector. The cache lookup/embed logic moves to embedding.vectors_for_texts(), shared by the full build and the incremental path. The SQLite cache waits up to 30 s for a concurrent writer (a full build and an incremental update can overlap). Checked locally with SynBioHub's payload format and the real model: the new private part ranks first in kNN for its owner and is not returned for other users. Elasticsearch 9 leaves vectors out of _source by default; partial updates (e.g. the later facet refresh) keep them.
GET /vector_search embeds the query with the same model as the parts and returns the nearest parts by cosine similarity. Visibility follows /facets: only public parts unless the caller passes its own graphs. vector_search.py holds the query; explorer.py adds the route, which returns 503 while vector_search is off in the config. Unit tests check the request body (k, num_candidates, the graph filter).
/update?embeddings_only=true adds the embedding mapping to the existing index if it is missing, then embeds every part already in it. It does not read Virtuoso, rebuild clustering, PageRank or facets, since the existing documents keep those fields. Checked on the local index (ES 9.5.3): 78,262 parts embedded in 671 s with the cache from earlier builds; /vector_search then returns the expected parts for 'ribosome binding site' and 'constitutive promoter'.
Also records the last update times and turns vector_search on.
SynBioHub emits a bare apostrophe keyword as lcase(\') with no string
literal, which Virtuoso rejects with 400. Rewrite it to the valid
lcase('\'') form before the query reaches search.search.
Selected facet values now narrow the kNN pre-filter, as they do on /facets, so the nearest parts are chosen among those that match. Passing facet_by returns counts for each facet over the returned hits.
added 3 commits
October 27, 2026 20:42
/vector_search takes an offset and returns the parts from there on, so the search by meaning results can be paged. A page is the first offset + size nearest parts with the first offset dropped, and the deepest result is MAX_RESULTS (100). The response has a 'count': the Top N, the number of visible parts that have an embedding, capped at MAX_RESULTS. kNN always has a nearest part to give, so the count is not a relevance threshold. Also adds flask/check_facet_refresh.py, which sends a burst of fake submissions to /incrementalupdate and counts the Facet refresh lines.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds semantic (kNN) search over parts. Each indexed part gets one embedding vector, computed from its displayId, name, description and SO role names. GET /vector_search?query=... embeds the query with the same model and returns the nearest parts by cosine similarity, so a query can match parts that share no words with it. Optional facet_by returns facet counts over the hits, and selected facets narrow the search as they do on /facets.
It also includes one unrelated fix: normalizing a bare apostrophe keyword in the SPARQL search endpoint (c25caa5).
What changed
Model and mapping (flask/embedding.py): sentence-transformers/all-MiniLM-L6-v2 via fastembed (ONNX, no PyTorch), 384 dimensions. The embedding field is a dense_vector with cosine similarity and bbq_hnsw indexing. create_parts_index adds the field to new indexes.
Embedding text: part_text() joins displayId, name (if different), description and role names. Sequence parts are skipped, since their only text is the displayId.
Cache: vectors are stored in SQLite, keyed by a hash of model and text, so a rebuild only embeds parts whose text changed.
Full build: parts are embedded in order of text length, which keeps model batches even. Locally, a first build of 78,262 parts from an empty cache went from 1,198 s to 584 s, and peak memory from 859 MB to 532 MB. Vectors are identical to the unsorted run.
Incremental updates: when vector_search is on, parts from /incrementalupdate and /update?subject= are embedded before indexing, so a new private part is searchable right away. A failed embedding is logged and the part is still indexed without a vector.
Backfill: /update?embeddings_only=true adds the mapping if missing and embeds the parts already in the index, without reading Virtuoso or rebuilding clustering, PageRank or facets.
Search (flask/vector_search.py, flask/explorer.py): the kNN query uses k = size and num_candidates = max(100, 10 * size). Visibility follows /facets: public parts plus the caller's own graphs, applied as a pre-filter. The route returns 503 when vector_search is off in the config.
Tests: test_embedding.py and test_vector_search.py cover the cache, batching order, the request body, and the graph filter.
Testing
Local ES 9.5.3 with 78,262 parts: /vector_search returns the expected parts for "ribosome binding site" and "constitutive promoter".
A new private part, sent in SynBioHub's payload format, ranks first for its owner and is not returned to other users.
Unit tests: flask/tests/test_embedding.py, flask/tests/test_vector_search.py.
Review notes
Config: flask/config.json sets vector_search: true. Please check that these match your deployment.
Model: all-MiniLM-L6-v2 is an English model. Non-English names or descriptions will embed less well.
Search type: this is pure vector search. It is not merged with keyword ranking.
Index size: embeddings add a cache of about 1.5 KB per part on disk, and the index needs an embedding field.
Existing indexes: run /update?embeddings_only=true once to backfill them.