Repository navigation
Conversation
…oders only when a queue needs them The indexing helper always loaded the full multimodal model — the text model plus the 555 MB image/audio projector (mmproj-embeddinggemma-2-Q8_0.gguf, this box's reference model) — even when every queued file was text or code, and kept it loaded through the idle linger after the run (issue ARahim3#155). `SharedEmbedder` now starts text-only. Each run asks the store (`IndexStore.pendingKinds`, a DISTINCT-kind query on the queue) whether it holds a kind that embeds media (`FileKind.needsMediaEncoders`: images, screenshots, video, audio) before reading any file, and re-shapes the model then. A queue that said text and hands over media anyway — a scanned PDF's page, an unfinished long file — upgrades on the spot at the embedding call, where a model failure marks files the way it always has. Once media has been wanted the shape is sticky: encoders are not dropped and paid for twice. The text-only shape keeps the full 2048-token document budget: `LlamaEmbedder.init` takes `contextTokens` so skipping the media encoders no longer borrows the *query* encoder's 512-token cut. That decoupling is the whole safety of the change — vectors for long PDF pages, full doc chunks, long code stretches, and multilingual text sit past 512 tokens, and cutting them there changes their vectors under a fingerprint that cannot notice. Measured on this box with the two shapes side by side (9,840-char input, well past the cut, plus a short input, both embedders, one process, this box): at text-only-2048 the long and short vectors are byte-identical to the full model's; at the old text-only-512 the long vectors differ (from the first dimension) while the short ones match. Text-only indexing therefore changes no stored vector by construction; the query encoder keeps its 512 as always. (`contextTokens` is a llama.cpp-only knob: the mlx-vlm worker path in $DIGUP_WORKER_DIR sets its own width and ignores it — disclosed.) Measured on this box (Apple M1, 8 GB, macOS 26.5.1), a corpus of 60 text/code docs: peak resident size 1,045 MB → 499 MB (−546 MB), model load 1.47 s → 0.53 s, same 60 files done, same idle counts (60 searchable, 60 vectors); three meaning searches return identical hits and scores from indexes built by the old and the new helper. A mixed folder (images queued first, the usual order) starts full in both: 10.02 s vs 9.66 s wall, 1,003 MB vs 990 MB peak — within run-to-run noise, no regression. Text-only start was gated by the existing "waiting for model" check (the helper waits whenever the model set is incomplete, mmproj among them): that gate is unchanged here, disclosed. Also folded in from the pre-push review: a failed media-model swap now falls back to the text-only shape instead of stranding the session (`wantsFull` sticky + model gone made every later text embed answer "the model is closed"); `Kept.info` answers through the last-seen shape instead of force-unwrapping (a trap takes the whole helper down, and the embedder can close under a queued indexer); the `"model"` event joins the helper's protocol doc header; the sticky `wantsFull` read in `use` runs under the lock like every other access. The shape/upgrade logic lives in the executable target and stays testable only by box runs — disclosed. Suite: 127 green (base 125, +2: `pendingKindsSeeTheQueuedShapesOnly`, `mediaKindsAreTheOnesThatEmbedPicturesAndSound`). Shell JS tests 17 pass. The `digup index` one-shot CLI keeps loading the full model in-process: it has no idle linger to pay, and was left alone (disclosed).
Ocean572
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Indexing Helper: Start Text-Only and Load Media Encoders on Demand (#200)
Closes #155.
Problem
The indexing helper previously loaded the full multimodal model at startup, regardless of the types of files being indexed.
This included both the text model and the 555 MB image/audio projector (
mmproj-embeddinggemma-2-Q8_0.gguf, the reference model used for these measurements).For queues containing only text and code files, the media encoders were unnecessary overhead. However, they remained loaded throughout indexing and the helper's idle-linger period after a run completed.
This resulted in:
Changes
The indexing helper now starts with a text-only model configuration and loads the media encoders only when required.
1. Detect Media Requirements Before Indexing
At the beginning of each indexing run, the helper queries
IndexStore.pendingKindsto determine which file types are currently queued.This uses a
DISTINCT-kind query to identify whether the queue contains any file types requiring media encoders.The decision is based on
FileKind.needsMediaEncoders, which identifies:If the queue contains only text and code files, the helper remains text-only. If any queued file requires media encoders, the model is upgraded before file processing begins.
2. Handle Unexpected Media Files
Queue inspection is not always sufficient to determine every file's eventual embedding requirements.
For example:
To handle these cases, the embedding call also checks whether media encoders are available.
If a file unexpectedly requires media embeddings, the helper upgrades the model at that point.
If the model upgrade or embedding operation fails, the affected files are handled through the existing failure-reporting mechanism.
Once media encoders have been loaded, the model configuration remains in that state for the lifetime of the helper session. The encoders are not repeatedly unloaded and reloaded between runs.
3. Preserve the Full Document Token Budget
An important part of this change is separating the document embedding context from the query embedding context.
Previously,
LlamaEmbedder.initused a 512-token context when initializing the text-only model. That limit was associated with the query encoder rather than the document embedding requirements.The implementation now explicitly passes
contextTokenstoLlamaEmbedder.init, preserving the full 2048-token document embedding budget even when media encoders are omitted.This distinction is critical because:
Truncating these inputs would change their resulting embeddings, even though the indexing fingerprint would not detect the change.
With the new implementation, text-only indexing retains the same 2048-token document context as the full model. The query encoder continues using its existing 512-token context.
This ensures that removing the media encoders does not change the generated document vectors.
Performance Measurements
Measured on the development machine:
Hardware and environment: Apple M1, 8 GB RAM, macOS 26.5.1.
Test corpus: 60 text and code documents.
Text-Only Indexing
The new implementation reduces peak resident memory by 546 MB, while reducing model initialization time from 1.47 seconds to 0.53 seconds.
Both implementations indexed the same 60 documents and produced the same final indexing counts.
Search Result Equivalence
Three semantic searches were performed against indexes built by the old and new implementations.
All three searches returned identical results and scores.
This verifies that the text-only optimization does not change search behavior for the tested corpus.
Embedding Vector Identity
Both embedding configurations were tested within the same process using:
The resulting vectors were compared between the full multimodal model and the text-only configuration.
The results confirm that the new text-only configuration produces byte-identical vectors to the full model for both tested inputs.
The previous 512-token configuration produced different vectors for the long input, despite matching on the short input.
The new implementation therefore preserves document embedding identity by construction for the tested inputs. The query encoder retains its existing 512-token context.
Mixed Media Queue
A mixed folder containing media files was also tested, with images queued first (the typical processing order).
Both implementations loaded the full model configuration.
These differences are within normal run-to-run variation. There is no measurable performance regression when media encoders are actually required.
Testing
The two new tests cover the store-level file-kind behavior used to determine whether queued work requires media encoders.
The model-shape and upgrade behavior is implemented in the executable target and was verified through direct runs on the development machine rather than isolated unit tests.
Implementation Details and Limitations
The following details are important to understanding the scope of this change:
contextTokensis a llama.cpp-specific setting. The MLX-VLM worker path ($DIGUP_WORKER_DIR) manages its own context width and ignores this parameter.mmproj) is unavailable. This readiness gate is unchanged.digup indexone-shot CLI continues loading the full model in-process. Unlike the persistent indexing helper, it does not have an idle-linger period where keeping unused media encoders loaded creates ongoing overhead.Additional Review Fixes
The following improvements were incorporated from pre-push code review:
Kept.infonow responds using the last-known model configuration rather than force-unwrapping the current model state."model"event has been added to the indexing helper's protocol documentation header.