Skip to content

feat: Batch embeddings, upgrade model2vec #298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Pringled merged 3 commits into from
Oct 16, 2026
Merged

feat: Batch embeddings, upgrade model2vec #298

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5fbcd03
perf: Embed new chunks in batches across files with model2vec 0.10
Pringled Oct 11, 2026
2b7f0ed
Rerun semble benchmarks with model2vec 0.10 and batched embedding
Pringled Oct 11, 2026
325ac87
Check every embedding row across batch boundaries in incremental rein…
Pringled Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@

</div>

Semble is a code search library built for agents. It returns the exact code snippets they need instantly, using ~99% fewer tokens than grep+read. Indexing and searching a full codebase end-to-end takes under a second for most repos, matching the retrieval quality of a code-specialized transformer while indexing ~380x faster and querying ~17x faster (see [benchmarks](#benchmarks)). Everything runs on CPU with no API keys, GPU, or external services. Use it as an MCP server, a CLI tool via AGENTS.md, or a dedicated sub-agent, and any coding agent (Claude Code, Cursor, Codex, OpenCode, etc.) gets instant access to any repo.
Semble is a code search library built for agents. It returns the exact code snippets they need instantly, using ~99% fewer tokens than grep+read. Indexing and searching a full codebase end-to-end takes under a second for most repos, matching the retrieval quality of a code-specialized transformer while indexing ~480x faster and querying ~23x faster (see [benchmarks](#benchmarks)). Everything runs on CPU with no API keys, GPU, or external services. Use it as an MCP server, a CLI tool via AGENTS.md, or a dedicated sub-agent, and any coding agent (Claude Code, Cursor, Codex, OpenCode, etc.) gets instant access to any repo.

<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/MinishLab/semble/main/assets/images/demo-dark.gif">
Expand Down Expand Up @@ -78,7 +78,7 @@ semble install --agent claude --type mcp subagent --yes
## Main Features

- **Fast**: indexes an average repo in ~500 ms and answers queries in ~1 ms, all on CPU.
- **Accurate**: NDCG@10 of 0.854 on our [benchmarks](#benchmarks), on par with code-specialized transformer models, at a fraction of the size and cost.
- **Accurate**: NDCG@10 of 0.855 on our [benchmarks](#benchmarks), on par with code-specialized transformer models, at a fraction of the size and cost.
- **Token-efficient**: returns only the relevant chunks, using [~99% fewer tokens than grep+read](#benchmarks).
- **Zero setup**: runs on CPU with no API keys, GPU, or external services required.
- **MCP server**: works with Claude Code, Cursor, Codex, OpenCode, VS Code, and any other MCP-compatible agent.
Expand Down Expand Up @@ -259,7 +259,7 @@ We benchmark quality and speed across ~1,250 queries over 63 repositories in 19
</tr>
</table>

The quality benchmark (left) scores retrieval quality (NDCG@10) against total latency; semble matches the quality of the 137M-parameter [CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed) while indexing 380x faster. The token efficiency benchmark (right) measures how many tokens each method needs to reach a given recall level; semble uses 99% fewer tokens on average and hits 97% recall at only 2k tokens, while grep+read needs a full 100k context window to reach 85%. See [benchmarks](benchmarks/README.md) for per-language results, ablations, and full methodology.
The quality benchmark (left) scores retrieval quality (NDCG@10) against total latency; semble matches the quality of the 137M-parameter [CodeRankEmbed](https://huggingface.co/nomic-ai/CodeRankEmbed) while indexing 480x faster. The token efficiency benchmark (right) measures how many tokens each method needs to reach a given recall level; semble uses 99% fewer tokens on average and hits 96% recall at only 2k tokens, while grep+read needs a full 100k context window to reach 85%. See [benchmarks](benchmarks/README.md) for per-language results, ablations, and full methodology.

## How it works

Expand Down
Binary file modified assets/images/speed_vs_ndcg_cold.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/images/speed_vs_ndcg_warm.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/images/token_efficiency.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
62 changes: 31 additions & 31 deletions benchmarks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Quality and speed across all methods.

| Method | NDCG@10 | Index | Query p50 |
| -------------------- | --------: | ---------: | ----------: |
| **semble** | **0.854** | **306 ms** | **0.91 ms** |
| **semble** | **0.855** | **240 ms** | **0.69 ms** |
| CodeRankEmbed | 0.839 | 116 s | 16 ms |
| ColGREP | 0.693 | 5.4 s | 122 ms |
| BM25 | 0.673 | 47 ms | 0.17 ms |
Expand All @@ -33,7 +33,7 @@ Quality and speed across all methods.
| :-----------------------------------------------------------------: | :-----------------------------------------------------------------: |
| _Time to first result (index + query) vs NDCG@10_ | _Query latency on a warm index vs NDCG@10_ |

semble matches the NDCG@10 of the 137M-param CodeRankEmbed while winning index time by ~380x and query latency by ~17x.
semble matches the NDCG@10 of the 137M-param CodeRankEmbed while winning index time by ~480x and query latency by ~23x.

NDCG@10 is averaged across all queries. Speed numbers use one repo per language, CPU only: cold-start index time and warm query p50 (median across 5 consecutive runs).

Expand All @@ -50,15 +50,15 @@ For each query: tokens consumed at first relevant hit, or 32k if the method neve
| Method | Expected tokens | Savings |
| ------------------- | --------------: | ------------: |
| ripgrep + read file | 45,587 | baseline |
| **semble** | **348** | **99% fewer** |
| **semble** | **353** | **99% fewer** |

### Recall at fixed token budgets

A relevant file is "covered" once any retrieved unit comes from it.

| Method | 500 | 1k | 2k | 4k | 8k | 16k | 32k |
| ------------------- | --------: | --------: | --------: | --------: | --------: | --------: | --------: |
| **semble** | **0.842** | **0.923** | **0.967** | **0.988** | **0.995** | **0.995** | **0.995** |
| **semble** | **0.843** | **0.921** | **0.963** | **0.989** | **0.995** | **0.996** | **0.996** |
| ripgrep + read file | 0.001 | 0.008 | 0.037 | 0.086 | 0.207 | 0.374 | 0.583 |

<details>
Expand All @@ -74,47 +74,47 @@ NDCG@10 per language, sorted by semble. CRE = CodeRankEmbed, cbm = codebase-memo

| Language | semble | CRE | ColGREP | zvec-grep | ck | cbm | grepai | probe | cs | ripgrep |
| ----------- | --------: | --------: | --------: | ---------: | --------: | --------: | --------: | --------: | --------: | --------: |
| javascript | 0.917 | **0.925** | 0.823 | 0.760 | 0.772 | 0.770 | 0.675 | 0.588 | 0.171 | 0.176 |
| cpp | **0.915** | 0.897 | 0.626 | 0.677 | 0.687 | 0.630 | 0.731 | 0.375 | 0.262 | 0.126 |
| zig | **0.913** | 0.911 | 0.474 | 0.664 | 0.511 | 0.766 | 0.755 | 0.369 | 0.121 | 0.000 |
| scala | 0.909 | **0.925** | 0.765 | 0.799 | 0.717 | 0.704 | 0.330 | 0.392 | 0.111 | 0.180 |
| scala | 0.921 | **0.925** | 0.765 | 0.799 | 0.717 | 0.704 | 0.330 | 0.392 | 0.111 | 0.180 |
| javascript | 0.911 | **0.925** | 0.823 | 0.760 | 0.772 | 0.770 | 0.675 | 0.588 | 0.171 | 0.176 |
| go | **0.910** | 0.713 | 0.785 | 0.512 | 0.458 | 0.506 | 0.722 | 0.410 | 0.183 | 0.133 |
| ruby | **0.909** | 0.905 | 0.708 | 0.758 | 0.738 | 0.689 | 0.643 | 0.382 | 0.255 | 0.230 |
| go | **0.895** | 0.713 | 0.785 | 0.512 | 0.458 | 0.506 | 0.722 | 0.410 | 0.183 | 0.133 |
| elixir | **0.894** | 0.893 | 0.808 | 0.811 | 0.786 | 0.506 | 0.669 | 0.412 | 0.397 | 0.134 |
| csharp | **0.885** | 0.848 | 0.614 | 0.609 | 0.548 | 0.775 | 0.277 | 0.392 | 0.248 | 0.117 |
| python | 0.867 | **0.878** | 0.777 | 0.704 | 0.721 | 0.643 | 0.634 | 0.488 | 0.305 | 0.202 |
| swift | **0.860** | 0.845 | 0.710 | 0.709 | 0.672 | 0.630 | 0.429 | 0.280 | 0.151 | 0.160 |
| php | **0.858** | 0.847 | 0.663 | 0.583 | 0.615 | 0.608 | 0.402 | 0.340 | 0.180 | 0.123 |
| rust | **0.856** | 0.754 | 0.662 | 0.541 | 0.419 | 0.454 | 0.519 | 0.242 | 0.193 | 0.162 |
| java | **0.849** | 0.790 | 0.641 | 0.685 | 0.606 | 0.554 | 0.386 | 0.536 | 0.136 | 0.198 |
| bash | 0.825 | **0.834** | 0.706 | 0.725 | 0.677 | 0.768 | 0.723 | 0.226 | 0.170 | 0.000 |
| lua | 0.823 | **0.829** | 0.798 | 0.736 | 0.738 | 0.591 | 0.699 | 0.336 | 0.050 | 0.000 |
| kotlin | 0.821 | **0.823** | 0.637 | 0.628 | 0.587 | 0.611 | 0.478 | 0.335 | 0.170 | 0.166 |
| haskell | 0.765 | **0.811** | 0.683 | 0.689 | 0.733 | 0.624 | 0.483 | 0.313 | 0.160 | 0.000 |
| c | 0.741 | **0.771** | 0.676 | 0.598 | 0.606 | 0.655 | 0.555 | 0.384 | 0.175 | 0.000 |
| typescript | **0.706** | 0.671 | 0.430 | 0.476 | 0.456 | 0.455 | 0.394 | 0.354 | 0.145 | 0.128 |
| **overall** | **0.854** | 0.839 | 0.693 | 0.670 | 0.642 | 0.630 | 0.561 | 0.387 | 0.200 | 0.126 |
| elixir | **0.905** | 0.893 | 0.808 | 0.811 | 0.786 | 0.506 | 0.669 | 0.412 | 0.397 | 0.134 |
| zig | 0.899 | **0.911** | 0.474 | 0.664 | 0.511 | 0.766 | 0.755 | 0.369 | 0.121 | 0.000 |
| cpp | 0.881 | **0.897** | 0.626 | 0.677 | 0.687 | 0.630 | 0.731 | 0.375 | 0.262 | 0.126 |
| python | 0.876 | **0.878** | 0.777 | 0.704 | 0.721 | 0.643 | 0.634 | 0.488 | 0.305 | 0.202 |
| csharp | **0.872** | 0.848 | 0.614 | 0.609 | 0.548 | 0.775 | 0.277 | 0.392 | 0.248 | 0.117 |
| php | **0.864** | 0.847 | 0.663 | 0.583 | 0.615 | 0.608 | 0.402 | 0.340 | 0.180 | 0.123 |
| swift | **0.848** | 0.845 | 0.710 | 0.709 | 0.672 | 0.630 | 0.429 | 0.280 | 0.151 | 0.160 |
| bash | **0.840** | 0.834 | 0.706 | 0.725 | 0.677 | 0.768 | 0.723 | 0.226 | 0.170 | 0.000 |
| lua | **0.835** | 0.829 | 0.798 | 0.736 | 0.738 | 0.591 | 0.699 | 0.336 | 0.050 | 0.000 |
| java | **0.828** | 0.790 | 0.641 | 0.685 | 0.606 | 0.554 | 0.386 | 0.536 | 0.136 | 0.198 |
| rust | **0.826** | 0.754 | 0.662 | 0.541 | 0.419 | 0.454 | 0.519 | 0.242 | 0.193 | 0.162 |
| kotlin | 0.803 | **0.823** | 0.637 | 0.628 | 0.587 | 0.611 | 0.478 | 0.335 | 0.170 | 0.166 |
| haskell | 0.775 | **0.811** | 0.683 | 0.689 | 0.733 | 0.624 | 0.483 | 0.313 | 0.160 | 0.000 |
| c | 0.750 | **0.771** | 0.676 | 0.598 | 0.606 | 0.655 | 0.555 | 0.384 | 0.175 | 0.000 |
| typescript | **0.742** | 0.671 | 0.430 | 0.476 | 0.456 | 0.455 | 0.394 | 0.354 | 0.145 | 0.128 |
| **overall** | **0.855** | 0.839 | 0.693 | 0.670 | 0.642 | 0.630 | 0.561 | 0.387 | 0.200 | 0.126 |

## Ablations

`raw` returns retrieval scores directly; `+ ranking` feeds them through semble's hybrid ranker.

| Retrieval | Raw | + ranking |
| ---------------------- | ----: | --------: |
| BM25 | 0.675 | 0.834 |
| potion-code-16M-v2 | 0.650 | 0.821 |
| BM25 + potion-code-16M-v2 | — | **0.854** |
| BM25 | 0.686 | 0.838 |
| potion-code-16M-v2 | 0.652 | 0.824 |
| BM25 + potion-code-16M-v2 | — | **0.855** |

<details>
<summary>By query category</summary>

| Mode | Architecture | Semantic | Symbol |
| ---------------------------------- | -----------: | --------: | --------: |
| BM25 raw | 0.628 | 0.676 | 0.719 |
| potion-code-16M raw | 0.626 | 0.666 | 0.629 |
| semble BM25 (+ ranking) | 0.770 | 0.819 | 0.957 |
| semble potion-code-16M (+ ranking) | 0.757 | 0.808 | 0.943 |
| **semble hybrid** | **0.802** | **0.846** | **0.958** |
| BM25 raw | 0.642 | 0.680 | 0.748 |
| potion-code-16M raw | 0.636 | 0.671 | 0.620 |
| semble BM25 (+ ranking) | 0.794 | 0.817 | 0.962 |
| semble potion-code-16M (+ ranking) | 0.768 | 0.819 | 0.917 |
| **semble hybrid** | **0.808** | **0.847** | **0.954** |

</details>

Expand Down
6 changes: 3 additions & 3 deletions benchmarks/plot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,9 +103,9 @@ class _Method(TypedDict):
},
{
"name": "semble",
"ndcg10": 0.8544,
"index_ms": 305.6,
"query_p50_ms": 0.91,
"ndcg10": 0.8546,
"index_ms": 240.3,
"query_p50_ms": 0.69,
"color": "#1a5fa8",
"params_m": 16,
},
Expand Down
Loading
Loading