Skip to content

feat: Rank symbol queries with BM25 only - #308

Closed
flyinprogrammer wants to merge 1 commit into
MinishLab:mainfrom
flyinprogrammer:feat/symbol-queries-bm25-only
Closed

flyinprogrammer wants to merge 1 commit into
MinishLab:mainfrom
flyinprogrammer:feat/symbol-queries-bm25-only

Conversation

@flyinprogrammer

Copy link
Copy Markdown

Closes #307

Summary

Bare identifier queries (MyService, Sinatra::Base) currently give the dense list 30% of the fused score. Setting their weight to 0 ranks them by BM25 alone; natural-language queries are unchanged.

 # semble/ranking/boosting.py
-_ALPHA_SYMBOL = 0.3  # lean BM25 for exact keyword matching
+_ALPHA_SYMBOL = 0.0  # BM25 only: exact identifier lookups gain nothing from the dense list
 _ALPHA_NL = 0.5  # balanced semantic + BM25
search(symbol query)
  fused = 0.0 × dense_rrf + 1.0 × bm25_rrf     # dense-only chunks score 0 and are dropped
  apply boosts (definition boost etc.)         # unchanged
  rerank_topk(penalise_paths = alpha < 1.0)    # 0.0 < 1.0: test/example penalties still apply

Evidence

  • Test: test_resolve_alpha[MyService-None-0.0] fails on main's boosting.py and passes here. It checks the constant; the behaviour is covered by the benchmark below.
  • Full suite: make test 391 passed; make lint and make typecheck clean.
  • Benchmark annotations: NDCG@10 +0.12, 95% CI by repo [+0.02, +0.25]. The symbol category goes 0.959 → 0.962, and run_benchmark 0.852 → 0.854.
  • Commit-message queries (Benchmark: add a commit-message query set so ranking changes aren't tuned to the annotations' style #301): no change. They're natural language, so none takes the symbol path.

Merge Danger

Door: two-way

A one-constant ranking change with no index or cache change; reverting restores the old ranking exactly.

Blast Radius: ranking

Only queries that is_symbol_query classifies as identifiers. The dense search still runs for them; its scores are just weighted 0.

Bare identifier queries ("MyService", "Sinatra::Base") are exact lookups: the
dense list adds near-miss neighbours that the definition boost then has to
outrank. Set _ALPHA_SYMBOL from 0.3 to 0.0 so symbol queries fuse BM25 only;
NL queries are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rank bare symbol queries with BM25 only (_ALPHA_SYMBOL 0.3 → 0.0)

2 participants