Skip to content

fix: Match file stems with leading or trailing underscores in the NL stem boost - #304

Closed
flyinprogrammer wants to merge 1 commit into
MinishLab:mainfrom
flyinprogrammer:fix/stem-boost-underscore-names
Closed

flyinprogrammer wants to merge 1 commit into
MinishLab:mainfrom
flyinprogrammer:fix/stem-boost-underscore-names

Conversation

@flyinprogrammer

Copy link
Copy Markdown

Closes #303

Summary

The NL stem boost splits file stems and directory names with split_identifier, which returns ["__git"] for __git, so the keyword git never matches. Strip leading and trailing underscores first, in the stem boost only:

 _boost_stem_matches(boosted, query, max_score)
   for each candidate file
-    parts = split_identifier(path.stem)
-    parts += split_identifier(path.parent.name)
+    parts = _split_path_part(path.stem)          # split_identifier(name.strip("_") or name)
+    parts += _split_path_part(path.parent.name)
     boost by the share of query keywords matching parts

split_identifier and BM25 tokenization are unchanged.

Evidence

The new parametrized cases in test_apply_query_boost_nl_stem_match_boosts cover a leading-underscore case, ("git status", "lua/telescope/__git.lua"), and a trailing-underscore case, ("git status", "lua/telescope/git__.lua").

  • Before (main's boosting.py): FAILED test_apply_query_boost_nl_stem_match_boosts[git status-lua/telescope/__git.lua]. The score stays at 0.5. The trailing case already passes through the prefix match and is kept as a regression guard.
  • After: 4 passed.
  • Full suite: make test 393 passed; make lint and make typecheck clean.
  • Benchmark: run_benchmark NDCG@10 goes from 0.852 to 0.854. Few files are named this way, so the effect is small.

Merge Danger

Door: two-way

A ranking-only change with no index or cache format change; reverting restores the old ranking exactly.

Blast Radius: ranking

It only affects natural-language queries over files or directories whose names start or end with _.

…stem boost

split_identifier("__git") returns ["__git"], so a query mentioning "git" never
earned the stem boost for files like telescope.nvim's __git.lua. Strip leading and
trailing underscores from file stems and directory names before splitting them
for the stem boost. BM25 tokenization is unchanged.
@stephantul stephantul closed this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NL stem boost never matches files whose names start with underscores (e.g. __git.lua)

2 participants