[flyinprogrammer]

Closes #309

Summary

Every BM25 token goes through a light suffix stemmer, at index and query time alike, so "handlers" and EventHandler share the term handler.

 tokenize(text)
   for tok in identifiers(text)
-    result.extend(split_identifier(tok))
+    result.extend(stem(part) for part in split_identifier(tok))

stem(token)
  len <= 3, or has digits/underscores  -> unchanged
  -ies -> -y          dependencies -> dependency
  -es after s/x/z/ch/sh -> drop "es"    classes -> class
  -s (not -ss/-us/-is) -> drop "s"      handlers -> handler
  -ing (len > 5), -ed (len > 4) -> drop  parsing -> pars, parsed -> pars

CACHE_FORMAT_VERSION goes 2 → 3, so cached indexes are rebuilt instead of mixing stemmed and unstemmed terms.

Evidence

• Tests: the new tests/test_tokens.py (13 cases) can't import stem from main's tokens.py and passes here.

• Full suite: make test 404 passed; make lint and make typecheck clean.

• NDCG@10, positive but not statistically clear on its own (95% CIs resample repos):

  Query set	Δ	95% CI
  Benchmark annotations	+0.29	[−0.31, +0.89]
  Commit-message queries (Benchmark: add a commit-message query set so ranking changes aren't tuned to the annotations' style #301)	+0.22	[−0.14, +0.56]

  run_benchmark: 0.852 → 0.855.

• Known quirk: "uses" stems to "us"; deliberately not special-cased, to keep the stemmer small.

Merge Danger

Door: two-way, with a cost

Reverting restores the old tokenization, but every user's index is rebuilt once on upgrade, and again on revert.

Blast Radius: indexing

All BM25 scores change slightly. The first search after upgrading reindexes, so it's slower. Conflicts with any other change that bumps CACHE_FORMAT_VERSION.