[u7-u7]

Change Summary

Make Han keywords searchable inside unspaced descriptions in the lightweight BM25 retriever used by default skill discovery. A local skill described as 用于分析销售数据并生成报表 is not found by 销售数据; CSV cannot match 使用Python3处理CSV文件. Reproduced through the real SkillCatalog and default SkillSearchEngine, without retriever/model mocks.

Split BMP Han and Extension A characters individually, preserving other Unicode word runs, case normalization, digits and underscores. No dependency, model download, configuration or public API is added. Standalone BM25 and default/hybrid sparse skill search are affected; legacy hybrid_retriever.py is unchanged.

This is a character-level bag of words, not word segmentation or semantic retrieval. Partial single-character matches can increase; word order is not modeled. Shared-character distractor ranking is covered by a regression fixture, not a general quality benchmark.

Related issue number

No duplicate found in all-state BM25/CJK/Chinese token searches. #975 is unrelated cache invalidation.

Validation

Python 3.12.14 with normal SDK dependencies.

• Original four regressions on unchanged main: 3 failed, 1 passed.
• Final five regressions plus existing skill-feature/retriever/fusion tests: 39 passed, including shared-character distractors, Latin tokens, punctuation and no-match.
• Entire tests/skills: 117 passed, 4 skipped, 1 unrelated SDK download directory failure, also reproduced with unmodified BM25. Dedicated writable HOME used because default SDK cache paths are read-only.
• Changed-file pre-commit hooks and git diff --check passed. Full pre-commit was run and has existing unrelated lint/format failures; unrelated changes were discarded.

Not run: full repository suite, live Agent/LLM calls, browser UI, other Python/platform versions. Broad existing skill tests include network-backed downloads; new regressions are offline.

Checklist

• Title summarizes the change.
• Unit regressions exist.
• Full-repository pre-commit passes (baseline failures above).
• Tokenization behavior documented in code; no new option.

AI assistance: implemented and tested with OpenAI Codex; automated review, no human-review attestation. Draft, no reviewer request.