Repository navigation
fix(embedding): bound truncation estimate above real tokenizer costs - #6140
Open
akushonkamen wants to merge 1 commit into
Open
akushonkamen wants to merge 1 commit into
akushonkamen wants to merge 1 commit into
Conversation
estimate_embedding_input_tokens charged every non-CJK character 0.25 tokens (4 chars/token). Tokenizers of the embedding models served behind OpenAI-compatible endpoints spend far more characters per token: measured against llama-server POST /tokenize (issue volcengine#6129), Cyrillic prose needs ~2.7 chars/token and even English prose and shell code land near 3, so text truncated to max_input_tokens arrived with 1.3-1.5x the configured budget and the provider rejected it — failing the whole session_commit task and losing that session's memory. Charge per character in tenths of a token: CJK 1.0 (unchanged), Cyrillic 0.5, everything else 0.4 — rates at or above the measured worst cases. truncate_embedding_input now also reserves the budget that the truncation marker itself consumes, so the final payload stays within max_input_tokens instead of exceeding it by the marker's cost. The estimate/truncate expectations in test_vectorize_file_strategy are updated for the tighter rates, and tests/unit/test_embedding_input.py adds a mock tokenizer calibrated to the issue's measurements asserting that truncated input stays within the real-token budget. Fixes volcengine#6129 Signed-off-by: akushonkamen <akushonkamen@163.com>
Contributor
Author
|
Rebased onto current |
Contributor
Author
|
The failing
Local evidence on this PR's head (e7a561e), run just now:
Happy to re-run the lane once the credentials issue clears on the runner side. |
ZaynJarvis
added
the
kernel
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
embedding.max_input_tokens的截断预算按"非 CJK 一律 0.25 token/字符"估算，系统性低于真实 tokenizer 消耗：issue #6129 实测（llama-serverPOST /tokenize）西里尔文约 2.7 字符/token，英文散文与 shell 源码也接近 3 字符/token，远密于估算假设的 4 字符/token。文本被截断到max_input_tokens后实际仍携带 30-50% 超出的真实 token，被 provider 以超出上下文拒收，导致session_commit整任务失败、该会话记忆永远写不进。本 PR 把截断预算改为按脚本分级的保守估算（以十分之一 token 为整数单位），并为截断标记本身预留预算，保证最终载荷（含 marker）的估算值不超过
max_input_tokens，且不低于各脚本实测最差消耗。Human Involvement
Related Issue
Fixes #6129
Type of Change
Changes Made
openviking/utils/embedding_input.py：截断计费改为十分之一 token 的整数单位、按脚本分级——CJK 1.0 token/字符（不变）、西里尔 0.5、其余 0.4，均不低于 issue [Bug]: embedding.max_input_tokens assumes 4 chars/token, so text truncated to "4096 tokens" arrives with up to 6090 #6129 实测最差值（西里尔 0.372、PowerShell 0.36 token/字符）；truncate_embedding_input同时为截断标记本身预留预算，最终载荷（含 marker）估算值 ≤max_input_tokens。非 ASCII 文本按_SCRIPT_RUN（CJK∪西里尔）逐 run 计费，替换原"4 分之一 token 单位"的单一费率。openviking/models/embedder/base.py:321,331、openviking/models/embedder/openai_embedders.py:329、openviking/utils/embedding_utils.py:647,680、openviking/service/reindex_executor.py:1558。tests/unit/test_embedding_input.py：mock tokenizer 按 issue 实测费率（0.372/0.36 token/字符）校准，断言俄文/英文/PowerShell 样例与混合脚本样例截断后真实 token ≤ 4096。tests/unit/test_vectorize_file_strategy.py：入队前截断断言更新为新费率与 marker 预留下的期望前缀。Testing
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I have tested this on the following platforms:
python -m pytest tests/unit/test_embedding_input.py -q：5 passed（mock tokenizer 按实测费率校准）。E2E 复核（issue 实测费率 mock）：俄文/英文/PowerShell 样例截断后真实 token 3140/3686/3687，均 ≤ 4096；修复前同口径实测 5180/9180/7200，全部超限。
python -m pytest tests -k embedding -q：12 failed, 269 passed, 24 skipped；与干净main（84cc2b4）同命令（12 failed, 264 passed, 24 skipped）逐条对比，12 个失败完全一致（batch_write 1 + collection_schemas 1 + semantic_processor_lock_ownership 4 + gemini config 2 + vectorize_strategy 1 + ollama factory 3，均为既有失败），零新增失败；多出的 5 passed 即新增测试。python -m pytest tests -k embedding -x -q：test_content_batch_write...embedding_worker因FileNotFoundError: OpenViking configuration file not found失败，干净main上同样失败（环境性，非本 PR 引入）。相关子集（test_vectorize_file_strategy + test_embedding_vectorize_strategy + test_embedding_input + tests/unit/embedder + tests/storage/test_collection_schemas.py + tests/test_telemetry_runtime.py）：2 failed, 177 passed，2 个失败均为既有失败。
改动文件
ruff check、ruff format --check通过。Checklist
Additional Notes
estimate_embedding_input_tokens；本 PR 尊重该清理、不恢复该函数，仅收紧生产路径truncate_embedding_input的估算与 marker 预留。