Skip to content

refactor(embedding): 删除无调用的 Token 估算函数 - #6134

Merged
qin-ctx merged 1 commit into
volcengine:mainfrom
qin-ptr:refactor/remove-unused-embedding-estimator
Oct 15, 2026
Merged

qin-ctx merged 1 commit into
volcengine:mainfrom
qin-ptr:refactor/remove-unused-embedding-estimator

Conversation

@qin-ptr

@qin-ptr qin-ptr commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Description

删除截断重构后已无生产调用的 estimate_embedding_input_tokens,以及测试中对它的引用和断言。保留入队前截断的行为验证,无运行时行为变化。

Human Involvement

  • A human participated in the implementation or review loop
  • This PR was generated entirely by AI agents without human participation in the loop

Related Issue

Related #6129。本 PR 仅清理死代码,不修复真实 Token 超限问题。

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Changes Made

  • 删除无生产调用的估算函数。
  • 移除测试中的导入、expected_tokens 参数和估算断言,保留截断结果断言。

Testing

  • I have added tests that prove my fix is effective or that my feature works

  • New and existing unit tests pass locally with my changes

  • I have tested this on the following platforms:

    • Linux
    • macOS
    • Windows
  • pytest -q --no-cov tests/unit/test_vectorize_file_strategy.py -k bounds_content_before_enqueue:3 passed。

  • 修改文件的 ruff check、ruff format --check 和 git diff --check 通过。

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Screenshots (if applicable)

不涉及。

Additional Notes

无。

Remove the estimator left without production callers after the truncation
rewrite, along with its test-only assertions. Keep truncation coverage.

Co-authored-by: qin-ctx <qinhaojie.exe@bytedance.com>
@qin-ctx
qin-ctx merged commit 465b146 into volcengine:main Oct 15, 2026
8 of 9 checks passed
@github-project-automation github-project-automation Bot moved this from Backlog to Done in OpenViking project Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants