Skip to content

fix(discovery): state the document case in the discovery routing prompt - #566

Open
tietoumiao wants to merge 1 commit into
T-Auto:mainfrom
tietoumiao:fix/408-doc-reader-discovery
Open

tietoumiao wants to merge 1 commit into
T-Auto:mainfrom
tietoumiao:fix/408-doc-reader-discovery

Conversation

@tietoumiao

@tietoumiao tietoumiao commented Nov 1, 2026 •

Copy link
Copy Markdown

Related to #408

What was wrong

The on-demand read_pdf / read_docx / read_table readers are only reachable through
discovery, yet the only routing sentence a model sees before selecting them said "when needed" and
named no format — while the host office skill advertises reading Office files directly. A model
facing a .pdf therefore had no stated reason to discover the reader.

Fixed in this PR after review (see ## Review response).

The change

  • src/policy.ts — the discovery routing line now uses the condition the shipped docs already
    state: a .pdf / .docx / .csv / .tsv file whose reader is absent from the tool table is
    a reason to select that reader. The weaker "when a task needs one this table does not show" wording
    is gone, because the resident ops_read_files is in that table and reads CSV/TSV text, so a
    literal reading of the old sentence could conclude no discovery was needed. A second line concedes
    Office authoring to the host office skill and now also covers the failed-read fallback the
    issue asked for: a document read that fails falls back to that skill instead of retrying the
    reader.
  • test/policy.test.mjs — pins the two sentences, and keeps asserting that no unpublished reader
    name (ops_read_pdf / ops_read_docx / ops_read_table) is ever advertised.
  • test/docs-facts.test.mjs — derives the always-loaded trio cost (810 tokens / 3619 bytes) from
    the catalogue so the documented figure cannot drift; a stray whitespace-only line is removed.
  • docs/discovery.md + docs/工具发现.md — both carry the same condition and the fallback sentence.
  • docs/CHANGELOG.md — ## [Unreleased] entry (EN + ZH).

Deliberately not done: no reader schema change, no change to what is published on demand, and the
office skill boundary is stated rather than implemented.

Validation

$ node test/run.mjs policy
dsh-ops policy: all 14 checks passed          (pristine HEAD: 13)

$ node test/run.mjs docs-facts
dsh-ops docs-facts: all 7 checks passed

$ node test/run.mjs discovery
dsh-ops test: 10 suite(s) passed in 23s

$ node scripts/check-file-size.mjs
check-file-size: 597 code file(s) checked, all at or under 1000 lines

$ node scripts/validate-manifest.mjs
validate-manifest: all 89 checks passed

$ node test/run.mjs policy                     # after reverting only src/policy.ts
dsh-ops policy: 1 of 14 check(s) failed        (test/policy.test.mjs:70)

Rebased onto 32daf5b (main after #557) and re-ran everything above on that base; the CHANGELOG
conflict was resolved by keeping this entry on top of the existing ## [Unreleased] lines.

  • 撤掉本次实现后变红的检查:policy 的 discovery routing names the document case and the Office-authoring boundary (#408)
  • docs/CHANGELOG.md ## [Unreleased]:已加(EN + ZH 成对)
  • schema / 提示词字节影响:discovery 段 117 字符 / 22 tok → 396 字符 / 86 tok,即
    +279 字符 / +64 cl100k_base tokens,用套件同一个编码器实测;工具 schema 未变,
    file/discovery 的 261 tokens / 1196 bytes 预算与 114-token 目录条目 pin 仍然成立
  • CI:本 PR 由自己的 pull_request 事件触发 run(fork 首跑停在 action_required,需维护者批准)。
    下面 ## CI 段贴的是该 run 的 url/event/conclusion;如果之后 rebase 或补推,run 会换成新 head 的那一次,
    届时同步更新这一段。

Environment

  • OS / 架构:Windows 11 x64
  • Node:24.16.0
  • 基于上游 main 的 commit:32daf5b
  • 本机另有 v0.5.0-alpha4(e3edb27)的发布基线用于后续开发;本 PR 的目标分支是 main,
    因此没有夹带发布提交(git diff --stat origin/main..HEAD = 6 文件)
  • 安装通道:源码 tarball + npm ci --ignore-scripts

Review response

针对 2026-11-02 的 reviews(4 点全部处理):

  1. 触发条件与 docs 不一致 — prompt 改用 docs 的同一句("whose reader is absent from the tool
    table")。如实说明:这句只是把判据写得更明确,并没有消除 CSV/TSV 的歧义——常驻的
    ops_read_files 描述的仍是 "Read UTF-8 ranges… Binary needs a binary reader",文本 CSV 仍可能被
    读成「表里已有 reader」。若维护者希望彻底闭合,可加一个 format-aware 限定词(措辞与新增 token
    我按实测补,并同步 docs/CHANGELOG/两个 test)。
  2. 漏掉读取失败的回退 — 第二行补上:读失败回退到宿主 office 技能,而不是重试读取器;中英
    文档同步。
  3. Closes #408 已过期 — 改为 Related to #408(issue 已于 2026-11-01T17:30Z 以 COMPLETED 关闭)。
  4. test/docs-facts.test.mjs:69 行尾空格 — 已改为空行;git diff --check 现在干净。

数字更正:之前 PR body 写的 +202 字符 / +51 tokens 是本轮之前的措辞;本轮实测 +279 / +64,
CHANGELOG 与本文均已按实测更新。

未验证 / 风险披露

  • 没有模型在环数据(需要 [学术工程] 行有不得,反求诸己 #527 的 BM-04/05 harness);证据是装配后的提示词原文 + pin 住的断言。
  • 51→64 token 里仍有约 28 token 与同一提示词下方的目录段(已列 pdf/docx/csv/tsv)语义重叠,
    真正新增的是 Office 授权边界与失败回退。

提交前交叉循环检验

LOOP:  round 3 · verifier A 零阻断 · verifier B 零阻断 · 收敛 是
FIXED: 分支基线漂移到 release/0.5.0-alpha4(diff 6→19 文件)→ 复位到 origin/main,现 6 文件 +42−2
       正文「CSV/TSV 半边不再可能被读成表里已有 reader」属过度声称 → 改为如实披露
       草稿 ## CI 段缺 url/event/conclusion 却声称已贴 → 补真实 run 行、命令改真实 id
KNOWN: #408 当前状态(验证者无网络,本地快照早于关闭时间,按维护者 review 记为已关闭)
       discovery 10 套件(验证者会话不能 spawn native worker;提交侧实测 10/10 绿)

CI

$ gh run view 41384985254 --json url,event,conclusion --jq '"\(.url) \(.event) \(.conclusion)"'
https://github.com/T-Auto/dsh-ops/actions/runs/41384985254 pull_request action_required

该 run 由本 PR 自己的 pull_request 事件触发,停在 action_required(fork 首跑需维护者批准);
批准后把结论补在这一行。若后续 rebase 或补推,换成新 head 的那次 run。

@Haoming9527

Copy link
Copy Markdown
Collaborator

Compared this patch with current main 065cd7f and #542. The on-demand directory already supplies actual eligible names, format terms and task-keyword selection guidance through #536. The routing sentence still says “when needed”, so this patch's explicit document trigger and Office-authoring boundary are an additional editorial proposal, not a duplicate of #542.

The disclosed +51-token cost and lack of model-in-loop evidence should remain visible for maintainer review. #408's directory resolution does not establish that this additional guidance has no value. Leaving this PR open for that scope decision; no overlapping implementation or fork-CI approval was performed in this review.

@Nicholas023 Nicholas023 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

审查结论:REQUEST_CHANGES — 但这是 9 个 PR 中唯一不与既定工作重复的增量提案,修完可以合并

已核实为真的部分

  • token 数字精确:用 js-tiktoken cl100k_base 独立核算,117→319 字符(+202)、22→73 tokens(+51),与 PR 声称完全一致
  • policy 14/14、docs-facts 6/6 通过
  • revert-red(撤 src/policy.ts)→ 1/14 红
  • 0.5.0 基线的路由句确实仍是「Discover a capability when needed」,缺口真实存在

需要修改的 4 点

1. 触发条件比 docs 弱,CSV/TSV 半边可能不生效(阻断)
prompt 写的是 "a task needs one this table does not show, e.g. reading a .pdf …";但常驻的 ops_read_files 就在工具表里且能读 CSV/TSV 文本,模型按字面可以判定「表里已有 reader」→ 不 discover。docs/discovery.md 的写法("whose reader is absent from the tool table")更准确,请让 prompt 与 docs 使用同一句。

2. 漏掉 issue 明确要求的回退(阻断)
#408 最后一段要求提示「读失败就退回 office 技能」。本 PR 只让出了创建/编辑/渲染/导出,没有覆盖读取失败这条最需要的路径。

3. Closes #408 已过期
#408 已于 2026-11-01T17:30Z 以 COMPLETED 关闭(结论是 #536 已覆盖)。请改为 Related to #408。

4. test/docs-facts.test.mjs:69 行尾空格
git diff --check 直接报错,属仓库 PR 的常规检查项。

成本视角

51 token 里约 28 个与同一提示词下方的目录段(已列 pdf/docx/csv/tsv 及「按任务关键词查询」)重复,真正新增的是 23 token 的 Office 授权边界。方向合理,但证据只到「提示词文本正确」,没有模型在环数据(PR 已如实披露)。

处置建议

保留。 修完上述 4 点即可合并;这是本批次里唯一有独立价值的 PR。

@tietoumiao
tietoumiao force-pushed the fix/408-doc-reader-discovery branch from f05855c to 8ffbe37 Compare November 2, 2026 02:55
@tietoumiao

Copy link
Copy Markdown
Author

All four points from the review are addressed in 8ffbe37:

  1. Trigger condition now matches the docs. The prompt uses the same sentence as
    docs/discovery.md — a .pdf, .docx, .csv or .tsv file whose reader is absent from the
    tool table
    . The previous "a task needs one this table does not show" wording is gone, so the
    resident ops_read_files cannot be read as covering CSV/TSV.
  2. Failed-read fallback added. The second routing line now says a document read that fails falls
    back to the host office skill instead of retrying the reader; both language versions of
    docs/discovery.md carry it.
  3. Closes #408 → Related to #408. Correct — the issue was closed as completed on
    2026-11-01T17:30Z, not by this PR.
  4. Trailing whitespace removed at test/docs-facts.test.mjs:69; git diff --check is clean.

Re-measured with the encoder the suite pins (my earlier +202 / +51 was the pre-review wording):

$ node -e "...getEncoding('cl100k_base')..."
117 chars / 22 tokens   (baseline sentence)
396 chars / 86 tokens   (both routing lines now)
=> +279 characters, +64 cl100k_base tokens

node test/run.mjs policy → all 14 checks passed; docs-facts → all 6; discovery → 6 suites
passed. Reverting src/policy.ts alone turns policy into 1 of 14 failed at
test/policy.test.mjs:70.

On the duplication question: this is the only PR in the batch whose subject is a prompt-text
increment, so I have not touched #541 / #538 / #543 / #544 / #547 / #572. Happy to wait for your
re-review, and to fold the 28 tokens that duplicate the directory block if you would rather see the
line shortened.

@tietoumiao
tietoumiao force-pushed the fix/408-doc-reader-discovery branch from 8ffbe37 to 7ab74ac Compare November 2, 2026 03:01

@Nicholas023 Nicholas023 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — 技术实现已达标;token 成本的实际情况需后续验证

上一轮的 4 点意见已全部落实

  1. 触发条件 ✅ 改为「a .pdf / .docx / .csv / .tsv file whose reader is absent from the tool table」,与 docs/discovery.md 同句。CSV/TSV 半边不再可能被「表里已有 reader」误判。
  2. 读失败回退 ✅ 新增「a document read that fails falls back to it rather than retrying the reader」。
  3. issue 关联 ✅ Closes #408 → Related to #408。
  4. 行尾空格 ✅ git diff --check 现在 clean。

数字核对(独立复算)

新措辞实测 +279 字符 / +64 cl100k_base tokens(js-tiktoken 独立计算,与 PR body 第 60 行一致);body 第 83 行主动更正了旧数字(+202/+51),这点值得肯定。

policy 14/14 通过,中英文档成对,CHANGELOG 成对。

保留意见:token 成本需要真实场景验证

实现本身没有问题,但 +64 token / step 的收益目前只有提示词文本证据,没有模型在环数据:

  • 这 64 token 里约 28 与同一提示词下方的目录段(已列 pdf/docx/csv/tsv 与「按任务关键词查询」)语义重叠;
  • 真正新增的是 Office 授权边界与「读失败回退」这约 36 token;
  • 它到底能省掉多少次 discovery 往返,需要真实会话数据,而不是文本推理。

技术达标,可以合并。但成本收益请维护者在合并后的真实使用中验证,再决定是否长期保留这段路由提示。

@tietoumiao
tietoumiao force-pushed the fix/408-doc-reader-discovery branch from 7ab74ac to 420b1f4 Compare November 2, 2026 03:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants