Repository navigation
[PR-2] M4 EvidenceLedger full traceability checker #27
Description
本任务属于 Epic #23。共享上下文(整体目标、EvidenceLedger 完整 schema、Agent Execution Contract、Non-goals、回归基线)见 #23。
依赖:blocked by #24(需 PR-1 的 EvidenceLedger 最小 builder 先落地)。
代码定位按当前仓库版本核对;若行号漂移,以函数名 / 字段名 / 挂载点语义为准。
完成本 PR 需遵守 Epic #23 的 Agent Execution Contract 与 Test Execution Protocol:先写失败测试与反例夹具 → 最小实现 → 跑里程碑测试 → 跑回归基线。PR 描述需列出:changed files / tests added / tests run / before-after / non-goals。
目标
把展示层取数统一收拢到 EvidenceLedger,并实现 Abstract/Conclusion 数字可回溯校验、禁用词检查、schema validation、百分比归一。
Allowed changes
- EvidenceLedger builder / schema / validation 相关文件
- 展示层读取 evidence 的路径
- PR-2 fixtures
- PR-2 tests
Forbidden changes
- 不从已渲染文本 parse 数字来生成展示结论
- 不把
confidence当作 confidence interval - 不新增不存在的
confidence_interval{low,high,level}字段 - 不把
per_seed改造成新形状 - 不让 Abstract / Conclusion 使用 ledger 外数字
Minimal API Contract
在 PR-1 的 build_evidence_ledger 基础上补齐:
def validate_evidence_ledger(ledger: dict) -> list[dict]:
"""Return schema violations. Missing required fields must be reported explicitly."""
def assert_traceable(main_tex: str, ledger: dict) -> list[dict]:
"""Return evidence traceability violations for Abstract / Conclusion only."""Violation 结构至少含:
{
"rule": "unsourced_number | positive_claim_with_negative_verdict | schema_error",
"location": {"section": "abstract | conclusion | discussion", "line": 12},
"snippet": "...",
"value": "0.01"
}assert_traceable 是唯一允许从已渲染 tex 抽取数字的路径;只做校验,不参与生成结论。
契约规则
- 展示层取数只能读 ledger 字段;禁止从渲染文本 parse 数字。
- 进 Abstract/Conclusion 的每个数字必须能映射到某个 ledger 字段,映射不到则不得进这两节。
verdict ∈ {refuted, inconclusive}时,Abstract/Conclusion 不得出现肯定式断言。禁用词为闭合清单(配置文件交付,验收对的就是这份),默认:improves, improve, outperforms, outperform, significant, significantly, superior, beats, surpasses。
数字抽取与匹配规则(定死)
- 边界: Abstract =
\begin{abstract}...\end{abstract};结论段 =\section{Conclusion}/Conclusions/Discussion(确定性 fallback 模板用 Discussion,见paper_orchestra_pipeline.py:518)到下一\section之间。\input{content/conclusion}模块化布局不在本批范围。 - 数字 token 正则:
-?\d+\.?\d*%?,排除Table\s*\d+/Figure\s*\d+/Section\s*\d+编号与N seeds种子计数。 - 单位归一: ledger 存分数(0.705),Abstract 常写百分数(70.5%);带
%的 token 匹配前除以 100,两侧四舍五入到 ledger 字段精度。匹配p_value/effect_size/per_method[*][metric]/seed_variance[*].mean/repro_ci/kept_ci任一 → 可回溯,否则 violation。 - 夹具须显式标注数字格式(分数/百分数)。
验收用例
| # | 输入 | 期望输出 |
|---|---|---|
| M4-1 | ledger 缺 p_value,Abstract 写 "p=0.01" |
violation:数字无 ledger 来源 |
| M4-2 | ledger verdict=refuted,Abstract 写 "significantly improves" |
violation:命中禁用词 |
| M4-3 | ledger 完整,Abstract 数字均可映射 | 无 violation |
| M4-4 | 同一数字仅出现在 Method 节 | 允许(规则只约束 Abstract/Conclusion) |
| M4-5 | ledger 缺必填字段(无 verdict) |
schema 校验失败,明确报缺哪个字段 |
| M4-6 | Abstract 含 "Table 2" / "5 seeds" | 不报 violation(白名单内) |
反例夹具: fixtures/m4_unsourced_number.tex、fixtures/m4_refuted_positive_claim.tex、fixtures/m4_clean.tex + 对应 ledger json。
DoD
M4-1~6 全绿;validate_evidence_ledger / assert_traceable 落地;数字抽取正则随 PR;展示层取数迁移到 ledger;回归基线全绿。
Activity
Protocol-zero-0 commentedon Jun 4, 2026
Completed in PR #29 on branch
epic-23-evidence-ledger.Changed files:
agents/paper_completeness.pytests/test_paper_completeness_m4.pytests/fixtures/m4_*Tests added:
Commands actually run:
pytest tests/test_pipeline_contracts.py->12 passedpytest tests/test_vnext_manuscript.py->5 passedpytest tests/test_paper_completeness_m4.py->6 passedpytest tests/test_pipeline_contracts.py->12 passedpytest tests/test_vnext_manuscript.py->5 passedpytest tests/test_paper_completeness_m1.py tests/test_paper_completeness_m4.py->13 passedNon-goals respected:
assert_traceableonly validates rendered tex; it does not generate display conclusions from parsed text.confidence_interval{low,high,level}.\input{content/conclusion}traceability.Complete Epic #23 presentation evidence gates (#29)
Protocol-zero-0 commentedon Jun 6, 2026
Done via #29 (squash-merged into master 2026-06-06). 验收用例与回归基线均在 PR #29 全绿。