Skip to content

docs(benchmark): 落地 DeepSeek 24-cell 诊断报告 - #272

Open
627150795 wants to merge 1 commit into
openpi-dev:mainfrom
627150795:docs/issue-22-benchmark-report-upstream
Open

docs(benchmark): 落地 DeepSeek 24-cell 诊断报告#272
627150795 wants to merge 1 commit into
openpi-dev:mainfrom
627150795:docs/issue-22-benchmark-report-upstream

Conversation

@627150795

Copy link
Copy Markdown
Contributor

Problem

Issue #22 已经记录了 24-cell DeepSeek 0731 对照与收敛消融,但结果、冻结条件、机制命中情况和统计限制尚未进入仓库内的正式报告,后续引用和复核不方便。

Value

把已有公开收据版本化,保留结果边界和失败/超时信息,避免把本轮小样本观察误读成官方成绩、稳定提速或总体排名。

Approach

  • 新增 docs/research/OPENPI_BARE_PI_DEEPSEEK_24_CELL_DIAGNOSTIC.md
  • 记录实验契约、最终对照表、逐题结果和实际命中的收敛机制。
  • 记录 ActiveEvidence、recovery/trajectory、delayed projection 三组消融及被拒绝的原因。
  • 保留源码、runner、runtime guard hash、timeout 和 artifact 可审计边界。
  • 本 PR 只整理 Issue [Benchmark] OpenPI 10/12 vs Bare Pi 6/12:DeepSeek 0731 下执行动态持平,tokens -37.7% #22 已公开的材料,不重新跑实验、不修改运行时代码。

Validation

Impact

Fixes #22

@tt-a1i

tt-a1i commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

This Benchmark report remains open under #277. Its merge gate is source-evidence review against Issue #22: frozen revisions, task/model/verifier identities, accounting, failure classification, limitations, and evidence references must all read back correctly. Existing CI is repository validation, not independent validation of the Benchmark conclusion.

@tt-a1i tt-a1i left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking evidence finding on exact head a6112a0d58e83e6439cbc38c66de31425af2aed5:

The prose matches Issue #22, and the 24-cell local evidence directory still exists, but openpiContentSha256 alone is not a retrievable source revision and the PR intentionally omits the artifacts needed to inspect or recompute the 10/12 vs 6/12 tables. Under #278's Benchmark contract, keep this report unmerged until it is moved to docs/benchmarks/, given the governed metadata/status, linked to a retrievable credential-free evidence receipt/archive identity, and tied to a source commit or immutable source package. Do not bulk-publish raw Sessions, traces, credentials, or candidate workspaces.

Also replace the historical tt-a1i/openpi Issue links with the canonical openpi-dev/openpi identity (or explicitly label them as historical redirects).

No runtime finding; this is a publication/evidence-state gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Benchmark] OpenPI 10/12 vs Bare Pi 6/12:DeepSeek 0731 下执行动态持平,tokens -37.7%

2 participants