fix(test-e2e): 修复 Daily 执行隔离、阻断范围及报告通知

.agents/skills/nexent-test-assets/references/runtime-migration.md

@@ -6,8 +6,8 @@ The repository owns test behavior and prerequisites. Put explicit asset families
 
 Machine-local `config/pipeline.yaml` owns only timeouts, notification opt-in and explicit build/deploy/preflight argv hooks. It must not become another source of Case definitions or preparation logic. Secrets, fixed resource IDs and local paths stay out of Case scripts. New settings require a schema and safe example. Existing helpers still load users, models, assets and secrets from the selected test home.
 
-The candidate runner must account for every selected Case, distinguish blocked/retired/policy-skipped outcomes from PASS, finalize local reports after interruption, and preserve per-case evidence. A failing independent Case does not prevent unrelated Cases from running. Pending recovery stops business Cases until environment safety is verified. The next scheduled Daily may restore a validated Local configuration backup, synchronize develop and redeploy under the Daily/checkout locks; it must preserve the old evidence and pass service/mode and Local recovery checks before ordinary Cases. Active owners/locks, unsafe or damaged journals/configuration and pending D5 fault recovery remain hard stops. Targeted runs/resume do not bypass recovery. See [daily-validation.md](../../../../test-e2e/infra/tools/docs/daily-validation.md#retrying-a-failed-deployment).
+The candidate runner must account for every selected Case, distinguish blocked/retired/policy-skipped outcomes from PASS, finalize local reports after interruption, and preserve per-case evidence. A failing independent Case does not prevent unrelated Cases from running. Common configuration and runtime safety may stop a batch; unavailable local toolchains and known-Case execution declarations block only affected Cases. Doctor checks common readiness by default; explicit doctor --full includes dependency diagnostics, not an all-or-nothing business prerequisite. Pending recovery stops business Cases until environment safety is verified. The next scheduled Daily may restore a validated Local configuration backup, synchronize develop and redeploy under the Daily/checkout locks; it must preserve the old evidence and pass service/mode checks before ordinary Cases. Business recovery Cases and historical Case-result files do not gate that redeploy. Active owners/locks, unsafe configuration backups and pending D5 fault recovery remain hard stops. Targeted runs/resume do not bypass recovery. See [daily-validation.md](../../../../test-e2e/infra/tools/docs/daily-validation.md#retrying-a-failed-deployment).
 
-Daily never generates candidates, edits Excel or YAML, refreshes manifests, or promotes test assets. Update tests on the product branch. Generated registry/Excel remain read-only views during execution. Normal notifications require completed D6 and an accounted execution plan. Uncertain delivery is not automatically retried.
+Daily never generates candidates, edits Excel or YAML, refreshes manifests, or promotes test assets. Update tests on the product branch. Generated registry/Excel remain read-only views during execution. Enabled terminal notifications cover success, failure and catchable interruption, including preparation failure; complete shared-latest publication is a separate decision. Uncertain delivery is not automatically retried.
 
 Before claiming runtime migration complete, verify: locked Python/Node dependency closure, controlled HTTP/MCP services, per-case preparation declarations and behavior, isolated deployment, timeout cleanup/recovery, same-version resume, D1-D5 live runs, failure evidence and D6/report/notification parity. Consult `test-e2e/infra/migration/runtime-migration-status.md` for current gaps rather than assuming they are implemented.

test-e2e/DEVELOPER_GUIDE.zh-CN.md

@@ -87,9 +87,11 @@ Linux 的 Chromium 系统库需要另外由主机环境提供。依赖安装和
 
 使用 GitHub Daily 时，若本机 `daily.env` 显式设置了 `NEXENT_REPO`，它必须与 `.github/workflows/nexent-daily-full-test.yml` 中的 `REPO_ROOT` 指向同一个产品 Git 仓库。迁移仓库路径时不能只改其中一个，还需核对启动入口和 SDK 引用，避免从一个仓库启动、却加载另一个仓库的代码。工作流修改只有提交并在实际运行的分支生效后才算完成入口切换；此前不要恢复可能混用新旧入口的 Runner。普通本地命令仍应从实际产品仓库根目录执行，不能仅靠修改该环境变量替代切换工作目录。
 
-普通部署失败留下恢复标记后，下一次 GitHub Daily 可以在持有 Daily 和仓库部署锁、确认旧进程已退出及旧批次终态、核验原始备份和 Local 配置后，保存旧证据并恢复配置，再同步最新 develop、构建部署。标记不会因允许同步而直接清除：服务预检、实际部署模式和 Local 恢复用例必须通过，才能开始普通用例；最终恢复验证完成后才解除本次标记。损坏备份、非预期配置改动、活跃锁、D5 故障恢复标记或未确认的 D5 超时/清理失败仍会阻断。普通定向执行和 resume 不具备这个自动恢复权限。详见[Daily 恢复流程](infra/tools/docs/daily-validation.md#retrying-a-failed-deployment)。
+普通部署失败留下恢复标记后，下一次 GitHub Daily 可以在持有 Daily 和仓库部署锁、确认旧进程已退出及旧批次终态、核验原始备份和 Local 配置后，保存旧证据并恢复配置，再同步最新 develop、构建部署。标记不会因允许同步而直接清除：服务预检和实际部署模式检查通过后开始普通用例；结束后同样按环境检查结果解除恢复标记。知识库等业务用例只在正常分组执行一次，失败不再阻断其他独立用例。损坏备份、非预期配置改动、活跃锁、D5 故障恢复标记或活跃故障操作锁仍会阻断。普通定向执行和 resume 不具备这个自动恢复权限。详见[Daily 恢复流程](infra/tools/docs/daily-validation.md#retrying-a-failed-deployment)。
 
-启用通知的 Daily 在成功、失败或可捕获中断时均发送终态报告；同步、资产校验、doctor 阻断也生成启动异常通知。报告注明未完成原因及环境恢复状态，配置还原不等于恢复用例通过。启动与执行批次通过显式关联共用发送回执，`SENDING`/`UNKNOWN` 不盲目重发。未完成批次的通知不会覆盖正式共享 latest，也不会改变用例结果或解除恢复标记。强杀、断电或发送通道不可用时不能保证进程内通知到达，需另行监控兜底。详见[终态通知](infra/tools/docs/daily-validation.md#terminal-reports-and-notifications)。
+启用通知的 Daily 在成功、失败或可捕获中断时均发送终态报告；同步、资产校验、doctor 阻断也生成启动异常通知。报告注明未完成原因及环境恢复状态，配置还原不等于环境检查通过。启动与执行批次通过显式关联共用发送回执，`SENDING`/`UNKNOWN` 不盲目重发。未完成批次的通知不会覆盖正式共享 latest，也不会改变用例结果或解除恢复标记。强杀、断电或发送通道不可用时不能保证进程内通知到达，需另行监控兜底。详见[终态通知](infra/tools/docs/daily-validation.md#terminal-reports-and-notifications)。
+
+正常通知直接发送本批次的正式 `report.md`。报告生成时统一写入终态、未完成原因及环境恢复状态，并脱敏；已有报告在发送前补齐最新终态并核对脱敏。仅在正式报告缺失、为空、不可读取或无法完成发送准备时，才生成 `notification-report.md` 异常通知，并明确详细报告不可用。正常报告不再复制一份通知报告；原始结果、日志、截图及 Trace 仍独立保留。
 
 专用 Ubuntu 测试机迁移后的目录分工：产品 Git 仓库是 `/home/jason/projects/nexent`，test-home 仍为 `/home/jason/nexent-test-suite`。旧 `/root/project/nexent` 保留用于回退，不再作为新入口；`/home/jason/nexent` 是容器工作目录，不是新的产品 Git 仓库，不能覆盖或删除。配置、历史结果、浏览器缓存及持久化数据均保留在原 test-home。专用机的本地命令应先执行 `cd /home/jason/projects/nexent`；其他开发者使用自己的仓库位置，不把上述机器路径写进通用 Case。
 
@@ -107,7 +109,11 @@ Windows Docker Desktop 的容器访问宿主机通常使用 `host.docker.interna
 uv pip check --python $TestPython
 ```
 
-遇到缺失资产或校验冲突时，按命令结果处理；不要拿空文件代替。`doctor` 按所选用例检查实际 jsdom/CSS 依赖导入及对应工具链的无头浏览器启动，`--live` 另外检查服务可达性；这些检查不代表所有账号、模型、业务流程已经通过。离线用例可省略 `--live`，并用 `--case <Case-ID>` 缩小检查范围。
+遇到缺失资产或校验冲突时，按命令结果处理；不要拿空文件代替。`doctor` 默认只检查公共环境，`--live` 另外检查服务可达性；需要完整诊断时加 `--full`，检查所选用例的实际 jsdom/CSS 依赖导入及对应工具链的无头浏览器启动。这些检查不代表所有账号、模型、业务流程已经通过。离线用例可省略 `--live`，并用 `--case <Case-ID>` 缩小检查范围。
+
+完整诊断的全部检查不作为整批执行的前置阻断。运行前只检查公共配置、专用 Python 环境和必要的部署条件；浏览器或 jsdom 工具链不可用时，仅将依赖该工具链的用例标记为 BLOCKED，其他独立用例继续。账号、模型及自定义脚本依赖由实际使用它们的用例检查。已知用例的执行声明错误也只阻断该用例，严格开发校验仍要求修正。并发锁、真实 D5 故障恢复标记、危险路径和执行中源码或配置变更仍会停止整批。旧批次业务结果文件缺失不再阻断已核验备份的重新部署；旧证据不会被清除。
+
+同步 develop 与执行测试使用独立 Python 进程：启动器持续持有 Daily 和仓库部署锁，新执行进程校验同步后的提交、工作区及继承锁，再重新核验恢复上下文。恢复工具不再提前导入部署执行器；异常退出仍保留恢复标记和原始报告。详见 [Daily 进程隔离](infra/tools/docs/daily-validation.md#fresh-execution-after-synchronization)。
 
 ## 3. 收到失败结果后如何修改
 

test-e2e/infra/config/deployment-exceptions.schema.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "type": "object", "additionalProperties": false,
-  "required": ["schema_version", "profiles", "cases", "recovery_cases"],
+  "required": ["schema_version", "profiles", "cases"],
   "properties": {
     "schema_version": {"const": 1},
     "profiles": {"type": "object", "minProperties": 1,
@@ -21,7 +21,7 @@
       }
     },
     "cases": {"type": "object", "additionalProperties": {"type": "string"}},
-    "recovery_cases": {"type": "array", "minItems": 1, "uniqueItems": true,
+    "recovery_cases": {"type": "array", "deprecated": true, "uniqueItems": true,
       "items": {"type": "string", "minLength": 1}}
   }
 }

test-e2e/infra/config/deployment-exceptions.yaml

@@ -19,5 +19,3 @@ cases:
   API-AUTO-FECE331C989B771F: aidp
   AGT-AUTO-7A2120E630C381C7: aidp
   API-AUTO-7513FEF9904B9F3F: aidp
-recovery_cases:
-  - PW-KB-LOCAL-01

test-e2e/infra/scripts/daily-launch.py

@@ -8,15 +8,15 @@
 import os
 from pathlib import Path
 import signal
-import subprocess
 import sys
 import uuid
 
 TOOLS = Path(__file__).resolve().parents[1] / "tools"
 sys.path.insert(0, str(TOOLS))
-from suite_runtime import BatchLock, deployment_lock, logged, now, save
+from suite_runtime import BatchLock, deployment_lock, now, save
 from daily_git import run_git
 from daily_recovery import prepare_daily_retry
+from daily_process import notify_terminal, run_worker
 
 
 def git(repo, *args, log_dir=None):
@@ -44,7 +44,8 @@ def checked(*args):
 def main(argv=None):
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("--test-home", type=Path, required=True)
-    parser.add_argument("--branch", required=True, choices=("djb/test-baseline", "develop"))
+    parser.add_argument("--branch", required=True,
+                        help="Explicit remote branch, validated and fast-forwarded under the Daily lock")
     parser.add_argument("--execute", action="store_true")
     parser.add_argument("--publish-latest", action="store_true",
                         help="Publish complete Daily logs to test-home/runs-ubuntu; default disabled")
@@ -67,39 +68,23 @@ def main(argv=None):
     save(directory / "status.json", state)
     old_handler = signal.signal(signal.SIGTERM, lambda *_: (_ for _ in ()).throw(KeyboardInterrupt()))
 
-    def record_batch(batch):
-        state["batch"] = str(batch)
-        save(directory / "status.json", state)
-
     try:
-        with BatchLock(home / "state/daily.lock"):
+        with BatchLock(home / "state/daily.lock") as daily_lock:
             try:
-                with deployment_lock(repo), prepare_daily_retry(repo, home, directory):
+                with deployment_lock(repo) as checkout_lock, prepare_daily_retry(repo, home, directory):
                     state["head"] = sync_checkout(repo, args.branch, log_dir=directory / "git")
-                    env = os.environ.copy()
-                    # Daily reports local asset problems without rewriting tracked views.
-                    command = [str(runtime), str(repo / "test-e2e/infra/tools/validate_test_assets.py"),
-                               "--phase", "implementation", "--daily-report", str(directory / "asset-validation.json")]
-                    if logged(command, repo, env, directory / "validate.log", 180):
-                        raise RuntimeError("Formal asset validation failed; inspect local launcher log")
-                    if git(repo, "status", "--porcelain", "--untracked-files=normal", log_dir=directory / "git"):
-                        raise RuntimeError("Derived generation changed tracked assets; repair them on the product branch")
-                    # Import only after checkout synchronization; use this exact revision's code.
-                    import static_assets
-                    import suite
-                    static_assets.apply(home, repo / "test-e2e/infra/assets")
-                    records = suite.selection(repo, argparse.Namespace(case=None, stage=None, feature=None, change=None), daily=True)
-                    problems = suite.doctor(repo, home, records)
-                    if problems:
-                        save(directory / "doctor.json", {"ready": False, "problems": problems})
-                        raise RuntimeError("Runtime preflight failed; inspect local doctor.json")
-                    settings = suite.config(repo, home)
-                    state["status"] = "EXECUTING"
                     save(directory / "status.json", state)
-                    code = suite.execute(repo, home, records, settings, daily=True, publish_latest=args.publish_latest,
-                                         on_batch_created=record_batch)
-                    state["status"] = "PASSED" if code == 0 else "FAILED"
-                    return code
+                    code = run_worker(repo, home, directory, state["head"], args.publish_latest,
+                                      [daily_lock, checkout_lock])
+                    result = json.loads((directory / "status.json").read_text(encoding="utf-8"))
+                    state.update(result, worker_exit_code=code)
+                    if (state.get("status") not in {"PASSED", "FAILED", "INCOMPLETE", "INTERRUPTED"}
+                            or (code == 0 and state.get("status") != "PASSED")
+                            or (code != 0 and state.get("status") == "PASSED")):
+                        state.update(status="INCOMPLETE",
+                                     error=f"Execution process exited with code {code} without a matching terminal receipt")
+                        return 2
+                    return code if code >= 0 else 2
             except (Exception, KeyboardInterrupt) as exc:
                 state.update(status="INTERRUPTED" if isinstance(exc, KeyboardInterrupt) else "INCOMPLETE",
                              error="Interrupted" if isinstance(exc, KeyboardInterrupt) else
@@ -115,11 +100,17 @@ def record_batch(batch):
         return 3 if isinstance(exc, RuntimeError) else 2
     finally:
         signal.signal(signal.SIGTERM, old_handler)
+        # Preserve any batch link already written by an interrupted executor.
+        try:
+            persisted = json.loads((directory / "status.json").read_text(encoding="utf-8"))
+            if persisted.get("batch"):
+                state["batch"] = persisted["batch"]
+        except (OSError, ValueError, AttributeError):
+            pass
         state["finished_at"] = now()
         save(directory / "status.json", state)
         print(f"Daily launch: {directory}; status={state['status']}")
-        from suite_notify import notify_launch
-        notify_launch(directory, home)
+        notify_terminal(repo, home, directory)
 
 
 if __name__ == "__main__":