Reduce research rebuilds and isolate DuckDB download dependencies

[proerror77]

Change

CEX production builds included an unused standalone offline diagnostic. Package eight executables through five recipes, down from nine through six. Keep the scientific worker, operator, backtest, four materializers and preparation program. The diagnostic remains available through its explicit local build.

Bundled DuckDB also compiled an unused HTTP downloader. Vendor the checksum-verified libduckdb-sys 1.10504.0 package and gate reqwest/zip behind download. Research-core and data-pipelines use the patch; Prediction retains its distinct locked version. System linking and explicit downloading remain supported. A download request without its feature fails clearly. Native C++ source, compiler backends and binding code retain upstream behavior.

Bind every native patch file, resolved dependencies, compiler, ABI, flags and profiles into cache compatibility. Preserve this exact dependency during cleanup and remove application artifacts. Prepare clean native source timestamps before restoring fresh CI targets. Apply the same contract to Loop nextest and regular Rust caches. Only successful cleaned main jobs save trusted dependencies.

Separate dependency compilation identity from packaging text. Reuse identical Cargo metadata commands without combining package-qualified feature selections. Grant the final remaining owner the admitted CPU budget for its next recipe. Preserve running jobservers, owner serialization, memory limits and cancellation cleanup.

Audit and provenance: DuckDB cache audit, native patch notes.

Issue relationship

None

Validation

Current head: 6dbe06253bef3378a3e71b942959ed0c75fe32f6, including main 4310dc7eb65f0994f8fe647e9536e5a407fe6121 without conflicts.

Focused local checks passed:

• Bundled, system-linked and explicitly downloaded DuckDB each executed SELECT 42 on macOS arm64.
• Bundled dependency graph and the actual native cache closure exclude reqwest, rustls and zip. The native closure contains 33 packages.
• Emulating checkout mtimes and preparing unchanged source kept Cargo's native build script and library fresh=true. This is local freshness evidence, not cross-run cache evidence.
• Research cache identity/cleanup, regular Rust owner cache, Loop nextest, CI selector, research product/scheduler, and Docker workspace contracts passed.
• Actionlint, Bash syntax and git diff --check passed.

Current CI:

• Prediction Markets CI.
• Monorepo CI.
• Security.
• Docker smoke.

All selected current-head checks passed, including ABI verification, strict Clippy and offline image smoke. The PR remains Open/Draft. Main is unchanged at 4310dc7e; the tested Prediction PR merge source is 71811449ae798c5bdd4327add882e62ecdc83378.

Current run evidence	Duration / result
Complete Prediction CI	40m09s
Complete Monorepo CI	32m04s
Security / Docker smoke	2m03s / 6m19s; passed
Research image binaries job	38m24s
Release build step / recipe wall phase	37m06s / 37m02s
Operator recipe	28m58s; two Cargo jobs; DuckDB build script/library fresh=false
Scientific worker recipe	5m12s; four Cargo jobs; native build script/library fresh=true
Offline smoke job / step	48s / 33s; passed
Rust Workspace job	30m04s
Rust focused build / research Clippy	8m36s / 7m53s
Loop archive / longest shard	7m35s / 9m33s; all shards passed

Durations overlap or nest; do not add them. The image runner was AMD EPYC 7763 with four logical CPUs. The prior head used Intel Xeon 6973P-C. These are not controlled performance comparisons.

Research cache exact and compatible keys both missed. Compatibility identity: b6bc2281bf90e4897703b0254b0e877c7d826f4475c2029f3843d1d375de2930; exact suffix: ca8e73ad3d7ab3fdd3cdc34a728b5bf6abf614c0c95786ccb9d3136e17626fcb. Dev/test and Loop v3 caches also missed. The worker's fresh native artifacts prove same-run reuse, not a cross-run hit. Main-only trusted saves were skipped on the PR ref.

The 10–15 minute target is not met. Cold operator compilation and broad cold dev/test build/Clippy remain the measured bottlenecks. A successful authorized main producer and a later compatible restore are still needed for cross-run warm-cache measurement. No merge, publication or deployment was performed.

The first attempt at 2d803fdd failed in run 38895793629: the policy build script correctly rejected two stale owning-manifest input hashes. Commit 6dbe0625 refreshes those hashes and the canonical digest. Re-projecting the locked Linux normal dependency tree reproduces all 174 package records and features. All source/checksum identities match the lockfile. Regeneration from the 19 original input definitions reproduces the complete artifact byte-for-byte. The original policy validator and both original identity unit tests pass, including stale/forbidden counterexamples. The validator and gate logic are unchanged.

A controlled local test changed only the normal-target Rustls package from 0.23.45 to 0.23.46. The actual patched DuckDB build script and library remained fresh=true; the active native static archive SHA-256 stayed identical, and both SELECT 42 consumers passed. The emitted artifact package set confirms exactly one package-version substitution. This verifies the intended incremental invalidation boundary, not a full-CI speedup.

The two image runs have identical compiler/native/flag hashes, builder digest, profile, target, recipes and job allocations. Both image caches missed. Of the 11m44s full-run increase, 11m30s occurs inside the build step: backtest +54s, operator +9m18s, worker +1m18s. All eight recipes take longer; CPU models differ, and main also updates normal Rustls. Hardware contribution is plausible but not quantified.

The Monorepo baseline restored v2 dependencies and had a full Loop cache hit. The current v3/Loop migration is cold. Workspace manifest/lock changes also select seven additional check/build stages totaling 18m43s that were absent in the narrow baseline. This is not a like-for-like benchmark. Existing migration coverage and the unchanged policy gate remain enforced.

All recoverable failures found here are fixed. No unchanged-input PR rerun is warranted. The next necessary validation is a successful authorized main cache producer followed by a separate compatible restore and representative small-change CI. That main-seed path remains unexecuted because this task forbids PR merge; no cache permission, cloud resource or deployment change was made.

The previous head f82a1ad2 passed Prediction CI in 28m25s, with a 25m36s build step and missed caches. That result did not meet the 10–15 minute target. The new identities require a trusted producer before measuring cross-run reuse. Local bundled compilation took 7m18s on macOS; this is neither complete Linux CI nor a controlled speed comparison. The current successful full run remains above the target; no acceleration percentage is established.

Runtime impact and rollback

Build and dependency changes only. ABI verification, executable hashing and offline image smoke remain required. SQLx CLI scripts, composite action and cache policy are unchanged. This draft is not merged or deployed. Reverting the native-isolation commit restores its previous dependency/cache behavior. Any later merge may trigger existing artifact publication; this PR grants no deployment authority.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: d4166929-4005-459c-8f73-5c214995c6d7

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[proerror77]

本评论记录 head 6dbe06253bef3378a3e71b942959ed0c75fe32f6 的逐条安全审查证据。下文状态是审查完成时快照；用户随后已批准附上证据、处理这些讨论并在门禁通过后 squash 合并。不会弱化规则或改变权限。

#1430 的 10 条 Semgrep unsafe 讨论审查

审查对象：PR #1430 head 6dbe06253bef3378a3e71b942959ed0c75fe32f6，base 4310dc7eb65f0994f8fe647e9536e5a407fe6121。2026-10-16 UTC。本次只审查这 10 个告警及相关调用边界，不是 DuckDB 全库安全认证。

风险结论与变更决定

这 10 条告警未证明当前补丁引入了可利用的安全缺陷。无需为了消除告警修改代码。三个是 cfg(test) 测试，六个是需要明确调用者契约的 FFI 字符串封装，一个是单线程构建进程的环境变量写入。

它们确实是 unsafe 使用提示，不能说扫描器没有命中 unsafe。GitHub 上规则是 rust.lang.security.unsafe-usage.unsafe-usage，severity 为 note，信息要求人工审查；它不提供污点流或具体破坏安全不变量的输入。逐条结论是“有效的信息性提示；未证实漏洞”，而不是未经审查地忽略规则。

本 PR 把原来的 registry 依赖源码纳入仓库，因此这些代码是本次新增的扫描位置，但 unsafe 操作并非本次新写。base 的 research-core 和 data-pipelines lock 都已固定 libduckdb-sys 1.10504.0 与相同 registry 校验和。src/lib.rs、src/string.rs 和 C++ archive 与认证的上游 crate 逐字节一致；build.rs 的 env::set_var 语句与周边执行次序未改变。上游来源一致只是证据的一部分，下面同时核查安全前置条件。

没有修改生产代码、缓存/安全策略或扫描规则，没有提交、推送或新远端 CI。保留 Draft，10 条讨论均未标记解决，告警未关闭。测试文件仅位于已有忽略的本地验证目录，副本在 evidence。

逐条审查

告警 / 讨论	位置	来源、执行范围与安全证据	结论及前置条件
2036	build.rs:225	上游已有的 env::set_var(PKG_CONFIG_PATH, …)。只在非 bundled 的系统链接路径且显式 DUCKDB_LIB_DIR 存在时执行。build main 直接进入 find_duckdb；此前只有环境读取/路径计算/打印，没有启动线程、HTTP client 或 bindgen。pkg-config probe 和 bindgen 都在写入之后；download 分支在显式库路径分支之后。	正常 Cargo 单线程 build-script 调用满足 set_var 契约。当前 Monday bundled 路径不编译执行此模块。安全前提是没有外部注入的线程/构造器在同进程并发访问环境，且库路径为构建配置。不能搬到多线程运行服务中使用；此次没有这样改变。
2037	src/lib.rs:47–60；讨论锚点 60	上游 print_int_result，位于 cfg(test)。唯一调用来自 basic_api_usage，且只在 duckdb_query / execute_prepared 成功之后、destroy_result 之前调用。行列索引均来自当前 result 的 count 范围，列名是活跃结果的借用 C 字符串。	无生产入口或攻击者提供裸指针。前提：有效、未销毁的 result、匹配的 C ABI。静态调用链满足；基础测试补跑。
2038	src/lib.rs:65–138；锚点 138	上游 test_query_arrow，cfg(test)，固定内存数据库/SQL。open/connect/query/schema/array 均检查成功。FFI_ArrowArray / Schema 初始化为空占位，转换函数接收结构指针的地址。C++ arrow-c.cpp:175–186、228–244 明确解引用外层指针并填写 Arrow CData。from_ffi 接管 array，Rust Arrow 对象管理释放，最后销毁 DuckDB 查询/连接/数据库。	未发现指针层级或生命周期缺陷。前提：Arrow CData ABI 与 native 库兼容、仅在成功填充后转换、释放回调有效。固定列数/行数和类型断言约束该测试。补跑原测试，非生产代码。
2039	src/lib.rs:143–210；锚点 210	上游 basic_api_usage，cfg(test)。CString 覆盖每次 C 调用生命周期；open/connect/prepare/bind/query/execute 都检查返回值；result 使用前初始化，三次结果依次 destroy；statement 在所有执行后 destroy，最后 disconnect/close。错误消息只在 query 错误分支立即读取。	未发现 use-after-free 或重复释放。前提：有效库 ABI 和成功状态对应有效句柄，固定合法 SQL。测试失败最多 panic/测试资源泄漏，不形成生产输入通道。
2040	src/string.rs:25	上游 CStr::from_ptr。from_ptr 本身是 unsafe API，先拒绝 null；调用者必须提供同一有效分配内 NUL 终止的 DuckDB 字符串、长度在 isize::MAX 内、保持不可变并转移唯一所有权。实际上游 duckdb Rust 调用来自 parameter_name、open_ext error、get_varchar、struct/union child name、alias 等分配字符串接口。	内容可来自数据，但该内容不直接构造裸地址/长度。检查 C++ 分配实现及 C API 契约后，未发现本补丁破坏前提。任意悬空/借用指针仍禁止；null 检查不被当作完整指针校验。
2041	src/string.rs:26	上游 from_ptr → from_raw_parts。len 直接取已验证 CStr 的字节长度；没有外部提供的长度。该调用转移同一分配所有权，没有复制所有者。	继承上一行前提；不会自行制造错误长度。未发现缺陷。
2042	src/string.rs:43	上游 from_nullable_ptr 是 unsafe API。null 提前返回 None；非 null 委托 from_ptr。实际 nullable C API 调用者保留句柄并立即接管独立分配。	null 分支不读/释放地址；非空仍需完整 from_ptr 契约，不能仅凭非空断言安全。已核对调用链，未发现本补丁新增不合法调用。
2043	src/string.rs:65	上游 slice::from_raw_parts，唯一安全入口是私有方法。ptr/len 字段私有；构造器均 unsafe。from_ptr 计算准确长度；直接 from_raw_parts 的文档要求有效终止串、准确长度、DuckDB 分配、独占所有权及不变性。无安全字段写入或安全裸指针构造。	前提是整段 len+1 在同一活分配内、可读且不超过 isize::MAX；保证 len+1 不溢出。违反前提需调用者先误用 unsafe；没有观察到这样的当前调用。
2044	src/string.rs:74	上游 CStr::from_bytes_with_nul_unchecked 使用上一行切片。初始长度止于第一个 NUL，或 unsafe raw-parts 调用者保证准确 C 字符串；所有权禁止外部改写。返回借用受 &self 生命周期约束。	不要求 UTF-8；不存在 unchecked UTF-8 转换。空串、非 UTF-8 和包含更早 NUL 的实际分配均有本地正例测试。raw_parts 调用者必须传首个 NUL 的实际长度。
2045	src/string.rs:81	上游 Drop 调用 duckdb_free。类型没有 Clone/Copy 或安全所有者复制入口。C++ helper-c.cpp:498–504 为 malloc/free；get_varchar 为独立 malloc+NUL，parameter_name、child name、alias 为 strdup，C API 明确要求 duckdb_free。	前提：只接管可由同一 DuckDB ABI/allocator 释放的分配，不能传 borrowed column_name、CString::as_ptr 或重复接管同一指针。实际已检查调用均是声明独立分配的 API。本地测试在原 value/type/statement 销毁后读取字符串并 drop，均通过。

Rust 标准库安全契约：set_var、CStr::from_ptr、slice::from_raw_parts。不将测试通过当作对任意裸指针的证明。

来源和调用链证据

• crates.io archive SHA-256：94e67523a40ee3da30411e00e243990b3760d91877202ab2f39c03ab34f8dbfc。重新读取本地 registry archive 并核验，非只相信说明文件。
• lib.rs SHA-256：b4c8b54124c958a244c38a469a2d48f4c13182e2d5ee4fff05638139fd680164，与认证上游相同。
• string.rs SHA-256：dc38e02b8c7bc2c34a384ff944c3e491de486952f5d2e0a224429bb19ba6527d，与认证上游相同。
• duckdb.tar.gz SHA-256：53398a1a9ac6b8c0bd9314a07db61d4b0473cc2ad3a567e323f820cc18fe4ba8，与认证上游相同。
• build.rs 差异仅 downloader 的 optional imports/cfg/error 与未使用参数命名；225 行操作未改。上游文件副本 upstream-libduckdb-build.rs。
• rust_hft 的非 native 生产源码没有直接 DuckDbString/raw sys API 调用。已检查 registry duckdb 1.10504.0 的 raw_statement.rs:353–372、inner_connection.rs:89–99、profiling.rs:101–104、vtab/value.rs:101–109、core/vector.rs:433–441、core/logical_type.rs:346–360/404–410 的字符串所有权边界。
• 直接 from_raw_parts 的完整正确性始终属于其 unsafe 调用者；不能据此宣布所有下游、所有 DuckDB API、系统库版本或原生内存错误均安全。外部系统库必须与绑定 ABI/allocator 兼容。没有进行恶意数据模糊测试、Miri 或 sanitizer 全库扫描。

验证与状态

• cargo test --manifest-path .native-validation/Cargo.toml --locked --offline --features bundled --test unsafe_contracts：4 passed，0 failed，执行 0.03 秒。实际链接原生 DuckDB，覆盖 null/空串/中文/非 UTF-8/首个 NUL、三个构造器、1500 次分配和 Drop、独立 value/alias/parameter 字符串所有权、查询/prepare/result 生命周期。测试不故意构造 UB。
• 原上游 lib.rs 的 cfg(test) 模块提取为 integration test，除 harness 导入外正文逐字一致；使用原要求的 Arrow 58/ffi 和当前 native crate。2 passed，0 failed，执行 0.04 秒；包含 basic_api_usage 和 test_query_arrow。首次编译 2 分 29 秒，属于本地验证，非 CI 性能对比。合并复验日志 unsafe-final-tests.log。
• git diff --check 通过；没有受版本控制文件变更。
• 旧 head 的既有 CI 证据仍适用；此次没有生产代码变化，不重复触发远端 CI，也不把本地结果表述为新 CI 通过。
• 所有 10 条讨论保留未解决，PR 仍为 Draft。用户/审查者可逐条检查以上证据，再决定讨论处置；本次没有自动标记解决或变更合并门禁。

原始证据：unsafe-threads-locations.json、unsafe-alerts-readonly.json、unsafe-reviewed-alerts.json、unsafe-upstream-provenance.json；测试：unsafe_contracts.rs、upstream_unsafe.rs、unsafe-contracts-tests.log、unsafe-upstream-tests.log、unsafe-final-tests.log；原生源码节选：unsafe-capi-source/。

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-16T22:46:27.561517Z	6dbe062	Draft marked ready
🔒 Security Review	✅ Completed 2026-10-16T22:44:23.248764Z	6dbe062	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6dbe06253b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

同步删除旧的发布镜像说明

这里说明生产 cex-runner 不再包含该诊断程序，但同一文档第 69 行仍称发布目录和 CEX Dockerfile 会构建独立的 monday-offline-baseline 配方和二进制。完整阅读文档的操作者会得到相互矛盾的发布契约；请同步更新或删除第 69 行的旧说明。

AGENTS.md reference: AGENTS.md:L207-L209

Useful? React with 👍 / 👎.