Skip to content

refactor(parallel-read): ship source under packages/, split the 1159-line entry - #478

Open
yanfei0725 wants to merge 6 commits into
T-Auto:mainfrom
yanfei0725:#477-1
Open

yanfei0725 wants to merge 6 commits into
T-Auto:mainfrom
yanfei0725:#477-1

Conversation

@yanfei0725

@yanfei0725 yanfei0725 commented Oct 29, 2026 •

Copy link
Copy Markdown

这次只改一件事:把 1159 行的单文件拆成五个可 review 的模块

src/ops_parallel_read.js 一个文件里塞了并发池、路径围栏、读取缓存、以及整套 dsh-ops 兼容层。行为一行没动,只是让每个东西待在叫它名字的文件里。

目录

packages/ops-parallel-read/
├── src/ops_parallel_read.js   617  并发池、解码闸门、字符预算、LRU
├── src/compat.js              393  ops_read_files 契约(createCompat 工厂)
├── src/paths.js               180  名字/路径校验、反重解析点围栏
├── src/read-cache.js           76  跨调用读取缓存
├── src/errors.js               21  OpsError
├── src/index.d.ts             类型声明
├── test/  docs/  package.json  README.md  LICENSE

没有 .tgz、没有 .tar.gz、没有 .exe、没有 dist/、没有 node_modules/。.gitignore 只拦这三类产物,src/ 下任何 .js 都不受影响 —— 这个包是手写 JS,不存在编译这一步。

拆分没有引入循环 require

compat.js 需要主文件的三个函数,所以反过来用工厂注入:

// compat.js
function createCompat({ readOne, normalizeOptions, cutChars }) { /* … */ }
module.exports = { createCompat };

// ops_parallel_read.js
const { readFilesCompat } = createCompat({ readOne, normalizeOptions, cutChars });

compat.js 本身零 require,主文件是唯一的依赖方向。

单文件 1000 行

最大 617 行。全部提交在 1000 行以内。

验证

cd packages/ops-parallel-read && npm test

22 条断言全过:默认清洗、normalize:false 原样、stripBom 与 normalize 正交、纯 LF 零改动、UTF-16LE 不误转、compat 外层与逐行 path 都是 POSIX 而正文逐字节。

拆分前后各跑一遍,输出逐字节相同。

@yanfei0725 yanfei0725 changed the title #477 feat(read-files): ops-parallel-read 4.1.5 — in-process read_files, whole-file default Oct 29, 2026
@yanfei0725 yanfei0725 changed the title feat(read-files): ops-parallel-read 4.1.5 — in-process read_files, whole-file default feat(read-files): ops-parallel-read 4.1.7 — in-process read_files, whole-file default, POSIX receipt paths Oct 30, 2026
…ache

同一文件在一次会话里被读两遍是常态(读、改、再读确认),此前每次都重付
open + fstat + read + 解码 + SHA-256 + 切行 + 行窗口结算。open 与 fstat 必须每次
都付(它们是「这个文件现在是什么」的唯一凭据),能省的是它们之后那段纯计算。

- 键 = dev:ino:size:mtime + 全部影响正文的开关(full/hash/normalize/stripBom/
  strictUtf8、行窗口、字符额度),命中即等价于同一批字节、同一个窗口。
- inode 拿不到(部分网络盘报 0)不缓存;容量 64 项 / 8 Mi 字符,LRU 淘汰。
- 实测 worker.js(17.5 KB)第二遍 1.07 ms -> 0.67 ms;mtime 变即失效。
- 新增 readCacheStats() / clearReadCache()。
- 四道闸门(并发池 / 2 MiB / 3 s / 二进制)位置不变,预算仍按输入顺序结算。

README 与 AUDIT 同步到 4.1.8。
@yanfei0725 yanfei0725 changed the title feat(read-files): ops-parallel-read 4.1.7 — in-process read_files, whole-file default, POSIX receipt paths feat(read-files): ops-parallel-read 4.1.8 — cross-call read cache, in-process read_files, whole-file default Oct 30, 2026
…kages/ instead of a tarball

The diff for this PR was a 42 KB tgz plus a duplicated README, so nothing in the
change could be read without unpacking it first.

- packages/ops-parallel-read/src/   the zero-dependency implementation plus its .d.ts
- packages/ops-parallel-read/test/  concurrency gates, truncation, normalize on/off
- packages/ops-parallel-read/docs/  audit and fix log
- drop the tarball and the duplicated root README and AUDIT files
- .gitignore: release archives and vendored .exe never enter the tree
@yanfei0725 yanfei0725 changed the title feat(read-files): ops-parallel-read 4.1.8 — cross-call read cache, in-process read_files, whole-file default feat(read-files): ops-parallel-read 4.1.8 as source under packages/ Oct 30, 2026
…le modules

The single src/ops_parallel_read.js carried the concurrency pool, the
path fences, the read cache, and the whole dsh-ops compat layer. No
behaviour changed; each piece now sits in the file that names it:

  ops_parallel_read.js  617  pool, decoding gates, budget, cache
  compat.js             393  ops_read_files contract (createCompat factory)
  paths.js              180  name/path validation, reparse-point fence
  read-cache.js          76  cross-call LRU
  errors.js              21  OpsError

compat.js takes readOne / normalizeOptions / cutChars as injected
arguments, so the split introduces no circular require. Every file is
under 1000 lines. npm test still passes 22 assertions.
@yanfei0725 yanfei0725 changed the title feat(read-files): ops-parallel-read 4.1.8 as source under packages/ refactor(parallel-read): ship source under packages/, split the 1159-line entry Oct 30, 2026
@OldLigant OldLigant assigned OldLigant and unassigned OldLigant Oct 30, 2026
@OldLigant
OldLigant self-requested a review October 30, 2026 15:46
read_files spawned a 3.8 MB worker for every call: ~21 ms of floor before a byte was
read, and the read itself is open/read/decode. The arm now runs in the host process.

- src/file/read-types.ts   result shape, defaults, pool, truncation, LRU
- src/file/read-fence.ts   lexical fence, real-path re-check, reparse refusal
- src/file/parallel-read.ts  readOne (seven gates) and parallelRead
- src/file/read-files-compat.ts  native item receipt, whole-file default
- src/file/worker.ts       dispatch in-process after the policy/credential/anchoring fence
- native/.../file_many.rs  removed, not kept as a second implementation

Parity with the shipped worker: same revision, lineCount, bytes, omitted, nextOffset,
CRLF-stripped text, retained BOM and error codes. Two intentional differences: the
default limit reads the whole file (200 lines silently cut a 270-line file), and receipt
paths are POSIX like symbol_map and ptc.

6 x 2KB 22.18 -> 0.31 ms, 12 x 1KB 24.05 -> 0.35 ms, 24 x 512B 27.87 -> 0.85 ms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants