feat(storage): implement multi-write v2 metadata partitioning by baojun-zhang · Pull Request #5741 · volcengine/OpenViking

baojun-zhang · 2026-10-08T07:13:47Z

Description

Implement Multi-Write V2 metadata partitioning for OpenViking storage. This replaces the legacy directory-sidecar and global-sequence model with account-scoped partition routing, append-only segments, checkpoints, background catch-up, and metadata GC.

This PR also removes legacy sync-status / sync-retry surfaces and updates API, CLI, configuration, and documentation to match the V2 async backup model.

Human Involvement

A human participated in the implementation or review loop
This PR was generated entirely by AI agents without human participation in the loop

Related Issue

Architecture Discussion: #5863
Detailed Design: #5865

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)
Performance improvement
Test update

Changes Made

Implemented Multi-Write V2 partitioned metadata, including routing, segments, checkpoints, catch-up, migration, provider abstraction, and metadata GC.
Added filesystem and cache-backed multi-write providers, including Redis-based head / manifest state handling for cache mode.
Removed legacy sync-status / sync-retry APIs, CLI commands, retry routing code, and obsolete multi-backend configuration semantics.
Updated English and Chinese docs, API references, configuration examples, and validation tests for the V2 async backup model.
Added and updated Rust and Python tests covering multi-write runtime, model, codec, catch-up, provider behavior, config validation, server APIs, and auth flows.

Testing

I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I have tested this on the following platforms:
- Linux
- macOS
- Windows

Verified by GitHub Actions:

API & CLI Integration Tests: passed
Docs build: passed
Rust CLI builds for Linux, macOS, and Windows: passed
Plugin checks and plugin version checks: passed
Source and binary package build jobs: passed
cuVS lite tests: passed

Checklist

My code follows the project's coding style
I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
Any dependent changes have been merged and published

Screenshots (if applicable)

N/A

Additional Notes

This is a breaking storage-layer change for legacy multi-write management surfaces. Normal file read/write APIs remain available, but the old sync-status / sync-retry APIs and related CLI commands are removed. V2 uses async backup catch-up and primary-only metadata durability by design.

…tion policy

…length from 4096 to 8192

…heckpoint chunks and sealed segment

baojun-zhang · 2026-10-10T08:16:56Z

本测试使用 NAS 服务，通过配置 fuse 将 NAS 服务 mount 到 localfs
因此文件读写会存在网络 IO，导致文件 IOPS 性能低于本地磁盘

基线服务 write 压测

测试口径

测试时间：2026-10-10 15:53:05 CST
服务：基线服务 openviking v0.5.0
本地地址：http://127.0.0.1:19331
接口：POST /api/v1/content/write
URI 前缀：viking://resources/multiwrite-iops/<prefix>/file-xxxxxx.md
TOTAL：10000
CONCURRENCY：15
PAYLOAD_BYTES：1024
wait：false
processing_mode：vectors_only
mode：create

本轮测试衡量 OpenViking 写请求前台返回速度。请求不等待 semantic /
embedding worker 完成模型处理，但包含文件写入、刷新任务构造和 QueueFS
入队耗时。

测试期间服务日志出现 Volcengine embedding AccountRateLimitExceeded 429 重试。
该现象会影响本轮 content/write 请求返回延迟，后续对比新版服务时需要用同一
口径记录。

三轮结果

轮次	success	errors	seconds	qps	avg_ms	p50_ms	p95_ms	p99_ms	max_ms
1	10000	`{}`	1170.523	8.54	1754.06	1662.30	2596.00	2750.06	2994.17
2	10000	`{}`	1196.662	8.36	1793.58	1732.15	2577.52	2705.22	2885.17
3	10000	`{}`	1205.739	8.29	1806.83	1744.65	2622.63	2765.06	3370.63

平均值

rounds	total_each_round	concurrency	payload_bytes	success_each_round	errors_each_round	seconds	qps	avg_ms	p50_ms	p95_ms	p99_ms	max_ms
3	10000	15	1024	`[10000, 10000, 10000]`	`[{}, {}, {}]`	1190.97	8.40	1784.82	1713.03	2598.72	2740.11	3083.32

原始结果

{
  "average": {
    "seconds": 1190.97,
    "qps": 8.4,
    "avg_ms": 1784.82,
    "p50_ms": 1713.03,
    "p95_ms": 2598.72,
    "p99_ms": 2740.11,
    "max_ms": 3083.32,
    "rounds": 3,
    "total_each_round": 10000,
    "concurrency": 15,
    "payload_bytes": 1024,
    "success_each_round": [
      10000,
      10000,
      10000
    ],
    "errors_each_round": [
      {},
      {},
      {}
    ]
  },
  "rounds": [
    {
      "round": 1,
      "service": "baseline",
      "base_url": "http://127.0.0.1:19331",
      "prefix": "base-round1-1791615199",
      "total": 10000,
      "concurrency": 15,
      "payload_bytes": 1024,
      "success": 10000,
      "errors": {},
      "seconds": 1170.523,
      "qps": 8.54,
      "avg_ms": 1754.06,
      "p50_ms": 1662.3,
      "p95_ms": 2596.0,
      "p99_ms": 2750.06,
      "max_ms": 2994.17
    },
    {
      "round": 2,
      "service": "baseline",
      "base_url": "http://127.0.0.1:19331",
      "prefix": "base-round2-1791616369",
      "total": 10000,
      "concurrency": 15,
      "payload_bytes": 1024,
      "success": 10000,
      "errors": {},
      "seconds": 1196.662,
      "qps": 8.36,
      "avg_ms": 1793.58,
      "p50_ms": 1732.15,
      "p95_ms": 2577.52,
      "p99_ms": 2705.22,
      "max_ms": 2885.17
    },
    {
      "round": 3,
      "service": "baseline",
      "base_url": "http://127.0.0.1:19331",
      "prefix": "base-round3-1791617566",
      "total": 10000,
      "concurrency": 15,
      "payload_bytes": 1024,
      "success": 10000,
      "errors": {},
      "seconds": 1205.739,
      "qps": 8.29,
      "avg_ms": 1806.83,
      "p50_ms": 1744.65,
      "p95_ms": 2622.63,
      "p99_ms": 2765.06,
      "max_ms": 3370.63
    }
  ]
}

新版服务 write 压测

测试口径

测试时间：2026-10-10 16:10:20 CST
服务：新版服务 OpenViking 当前分支
接口：POST /api/v1/content/write
URI 前缀：viking://resources/multiwrite-iops/<prefix>/file-xxxxxx.md
TOTAL：10000
CONCURRENCY：15
PAYLOAD_BYTES：1024
wait：false
processing_mode：vectors_only
mode：create

本轮测试参数与基线服务一致。

三轮结果

轮次	success	errors	seconds	qps	avg_ms	p50_ms	p95_ms	p99_ms	max_ms
1	10000	`{}`	299.973	33.34	449.56	441.07	526.48	618.07	1277.68
2	10000	`{}`	305.607	32.72	458.01	445.91	558.19	692.40	1077.31
3	10000	`{}`	302.298	33.08	453.02	444.79	536.53	624.73	1068.83

平均值

rounds	total_each_round	concurrency	payload_bytes	success_each_round	errors_each_round	seconds	qps	avg_ms	p50_ms	p95_ms	p99_ms	max_ms
3	10000	15	1024	`[10000, 10000, 10000]`	`[{}, {}, {}]`	302.63	33.05	453.53	443.92	540.40	645.07	1141.27

原始结果

{
  "average": {
    "seconds": 302.63,
    "qps": 33.05,
    "avg_ms": 453.53,
    "p50_ms": 443.92,
    "p95_ms": 540.4,
    "p99_ms": 645.07,
    "max_ms": 1141.27,
    "rounds": 3,
    "total_each_round": 10000,
    "concurrency": 15,
    "payload_bytes": 1024,
    "success_each_round": [
      10000,
      10000,
      10000
    ],
    "errors_each_round": [
      {},
      {},
      {}
    ]
  },
  "rounds": [
    {
      "round": 1,
      "service": "v2",
      "base_url": "http://127.0.0.1:19332",
      "prefix": "v2-round1-1791618898",
      "total": 10000,
      "concurrency": 15,
      "payload_bytes": 1024,
      "success": 10000,
      "errors": {},
      "seconds": 299.973,
      "qps": 33.34,
      "avg_ms": 449.56,
      "p50_ms": 441.07,
      "p95_ms": 526.48,
      "p99_ms": 618.07,
      "max_ms": 1277.68
    },
    {
      "round": 2,
      "service": "v2",
      "base_url": "http://127.0.0.1:19332",
      "prefix": "v2-round2-1791619198",
      "total": 10000,
      "concurrency": 15,
      "payload_bytes": 1024,
      "success": 10000,
      "errors": {},
      "seconds": 305.607,
      "qps": 32.72,
      "avg_ms": 458.01,
      "p50_ms": 445.91,
      "p95_ms": 558.19,
      "p99_ms": 692.4,
      "max_ms": 1077.31
    },
    {
      "round": 3,
      "service": "v2",
      "base_url": "http://127.0.0.1:19332",
      "prefix": "v2-round3-1791619503",
      "total": 10000,
      "concurrency": 15,
      "payload_bytes": 1024,
      "success": 10000,
      "errors": {},
      "seconds": 302.298,
      "qps": 33.08,
      "avg_ms": 453.02,
      "p50_ms": 444.79,
      "p95_ms": 536.53,
      "p99_ms": 624.73,
      "max_ms": 1068.83
    }
  ]
}

结论

在相同测试条件下，新版服务的写请求前台返回速度明显优于基线服务。

测试条件：

接口：POST /api/v1/content/write
TOTAL：10000
CONCURRENCY：15
PAYLOAD_BYTES：1024
wait：false
processing_mode：vectors_only
每组 3 轮，全部请求成功，无错误。

平均值对比：

服务	qps	avg_ms	p95_ms	p99_ms
基线	8.40	1784.82	2598.72	2740.11
新版	33.05	453.53	540.40	645.07

提升幅度：

新版 QPS 约为基线的 3.93x。
新版 avg_ms 较基线降低约 74.59%。
新版 p95_ms 较基线降低约 79.21%。
新版 p99_ms 较基线降低约 76.46%。

本轮结果说明：在当前 content/write 测试口径下，新版服务每秒可完成的前台
写请求更多，且平均延迟和尾延迟都低于基线服务。

zhoujh01

V2 的分区化与 primary-only 异步写路径方向合理，但当前发现两处会使已成功的 primary 变更永久无法收敛到 backup 的可靠性缺口，因此请求修改。

qin-ptr

分区化、前台写入与备份追平解耦的方向合理，现有 NAS 压测也支持前台延迟改善。但当前正常路径仍有四处需要在合并前解决的问题：单文件改名重复复制、离线备份阻止已吸收日志回收、filesystem provider 全量重写日志头，以及无变化的系统目录被反复全量复制。

六条行级评论包含四条必须修复和两条非阻塞设计建议。此前已解释并关闭的 flush 失败处理、Redis 持久性讨论不重复提出。

qin-ctx

账户删除后复用同一 ID 的正常路径会在备份中残留旧账户文件，需要在合并前处理。

qin-ctx

此前的修复和相关取舍说明已核对。本次仅针对账号删除修复 f74f07e 新引入的三处问题请求修改：延迟删除事件会清掉重建账号已同步的新文件、系统分区跳过检查点后日志无法回收，以及测试文件多余括号导致库单测编译失败。具体证据见三条行内评论。

1

feat(storage): implement multi-write v2 metadata partitioning

a35db4e

github-project-automation Bot added this to OpenViking project Oct 8, 2026

baojun-zhang marked this pull request as draft October 8, 2026 07:13

baojun-zhang added 15 commits October 8, 2026 19:08

feat(storage): multiwrite meta data file should follow backend encryp…

fd4b694

…tion policy

feat(storage): remove redundant Redis implementation. adjust segment …

cd8a7f9

…length from 4096 to 8192

feat(storage): remove unused code

bf2c195

feat(storage): uniform const manager && using stream read/write for c…

a9ffe51

…heckpoint chunks and sealed segment

feat(storage): uniform const manager && using stream read/write for c…

3384f2e

…heckpoint chunks and sealed segment

feat(storage): remove meaningless const

28393b3

feat(storage): remove meaningless const

1943587

feat(storage): remove dead code

75438df

fix(storage): fix checkpoint empty directory issue

05dc38f

fix(storage): fix checkpoint remain unused directory issue

bf43254

feat(storage): remove unused const

41bbd84

feat(storage): route V1 cleanup through wrapped filesystem

578d937

feat(storage): refine multiwrite redis key

d6808e3

feat(plugins): bump memory plugin versions

37704b7

chore: merge main into multiwrite-v2

04a9f47

baojun-zhang marked this pull request as ready for review October 10, 2026 08:17

zhoujh01 previously requested changes Oct 10, 2026

View reviewed changes

Comment thread crates/ragfs/src/multibackend/runtime.rs

Comment thread crates/ragfs/src/multibackend/provider/cache.rs

ZaynJarvis added the kernel Core server, runtime, storage, retrieval, SDK, and CLI. label Oct 12, 2026

chore: merge main into multiwrite-v2

08b9dc2

qin-ptr previously requested changes Oct 12, 2026

View reviewed changes

baojun-zhang added 2 commits October 12, 2026 17:25

fix(plugins): 普通文件rename(a.md -> b.md) 不再生成MoveTree

6197c8f

fix(plugins): 将根/_system 建成一个显式 system account/scope，避免定时全量复制成本

246e93b

qin-ctx requested changes Oct 12, 2026

View reviewed changes

Comment thread crates/ragfs/src/multibackend/runtime.rs Outdated

baojun-zhang added 3 commits October 13, 2026 12:21

fix(storage): preserve recreated account writes after delete

f74f07e

chore: merge upstream main into multiwrite-v2

7022f83

chore(plugins): bump memory plugin versions

a10b2c7

qin-ctx requested changes Oct 13, 2026

View reviewed changes

Comment thread crates/ragfs/src/multibackend/catch_up.rs

Comment thread crates/ragfs/src/multibackend/checkpoint.rs Outdated

Comment thread crates/ragfs/src/multibackend/tests/test_runtime.rs Outdated

baojun-zhang added 3 commits October 13, 2026 15:57

fix(storage): remove stray multibackend test brace

d4122b6

fix(storage): make _system dir also execute checkpoint worker

a20757a

fix(storage): replay recreated account data after scope delete

f967a69

qin-ctx approved these changes Oct 13, 2026

View reviewed changes

qin-ctx requested a review from zhoujh01 October 13, 2026 09:25

qin-ctx merged commit ff79020 into volcengine:main Oct 13, 2026
10 of 11 checks passed

github-project-automation Bot moved this from Backlog to Done in OpenViking project Oct 13, 2026

baojun-zhang deleted the multiwrite-v2 branch October 13, 2026 09:26

Navigation Menu

Repository navigation

feat(storage): implement multi-write v2 metadata partitioning - #5741

feat(storage): implement multi-write v2 metadata partitioning#5741
qin-ctx merged 25 commits into
volcengine:mainfrom
baojun-zhang:multiwrite-v2

baojun-zhang commented Oct 8, 2026 •

edited

Loading

Uh oh!

baojun-zhang commented Oct 10, 2026 •

edited

Loading

Uh oh!

zhoujh01 left a comment

Uh oh!

Uh oh!

Uh oh!

qin-ptr left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qin-ctx left a comment

Uh oh!

Uh oh!

qin-ctx left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

baojun-zhang commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Human Involvement

Related Issue

Type of Change

Changes Made

Testing

Checklist

Screenshots (if applicable)

Additional Notes

Uh oh!

baojun-zhang commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

基线服务 write 压测

测试口径

三轮结果

平均值

原始结果

新版服务 write 压测

测试口径

三轮结果

平均值

原始结果

结论

Uh oh!

zhoujh01 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

qin-ptr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qin-ctx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

qin-ctx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

baojun-zhang commented Oct 8, 2026 •

edited

Loading

baojun-zhang commented Oct 10, 2026 •

edited

Loading