Skip to content

feat(services): add 3FS USRBIO support - #8434

Draft
MrCroxx wants to merge 6 commits into
apache:mainfrom
MrCroxx:codex/hf3fs-usrbio
Draft

MrCroxx wants to merge 6 commits into
apache:mainfrom
MrCroxx:codex/hf3fs-usrbio

Conversation

@MrCroxx

@MrCroxx MrCroxx commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

No linked issue. This draft proposes native 3FS data I/O for review.

Rationale for this change

Using Fs on a 3FS mount sends file data through the regular FUSE read/write path. Add an opt-in service that uses 3FS's USRBIO shared-memory interface for data I/O while keeping filesystem metadata operations on the mount.

What changes are included in this PR?

  • Add opendal-service-hf3fs, the services-hf3fs feature, and service registration.
  • Load the public USRBIO C API at runtime. Use bounded read/write drivers, shared-memory read buffers, and one copy into shared memory for writes.
  • Add real-service tests, usage documentation, and a read benchmark against Fs.
  • Add a single-node, single-replica 3FS fixture to the existing behavior CI. It uses SoftRoCE and ordinary directories on ubuntu-22.04, with a cached, pinned 3FS build.

Validation: the existing behavior workflow passed on a GitHub-hosted Ubuntu runner using the implementation and fixture from this draft: 5 focused USRBIO tests and 116 behavior tests passed against real 3FS. The validation ran from a separate probe branch. Focused Clippy, formatting, and 12 behavior-planner tests also passed.

Are there any user-facing changes?

Yes. Linux users can enable services-hf3fs and construct Hf3fs with an existing 3FS directory and a compatible libhf3fs_api_shared.so. Building OpenDAL does not require the 3FS SDK. Running the service requires a real 3FS deployment and its FUSE client.

Reads avoid a copy while shared-memory slots are available; retained buffers trigger a copy fallback to prevent pool exhaustion. Writes are not atomic. USRBIO has no cancellation fence, so fatal driver failures can retain in-flight resources until process exit. These constraints are documented.

The CI fixture selects Ubuntu 22.04 because the tested Ubuntu 24.04 Azure kernel lacks RXE support. The initial 3FS build is expensive; subsequent jobs restore it from cache.

Breaking changes

None.

AI Usage Statement

  • Harness: Codex desktop.
  • Model: GPT-6; a more specific model identifier was not exposed.
  • Effort: Not exposed by the harness to the agent.
  • Role: Assisted with the service implementation, native API investigation, real-service validation, CI fixture, documentation, and this draft. The contributor directed the use case and validation requirements and requested this draft for diff review. Compatibility with other 3FS client-library versions and future hosted-runner kernels remains unverified.

@github-actions github-actions Bot added releases-note/feat The PR implements a new feature or has a title that begins with "feat" services/hdfs size:XXL This PR changes 1000+ lines, ignoring generated files. labels Oct 26, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat" services/hdfs size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant