feat(load): add bounded Arrow IPC ingestion path

.golangci.yml

@@ -35,6 +35,13 @@ linters:
             - $all
             - '!$test'
             - '!pkg/iceberg/adapter/iceberggo/**/*.go'
+            # Arrow IPC is isolated to the ingestion bridge and its external
+            # scan reader. Keep the dependency out of all other production
+            # packages, including the Iceberg-facing layers.
+            - '!**/pkg/container/arrowbridge/*.go'
+            - '!**/pkg/sql/colexec/external/arrowio/*.go'
+            - '!**/pkg/sql/colexec/external/reader_arrow.go'
+            - '!**/pkg/sql/compile/compile.go'
           deny:
             - pkg: github.com/apache/iceberg-go
               desc: Iceberg dependency must stay behind pkg/iceberg/adapter/iceberggo

cmd/mo-service/config.go

@@ -177,7 +177,8 @@ func NewConfig() *Config {
 		CN: cnservice.Config{
 			AutomaticUpgrade: true,
 			Frontend: config.FrontendParameters{
-				MongoDB: *config.NewMongoDBParameters(),
+				MongoDB:   *config.NewMongoDBParameters(),
+				ArrowLoad: *config.NewArrowLoadParameters(),
 			},
 		},
 	}

cmd/mo-service/config_test.go

@@ -386,6 +386,46 @@ func TestMongoDBEnablementConfigDefaults(t *testing.T) {
 	}
 }
 
+func TestArrowLoadConfigDefaults(t *testing.T) {
+	for _, test := range []struct {
+		name               string
+		input              string
+		enabled            bool
+		s3Enabled          bool
+		distributedEnabled bool
+	}{
+		{name: "omitted"},
+		{
+			name: "S3 opt in", input: "[cn.frontend.arrow-load]\nenabled = true\ns3-enabled = true\n",
+			enabled: true, s3Enabled: true,
+		},
+		{
+			name: "all opt in", input: `[cn.frontend.arrow-load]
+	enabled = true
+s3-enabled = true
+distributed-enabled = true
+`,
+			enabled: true, s3Enabled: true, distributedEnabled: true,
+		},
+	} {
+		t.Run(test.name, func(t *testing.T) {
+			cfg := NewConfig()
+			require.NoError(t, parseFromString(test.input, cfg))
+			require.NoError(t, cfg.setDefaultValue())
+			require.Equal(t, test.enabled, cfg.CN.Frontend.ArrowLoad.Enabled)
+			require.Equal(t, test.s3Enabled, cfg.CN.Frontend.ArrowLoad.S3Enabled)
+			require.Equal(t, test.distributedEnabled, cfg.CN.Frontend.ArrowLoad.DistributedEnabled)
+
+			// CN construction validates the frontend configuration again. Keep
+			// every explicit rollback value stable across that second pass.
+			cfg.CN.SetDefaultValue()
+			require.Equal(t, test.enabled, cfg.CN.Frontend.ArrowLoad.Enabled)
+			require.Equal(t, test.s3Enabled, cfg.CN.Frontend.ArrowLoad.S3Enabled)
+			require.Equal(t, test.distributedEnabled, cfg.CN.Frontend.ArrowLoad.DistributedEnabled)
+		})
+	}
+}
+
 func TestClockOffsetBoundRetainedWhenMonitoringDisabled(t *testing.T) {
 	validators := []struct {
 		name string

docs/design/23684_arrow_load_design.md

@@ -0,0 +1,120 @@
+# #23684 Arrow LOAD design
+
+Status: implementation proposal; independent design approval is pending. This
+is the stable, versioned design artifact for implementation PR #28145, not
+evidence of approval. Until approval is recorded, all Arrow LOAD modes are
+fail-closed by default. The release-readiness matrix is maintained in
+[`evidence/23684_arrow_load_release_readiness.md`](evidence/23684_arrow_load_release_readiness.md).
+The separately versioned
+[`implementation delivery decision`](evidence/23684_arrow_load_delivery_decision.md)
+defines the mergeable shared substrate and deliberately separates it from
+deployment enablement.
+
+## Problem and scope
+
+`LOAD DATA` previously accepted only text-oriented formats.  Arrow IPC File and
+Stream inputs need a bounded, transactional ingestion path that preserves SQL
+mapping and error semantics.  This design adds Arrow only to `LOAD DATA`; it
+does not make Arrow a general external-table, result-scan, or Flight protocol.
+
+The invariant is: every accepted Arrow input is decoded against the planned
+schema, charged to the executing statement, and either becomes one ordinary
+transactional LOAD result or leaves no visible partial result.  A remote source
+or distributed worker is admitted only when **that executing CN** has enabled
+the applicable rollout gate before it opens I/O.
+
+Non-goals are aggregate cluster quota/range planning, automatic cloud-provider
+enablement, and a new long-lived cache or controller.  Those are deliberately
+separate release gates, not implied by local Arrow support.
+
+## Contract and alternatives
+
+The input contract follows Apache Arrow IPC (File and Stream container framing)
+as implemented by Arrow-Go v18.  File footer/message metadata and decoded-size
+claims are checked before allocation.  Schema binding is compile-time and the
+execution reader rejects schema/type drift.  Arrow fields map by name by
+default, or by the explicit positional option; normal LOAD target rules still
+own casts, generated columns, constraints, and transaction commit.
+
+We considered (1) materializing every Arrow value, (2) accepting Arrow only
+through a conversion sidecar, and (3) bounded borrowing with materialization
+fallback.  Always materializing is simpler but makes large varlen loads pay an
+avoidable copy; a sidecar adds an operational data-plane boundary and retry
+contract.  The selected option borrows only reference-counted Arrow backing
+that passes a pin-amplification bound, otherwise materializes.  `force-
+materialize` is an explicit rollback/diagnostic switch, not a different SQL
+semantics mode.
+
+## Configuration, rollout, and compatibility
+
+`frontend.arrow-load.enabled`, `s3-enabled`, and `distributed-enabled` all
+default to false. Planning samples the
+settings for its scope, but every `External.Prepare` repeats the gate using the
+worker CN's `ParameterUnit` before constructing an Arrow reader:
+
+| Source/execution | Required worker settings | Default |
+| --- | --- | --- |
+| local File or Stream | `enabled` | rejected |
+| direct S3 or S3-backed stage | `enabled`, `s3-enabled` | rejected |
+| distributed Arrow scope | `enabled`, `distributed-enabled` | rejected |
+| distributed S3 scope | all three | rejected |
+
+This prevents a coordinator's stale or more-permissive configuration from
+bypassing a worker's fail-closed policy during a rolling change.  The execution
+scope remains a positive compile authorization. Arrow fanout additionally
+serializes `arrow_distributed_execution`; it is independent of the requested
+`Parallel` flag because already-planned shard scopes must clear that flag to
+avoid a second split. `External.Prepare` applies the worker's distributed gate
+to this execution fact before it opens I/O. MORPC v55 remains the receiver
+compatibility gate. v55 is `up/main` v54 plus one on the delivery
+rebase; older peers reject the additive Arrow pipeline fields, so mixed-version
+deployments must drain or keep remote Arrow modes disabled.  Downgrade is safe
+under the same gates because local Arrow does not advertise a remote capability.
+
+## Ownership and failure model
+
+The compiler owns scope construction and immutable object identity snapshots.
+`External.Prepare` owns worker admission; `ArrowReader` owns one open IPC
+reader; FileService owns provider response closure; and the statement allocation
+account owns capacity reservations.  A range is published only after bytes are
+read **and** the conditional provider `Close` succeeds.  Read, probe, close,
+parse, conversion, cancellation, and commit failure all abort the reservation;
+the released lease returns its exact charge once.
+
+For a record batch, the reader validates immutable shape and validity once at
+record admission, including dictionary values. It then budgets and converts
+windows.  Budgeting performs only O(columns) structural validation; each
+conversion checks its selected indices, validity window, and cancellation
+checkpoint without rescanning immutable dictionary values.  Thus a record split
+into K output batches has linear total validity work rather than K full-record
+scans.  The
+statement account bounds retained range and vector capacity; pin amplification
+forces materialization instead of retaining an oversized source allocation.
+
+Terminal ownership is explicit: EOF calls `ArrowReader.Close`, which releases
+IPC references and returns the underlying stream close result.  `External.Call`
+propagates that terminal error rather than reporting successful completion.
+`Reset`/`Free` remain best-effort cleanup only after the execution result is
+already determined.
+
+## Validation and acceptance
+
+Focused unit coverage proves worker-side local/S3/distributed gates, positive
+compile scope propagation, conditional read success plus close-only
+`ErrObjectChanged` with zero committed capacity, terminal external close-error
+propagation, malformed IPC/schema/null cases, and budget behavior.  Existing
+Arrow File/Stream, identity, MinIO, multi-CN, rollback, and SQL BVT cases cover
+the consumer and public paths.  The immediate predecessor compatibility test
+is retained with the MORPC v55 gate.
+
+Before remote production enablement, the readiness record requires a bounded
+cross-worker aggregate admission design, real-provider evidence, exact-release
+mixed-version validation, deployment A/B, and independent SQL/execution/
+resource/FileService/storage/security-release decisions.  Until then the
+defaults above are the conservative rollback and failure-containment plan.
+
+## Open decisions
+
+Independent owners must approve the proposed remote rollout only after the
+listed acceptance evidence exists.  No product choice is needed to ship this
+documented fail-closed default: every mode requires explicit deployment opt-in.

docs/design/evidence/23684_arrow_bridge_benchmark.md

@@ -0,0 +1,64 @@
+# #23684 Arrow bridge A/B benchmark
+
+This record compares identical Arrow records with ordinary borrow policy and
+`ForceMaterialize=true`. It contains both a bridge-level attribution benchmark
+and a local end-to-end SQL LOAD benchmark; neither is a cloud-provider or
+deployment acceptance benchmark.
+
+Command, run on 2026-09-03 on Darwin/arm64 Apple M4:
+
+```text
+.agents/skills/mo-dev/scripts/mo-cgo-test -run '^$' \
+  -bench '^BenchmarkArrowBridgeMaterializeAB$' -benchmem \
+  -benchtime=500ms -count=3 ./pkg/container/arrowbridge
+```
+
+## Results
+
+| Fixture | Mode | ns/op, three runs | Borrowed / copied payload | Interpretation |
+| --- | --- | --- | --- | --- |
+| numeric + decimal | borrow | 982 / 991 / 990 | 98,304 / 0 bytes | exact fixed layouts retain the Arrow buffers |
+| numeric + decimal | materialize | 19,951 / 20,053 / 20,171 | 0 / 98,304 bytes | forced mode performs the attributed payload copy |
+| timestamp + short string | borrow | 346,201 / 346,379 / 345,458 | 0 / 73,728 bytes copied | no payload is borrow-eligible; temporal conversion and canonical inline strings dominate |
+| timestamp + short string | materialize | 346,088 / 363,055 / 361,882 | 0 / 73,728 bytes copied | modes are intentionally equivalent for an ineligible schema |
+| long binary | borrow | 78,241 / 76,979 / 77,033 | 1,048,576 / 0 bytes | long varlen payload is retained, with descriptor work still performed |
+| long binary | materialize | 560,541 / 607,918 / 545,651 | 0 / 1,048,576 bytes | forced mode copies the full payload |
+
+The counters prove attribution: eligible fixed/long-varlen cases report the
+same logical payload in either mode, while ownership changes from borrowed to
+copied. The short-string/temporal case correctly reports no eligible bytes and
+therefore no artificial zero-copy win.
+
+## End-to-end LOAD A/B
+
+The same candidate binary was also exercised through the MySQL frontend,
+parser/planner, External reader, conversion, transaction commit, storage, and
+result acknowledgement. Each sample loaded 100,000 rows; table truncation was
+outside the timer. The benchmark asserts the expected borrowed/copy counters so
+the two modes cannot silently collapse to the same policy.
+
+```text
+.agents/skills/mo-dev/scripts/mo-cgo-test -p=1 -run '^$' \
+  -bench '^BenchmarkArrowLoadEndToEndMaterializeAB$' -benchmem \
+  -benchtime=3x -count=3 -timeout=2400s ./pkg/tests/arrowload
+```
+
+| Mode | ns/op, three runs | rows/s, three runs | B/op | allocs/op |
+| --- | --- | --- | --- | --- |
+| borrow | 99,915,126 / 110,736,000 / 77,651,930 | 1,000,849 / 903,049 / 1,287,798 | 18,472,338 / 21,284,285 / 16,848,141 | 149,529 / 220,473 / 139,554 |
+| materialize | 86,487,403 / 99,493,569 / 94,644,931 | 1,156,238 / 1,005,090 / 1,056,581 | 16,791,154 / 16,896,400 / 16,789,093 | 139,463 / 139,928 / 139,789 |
+
+On the final rebased candidate, the median forced-materialize run was 5.3%
+faster and delivered 5.6% more rows/s than the median borrow run. Both modes
+showed substantial run-to-run noise, including one borrow sample with elevated
+allocation counts, so this result proves the two policies execute and remain
+within the local reference gate; it does not prove a repeatable performance win
+for either policy. Materialize remains a correctness diagnostic and emergency
+fallback because it deliberately gives up eligible zero-copy ownership.
+
+## Remaining deployment evidence
+
+The local A/B does not replace provider-specific object-store measurements,
+ordinary Parquet/INSERT controls on customer data, cache/pin pressure testing,
+or production CPU/RSS profiles. Deployment owners must run those controls on
+the exact release artifact before enabling S3 or distributed execution.

docs/design/evidence/23684_arrow_go_supply_chain.md

@@ -0,0 +1,74 @@
+# #23684 Arrow-Go supply-chain review
+
+Review date: 2026-09-04. Candidate base: `up/main@bf63172c06`. The dependency is
+pinned to `github.com/apache/arrow-go/v18 v18.7.0`; no floating version is used.
+
+## Arrow runtime closure
+
+`go list -deps` for the imported `arrow/array` and `arrow/ipc` packages reports
+eight non-standard modules:
+
+| Module | Version | Detected license |
+| --- | --- | --- |
+| `github.com/apache/arrow-go/v18` | `v18.7.0` | Apache-2.0; upstream NOTICE present |
+| `github.com/goccy/go-json` | `v0.10.6` | MIT |
+| `github.com/google/flatbuffers` | `v25.12.19+incompatible` | Apache-2.0 |
+| `github.com/klauspost/compress` | `v1.19.0` | BSD-3-Clause |
+| `github.com/pierrec/lz4/v4` | `v4.1.27` | BSD-2-Clause |
+| `github.com/zeebo/xxh3` | `v1.1.0` | BSD-2-Clause |
+| `golang.org/x/exp` | `7ab1446f8b90` | BSD-3-Clause |
+| `golang.org/x/sys` | `v0.47.0` | BSD-3-Clause |
+
+No copyleft or source-availability license was found in this closure. Final
+NOTICE aggregation remains a release-packaging owner action.
+
+## SBOM and size
+
+CycloneDX `cyclonedx-gomod v1.12.0` generated a 1.6 binary SBOM from the local
+Darwin/arm64 `mo-service`: 219 components and 220 dependency entries. The Arrow
+component and all eight closure modules were present. Reproducible command:
+
+```text
+cyclonedx-gomod bin -json -std -version <release-version> \
+  -output mo-service.cdx.json ./mo-service
+```
+
+The final local SBOM SHA-256 is
+`ab787ad79e00c4f048de8390ebfaf7b91ab9ff5bfd13e3be2fd2ce02bd531282`.
+The candidate and identically built baseline binary SHA-256 values are
+`7db369ce401480e4dcca98d1463646cdebb86db3d3dfd438adf37568c291640c` and
+`308cb694d524206e68ff6957cd54cf74ab60e5f9d778968524397660b7e1f9d8`,
+respectively. These hashes identify local evidence artifacts, not signed
+release artifacts.
+
+The local candidate binary was 261,955,890 bytes (249.82 MiB), versus
+254,982,450 bytes (243.17 MiB) for an identically built `up/main` binary. The
+delta is 6,973,440 bytes, or 2.73%. Packaging owners must decide whether that
+increase is acceptable for the release image.
+
+## Vulnerability review
+
+`govulncheck -mode=binary` found 17 reachable advisories in the candidate and
+18 in the identical `up/main` baseline. The candidate introduced no new
+advisory ID and removed the baseline's `GO-2026-4762` result through its module
+graph update. The remaining results are repository/toolchain debt in the Go
+1.26.4 standard library, gRPC, `x/net`, `x/text`, AWS SDK v2, Avro, and pgx.
+
+A source scan of `arrowbridge`, `arrowipc`, and `external/arrowio` still reaches
+`GO-2026-5764` through the existing FileService AWS SDK (`service/s3 v1.68.0`,
+fixed in `v1.97.3`). Arrow-Go itself was not named by a reachable advisory.
+Because the full candidate scan is non-zero, security approval is **blocked**
+until the repository/toolchain findings are upgraded, waived by the security
+owner with scope and rationale, or proven unreachable in the release build.
+
+## Platforms and compatibility
+
+MatrixOne's native build matrix names Linux amd64 and Linux arm64; the local
+host is Darwin arm64. The imported Arrow `array` and `ipc` packages compile for
+all three targets with `CGO_ENABLED=0`. The complete Darwin/arm64 `mo-service`
+build and affected CGo test closure pass. Linux image builds remain an exact
+release-CI artifact requirement because MatrixOne's native dependencies cannot
+be proven by a pure-Go cross compile alone.
+
+This review is complete as evidence, but its result is not an approval: CVE,
+binary-size, Linux artifact, NOTICE, and owner sign-off gates remain explicit.

docs/design/evidence/23684_arrow_load_consumer_inventory.md

@@ -0,0 +1,34 @@
+# #23684 Arrow LOAD consumer inventory
+
+This inventory closes the LOAD-reachable borrowed-vector surface. It does not
+claim that arbitrary external-table SELECT or every stateful operator can
+consume borrowed backing.
+
+| Reachable boundary | Borrowed-input behavior | Ownership or copy boundary | Evidence |
+| --- | --- | --- | --- |
+| External Arrow reader | Publishes a generation-scoped batch; fixed and long-varlen payloads may be borrowed | each output vector retains its Arrow ArrayData/range backing; the reader may advance after publication | `pkg/sql/colexec/external/reader_arrow.go`, Arrow reader lifecycle tests |
+| PreInsert | Reads source columns synchronously; generated, auto, const, row-id, and transformed columns are copied with `UnionBatch` | destination vectors are MPool-owned; no source lease is detached | `pkg/sql/colexec/preinsert/preinsert.go`, vector `UnionBatch` borrowed-source tests |
+| Constraint, dedup, and lock operators | Read input during the call and retain no untracked source pointer | mutation routes through vector materialization; terminal batch owner keeps the generation alive | vector borrowed/COW tests and affected operator package tests |
+| Conditional local dispatch / pSpool | Async fan-out is reachable when one source feeds multiple writers | `RetainedReadonlyViewWithMP` copies descriptors where required and retains immutable payload owners per slot; abort and late release are idempotent | `pkg/container/pSpool/copy.go`, `pkg/container/pSpool/sender_test.go` |
+| Insert / MultiUpdate | Builds owned write batches with `UnionBatch`, or synchronously delegates a unique batch | copied destinations own MPool data; the direct S3 handoff is accepted only after an owned check | `pkg/sql/colexec/multi_update/insert.go`, multi-update tests |
+| Sinker direct stage | Rejects borrowed or `NeedDup` vectors from `WriteOwned` | callers use copying `Write` when ownership cannot move | `pkg/objectio/ioutil/sinker.go`, `pkg/objectio/ioutil/sinker_test.go` |
+| Vector mutation | A borrowed vector is readonly until materialized | mutators call `MaterializeOwned`; reserve/allocate/convert/swap is transactional | `pkg/container/vector/buffer_lease.go`, buffer-lease and allocation-account tests |
+| Remote marshal / object encoding | Borrowed process pointers never cross the wire or storage boundary | canonical marshal copies fixed data and compacts/rebases only referenced varlen area | `pkg/container/vector/vector.go`, borrowed varlen marshal tests |
+
+## Required invariants
+
+- a next `External.Call` may release the reader's record only after every async
+  consumer has retained a readonly view or made an owned copy;
+- `NeedDup` is not a lease carrier and cannot authorize ownership transfer;
+- `WriteOwned` is move-only and rejects borrowed backing;
+- every materialization is charged to the statement allocation account;
+- every retained backing remains charged by physical capacity until its final
+  release, including cancellation and late consumer cleanup;
+- unsupported or unknown mutators materialize before write.
+
+## Deliberate exclusions
+
+General external-table SELECT, arbitrary joins/aggregations, spill paths not
+reachable from LOAD, and future Python UDF operators are outside this closure.
+They must perform their own ownership audit before accepting borrowed Arrow
+backing.