*: bump arrow-go and update replacement | plugin=pr/246

[joechenrh]

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

TiDB currently replaces github.com/apache/arrow-go/v18 with a forked module. Now the fork is no longer needed.

What changed and how does it work?

• Update github.com/apache/arrow-go/v18 to v18.6.1-0.20260713210906-efc145c067b6, which includes feat(parquet): opt-in streaming reads for large data pages apache/arrow-go#880.
• Remove the replacement of the upstream module with github.com/joechenrh/arrow-go/v18.
• Enable page streaming in the TiDB Parquet parser while retaining the existing 1 KiB per-column read-ahead.
• Keep cloud.google.com/go/storage pinned to v1.39.1 to avoid the current Bazel protobuf toolchain incompatibility.
• Align enterprise plugins in pingcap-inc/enterprise-plugin#246.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Page-streaming benchmark

• Launch a fresh c6i.2xlarge in the same region as the S3 bucket.
• Compare master, an earlier prototype with 8 MiB per column, an 8 MiB-total prototype, and the current PR implementation with 1 KiB per column.
• Use PLAIN, uncompressed, dictionary-disabled Parquet files for the four shapes below.
• Run every sample in a separate process after dropping the OS page cache. Balance both case and implementation order to reduce ordering bias.
• Exclude one warm-up per combination, run 10 measured samples, and extend combinations whose CV exceeds 10% to 20 measured samples.
• Measure parse-only throughput and peak RSS; collect CPU and heap profiles in separate runs so profiling overhead does not affect throughput samples.

The cells below are median parse throughput in MiB/s (CV; median peak RSS in MiB):

Parquet shape	master	Earlier prototype: 8 MiB/column	Prototype: 8 MiB total	Current PR: 1 KiB/column
A: 100 columns x one ~2 MiB page	180 (6.6%; 495)	179 (14.2%; 399)	325 (20.9%; 206)	314 (20.3%; 197)
B: 100 columns x 128 ~16 KiB pages	455 (22.1%; 106)	185 (13.0%; 310)	403 (8.3%; 116)	449 (4.3%; 108)
C: 10 columns x one ~16 MiB page	176 (27.0%; 396)	260 (14.9%; 261)	688 (19.3%; 151)	677 (26.7%; 137)
D: 1 column x one ~200 MiB page	91.5 (4.2%; 475)	96.4 (9.7%; 116)	96.4 (0.3%; 116)	96.4 (0.2%; 95)

Profiles show that the PR's 8 MiB-per-column setting allocates about 196 MiB in Arrow's bufferedReader.resetBuffer in case B; representative total allocation rose from 61 MB on master to 268 MB. This explains the small-page regression. In case A, page streaming reduces representative total allocation from 466 MB on master to 157 MB with an 8 MiB total budget and 150 MB with 1 KiB per column. CPU samples were dominated by TLS/syscall and runtime.memmove; no new decoder hotspot appeared.

The result supports page streaming, but not an 8 MiB buffer per column. The current implementation therefore keeps the existing 1 KiB read-ahead, which gives the best overall balance in these workloads; an 8 MiB total budget still regresses the small-page case by about 11% at the median.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Summary by CodeRabbit

• Improvements

  • Improved Parquet reading for both streaming and in-memory workloads.
  • Added more accurate peak-memory estimation, including read-ahead buffers.
  • Reduced memory usage when processing large Parquet data pages.

• Bug Fixes

  • Improved handling of large uncompressed Parquet pages while preserving row values and end-of-file behavior.

• Maintenance

  • Updated Go and build dependencies, including cloud, compression, gRPC, and OpenTelemetry libraries.
  • Expanded Parquet test parallelization.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR refreshes Go and Bazel dependencies, adds repository pins, and updates replacement directives. It also separates Parquet streaming and preload reader properties, includes read-ahead buffers in memory estimates, and adds large-page peak-memory coverage.

Changes

Dependency refresh

Layer / File(s)	Summary
Go module declarations go.mod	Direct and indirect modules, OpenTelemetry components, Google libraries, and replacement directives are updated.
Bazel dependency pins DEPS.bzl	Repository additions, removals, and version or checksum refreshes are applied across supporting, Google Cloud, protocol, OpenTelemetry, and modernc dependencies.

Parquet memory controls

Layer / File(s)	Summary
Reader mode selection pkg/dumpformat/parquetfile/parser.go	Separate stream and preload reader properties are created and selected for each row group.
Memory estimation and validation pkg/dumpformat/parquetfile/parser.go, pkg/dumpformat/parquetfile/parser_test.go, pkg/dumpformat/parquetfile/BUILD.bazel	Read-ahead buffers are included in peak-memory estimation, with large-page coverage and updated test sharding.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant NewParser
  participant Parser
  participant RowGroupBuilder
  participant ParquetReader
  NewParser->>Parser: create stream and preload properties
  Parser->>RowGroupBuilder: select properties by row-group size
  RowGroupBuilder->>ParquetReader: construct reader with selected properties

Loading

Possibly related PRs

• pingcap/tidb#69320: Updates overlapping dependency-pinning surfaces and module versions.
• pingcap/tidb#69843: Changes tooling associated with regenerating DEPS.bzl.
• pingcap/tidb#69844: Changes dependency resolution used to generate DEPS.bzl and go.mod.

Suggested reviewers: d3hunter

Poem

I’m a rabbit with buffers in tow,
Through Parquet pages I hop and go.
Stream here, preload there,
Memory counts with care—
Fresh pins make the burrow glow!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title names the Arrow-go replacement and version bump, which matches the primary scope of the PR.
Description check	✅ Passed	The description follows the template with problem, change summary, checklist, and release note sections.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[joechenrh]

/test all

[codecov]

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.8191%. Comparing base (b8d04e1) to head (42e3df9).
⚠️ Report is 31 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #69461        +/-   ##
================================================
- Coverage   76.3194%   73.8191%   -2.5003%     
================================================
  Files          2041       2058        +17     
  Lines        560092     583920     +23828     
================================================
+ Hits         427459     431045      +3586     
- Misses       131732     152028     +20296     
+ Partials        901        847        -54     

Flag	Coverage Δ
integration	40.7244% <0.0000%> (+1.0191%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	60.4471% <ø> (ø)
parser	∅ <ø> (∅)
br	47.4060% <ø> (-15.3154%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[joechenrh]

/test all

[joechenrh]

/test all

[ingress-bot]

🔍 Starting code review for this PR...

[ingress-bot]

This review was generated by AI and should be verified by a human reviewer.
Manual follow-up is recommended before merge.

Summary

• Total findings: 3
• Inline comments: 2
• Summary-only findings (no inline anchor): 1

Findings (highest risk first)

⚠️ [Major] (1)

1. In-memory-vs-streaming threshold predicate duplicated a third time instead of extracted into a shared helper (pkg/dumpformat/parquetfile/parser.go:440, pkg/dumpformat/parquetfile/parser.go:906, pkg/dumpformat/parquetfile/parser.go:927)

🟡 [Minor] (2)

1. Per-column 8 MiB stream read-ahead scales with column count and regresses memory for wide parquet files (pkg/dumpformat/parquetfile/parser.go:79, pkg/dumpformat/parquetfile/parser.go:392, pkg/dxf/importinto/task_executor.go:218)
2. Peak-memory regression test asserts an unexplained 4<<20 slack constant (pkg/dumpformat/parquetfile/parser_test.go:1401)

Unanchored findings

⚠️ [Major] (1)

1. In-memory-vs-streaming threshold predicate duplicated a third time instead of extracted into a shared helper
   • Request: Extract a single helper, e.g. func isRowGroupInMemory(rangeSize int64) bool { return rangeSize <= int64(rowGroupInMemoryThreshold) }, and call it from getBuilder, estimateInMemoryRowGroupBufferBytes, and estimateReadAheadBufferBytes so the three sites cannot drift independently.

[joechenrh]

/hold

This shold be merged simultaneously with plugin PR.

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-07-16 04:15:53.773435419 +0000 UTC m=+859939.809530465: ☑️ agreed by D3Hunter.
• 2026-07-16 10:13:09.997986698 +0000 UTC m=+881376.034081784: ☑️ agreed by GMHDBJD.

[bb7133]

LGTM

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bb7133, D3Hunter, GMHDBJD

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [D3Hunter,GMHDBJD,bb7133]
• pkg/dumpformat/OWNERS [D3Hunter,GMHDBJD]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[joechenrh]

/retest

[joechenrh]

/unhold

[joechenrh]

/cherry-pick release-nextgen-202603

[ti-chi-bot]

@joechenrh: new pull request created to branch release-nextgen-202603: #70075.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherry-pick release-nextgen-202603

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.