*: bump arrow-go and update replacement (#69461) | plugin=pr/248

[ti-chi-bot]

This is an automated cherry-pick of #69461

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

TiDB currently replaces github.com/apache/arrow-go/v18 with a forked module. Now the fork is no longer needed.

What changed and how does it work?

• Update github.com/apache/arrow-go/v18 to v18.6.1-0.20260713210906-efc145c067b6, which includes feat(parquet): opt-in streaming reads for large data pages apache/arrow-go#880.
• Remove the replacement of the upstream module with github.com/joechenrh/arrow-go/v18.
• Enable page streaming in the TiDB Parquet parser while retaining the existing 1 KiB per-column read-ahead.
• Keep cloud.google.com/go/storage pinned to v1.39.1 to avoid the current Bazel protobuf toolchain incompatibility.
• Align enterprise plugins in pingcap-inc/enterprise-plugin#246.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Page-streaming benchmark

• Launch a fresh c6i.2xlarge in the same region as the S3 bucket.
• Compare master, an earlier prototype with 8 MiB per column, an 8 MiB-total prototype, and the current PR implementation with 1 KiB per column.
• Use PLAIN, uncompressed, dictionary-disabled Parquet files for the four shapes below.
• Run every sample in a separate process after dropping the OS page cache. Balance both case and implementation order to reduce ordering bias.
• Exclude one warm-up per combination, run 10 measured samples, and extend combinations whose CV exceeds 10% to 20 measured samples.
• Measure parse-only throughput and peak RSS; collect CPU and heap profiles in separate runs so profiling overhead does not affect throughput samples.

The cells below are median parse throughput in MiB/s (CV; median peak RSS in MiB):

Parquet shape	master	Earlier prototype: 8 MiB/column	Prototype: 8 MiB total	Current PR: 1 KiB/column
A: 100 columns x one ~2 MiB page	180 (6.6%; 495)	179 (14.2%; 399)	325 (20.9%; 206)	314 (20.3%; 197)
B: 100 columns x 128 ~16 KiB pages	455 (22.1%; 106)	185 (13.0%; 310)	403 (8.3%; 116)	449 (4.3%; 108)
C: 10 columns x one ~16 MiB page	176 (27.0%; 396)	260 (14.9%; 261)	688 (19.3%; 151)	677 (26.7%; 137)
D: 1 column x one ~200 MiB page	91.5 (4.2%; 475)	96.4 (9.7%; 116)	96.4 (0.3%; 116)	96.4 (0.2%; 95)

Profiles show that the PR's 8 MiB-per-column setting allocates about 196 MiB in Arrow's bufferedReader.resetBuffer in case B; representative total allocation rose from 61 MB on master to 268 MB. This explains the small-page regression. In case A, page streaming reduces representative total allocation from 466 MB on master to 157 MB with an 8 MiB total budget and 150 MB with 1 KiB per column. CPU samples were dominated by TLS/syscall and runtime.memmove; no new decoder hotspot appeared.

The result supports page streaming, but not an 8 MiB buffer per column. The current implementation therefore keeps the existing 1 KiB read-ahead, which gives the best overall balance in these workloads; an 8 MiB total budget still regresses the small-page case by about 11% at the median.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Summary by CodeRabbit

• Bug Fixes
  • Improved Parquet parsing for large pages by enabling page streaming to avoid loading entire pages into memory, lowering peak memory usage for large values.
• Dependency Updates
  • Refreshed and expanded the pinned Go dependency set, including Arrow, OpenTelemetry, Google Cloud, and gRPC libraries, along with related module version and checksum updates.
• Tests
  • Added a unit test that validates large-page parsing succeeds while keeping allocator peak usage within a tight streaming-focused limit.

[ti-chi-bot]

@joechenrh This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

[ti-chi-bot]

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[coderabbitai]

📝 Walkthrough

Walkthrough

Dependency versions and Bazel repository pins are refreshed across the Go ecosystem. The Parquet parser enables page streaming, with a large-page test verifying complete reads and bounded peak allocation.

Changes

Dependency refresh and Parquet streaming

Layer / File(s)	Summary
Go module requirements and replacements go.mod	Direct and indirect dependencies are upgraded, Google Cloud auth modules are added, github.com/rivo/uniseg is removed, and replacement entries are updated.
Bazel third-party dependency pins DEPS.bzl	Arrow and numerous third-party repository pins are added, removed, or refreshed.
Cloud and observability dependency pins DEPS.bzl	Cloud, Google, OpenTelemetry, PDF, GUI, and related repository pins are updated.
Go and modernc dependency pins DEPS.bzl	golang.org/x, Gonum, Modernc, and supporting repository pins are refreshed.
Parquet streaming integration and validation pkg/lightning/mydump/parquet_parser.go, pkg/lightning/mydump/parquet_parser_test.go	Page streaming is enabled and large-page parsing is tested for successful reads and bounded allocator peak usage.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Test as Large-page test
  participant Parser as NewParquetParser
  participant Reader as Parquet reader
  participant Allocator as trackingAllocator
  Test->>Parser: create parser with streaming enabled
  Parser->>Reader: configure PageStreamingEnabled
  Reader->>Allocator: use streaming buffers
  Reader-->>Test: read rows and return io.EOF
  Test->>Allocator: verify bounded peak allocation

Loading

Possibly related PRs

• pingcap/tidb#69959: Refreshes overlapping Bazel and Go dependency pins, including Google, gRPC, and OpenTelemetry modules.
• pingcap/tidb#70193: Overlaps with the Parquet streaming behavior and large-page memory test.

Poem

A rabbit hops through pins refreshed,
While Parquet pages stream their best.
Big pages pass with memory light,
Rows reach EOF in tidy flight.
“Neat!” says Bun, and twitches nose.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and clearly describes the main change: bumping arrow-go and removing the replacement.
Description check	✅ Passed	The description follows the template and includes the required sections with detailed change, test, and release-note info.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

go.mod (1)

146-170: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

Remove all unresolved Git conflict markers before merging.

The <<<<<<<, =======, and >>>>>>> markers make go.mod syntactically invalid, so commands such as go mod tidy, go list, and builds that resolve this module will fail. Resolve each dependency choice and remove the markers from all four regions.

Also applies to: 229-236, 243-253, 364-399

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@go.mod` around lines 146 - 170, Resolve every remaining merge conflict in
go.mod, including the dependency blocks around the visible golang.org/x entries
and the additional referenced regions. Choose the correct dependency versions
for each conflict, then remove all <<<<<<<, =======, and >>>>>>> markers so
go.mod is valid and module commands can run.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@DEPS.bzl`:
- Around line 1050-1060: Resolve every remaining merge conflict in DEPS.bzl,
including the go_repository entries around com_github_cncf_xds_go and the other
listed dependency blocks, removing all conflict markers so the file is valid
Starlark. Reconcile each version and checksum deliberately, and merge both
sides’ go_repository additions in the OpenTelemetry detectors and
org_golang_google_grpc sections rather than discarding either side.

---

Outside diff comments:
In `@go.mod`:
- Around line 146-170: Resolve every remaining merge conflict in go.mod,
including the dependency blocks around the visible golang.org/x entries and the
additional referenced regions. Choose the correct dependency versions for each
conflict, then remove all <<<<<<<, =======, and >>>>>>> markers so go.mod is
valid and module commands can run.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 68e15f4a-0ab5-47be-b88d-c6848956ac90

📥 Commits

Reviewing files that changed from the base of the PR and between 09aa743 and 598425c.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (5)

• DEPS.bzl
• go.mod
• pkg/dumpformat/parquetfile/BUILD.bazel
• pkg/lightning/mydump/parquet_parser.go
• pkg/lightning/mydump/parquet_parser_test.go

[ti-chi-bot]

Cherry-pick conflicts appear resolved; removing the do-not-merge/hold label.

[joechenrh]

/hold

[joechenrh]

/retest

[joechenrh]

/retest

[joechenrh]

/retest

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (release-nextgen-202603@8dffb79). Learn more about missing BASE report.

Additional details and impacted files

@@                     Coverage Diff                     @@
##             release-nextgen-202603     #70075   +/-   ##
===========================================================
  Coverage                          ?   76.1896%           
===========================================================
  Files                             ?       1937           
  Lines                             ?     541297           
  Branches                          ?          0           
===========================================================
  Hits                              ?     412412           
  Misses                            ?     128885           
  Partials                          ?          0           

Flag	Coverage Δ
unit	76.1896% <100.0000%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	61.5065% <0.0000%> (?)
parser	∅ <0.0000%> (?)
br	48.7808% <0.0000%> (?)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-07-29 07:33:45.878794085 +0000 UTC m=+1995011.914889141: ☑️ agreed by joechenrh.
• 2026-07-29 10:25:58.746063678 +0000 UTC m=+2005344.782158734: ☑️ agreed by D3Hunter.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cfzjywxk, D3Hunter, joechenrh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [D3Hunter,cfzjywxk,joechenrh]
• pkg/lightning/OWNERS [D3Hunter]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[D3Hunter]

/unhold

[D3Hunter]

/hold

[joechenrh]

Will merge with https://github.com/pingcap-inc/enterprise-plugin/pull/248

[joechenrh]

/unhold