importinto: log chunk count and size (#70271)

[ti-chi-bot]

This is an automated cherry-pick of #70271

What problem does this PR solve?

Issue Number: ref #61702

Problem Summary:

Import Into encode subtasks do not currently expose how many chunks they need to process or the size of each chunk, making chunk-related failures harder to diagnose.

What changed and how does it work?

• Log the number of chunks after the encode subtask metadata is fully loaded.
• Add the chunk size, calculated as EndOffset - Offset, to the per-chunk logger.
• Verify that the chunk processing log contains the expected chunk size.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

parquet

[2026/08/03 10:38:30.868 +08:00] [INFO] [task_executor.go:285] ["start processing chunks"] [keyspaceName=SYSTEM] [task-id=1] [task-key=SYSTEM/ImportInto/1] [step=encode] [subtask-id=1] [chunkCount=1]
[2026/08/03 10:38:30.871 +08:00] [INFO] [chunk_process.go:403] ["process chunk start"] [keyspaceName=SYSTEM] [task-id=1] [task-key=SYSTEM/ImportInto/1] [step=encode] [subtask-id=1] [key=1col-1row.parquet:0] [chunkSize=254]
[2026/08/03 10:38:30.871 +08:00] [INFO] [chunk_process.go:407] ["process chunk completed"] [keyspaceName=SYSTEM] [task-id=1] [task-key=SYSTEM/ImportInto/1] [step=encode] [subtask-id=1] [key=1col-1row.parquet:0] [chunkSize=254] [readDur=65.583µs] [encodeDur=53.25µs] [checksum="{cksum=12129104817369816750,size=33,kvs=1}"] [deliverDur=59.375µs] [type=file] [takeTime=215.5µs] []

csv

[2026/08/03 10:40:01.161 +08:00] [INFO] [task_executor.go:285] ["start processing chunks"] [keyspaceName=SYSTEM] [task-id=2] [task-key=SYSTEM/ImportInto/2] [step=encode] [subtask-id=4] [chunkCount=1]
[2026/08/03 10:40:01.163 +08:00] [INFO] [chunk_process.go:403] ["process chunk start"] [keyspaceName=SYSTEM] [task-id=2] [task-key=SYSTEM/ImportInto/2] [step=encode] [subtask-id=4] [key=1.csv:0] [chunkSize=10]
[2026/08/03 10:40:01.164 +08:00] [INFO] [chunk_process.go:407] ["process chunk completed"] [keyspaceName=SYSTEM] [task-id=2] [task-key=SYSTEM/ImportInto/2] [step=encode] [subtask-id=4] [key=1.csv:0] [chunkSize=10] [readDur=39.79µs] [encodeDur=64.873µs] [checksum="{cksum=13668931463816667584,size=165,kvs=5}"] [deliverDur=87.375µs] [type=file] [takeTime=241.833µs] []

• No need to test

  • I checked and no code files have been changed.

Targeted validation:

./tools/check/failpoint-go-test.sh pkg/dxf/importinto -run 'TestImportStepExecutorCleanupAllLocalEnginesOnRetry' -count=1
./tools/check/failpoint-go-test.sh pkg/executor/importer -run 'TestFileChunkProcess/process_success' -count=1
make lint

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Summary by CodeRabbit

• New Features

  • Import processing logs now include the total number of chunks and the size of each chunk.
  • Chunk size reporting is available for both Parquet files and byte-range imports.

• Tests

  • Added coverage verifying that chunk size details are included in processing logs.

[coderabbitai]

📝 Walkthrough

Walkthrough

The importer now logs subtask chunk counts and processor chunk sizes. Chunk.GetSize provides sizes for Parquet and byte-range chunks. Tests verify the processor log output.

Changes

Importer observability

Layer / File(s)	Summary
Chunk size contract pkg/executor/importer/table_import.go	Chunk.GetSize returns FileSize for Parquet chunks and the byte range length for other chunks.
Processor chunk telemetry pkg/executor/importer/chunk_process.go, pkg/executor/importer/chunk_process_testkit_test.go, pkg/executor/importer/BUILD.bazel	The processor logs chunkSize. Tests use a Zap observer and verify one "process chunk start" log with the expected size.
Subtask chunk-count logging pkg/dxf/importinto/task_executor.go	RunSubtask logs the number of chunks before processing begins.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• pingcap/tidb#68652: Both changes modify importer chunk sizing.
• pingcap/tidb#70233: Both changes modify chunk size handling.
• pingcap/tidb#70271: The changes cover matching chunk-count logging, size computation, processor logging, tests, and Bazel dependencies.

Suggested reviewers: d3hunter

Poem

A rabbit logs each chunk in sight,
Its size is measured clear and right.
Parquet keeps its file-sized claim,
Byte ranges join the logging game.
Tests watch the messages hop.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: adding Import Into chunk count and size logging.
Description check	✅ Passed	The description includes the issue reference, problem, implementation details, tests, validation commands, side effects, documentation impact, and release note.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

pkg/executor/importer/chunk_process_testkit_test.go (1)

154-160: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert the completion log as well.

The PR objective requires chunkSize on both the chunk start and completion logs. These assertions inspect only "process chunk start". Add an assertion for the completion log and its chunkSize field.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/executor/importer/chunk_process_testkit_test.go` around lines 154 - 160,
Extend the test after processing in the relevant chunk processor test to also
filter the completion log, assert exactly one completion entry, and verify its
ContextMap()["chunkSize"] equals int64(len(sourceData)), matching the existing
start-log assertions.

pkg/executor/importer/table_import.go (1)

121-128: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Verify coverage for both Chunk.GetSize branches.

Chunk.GetSize has two different size contracts. The supplied processor test covers only the non-Parquet path with a zero offset. Add or confirm deterministic tests for a Parquet chunk and a non-zero Offset.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/executor/importer/table_import.go` around lines 121 - 128, Expand tests
for Chunk.GetSize to cover both branches: verify a Parquet chunk returns
FileSize regardless of Offset/EndOffset, and verify a non-Parquet chunk with a
non-zero Offset returns EndOffset minus Offset. Keep the test inputs
deterministic and assert each branch’s distinct size contract.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pkg/executor/importer/chunk_process_testkit_test.go`:
- Around line 154-160: Extend the test after processing in the relevant chunk
processor test to also filter the completion log, assert exactly one completion
entry, and verify its ContextMap()["chunkSize"] equals int64(len(sourceData)),
matching the existing start-log assertions.

In `@pkg/executor/importer/table_import.go`:
- Around line 121-128: Expand tests for Chunk.GetSize to cover both branches:
verify a Parquet chunk returns FileSize regardless of Offset/EndOffset, and
verify a non-Parquet chunk with a non-zero Offset returns EndOffset minus
Offset. Keep the test inputs deterministic and assert each branch’s distinct
size contract.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ca5ca959-60ea-4b1b-9b18-78c9084c5715

📥 Commits

Reviewing files that changed from the base of the PR and between e97fbba and cc8ed55.

📒 Files selected for processing (5)

• pkg/dxf/importinto/task_executor.go
• pkg/executor/importer/BUILD.bazel
• pkg/executor/importer/chunk_process.go
• pkg/executor/importer/chunk_process_testkit_test.go
• pkg/executor/importer/table_import.go

[codecov]

Codecov Report

❌ Patch coverage is 63.63636% with 4 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (release-nextgen-202603@e97fbba). Learn more about missing BASE report.

Additional details and impacted files

@@                     Coverage Diff                     @@
##             release-nextgen-202603     #70338   +/-   ##
===========================================================
  Coverage                          ?   76.3185%           
===========================================================
  Files                             ?       1937           
  Lines                             ?     542103           
  Branches                          ?          0           
===========================================================
  Hits                              ?     413725           
  Misses                            ?     128378           
  Partials                          ?          0           

Flag	Coverage Δ
unit	76.3185% <63.6363%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	61.5065% <0.0000%> (?)
parser	∅ <0.0000%> (?)
br	48.8016% <0.0000%> (?)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, joechenrh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/dxf/OWNERS [D3Hunter]
• pkg/executor/importer/OWNERS [D3Hunter]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-08-04 10:02:45.561519728 +0000 UTC m=+2522351.597614783: ☑️ agreed by D3Hunter.
• 2026-08-05 07:45:35.39528408 +0000 UTC m=+2600521.431379146: ☑️ agreed by joechenrh.