GH-51471: [C++] Disable simdjson threading in chunker

[taepper]

Rationale for this change

Resolves #51471:

When invoking simdjson's parse_many, it uses threading to perform its stage 1 parsing on the next batch when while parsing the current batch:
https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md?plain=1#L96-L104

In most cases, we do not need to parse any batches after the first one to find delimiters (in fact, we currently would even error, when the first batch does not contain the whole first document). By disabling the parser's threading we can expect performance improvements

What changes are included in this PR?

This disables threading in the parser by setting parser_.threaded = false;

Are these changes tested?

Yes

before (origin/main bb83012743):

ChunkJSONPrettyPrintedMultipleBlocks            187462 ns       158760 ns         4415 block_size=27.344k
bytes_per_second=1.28328Gi/s json_size=218.757k

after (taepper:GH-51471 fb56ccb2a1):

ChunkJSONPrettyPrintedMultipleBlocks             98726 ns        98721 ns         7144 block_size=27.344k
bytes_per_second=2.06374Gi/s json_size=218.757k

Are there any user-facing changes?

No

Was AI used for this PR?

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human
• AI
• Not reviewed

• GitHub Issue: [C++] Disable simdjson threading in chunker #51471

[github-actions]

⚠️ GitHub issue #51471 has been automatically assigned in GitHub to PR creator.

[taepper]

If you would have time for review, I think this is a good improvement to the chunker @pitrou (or please @ others that might be a good fit to review this)

[pitrou]

LGTM, can you rebase/merge so that we get updated CI?

[taepper]

LGTM, can you rebase/merge so that we get updated CI?

Thank you for the review, and also #51472! I found these while trying to investigate possible improvements we can achieve with #51463. While studying the docs, I noticed that we definitely still lack some familiarity, and a few simdjson code sections that were recently introduced, use the library in suboptimal ways.

I merged main into the branch to retrigger CI with all latest changes (in particular including #51472)

[pitrou]

@taepper Can you perhaps update the PR description (especially benchmark numbers) now that #51470 has been merged?

[taepper]

@taepper Can you perhaps update the PR description (especially benchmark numbers) now that #51470 has been merged?

I updated the benchmarks, the PR description is still valid