Repository navigation
[C++] Allocate input buffers with sufficient capacity for SIMDJSON padding #51463
Description
Describe the enhancement requested
Parse currently creates a simdjson::padded_string when the input buffer does not have enough capacity for SIMDJSON's required padding. This requires copying the input buffer.
simdjson::padded_string padded_json(reinterpret_cast<const char*>(json->data()),
json->size());
return parse(padded_json);Investigate whether we can allocate input buffers with enough capacity initially so that this copy can be avoided.
Context: #51038 (comment)
Component(s)
C++
Activity
Reranko05 commentedon Sep 22, 2026
Wait for #51038 to be merged before working on this, if someone wants to contribute.
pitrou commentedon Sep 23, 2026
I don't think this frames the question correctly. If there is not enough padding in the buffer capacity, then we must copy.
The question is whether we can ensure that we allocate the buffer with enough padding initially, to avoid having to copy later.
pitrou commentedon Sep 23, 2026
A related question is what happens statistically in real usage. For example, take some real-world JSONL file(s), and determine whether most buffers have enough padding when reading it from Arrow, or not.
taepper commentedon Sep 23, 2026
My following comment on a previous PR might also be relevant:
#50945 (comment)
simdjson::padded_inputwill provide a (possibly) copy-free version, but it is not yet in a released version:simdjson/simdjson@85cadf4
[-][C++] Avoid copying input buffer when SIMDJSON padding is unavailable[/-][+][C++] Allocate input buffers with sufficient capacity for SIMDJSON padding[/+]Reranko05 commentedon Sep 23, 2026
@pitrou Updated the title and desc, is this fine now?
taepper commentedon Sep 23, 2026
Ah, this reframing makes my previous comment mostly obsolete, as it is about allocating sufficiently large buffer sizes in the first place
pitrou commentedon Sep 23, 2026
I wonder how that will interact with AdressSanitizer or Valgrind.