[C++] can we use simdjson to replace rapidjson

Describe the enhancement requested

As the performance result mentioned in simdjson community: the simdjson library uses three-quarters less instructions than state-of-the-art parser RapidJSON. And the throughput of simdjson is much higher than that of rapidjson:

So can we replace rapidjson with simdjson to implement json parser?

Component(s)

C++

If simdjson is faster than RapidJSON for our use case too, I'm OK with this.

Could you try this and share our benchmark result?
https://github.com/apache/arrow/blob/main/cpp/src/arrow/json/parser_benchmark.cc

We are available to help.

Note that simdjson is used by Apache Doris and ClickHouse.

Great!

Seems that writer can still use original logic, but parser can make full use of simdjson?

Is there any merit to use both RapidJSON and simdjson?

I think that using either RapidJSON or simdjson will reduce our maintenance cost.

@kou

Agreed with @kou , we probably want to avoid depending on two different JSON libraries.

Interested people should try working on a PR.

I'm skeptical switching to simdjson would improve performance a lot, btw. Parsing is only a small part of the work necessary to convert JSON to Arrow.

I just want to revive this discussion again. #45459 will make RapidJson as a required dependency to parquet. However, RapidJson has been poorly maintained since 2016: Tencent/rapidjson#2321. It also seems to be a blocker to support CMake 4: #45985

Let's try simdjson for better maintainability!

For what it's worth, I think DuckDB uses yyjson, although I don't know if simdjson was considered. I seem to remember some past issues with homebrew packaging with respect to buildtime vs. runtime SIMD support (although I imagine simdjson has considered this as well!).

I seem to remember some past issues with homebrew packaging with respect to buildtime vs. runtime SIMD support (although I imagine simdjson has considered this as well!).

The simdjson library uses runtime dispatching. It is currently used by Node.js, ClickHouse and other important systems. In turn, Node.js is part of several important systems.

Optionally, we also support fancy C++20 features, and we will be integrating C++26 features (e.g., static reflection and the like) as soon as possible.

Suggested reference:

On-Demand JSON: A Better Way to Parse Documents? Software: Practice and Experience 54 (6), 2024

Documentation: https://github.com/simdjson/simdjson/blob/master/doc/basics.md

I agree we need to do something about our RapidJSON dependency. simdjson is a reasonable contender. We should just have to check it doesn't reduce the current platform compatibility (especially when SIMD isn't available/supported by simdjson).

@Reranko05

@Reranko05 on main we're getting this:

/usr/bin/ld: /build/arrow-install/lib/libparquet.so.2600.0.0: undefined reference to `arrow::json::JsonWriter::String(std::basic_string_view<char, std::char_traits<char> >)'
/usr/bin/ld: /build/arrow-install/lib/libparquet.so.2600.0.0: undefined reference to `arrow::json::JsonWriter::GetString() const'

Is this something one of these issues will address or should we open a new one?

Is this something one of these issues will address or should we open a new one?

There's a PR open at #50900 but I don't think it's a good solution - see comments.

Edit: and it seems the corresponding issue is #50859

@Krishnanand-G

Is this something one of these issues will address or should we open a new one?

@Krishnanand-G is working on the issue, we can try the approach suggested by @pitrou #50900 (comment)

Merge branch 'main' into apachegh-35460-integration2

Merge branch 'main' into apachegh-35460-parser

Navigation Menu

Repository navigation

[C++] can we use simdjson to replace rapidjson #35460

Describe the enhancement requested

Component(s)

64 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[C++] can we use simdjson to replace rapidjson #35460

Description

Describe the enhancement requested

Component(s)

Activity

kou commented on May 6, 2023

lemire commented on May 12, 2023

kou commented on May 12, 2023

mapleFU commented on May 12, 2023

kou commented on May 12, 2023

pitrou commented on May 24, 2023

pitrou commented on May 24, 2023

wgtmac commented on Apr 11, 2025

kou commented on Apr 11, 2025

paleolimbot commented on Apr 11, 2025

lemire commented on Apr 11, 2025

pitrou commented on Apr 16, 2025

64 remaining items

rok commented on Aug 25, 2026

pitrou commented on Aug 25, 2026

Reranko05 commented on Aug 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions