Repository navigation
[C++][Python] Support NaN/Infinity in JSON parsing and accept DictionaryEncoding without indexType in IPC reader - #53871
Asthenia0412 wants to merge 1 commit into
Conversation
…pt DictionaryEncoding without indexType in IPC reader Two related Arrow format compatibility fixes: 1. Bump embedded simdjson from v4.6.4 to v5.0.5 and enable SIMDJSON_ENABLE_NAN_INF so JSON parsing accepts NaN, Infinity, -Infinity, Inf and -Inf (case-insensitive). The Arrow JSON reader already had NonFiniteDoubleFromRawToken as a partial workaround, but simdjson v5.0 gives first-class, case-insensitive support. 2. In the IPC reader, treat DictionaryEncoding.indexType being null as signed int32 indices per the Arrow format spec. Previously metadata_internal.cc used CHECK_FLATBUFFERS_NOT_NULL and rejected well-formed IPC streams written by e.g. apache-arrow (JavaScript) when the JS writer omitted indexType. Tests: - New C++ unit test IPC DictionaryEncodingWithoutIndexType round-trips through metadata_internal::GetSchema with a hand-built flatbuffer. - All existing JSON from_string tests (including NaN/Inf coverage) continue to pass on simdjson v5.0.5.
|
Thanks for opening a pull request! This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format. If this is not a minor PR, could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or After updating the title, you can mark the pull request as ready for review. See also: |
|
Wouldn't we rather bump to version 5 first and then change the functionality afterwards? |
| } | ||
| } | ||
|
|
||
| // ARROW-XXXXX: DictionaryEncoding.indexType may be null per the Arrow format |
There was a problem hiding this comment.
// ARROW-XXXXX: DictionaryEncoding.indexType may be null per the Arrow format
I think you forgot to fill this one out
2 participants
Two Arrow C++ format-compatibility fixes: 1) bump simdjson to v5.0.5 with SIMDJSON_ENABLE_NAN_INF=ON so JSON accepts NaN/Infinity/Inf literals; 2) in IPC metadata_internal.cc, default null DictionaryEncoding.indexType to signed int32 per Schema.fbs. Includes a new C++ unit test DictionaryEncodingWithoutIndexType. All 432 IPC tests and 242 JSON tests pass.