Repository navigation
[Ruby] Add builder API for red-arrow-format #50382
Description
Describe the enhancement requested
red-arrow-format is a pure Ruby Apache Arrow reader/writer implementation. It can write Apache Arrow IPC file/streaming format data but users need to build Apache Arrow in-memory layout data by themselves. For example, [1, 2, 3].pack("l") + padding for int32 array. It's inconvenient. We should provide builder API for easy to build Apache Arrow in-memory layout data.
Component(s)
Ruby
Activity
emecii commentedon Sep 9, 2026
take
I’d like to work on this. Since
ArrowFormat::ArrayBuilderalready exists internally and #51214 added schema-awareRecordBatchconstruction, would exposing a public builder for primitive, string, and null-containing arrays first—while deferring nested, dictionary, and union arrays—match the intended scope?kou commentedon Sep 9, 2026
I want to add support for all types (including nested, dictionary and so on) before we work on this because we may need different API for non primitive types. If we defer these types, we may design inconsistent API.
Could you work on other types before this?
emecii commentedon Sep 9, 2026
Yes. I’ll map the builder behavior across nested, dictionary, union, and primitive types first, then propose one consistent public API and test matrix here before implementing it.
emecii commentedon Sep 9, 2026
I mapped the current
ArrowFormattype families. I suggest one explicit-type path for every represented type:Inference would stay conservative; empty/all-null values and ambiguous
Hashvalues would requiretype:. Dictionary arrays also need a statefulArrayBuilder.new(type:)API so memo/delta state can survive multiplefinishcalls.Two scope questions before I implement: does “all types” mean the existing
ArrowFormat::Typesubclasses, or also currently missing IPC types such as run-end encoded and view arrays? And should the stateful dictionary API ship together with the batch API?kou commentedon Sep 9, 2026
I want to use
ArrowFormat::Int32Array.new(values)like API instead ofArrowFormat::Array.build(values, type: :int32)like API for explicit type build API because Ruby is an object oriented language.The former.
For example, we don't have
ArrowFormat::ListArray.new(type, values)like API for now. We need to work on it as a separated task (separated issue and separated PR) before this.We can work on them as separated tasks.
emecii commentedon Sep 9, 2026
Opened and claimed #51262 for the separate
ArrowFormat::ListArray.new(type, values)prerequisite. I’ll keep the dictionary state API separate as suggested.GH-51262: [Ruby] Add ListArray values constructor (#51263)
GH-51273: [Ruby] Add FixedSizeListArray values constructor (#51274)