SegFault with fill_null_forward() on Boolean ChunkedArray

Describe the bug, including details regarding any error messages, version, and platform.

I've encountered a consistent segfault when trying to use Pandas .ffill() with a boolean ChunkedArray on Windows ~~(no problems on Linux)~~:

import pandas as pd
import pyarrow as pa
print("Pandas:", pd.__version__)
print("PyArrow:", pa.__version__)

for i in range(1, 32):
    mx = 2 ** i
    print(i, mx)
    x = pd.Series(pd.NA, index=range(mx), dtype='boolean[pyarrow]')
    x[mx // 2] = True
    x.ffill()

This always crashes on the 12th iteration, when the array has 4096 values, across multiple computers with the following error:

Process finished with exit code -1073741819 (0xC0000005)

I dug under the hood to see what Pandas was doing and was able to create this pure PyArrow example (though I'm not sure if I'm using the API correctly):

import pyarrow as pa 
import pyarrow.compute as pc

pad = [None] * 4000
a = pa.chunked_array([pad, [True], pad], type=pa.bool_())
pc.fill_null_forward(a)

Additionally, I also tried to reproduce this with just pa.array() but that worked fine.

Operating System: Windows 10 (10.0.19045 Build 19045)
Processor: Intel Core i7-12850HX
Python Version: 3.12.7 (additionally confirmed by co-workers on 3.11 & 3.10)
PyArrow Versions Tested: 16.1, 17.0, 18.0, 18.1

Component(s)

Python

While running some other benchmarks, I ended up reproducing the issue on my personal Linux PC with Python 3.13.1 and PyArrow 18.1:

import pandas as pd
import numpy as np

data = pd.Series(np.random.random(100000) > 0.5, dtype='bool[pyarrow]')
data[0] = pd.NA
data[50000] = pd.NA

data.ffill()  # segfaults


import pyarrow.compute as pc

pc.fill_null_forward(data.values._pa_array)  # Also segfaults

System Info

OS: Arch Linux
Kernel: 6.12.8-zen1-1-zen
Python: 3.13.1
Installed Packages (venv):

numpy==2.2.1
pandas==2.2.3
pyarrow==18.1.0
python-dateutil==2.9.0.post0
pytz==2024.2
six==1.17.0
tzdata==2024.2

Additionally, I was able to reproduce it on a RHEL8 server in the current Docker python:3.12 image. On that same server, outside of the container, I was unable to reproduce the problem in the system Python 3.8 installation with PyArrow 14.0.1.

I found the bug! It appears allocations for new chunked arrays don't account for the BooleanType.byte_width() returning 0:

arrow/cpp/src/arrow/type.h

Lines 608 to 609 in 6a0414b

    
           // BooleanType within arrow use a single bit instead of the C 8-bits layout. 
        
           int bit_width() const final { return 1; }

arrow/cpp/src/arrow/type.h

Lines 186 to 192 in 6a0414b

    
           /// \brief Returns the type's fixed byte width, if any. Returns -1 
        
           /// for non-fixed-width types, and should only be used for 
        
           /// subclasses of FixedWidthType 
        
           virtual int32_t byte_width() const { 
        
             int32_t num_bits = this->bit_width(); 
        
             return num_bits > 0 ? num_bits / 8 : -1; 
        
           }

This results in the allocation of a chunk with 0 bytes in it, but with the length declared to that of the original chunk.

Example in ForwardFillChunked.Exec():

arrow/cpp/src/arrow/compute/kernels/vector_replace.cc

Lines 693 to 699 in 6a0414b

    
           if (is_fixed_width(out->type()->id())) { 
        
             ArrayData* output = out->mutable_array(); 
        
             ARROW_ASSIGN_OR_RAISE(output->buffers[0], ctx->AllocateBitmap(chunk->length())); 
        
             ARROW_ASSIGN_OR_RAISE( 
        
                 output->buffers[1], 
        
                 ctx->Allocate(out->type()->byte_width() * chunk->length())); 
        
           }

apacheGH-45086: [C++] Fix heap buffer overflow in FillNullForward/Bac…

GH-45086: [C++] Fix heap buffer overflow in FillNullForward/Backward … (

Issue resolved by pull request 50843
#50843

Navigation Menu

Repository navigation

SegFault with fill_null_forward() on Boolean ChunkedArray #45086

Describe the bug, including details regarding any error messages, version, and platform.

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	// BooleanType within arrow use a single bit instead of the C 8-bits layout.
	int bit_width() const final { return 1; }

	/// \brief Returns the type's fixed byte width, if any. Returns -1
	/// for non-fixed-width types, and should only be used for
	/// subclasses of FixedWidthType
	virtual int32_t byte_width() const {
	int32_t num_bits = this->bit_width();
	return num_bits > 0 ? num_bits / 8 : -1;
	}

	if (is_fixed_width(out->type()->id())) {
	ArrayData* output = out->mutable_array();
	ARROW_ASSIGN_OR_RAISE(output->buffers[0], ctx->AllocateBitmap(chunk->length()));
	ARROW_ASSIGN_OR_RAISE(
	output->buffers[1],
	ctx->Allocate(out->type()->byte_width() * chunk->length()));
	}

SegFault with fill_null_forward() on Boolean ChunkedArray #45086

Description

Describe the bug, including details regarding any error messages, version, and platform.

Component(s)

Activity

aiudirog commented on Jan 7, 2025

aiudirog commented on Jan 7, 2025

zanmato1984 commented on Sep 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions