GH-54903: [C++][OSS-Fuzz fuzzing] Update csv fuzzer and corpora/dict preparation by arthurscchan · Pull Request #54912 · apache/arrow

arthurscchan · 2026-10-21T14:01:12Z

This PR updates the csv fuzzer with new corpora/dict, together with google/oss-fuzz#16332 to improve the fuzzer for project arrow. This PR is the 3rd PR aiming to improve the overall fuzzing coverage and effectiveness of the arrow project, mentioned in #54903.

GitHub Issue: [C++][CI] Adding more OSS-Fuzz fuzzers for the arrow project #54903

Signed-off-by: Arthur Chan <arthur.chan@adalogics.com>

pitrou · 2026-10-22T09:03:38Z

Thanks for the PR @arthurscchan . Is there any reason this doesn't reuse our standard PR description template?

pitrou

Thanks for doing this. On the principle this is interesting, but I think there are number of problems with the current implementation.

pitrou · 2026-10-22T09:05:03Z

 # Add examples from Pandas test suite
 find "${PANDAS_DIR}/" -name "*.csv" -exec cp --backup=numbered '{}' "${CORPUS_DIR}" \;
+# Some Pandas CSV files are exact duplicates of each other under different
+# names, and pack_corpus.py fails on duplicate hashes; keep only one copy.


Hmm, can pack_corpus.py be changed to not fail on duplicate hashes then? Perhaps it should just emit a warning instead of failing?

pitrou · 2026-10-22T09:06:52Z

@@ -0,0 +1,60 @@
+# Tokens for arrow-csv-fuzz: CSV dialect characters, null/bool spellings,
+# numeric edge cases and timestamp formats understood by the Arrow CSV reader.


Ok, it's not clear to me how dictionaries are used by the fuzzing engine.

Are inputs restricted to a combination of tokens from the dictionaries? Or are random mutations still applied on top?

pitrou · 2026-10-22T09:09:25Z

+  opts.parse.delimiter = kDelimiters[header[0] % 5];
+  opts.parse.quoting = !(header[1] & 1);


Hmm, instead of all the manual bit-twiddling here, can we use a dedicated packed structure with bit-fields?

Can get inspiration from the Parquet encoding fuzzer:

arrow/cpp/src/parquet/arrow/fuzz_encoding_internal.cc

Lines 70 to 88 in 373d64c

constexpr auto kPackedEncodingHeaderSize = 64;

ARROW_PACKED_START(struct, PackedEncodingHeader) {

ARROW_PACKED_START(struct, Header) {

uint8_t source_encoding_id;

uint8_t roundtrip_encoding_id;

uint8_t type_id;

int32_t type_length;

int32_t num_values;

};

ARROW_PACKED_END

static_assert(sizeof(Header) == 3 * 1 + 2 * 4);

using Reserved = std::array<uint8_t, kPackedEncodingHeaderSize - sizeof(Header)>;

Header header;

Reserved reserved;

};

ARROW_PACKED_END

pitrou · 2026-10-22T09:10:20Z

+  ConvertOptions convert;
+};
+
+FuzzCsvOptions MakeFuzzCsvOptions(const uint8_t* header) {


Just like in

arrow/cpp/src/parquet/arrow/fuzz_encoding_internal.cc

Line 119 in 373d64c

::arrow::Result<FuzzEncodingHeader::ParseResult> FuzzEncodingHeader::Parse(

, it would be nicer for this function to take a std::span<const uint8_t> IMHO.

pitrou · 2026-10-22T09:11:56Z

-                                            parse_options, convert_options));
+    ARROW_ASSIGN_OR_RAISE(
+        auto table_reader,
+        TableReader::Make(io_context, input, read_options, opts.parse, opts.convert));


At this point, the input has been exhausted by the previous table reader call, so you'd just getting an empty table here?

pitrou · 2026-10-22T09:12:56Z

-    return table->ValidateFull();
-  };
+    st &= table->ValidateFull();
+  }


Note: if all three table reads are successful, we may want to add differential fuzzing, i.e. test that all three tables are equal (but there may be legitimate reasons why they are not?).

pitrou · 2026-10-22T09:15:28Z

+  // Small batch sizes exercise the writer's chunking and produce files that
+  // make the reader merge many batches.


I'm not sure we care about exercising the writer here. We're just generating files for the reader.

And since those are CSV files, there is no inherent chunking in the file contents, so I don't understand what this is achieving.

pitrou · 2026-10-22T09:17:13Z

+    // Legacy behavior for inputs without a full header: the original
+    // hardcoded options. Small block sizes make chunking more likely to be
+    // exercised, as most legacy corpus entries are in the 4-10 kB range.


Well, the header is small (8 bytes), and most legacy inputs will actually go through the other code path, and get a non-sensical header as well as an unexpected non-header payload.

A possible workaround would be to have a magic number in front of the new header. If the magic number matches, parse the header, otherwise consider the input legacy.

pitrou · 2026-10-22T09:17:51Z

+      ARROW_ASSIGN_OR_RAISE(auto sample_fn, dir_fn.Join(sample_name()));
+      std::cerr << sample_fn.ToString() << std::endl;
+      ARROW_ASSIGN_OR_RAISE(auto file, io::FileOutputStream::Open(sample_fn.ToString()));
+      RETURN_NOT_OK(file->Write(buffer));


We should also be writing the serialized FuzzCsvOptions header, no?

See the Parquet encoding fuzzer for how this can be done.

pitrou · 2026-10-22T09:19:31Z

cc @addisoncrump if you want a bit more of fuzzing-related fun :)

OSS-Fuzz: Update csv fuzzer and corpora/dict preparation

c7662ab

Signed-off-by: Arthur Chan <arthur.chan@adalogics.com>

arthurscchan changed the title ~~OSS-Fuzz: Update csv fuzzer and corpora/dict preparation~~ GH-54903: [OSS-Fuzz fuzzing] Update csv fuzzer and corpora/dict preparation Oct 21, 2026

github-actions Bot added the awaiting review Awaiting review label Oct 21, 2026

arthurscchan changed the title ~~GH-54903: [OSS-Fuzz fuzzing] Update csv fuzzer and corpora/dict preparation~~ GH-54903: [C++][OSS-Fuzz fuzzing] Update csv fuzzer and corpora/dict preparation Oct 22, 2026

pitrou requested changes Oct 22, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Oct 22, 2026

Navigation Menu

Repository navigation

GH-54903: [C++][OSS-Fuzz fuzzing] Update csv fuzzer and corpora/dict preparation - #54912

GH-54903: [C++][OSS-Fuzz fuzzing] Update csv fuzzer and corpora/dict preparation#54912
arthurscchan wants to merge 1 commit into
apache:mainfrom
arthurscchan:fuzz/csv-options

arthurscchan commented Oct 21, 2026 •

edited by github-actions Bot

Loading

Uh oh!

pitrou commented Oct 22, 2026

Uh oh!

pitrou left a comment

Uh oh!

pitrou Oct 22, 2026

Uh oh!

pitrou Oct 22, 2026

Uh oh!

pitrou Oct 22, 2026

Uh oh!

pitrou Oct 22, 2026

Uh oh!

pitrou Oct 22, 2026

Uh oh!

pitrou Oct 22, 2026

Uh oh!

pitrou Oct 22, 2026

Uh oh!

pitrou Oct 22, 2026

Uh oh!

pitrou Oct 22, 2026

Uh oh!

pitrou Oct 22, 2026

Uh oh!

pitrou commented Oct 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,60 @@
		# Tokens for arrow-csv-fuzz: CSV dialect characters, null/bool spellings,
		# numeric edge cases and timestamp formats understood by the Arrow CSV reader.

		opts.parse.delimiter = kDelimiters[header[0] % 5];
		opts.parse.quoting = !(header[1] & 1);

	constexpr auto kPackedEncodingHeaderSize = 64;

	ARROW_PACKED_START(struct, PackedEncodingHeader) {
	ARROW_PACKED_START(struct, Header) {
	uint8_t source_encoding_id;
	uint8_t roundtrip_encoding_id;
	uint8_t type_id;
	int32_t type_length;
	int32_t num_values;
	};
	ARROW_PACKED_END

	static_assert(sizeof(Header) == 3 * 1 + 2 * 4);
	using Reserved = std::array<uint8_t, kPackedEncodingHeaderSize - sizeof(Header)>;

	Header header;
	Reserved reserved;
	};
	ARROW_PACKED_END

		// Small batch sizes exercise the writer's chunking and produce files that
		// make the reader merge many batches.

Conversation

arthurscchan commented Oct 21, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou commented Oct 22, 2026

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou commented Oct 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

arthurscchan commented Oct 21, 2026 •

edited by github-actions Bot

Loading