Skip to content

String-to-timestamp cast rejects explicit positive years accepted by Spark #5716

Description

@peterxcli

Describe the bug

Spark accepts a leading + on a timestamp year, but Comet's native string-to-timestamp parser returns NULL. Widening the timestamp fuzz alphabet in PR #5682 exposed this on Spark 4.1.3. Observed inputs include +7528, +00463, +79821, and +2976.

To reproduce

Use Parquet-backed column input to avoid constant folding:

SET spark.sql.session.timeZone=UTC;
SET spark.sql.ansi.enabled=false;
CREATE TABLE positive_year_probe USING parquet AS SELECT '+7528' AS s;
SELECT s, CAST(s AS TIMESTAMP) FROM positive_year_probe;

With native execution disabled, Spark returns the start of year 7528. With Comet enabled, the cast returns NULL. The failing differential run on PR head 0a1fc76a0 plus the review follow-up used 10,000 seeded random strings and exposed nine such bare positive-year inputs. No other mismatch appeared in that run.

Expected behavior / proposed fix

Accept explicit positive years according to Spark's timestamp segment rules. Current patterns use -?[0-9]{4,6}, excluding +. Audit both timestamp and timestamp_ntz paths and numeric decoding before broadening the sign grammar; add column-input Legacy/TRY/ANSI coverage, including signed dates and malformed signs. Remove the narrow positive-bare-year fuzz exclusion in CometNativeCastSuite once fixed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions