Repository navigation
[C++] strptime rolls-over dates not in range for current month #31374
Description
I noticed some potentially unexpected behaviour when converting from string to date. Days that are out of bounds for the given month are rolled over into the following month.
I think the expected behaviour would be to either error (Python) or return NULL/NA (R), but not to roll over dates in the following month.
library(arrow, warn.conflicts = FALSE)
library(lubridate, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
df <- tibble::tibble(string_date = "1999-02-30")
# base R returns NA
df %>%
mutate(date = strptime(string_date, format = "%Y-%m-%d"))
#> # A tibble: 1 × 2
#> string_date date
#> <chr> <dttm>
#> 1 1999-02-30 NA
# arrow rolls over the 30th of February into the 2nd of March
df %>%
arrow_table() %>%
mutate(date = strptime(string_date, format = "%Y-%m-%d")) %>%
collect()
#> # A tibble: 1 × 2
#> string_date date
#> <chr> <dttm>
#> 1 1999-02-30 1999-03-02 00:00:00Thanks Alenka, Joris and Rok for helping me with the Python examples:
pandas:
>>> import pandas as pd
>>> pd.to_datetime("1999-02-30", format="%Y-%m-%d")
...
ValueError: time data 1999-02-30 doesn't match format specifieddatetime:
>>> import datetime
>>> from datetime import datetime
>>> datetime.strptime("1999-02-30", "%Y-%m-%d")
...
ValueError: day is out of range for montharrow:
>>> import pyarrow.compute as pc
>>> print(pc.strptime("1999-02-30", format="%Y-%m-%d", unit="s"))
1999-03-02 00:00:00Reporter: Dragoș Moldovan-Grünfeld / @dragosmg
Watchers: Rok Mihevc / @rok
Related issues:
- [C++] Add error handling option to StrptimeOptions (is related to)
Note: This issue was originally created as ARROW-15948. Please see the migration documentation for further details.
Activity
asfimport commentedon Mar 16, 2022
Joris Van den Bossche / @jorisvandenbossche:
Copying my comment from the other issue: personally, I don't like that behaviour, but I suppose we get this from the system strptime? (so that might even depend on your OS?)
It might be interesting to check what date.h's version of strptime does.
I would say it is both: by default error, and optionally return null (after ARROW-15665)
asfimport commentedon Mar 16, 2022
Rok Mihevc / @rok:
Two notes:
Perhaps system strptimes have an option to disable this behaviour?
Switching from system to date.h strptime might have performance implications so we should probably benchmark as we change this.
asfimport commentedon Jun 29, 2022
Antoine Pitrou / @pitrou:
Does date.h have a strptime? We use a vendored implementation from musl on Windows.
asfimport commentedon Jun 29, 2022
Rok Mihevc / @rok:
It does, but there's this comment: https://github.com/apache/arrow/blob/master/cpp/src/arrow/util/value_parsing.h#L784
asfimport commentedon Jun 29, 2022
Antoine Pitrou / @pitrou:
Ah, well, let's not use it, then :)
asfimport commentedon Jun 29, 2022
Rok Mihevc / @rok:
But maybe it's nice to dates? :D
thisisnic commentedon Jul 30, 2026
Still an issue as of 25.0.0 - potential fix is to call the
.ok()method to check.apacheGH-31374: [C++] Reject invalid calendar dates parsed by strptime
apacheGH-31374: [C++] Prevent strptime from normalizing invalid dates
apacheGH-31374: [C++] Prevent strptime from normalizing invalid dates