Repository navigation
Time mapper doesn't handle time zones from dsgrid #31
Description
dsgrid creates geography records with time zones like EasternPrevailing, CentralPrevailing, etc: https://github.com/dsgrid/dsgrid-project-StandardScenarios/blob/main/dsgrid_project/dimensions/counties.csv
chronify uses pandas for time zone conversions and doesn't know about these. What should we do?
E pytz.exceptions.UnknownTimeZoneError: 'PacificPrevailing'
env/lib/python3.11/site-packages/pytz/__init__.py:188: UnknownTimeZoneError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/dthom/repos/secondary-dsgrid/env/lib/python3.11/site-packages/pytz/__init__.py(188)timezone()
-> raise UnknownTimeZoneError(zone)
(Pdb) up
> /Users/dthom/repos/secondary-dsgrid/timezones.pyx(170)pandas._libs.tslibs.timezones.maybe_get_tz()
(Pdb)
> /Users/dthom/repos/secondary-dsgrid/timezones.pyx(145)pandas._libs.tslibs.timezones.maybe_get_tz()
(Pdb)
> /Users/dthom/repos/secondary-dsgrid/env/lib/python3.11/site-packages/pandas/core/arrays/datetimes.py(911)tz_convert()
-> tz = timezones.maybe_get_tz(tz)
(Pdb)
> /Users/dthom/repos/secondary-dsgrid/env/lib/python3.11/site-packages/pandas/core/indexes/datetimes.py(283)tz_convert()
-> arr = self._data.tz_convert(tz)
(Pdb)
> /Users/dthom/repos/secondary-dsgrid/env/lib/python3.11/site-packages/pandas/core/indexes/accessors.py(132)_delegate_method()
-> result = method(*args, **kwargs)
(Pdb)
> /Users/dthom/repos/secondary-dsgrid/env/lib/python3.11/site-packages/pandas/core/accessor.py(112)f()
-> return self._delegate_method(name, *args, **kwargs)
(Pdb)
> /Users/dthom/repos/chronify/src/chronify/representative_time_range_generator.py(64)create_tz_aware_mapping_dataframe()
-> dft["timestamp_tmp"] = dft[timestamp_col].dt.tz_convert(tz)
(Pdb) ll
54 def create_tz_aware_mapping_dataframe(
55 self,
56 df: pd.DataFrame,
57 timestamp_col: str,
58 time_zones: list[str],
59 ) -> pd.DataFrame:
60 """Create time zone-aware time mapping dataframe."""
61 dfm = []
62 for tz in time_zones:
63 dft = df.copy()
64 -> dft["timestamp_tmp"] = dft[timestamp_col].dt.tz_convert(tz)
65 dft = self._handler.add_time_attribute_columns(dft, "timestamp_tmp")
Activity
lixiliu commentedon Jan 17, 2025
Either:
[1] Remove the use of these enums from dsgrid and rely instead on a tz library that pandas uses, which is dateutil: https://dateutil.readthedocs.io/en/stable/tz.html
[2] Convert the enums to pandas-friendly field values
[3] Build in Chronify capability to accept these enums.
These enums contain values for the pyspark API and can be used to go between pairs of standard time and prevailing time. They were useful for mapping industrial time. Let's see what @JensZack ends up implementing. I am inclined to go with the cleanest solution in that if these enums are no longer needed, we go with [1].
lixiliu commentedon Feb 13, 2025
Decision:
@elainethale - does this work for you?
elainethale commentedon Feb 13, 2025
Yes, that works. Thanks!