Refactoring all providers into a single script#413
Refactoring all providers into a single script#413siddharth7113 wants to merge 1 commit intoopenclimatefix:mainfrom
Conversation
| All providers follow the same shape: | ||
| open zarr -> normalise dim/coord names -> shared post-processing. |
There was a problem hiding this comment.
| All providers follow the same shape: | |
| open zarr -> normalise dim/coord names -> shared post-processing. | |
| All providers follow the same pipeline: | |
| open zarr -> standardise dim/coord names -> shared post-processing. |
Just a few suggestions on wording so it doesn't get confusing since shape often refers to the shape of the data and normalise to normalisation of data which we do elsewhere in this repo so avoiding those words in here to avoid confusion
| """Opens ECMWF IFS / MetOffice Global NWP data.""" | ||
| ds = open_zarr_paths(zarr_path, backend="tensorstore") | ||
| # LEGACY SUPPORT - older zarrs use "init_time"/"variable" dim names | ||
| ds = ds.rename({"init_time": "init_time_utc", "variable": "channel"}) |
There was a problem hiding this comment.
So open_ifs and open_gdm are almost identical apart from the renaming, they could just be one generic open standard_lat_long grid with some checks to see if init_time or variable are in the ds and in each case if they are then rename them or use the rename map pattern which is in open_cloudcasting
There was a problem hiding this comment.
I think there is one issue actually which is when using open_zarr_paths you have to specify whether to use init_time or init_time_utc but we could have a try/except in there which tries init_time and if not then init_time_utc
|
@siddharth7113 sorry for being slow to review this but it seems like a solid start, left some suggestions and ideas to simplify a bit more, thanks |
|
I think you'll need to pull some of the latest changes in the library into your branch as well |
|
@siddharth7113 just an FYI we are just updated access to raise PRs against some repos in OCF so we've sent you an invite to a team within OCF's github which will give you the correct access |
Thanks Sukh, for the review as well as the invite, I'll make the changes soon. |
2 participants
Pull Request
Fixes #147
Checklist: