Repository navigation
deduplicate met writes - #71
Conversation
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The deduplication matches the grid-based input and output paths without altering downstream behavior.
Review effort: Balanced
Findings: None
What changed in this PR
Deduplicates ERA5 meteorology conversion jobs by grid cell and ensemble member.
Changes:
- Retains one location per ERA5 grid cell before expanding ensemble IDs.
| File | Description |
|---|---|
workflow/01_ERA5_nc_to_clim.R |
Prevents duplicate clim-file writes for sites sharing a grid cell. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
met2model.SIPNET is run with overwrite = FALSE here, so duplicated writes are skipped already. The original design here was that to support both site-specific and gridded use cases with the same script. To convert a convert a whole grid users would pass an already deduplicate list of cell locations; to convert a few sites users would pass the site_info and it would write each output by site_id without checking for duplicate file content. Now that we write by cell name instead, I guess that means the site-specific use case was already removed and we can move the deduplication a step earlier. |
Convert weather once per ERA5 grid and ensemble member.
This eliminates ~90% of the duplicated writes, may avoid potential issues with concurrent writes.