Skip to content

deduplicate met writes - #71

Merged
infotroph merged 1 commit into
projectionsfrom
projection-met-fix
Oct 5, 2026
Merged

infotroph merged 1 commit into
projectionsfrom
projection-met-fix

Conversation

@dlebauer

Copy link
Copy Markdown
Contributor

Convert weather once per ERA5 grid and ensemble member.

This eliminates ~90% of the duplicated writes, may avoid potential issues with concurrent writes.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The deduplication matches the grid-based input and output paths without altering downstream behavior.

Review effort: Balanced
Findings: None

What changed in this PR

Deduplicates ERA5 meteorology conversion jobs by grid cell and ensemble member.

Changes:

  • Retains one location per ERA5 grid cell before expanding ensemble IDs.
File Description
workflow/​01_ERA5_nc_to_clim.R Prevents duplicate clim-file writes for sites sharing a grid cell.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@dlebauer
dlebauer requested a review from infotroph September 22, 2026 05:01
@infotroph

Copy link
Copy Markdown
Contributor

eliminates ~90% of the duplicated writes

met2model.SIPNET is run with overwrite = FALSE here, so duplicated writes are skipped already.

The original design here was that to support both site-specific and gridded use cases with the same script. To convert a convert a whole grid users would pass an already deduplicate list of cell locations; to convert a few sites users would pass the site_info and it would write each output by site_id without checking for duplicate file content. Now that we write by cell name instead, I guess that means the site-specific use case was already removed and we can move the deduplication a step earlier.

@infotroph
infotroph merged commit 7438b4f into projections Oct 5, 2026
1 check passed
@infotroph
infotroph deleted the projection-met-fix branch October 5, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants