Automate monthly incremental catalog updates via GitHub Actions

[NewGraphEnvironment]

Summary

• .github/workflows/update.yml: monthly incremental catalog update on a GitHub-hosted runner (cron 3rd of month + workflow_dispatch), authenticating via OIDC against role_gha_stac_dem_bc (applied in rtj PR #189). Detects objectstore changes, builds/validates STAC items for new URLs only, syncs to S3, and commits refreshed data/ caches back to main — a failed run persists nothing and self-heals on the next detection.
• CI-safety fixes to the pipeline: STAC_OUTPUT_DIR override (stac_utils.py), detect_changes.R exit contract (0/1/2) with a truncated-listing plausibility guard, s3_sync-ci.sh (never --delete; item JSONs before collection.json), pystac[validation] + requests deps.
• urls_reconcile.py: one-off cache trim so change detection re-flags the 2,107 URLs that entered the cache without ever producing a catalog item (incl. the 90 parenthesized files).
• Guard rails from review: deletions-only months still commit caches; an all-invalid batch stays green with a shortfall warning instead of wedging the cron; warn-only source-access check; rebase-before-push on the cache commit; re-run-safe log artifact.

Related Issues

• Relates to Automate monthly incremental catalog updates via GitHub Actions #23 (issue stays open for the post-merge catch-up + registration phase)
• Relates to NewGraphEnvironment/sred#8

Test plan

• Cold-path rehearsal on a stateless workspace (S3-fetched collection.json only, exact CI dep list via uv/py3.12): item create + dedupe, incremental validate, s3_sync-ci.sh --dryrun with zero deletions and items-before-collection order
• Live ngr fetch verified current listing format matches the cache (98,039 URLs, single-slash scheme uniform) — pinned that ngr SHA in DESCRIPTION
• update.yml YAML-parsed (16 steps, all run blocks literal strings), pure ASCII
• Five fresh-eyes review rounds across the two code commits; 4 real defects found and fixed pre-merge
• Post-merge (tracked in Automate monthly incremental catalog updates via GitHub Actions #23): reconcile seed → local catch-up build (~40k items — beyond the GHA timeout, per the documented oversized-batch path) → workflow_dispatch steady-state verification → pgstac registration on geoserv

Notes

• urls_fetch.R is deliberately bypassed in CI: detect_changes.R fetches the fresh listing itself and maintains the cache, so a separate fetch step would duplicate the ~90 s listing call without adding information.
• Merging arms the monthly cron. The first scheduled fire is Aug 3; the local catch-up is planned well before then, and an accidental early dispatch fails safe (commits nothing).
• Also archives the stale issue-13 planning files (that work closed 2026-02-17).

🤖 Generated with Claude Code

https://claude.ai/code/session_01HN1tMTLezPVkv9eT86FLSH

[claude]

Reviewed the workflow, detect_changes.R, s3_sync-ci.sh, stac_utils.py, and urls_reconcile.py. Overall the control flow is careful — no --delete in CI sync, items-before-collection upload order, validate-before-sync gating, plausibility guard against a truncated listing, and the commit-only-at-end-of-job design so a failed run doesn't persist partial state. No blocking correctness or security issues found.

One minor edge case worth a look, not blocking:

scripts/detect_changes.R: the comment (line 10-12) states the goal is "a missing package fails inside the tryCatch and exits 2 (error), not 1." But fs::dir_create("logs") and sink(log_file, split = TRUE) (lines 17-18) run before the tryCatch block, not inside it. If either of those calls fails (e.g. an unwritable workspace), Rscript exits with its default uncaught-error status (1), which the workflow's case statement reads as changes=true rather than 2 (error) — the exact ambiguity the tryCatch refactor was meant to eliminate. In practice this is low-risk (downstream steps still gate on urls_new.txt presence, so no bad commit results), but it's a gap in the stated exit-code contract.

Everything else — OIDC role usage, --exclude handling of hidden files in s3_sync-ci.sh, the BUCKET%/} trailing-slash strip, the zero-created-items gate, and the git add -A data/ / rebase / push commit sequence — looks correct.

[claude]

Reviewed the workflow, R change-detection rewrite, sync script, and reconcile helper. Didn't find correctness or security bugs — the exit-code contract (0/1/2), the plausibility guard, items-before-collection sync order, the zero-created gate, and the STAC_S3_BUCKET trailing-slash strip are all handled correctly and match what's described in the PR body.

A couple of non-blocking observations, both already implicitly acknowledged in the README triage notes:

• Validation-gate backlog growth: since item_validate.py fails the whole batch on a single invalid item, and a failed job commits nothing, one bad item each month blocks all new items from that run (not just the bad one), and the unprocessed backlog compounds with next month's growth until someone remediates via item_reprocess.py. This is intentional fail-loud design per the PR notes, just flagging that the failure mode is "whole month's batch delayed," not "one item delayed," in case that's sharper than intended.
• Cache absorbs failed URLs without a retry path: detect_changes.R writes the full fresh listing to urls_list.txt every run regardless of whether item_create.py successfully produced an item for each URL. A URL whose GeoTIFF read fails gets cached as "not new" going forward and won't be retried automatically — this is the same class of issue urls_reconcile.py was written to fix once, but nothing prevents it from recurring incrementally every month. Worth keeping urls_reconcile.py runnable as an ongoing (not just one-off) remediation tool, which it looks like it already is.

Both are design tradeoffs rather than bugs, and both have documented remediation paths in scripts/README.md. No blocking findings.