Refactor for hpc batch api pull and outbreak extraction

.Rbuildignore

@@ -0,0 +1,2 @@
+^.*\.Rproj$
+^\.Rproj\.user$

.gitignore

@@ -0,0 +1,14 @@
+.Rproj.user
+.Rhistory
+.RData
+.Ruserdata
+
+# R check artifacts
+*.Rcheck/
+
+# Generated analysis outputs — large binaries, not source
+analysis/generated_data/
+analysis/worldpop/
+
+# SLURM logs
+logs/

CLAUDE.md

@@ -0,0 +1,141 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Package Overview
+
+**OutbreakExtractR** is an R package for epidemiological cholera outbreak surveillance. It extracts and identifies cholera outbreaks from surveillance data (sourced from a PostgreSQL/Global Cholera Taxonomy Database) using a configurable operational outbreak definition.
+
+## Development Commands
+
+```r
+# Regenerate documentation (run before installing after any @param/@export changes)
+devtools::document()
+
+# Install from source
+install.packages('.', repo=NULL, type="source")
+
+# Run all tests
+devtools::test()
+
+# Run a single test file
+testthat::test_file("tests/testthat/test-clean_psql_data.R")
+
+# Load package interactively without reinstalling
+devtools::load_all()
+
+# Check the full package (CRAN-style)
+devtools::check()
+```
+
+NAMESPACE is auto-generated by roxygen2 — never edit it by hand.
+
+## Architecture
+
+The pipeline has five sequential layers:
+
+### 1. Ingestion & Cleaning (`R/clean_*.R`, `R/observation_filter.R`)
+- `clean_psql_data()` — entry point for raw PostgreSQL CSV exports; standardizes types, identifies spatial/temporal scales, flags primary records
+- `clean_location_names()` — normalizes `::`-delimited location hierarchy strings (`country::admin1::admin2::admin3::admin4`)
+- `observation_filter()` — subsets by date range, WHO region, scale, and minimum case count
+
+### 2. Normalization (`R/observation_aggregator.R`, `R/fill_*.R`, `R/set_uniform_*.R`)
+- Aggregates daily data to weekly; aligns week-start days; fills phantom zero-case weeks (±8 weeks padding) and missing location-periods from linked records
+- Time periods are always represented as `TL` (start) / `TR` (end) date pairs
+
+### 3. Threshold Calculation (`R/get_outbreak_threshold.R`, `R/get_pop.R`, `R/calculate_population_density.R`)
+- Supports four threshold modes: `fixed`, `mean_weekly_incidence`, `outbreak_dependent`, `time_restricted`
+- Population data comes from bundled `WPP2022.rda` / `WPP2024.rda` (WorldPop) and `endemic_locs_gte50nz_3ydata.rda`
+
+### 4. Outbreak Detection (`R/identify_*.R`)
+`identify_outbreaks()` is the main orchestrator. It calls:
+- `identify_epidemic_start()` — two modes: `"consecutive"` (N weeks above threshold) or `"dual_window"` (sliding window + cumulative case trigger); key params: `min_weeks_above`, `window_weeks`, `cumulative_case_threshold_ratio`
+- `identify_epidemic_tail()` — marks outbreak end after `tail_period` (default 6) consecutive weeks below threshold
+- `identify_consecutive_outbreak_data()` / `identify_epidemic_period()` — group and extract bounded outbreak windows
+- `identify_washout_period()` — labels inter-epidemic gaps
+
+### 5. Alert Generation & Post-Processing (`R/trigger_alert*.R`, `R/format_alerts.R`, `R/label_*.R`)
+`trigger_alert()` fires 17 alert types across three families:
+- **Trend**: single/double/triple week exceeding 4-week rolling mean
+- **Case count**: 2/5/10/25/50/100/250 cases in 3 consecutive weeks
+- **Cumulative**: 5/10/25/50/100/500/1000 cumulative cases in 3 weeks
+
+Functions suffixed `_obs` operate on observatory (observed) data; others operate on modeled/threshold data.
+
+## Key Conventions
+
+- **Location strings** use `::` as hierarchy delimiter (e.g., `"Chad::Borkou"`). Helper `custom_paste()` handles safe concatenation.
+- **Spatial scales**: `country`, `admin1`–`admin4` (and `admin4+`)
+- **Temporal scales**: `daily`, `multiday`, `weekly`, `multiweek`, `monthly`, `multimonth`, `yearly`, `multiyear`
+- **WHO regions**: `AFR`, `EMR`, `AMR`, `SEAR`, `EUR`, `WPR`
+- Code uses R 4.1+ lambda syntax (`\(x)`) throughout; pipe is `%>%` (magrittr)
+- Rolling windows via `slider`; RLE-based grouping via `zoo`; per-location processing via `purrr::map`
+
+## Testing
+
+Test files live in `tests/testthat/` with CSV fixtures (`outbreak_testing_data.csv`, `clean_outbreak_testing_data.csv`). Tests use **testthat edition 3**. Coverage focuses on filtering boundaries, type coercion, NA handling, aggregation correctness, and de-duplication logic.
+
+---
+
+## HPC / SLURM Analysis Layer (`analysis/`)
+
+A config-driven pipeline that parallelizes over **countries × time windows** on Yggdrasil (University of Geneva HPC). Built on the same pattern as `mpox-uvira-sprint`.
+
+### Quick start
+
+```bash
+# 1. Generate YAML configs
+Rscript analysis/00_make_configs.R
+
+# 2. Set API credentials (these propagate to SLURM tasks via --export=ALL)
+export CHOLERA_API_USERNAME=<username>
+export CHOLERA_API_KEY=<api_key>
+
+# 3. Submit Batch 1 (data pull), chain Batch 2 (outbreak detection) after it
+BATCH1=$(sbatch --parsable analysis/bash/submit_01_pull_data.sh)
+sbatch --dependency=afterok:$BATCH1 analysis/bash/submit_02_detection.sh
+
+# 4. After all jobs finish, aggregate results locally
+Rscript analysis/03_aggregate_results.R
+```
+
+### Architecture
+
+**Two SLURM batches:**
+- **Batch 1** (`submit_01_pull_data.sh`): array over `analysis/configs/pull_set/` — one task per country × time window. Pulls via `taxdat::pull_taxonomy_data(source="api")`, runs the full normalization pipeline, writes:
+  - `stage1_geo_{run_id}.parquet` — GeoParquet with geometry (via `sfarrow`)
+  - `stage1_flat_{run_id}.parquet` — flat Parquet without geometry (input to Batch 2)
+- **Batch 2** (`submit_02_detection.sh`): array over `analysis/configs/detection_set/` — one task per country. Globs all Stage 1 flat files for that country, runs `identify_outbreaks()` + `trigger_alert()` per time window, writes `stage2_{region}_{iso3}.parquet`.
+
+### Config system
+
+- `analysis/config_defaults.yml` — full parameter schema with defaults
+- `analysis/00_make_configs.R` — editable country list and time windows; generates `pull_set/` and `detection_set/` configs
+- `analysis/utils.R` — shared helpers: `write_configs()`, `make_options_from_config()`, `make_taxdat_location()`, filename functions
+
+**taxdat location format:** `"CT-World::{WHO_REGION}::{ISO3}"` (e.g. `"CT-World::AFR::COD"`).  
+**Credentials:** always via env vars `CHOLERA_API_USERNAME` / `CHOLERA_API_KEY` — never in config files.
+
+### Local dry-run workflow
+
+```bash
+# Generate a 1-country test config
+# (already created by 00_make_configs.R as configs/test_pull/ and test_detection/)
+
+# Test Batch 1
+Rscript analysis/01_pull_data.R -c analysis/configs/test_pull/test_pull_1.yml
+
+# Test Batch 2
+Rscript analysis/02_run_outbreak_detection.R -c analysis/configs/test_detection/test_detection_1.yml
+
+# SLURM dry run (no submission)
+SLURM_ARRAY_TASK_ID=0 bash analysis/bash/submit_01_pull_data.sh
+```
+
+### SLURM array bounds
+
+After running `00_make_configs.R`, it prints the exact `--array` bounds to set in the submission scripts. **Edit the `#SBATCH --array=` lines** in `submit_01_pull_data.sh` and `submit_02_detection.sh` before submitting.
+
+### GeoParquet
+
+`get_shp()` now accepts an `output_parquet` parameter — when provided, it saves the returned sf object as GeoParquet via `sfarrow::st_write_parquet()`. All `analysis/` layer outputs use `.parquet` format (`sfarrow` for spatial, `arrow` for tabular).

DESCRIPTION

@@ -7,10 +7,40 @@ Description: This package
 License: GLP-2
 Encoding: UTF-8
 Roxygen: list(markdown = TRUE)
-RoxygenNote: 7.3.2
-Suggests: 
-    testthat (>= 3.0.0)
+Imports:
+    DBI,
+    RPostgres,
+    glue,
+    sf,
+    dplyr,
+    purrr,
+    tidyr,
+    stringr,
+    tibble,
+    lubridate,
+    zoo,
+    tidyselect,
+    rlang,
+    magrittr,
+    slider,
+    curl,
+    raster,
+    exactextractr
+Suggests:
+    testthat (>= 3.0.0),
+    sfarrow,
+    arrow,
+    yaml,
+    optparse,
+    furrr,
+    future,
+    here,
+    taxdat,
+    rgeoboundaries,
+    digest,
+    withr
 Config/testthat/edition: 3
-Depends: 
+Depends:
     R (>= 2.10)
 LazyData: true
+Config/roxygen2/version: 8.1.0

NAMESPACE

@@ -9,8 +9,11 @@ export(add_outcome_bin)
 export(add_pop3_adm_columns)
 export(add_pop3_adm_columns_outbreak)
 export(add_pop6_adm_columns)
+export(add_population)
 export(add_unique_alert_ids)
 export(average_duplicate_observations)
+export(band_adj_factor)
+export(build_composite_locations)
 export(calculate_cases)
 export(calculate_population_density)
 export(clean_location_names)
@@ -19,13 +22,17 @@ export(create_alert_groups)
 export(create_alert_groups2)
 export(custom_paste)
 export(define_postprocessed_alerts)
+export(detect_duplicate_geometries)
+export(estimate_pop_for_geometries)
 export(extract_agroup_case_outcomes)
 export(extract_agroup_pers_outcomes)
 export(extract_alert_outcomes)
 export(extract_binary_case_outcomes)
 export(fill_missing_lps)
 export(fill_phantom_zeroes)
+export(filter_ms_data)
 export(format_alerts)
+export(get_country_boundary)
 export(get_epiweek)
 export(get_ghs_pop)
 export(get_outbreak_threshold)
@@ -46,11 +53,14 @@ export(label_dimensions)
 export(observation_aggregator)
 export(observation_filter)
 export(remove_consecutive_alerts)
+export(resolve_composite_children)
 export(set_uniform_wday_start)
 export(trigger_alert)
 export(trigger_alert_caseratio)
 export(trigger_alert_cases)
 export(trigger_alert_rate)
 export(trigger_alert_trends)
+export(validate_population)
+export(verify_outbreak_definitions)
 import(magrittr)
 importFrom(magrittr,"%>%")

OutbreakExtractR.Rproj

@@ -0,0 +1,20 @@
+Version: 1.0
+
+RestoreWorkspace: Default
+SaveWorkspace: Default
+AlwaysSaveHistory: Default
+
+EnableCodeIndexing: Yes
+UseSpacesForTab: Yes
+NumSpacesForTab: 2
+Encoding: UTF-8
+
+RnwWeave: Sweave
+LaTeX: pdfLaTeX
+
+AutoAppendNewline: Yes
+
+BuildType: Package
+PackageUseDevtools: Yes
+PackageInstallArgs: --no-multiarch --with-keep.source
+PackageRoxygenize: rd,collate,namespace