Skip to content

Add SMOS preprocessing python scripts - #189

Merged
gmao-rreichle merged 19 commits into
developfrom
feature/qliu/smos_preproc
Sep 3, 2026
Merged

Add SMOS preprocessing python scripts#189
gmao-rreichle merged 19 commits into
developfrom
feature/qliu/smos_preproc

Conversation

@gmao-qliu

Copy link
Copy Markdown
Contributor

Summary

Adds a new standalone pipeline that preprocesses SMOS L1C brightness temperature (Tb) data for near-real-time ingestion into MERRA21C-land:

EE .zip --(smos-ee-to-nc.sh)--> NetCDF --(preprocess_nc)--> EASEv2 M36 "REG" binaries --(SCLF1C_reg2fit)--> Tb40 "FIT" binaries

Normal operation is a daily cron job processing the previous day's data; the driver also supports on-demand backfill of a single day or an inclusive date range.

Requirements / external dependencies

  • Runs on Discover/NCCS under module load python/GEOSpyD (numpy, scipy, netCDF4, PyYAML).
  • Depends on an external, out-of-repo converter script (smos-ee-to-nc.sh, path set in config.yaml).
  • Depends on static, pre-generated regridding weights (data/GEOSIT_to_EASEv2_M36.mat) and an AUX Gal SM file — not committed to the repo; linked from production Discover paths.

Usage

module load python/GEOSpyD
python SMOSproc_main.py # NRT: process yesterday's data
python SMOSproc_main.py --date 20260801 # Backfill a single day
python SMOSproc_main.py --start 20260801 --end 20260803 # Backfill a date range

Notes for reviewers

  • This is a Python port (2021, Q. Liu) of prior MATLAB SCLF1C preprocessing code (G. De Lannoy); the science logic mirrors the original.
  • No automated tests are included; validation has been manual, comparing REG/FIT binary outputs against the legacy MATLAB pipeline.

Testing done

  • Ran end-to-end on Discover for sample backfill days and confirmed REG and FIT binaries are produced with expected file naming/paths for both ascending and descending passes.

@gmao-qliu gmao-qliu self-assigned this Aug 14, 2026
@gmao-qliu gmao-qliu added enhancement New feature or request 0-diff trivial very, very obvious 0-diff change pre-processing labels Aug 14, 2026
@gmao-rreichle gmao-rreichle changed the title Add the SMOS preprocessing python scripts Add SMOS preprocessing python scripts Aug 31, 2026

@gmao-rreichle gmao-rreichle left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gmao-qliu : Sorry this slipped through the cracks a bit. I added a few commits in an effort to make the scripts more suitable for the repo.
I don't expect the scripts to run after my mods. (I didn't even try.)
See comments below.
Please let me know if you have any questions, thanks!

Comment thread GEOSldas_App/util/shared/python/EASEv2.py
Comment thread GEOSldas_App/util/shared/python/EASEv2.py
Comment thread GEOSldas_App/util/inputs/obs_preproc/SMOS_preproc/config.yaml Outdated
@gmao-rreichle

Copy link
Copy Markdown
Collaborator

Hi @gmao-qliu, thanks for fixing up the code after my previous changes. I cleaned up a bit more. Please test the revised code one more time to make sure everything works as before, and let me know. Thanks!

@gmao-qliu

Copy link
Copy Markdown
Contributor Author

Tested and works fine.

@gmao-rreichle
gmao-rreichle marked this pull request as ready for review September 3, 2026 19:38
@gmao-rreichle
gmao-rreichle requested a review from a team as a code owner September 3, 2026 19:38
@gmao-rreichle
gmao-rreichle merged commit 3875228 into develop Sep 3, 2026
12 checks passed
@gmao-rreichle
gmao-rreichle deleted the feature/qliu/smos_preproc branch September 3, 2026 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

0-diff trivial very, very obvious 0-diff change enhancement New feature or request pre-processing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants