Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
25d9d1f
Further optimizations in waterabtraction module
doc78 Oct 15, 2026
18703b4
vegSum optimizations and threading layer automatic selection to OpenM…
doc78 Oct 15, 2026
fd5c488
First optional float32 implementation in routing
doc78 Oct 16, 2026
6e2a82c
additional float32 model optional implementation
doc78 Oct 19, 2026
56c6490
small fix to force omp in numba layer
doc78 Oct 19, 2026
2f22beb
f32 meteomap loader optimization
doc78 Oct 20, 2026
f73296d
Added output write in threads with unit test
doc78 Oct 21, 2026
c2e9190
Global lock in output writing
doc78 Oct 22, 2026
63a4f1f
Moved HDF5 lock into strict read and write ops
doc78 Oct 23, 2026
46e9f01
Fix to NaN values in packed data writing
doc78 Oct 26, 2026
fc2679c
Optimized packing on compressed data
doc78 Oct 26, 2026
db2ca50
Add daily aggregation from subdaily
fuchsiger Oct 27, 2026
54b678b
Added prefetch of next chunk in meteo maps loader. Fix to not packed …
doc78 Oct 28, 2026
d3fb5d0
Merge pull request #235 from ec-jrc/feature/perf_day_agg
doc78 Oct 28, 2026
b9167b3
Added aggregate unit test for unpacked variables
doc78 Oct 28, 2026
036da16
Optional use of EarthKit package to replace PCRaster in tss reporting…
doc78 Oct 30, 2026
3ecd94a
Replace PCRaster downstream/ifthen in lakes
doc78 Nov 6, 2026
512eb2b
Replace PCRaster downstream/lddrepair in structures.initial with numpy
doc78 Nov 9, 2026
aac3c02
Removed unused ReservoirSitePcr map
doc78 Nov 9, 2026
a40e9b2
Fuse per-step UpsGauges into one batched earthkit upstream.sum
doc78 Nov 9, 2026
61c7564
Added option to set earthkit-hydro rayon threads number in xml numCPU…
doc78 Nov 10, 2026
b74a313
Log Packing of Q
fuchsiger Nov 11, 2026
0f61efa
Further threading optimizations and auto-tuning of routing-substep nu…
doc78 Nov 12, 2026
d18ee11
Removed numexpr to be replaced with numba, to simplify threading opti…
doc78 Nov 12, 2026
becbd65
Fix the Aggregation rm usage instead of dict key
fuchsiger Nov 12, 2026
ac734b7
Adding previos numexpr lines to numba
doc78 Nov 12, 2026
6c72a94
Correct uz2lz writing & daily aggregation packing allowance
Nov 13, 2026
4ead322
Fix aggregate time-axis overflow; add discharge log-packing tests
Nov 13, 2026
921813e
Set numba cores to auto in xml ref settings
doc78 Nov 16, 2026
6d6e7d2
Reverted previous fix for aggregate timestep issue with correct outpu…
doc78 Nov 16, 2026
40bad9b
Merge pull request #236 from ec-jrc/feature/packingQ
doc78 Nov 16, 2026
283f97e
Set useEarthkitHydro True by default. Updated docs
doc78 Nov 17, 2026
1db63a2
Added documentation for OutputDailyMean and OutputDailySum
doc78 Nov 17, 2026
c8cf083
Removed earthkit profiling
doc78 Nov 17, 2026
53d824e
Small change to reservoir channel check for safe NaN management
doc78 Nov 17, 2026
3f2566d
Inverted test config logic for useEarthKitHydro flag
doc78 Nov 17, 2026
0e53fe1
Updated unit test aggregation for daily mean and sum
doc78 Nov 17, 2026
41f46a2
Merge pull request #237 from ec-jrc/feature/replacePCR
doc78 Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
*.sublime-project
*.sublime-workspace
.kiro
.glm53flash
.agents
.bundle
.DS_Store
.jekyll-metadata
Expand Down
26 changes: 18 additions & 8 deletions docs/3_step2_preparing-setting-file/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -751,30 +751,40 @@ Within the 'lfoptions' element of the settings file, each option is defined usin

+ **NetCDFTimeChunks**: chunking size in the time dimension. This option was implemented to optimize the loading of the meteo maps. Recommended value is “auto" but chunking size can be specified manually (e.g. "10") or set to “-1" to load the whole time series into memory (very fast but expensive in terms of memory).
+ **MapsCaching** (True or False): option designed for lisflood calibration. If set to True, all the static maps and forcings will be stored in a cache so that they don't have to be loaded by each lisflood instance. This option sets the value of NetCDFTimeChunks to "-1", meaning that the whole time series in the NetCDF inputs is loaded into memory.
+ **NetCDFTimeChunksPrefetch** (1 or 0, default 0): when set to 1, the next forcing time chunk is loaded in a background thread while the model is still computing with the current one, so the chunk boundary no longer stalls the run. This only helps when forcing reads are slow compared to the computation (e.g. forcings on a network file system); on fast local disks the benefit is negligible. It requires `readNetcdfStack=1` and has no effect when `NetCDFTimeChunks` results in a single chunk (e.g. `-1` or `MapsCaching=1`), because there is no next chunk to load. Model outputs are identical with the option on or off.

### Options to manage output files

+ **OutputMapsChunks**: this option is used to write output maps every X steps (default 1).
+ **OutputMapsDataType**: this option sets the output data type and may take the following values: "float64" or "float32" (default float64). This option applies to all output maps.

### Options to control parallelization
### Option to select the reporting engine
+ **useEarthkitHydro** (1 or 0, default 1): selects the engine used for the river-network reductions of the reporting step, i.e. upstream accumulation (`accuflux`/`catchmenttotal`-type series) and the zonal means at gauge locations written to time series. When 1, these are computed with [earthkit-hydro](https://github.com/ecmwf/earthkit-hydro) directly on the model's compressed 1D arrays, which avoids the conversions to and from 2D PCRaster maps and the single-threaded PCRaster engine; it reduces run time compared to PCRaster. When 0, the original PCRaster operations are used. If the `earthkit-hydro` package is not installed, LISFLOOD falls back to the PCRaster path automatically. Install it with `pip install earthkit-hydro` (listed in `requirements.txt`). To force the PCRaster path (for example to reproduce results of earlier versions), add to the settings file:
```xml
<setoption name="useEarthkitHydro" choice="0"/>
```
The thread pool used by earthkit-hydro can be limited with `numCPUs_earthkit` (see below).


LISFLOOD uses several independent parallelization mechanisms. They all affect run time only, never results: model outputs are identical regardless of the values below. Each option accepts a positive integer (number of threads/processes), or `0` / `"all"` / `"auto"` meaning "use all available CPU cores".
LISFLOOD uses several independent parallelization mechanisms. They all affect run time only, never results: model outputs are identical regardless of the values below. Every `numCPUs_*` option shares the same value legend: **absent or `"auto"`** selects the automatic policy (the default; each option's policy is described below), **`"all"` or `"0"`** forces all available CPU cores, and a **positive integer** sets a fixed thread/process count. Left at `auto`, LISFLOOD sizes each pool automatically from the host core count and the domain size, so most users do not need to set these at all.

These are `lfuser`/`lfbinding` variables (add them like any other `textvar`), for example:

```xml
<textvar name="numCPUs_parallelNumba" value="0"/>
<textvar name="numCPUs_parallelNumexpr" value="auto"/>
<textvar name="numCPUs_parallelNumba" value="auto"/>
<textvar name="numCPUs_routingNumba" value="auto"/>
<textvar name="numCPUs_BLAS" value="auto"/>
<textvar name="numCPUs_soilInit" value="0"/>
<textvar name="numCPUs_soilInit" value="auto"/>
<textvar name="modelPrecision" value="float64"/>
```

+ **numCPUs_parallelNumba**: number of threads used by the Numba just-in-time compiled kernels (the kinematic-wave / Muskingum-Cunge routing and the soil/vegetation loop). Value `0` uses all cores (or the `NUMBA_NUM_THREADS` environment variable if set). On small catchments, thread fork/join overhead can make multi-threaded execution slower than serial, so LISFLOOD auto-tunes this to 1 (serial) for small domains when left on `0`; large domains benefit from more threads.
+ **numCPUs_parallelNumexpr**: number of threads used by the `numexpr` library (vectorized array maths in the kinematic-wave routing). Default `auto`: a domain-size-aware choice that runs serial on small catchments (where `numexpr`'s pool gives no speedup and oversubscribes against Numba/BLAS) and uses all cores on large/global domains (where it is measurably faster). You can force a value: `1` (serial), `0`/`all` (all cores), or a specific number. The large/small cutoff is the same valid-pixel threshold used by the Numba auto-tune. **Set this to `1` under an outer parallel driver** (e.g. [lisflood-calibration](https://github.com/ec-jrc/lisflood-calibration), one LISFLOOD per DEAP individual) so each instance stays single-core.
+ **numCPUs_BLAS**: number of threads used by the BLAS/OpenMP backend of NumPy/SciPy (e.g. the linear algebra in the soil-hydraulics fitting during initialization). Default `auto`, with the same domain-size-aware behaviour and overrides as `numCPUs_parallelNumexpr`; set it to `1` under an outer parallel driver.
+ **numCPUs_parallelNumba**: threads for the **land-phase** Numba kernels (the soil/vegetation loop, water abstraction, surface routing). `auto` is domain-size-aware: serial on small catchments (where thread fork/join overhead makes multi-threaded execution slower) and all cores on large/global domains. `0`/`all` forces all cores; a positive integer sets a fixed count.
+ **numCPUs_routingNumba**: threads for the **channel-routing substep** Numba kernels (kinematic-wave / Muskingum-Cunge), capped *independently* of the land-phase pool above. These kernels run many times per model step with a small per-call payload, so beyond a modest thread count the OpenMP fork/join cost dominates and more threads make routing *slower*. `auto` applies a host-relative cap (about half the cores, clamped to the 8–16 band, never above the land-phase count); `all`/`0` removes the routing-specific cap; a positive integer sets a fixed count. This is scheduling only — routing results are identical for any thread count.
+ **numCPUs_BLAS**: threads for the BLAS/OpenMP backend of NumPy/SciPy. In LISFLOOD, BLAS is used only by the cold-start soil-hydraulics least-squares solve (a scalar per-pixel fit, no matrix algebra) and is idle during the dynamic loop, so extra BLAS threads do not speed the model up and only risk oversubscribing against the soil-init process pool. `auto` is therefore a single thread (1); `0`/`all` forces all cores; a positive integer sets a fixed count.
+ **numCPUs_soilInit**: number of worker **processes** used to solve the soil layer-2 saturation during a cold start. This is a per-pixel SciPy solve that can dominate initialization time on large/global domains (turning it from serial minutes/hours into a few minutes with many cores). This is process-based parallelism, independent of the thread options above. Default `0` (use all available cores); the solve is bit-identical regardless of the worker count.
+ **numCPUs_earthkit**: thread count of the earthkit-hydro (rayon) pool used when `useEarthkitHydro` is 1. Default (absent or `auto`) is 1, which avoids oversubscribing the node against the routing kernel threads; a positive integer caps the pool to that many threads; `0` or `all` leaves it uncapped. Thread count affects scheduling only, not results.
- **Set `numCPUs_soilInit` to `1` when an outer tool already runs many LISFLOOD instances in parallel** — most importantly [lisflood-calibration](https://github.com/ec-jrc/lisflood-calibration), which runs one LISFLOOD per DEAP individual and expects each instance to use a single core. As a safeguard, this parallelization automatically disables itself (falls back to serial) when LISFLOOD detects it is running inside a forked/daemon worker, so an accidental high value cannot cause oversubscription under calibration.
+ **modelPrecision**: floating-point precision of the LISFLOOD model state and the kinematic-wave routing kernel. This is a performance/precision lever, separate from the thread options above. Default `float64` reproduces previous LISFLOOD output bit-for-bit (use it for calibration and any bitwise-reproducibility need). Setting it to `float32` makes the routing kernel faster by halving its memory traffic; results are **not** bit-identical but stay within the LISFLOOD reference tolerance (`atol=1e-4, rtol=1e-3`) on tested catchments, with discharge and channel storage differing only at the last reported digit. The Muskingum-Cunge-Todini (MCT) routing and all mass-balance reductions always stay `float64` regardless of this setting. An unknown value falls back to `float64` (precision is never silently degraded).

### Reference settings file
In order to facilitate the preparation of the settings file, a complete example is provided [here](https://github.com/ec-jrc/lisflood-code/tree/master/src/lisfloodSettings_reference.xml). The user is encouraged to update the paths, the names of the maps and of the tables in the provided template.
Expand Down
55 changes: 50 additions & 5 deletions docs/3_step6_model-output/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,15 +110,56 @@ Values falling outside the representable range are clipped to the nearest bound,
### Variables that are never packed

- **State and "end" maps** (used to initialise warm-start runs) are always written at full floating-point precision, regardless of the `OutputPacking` setting. This preserves the exact model state needed for a bit-reproducible restart.
- Variables whose value range is too wide to pack meaningfully (for example **discharge**, which spans several orders of magnitude globally) are intentionally left unpacked to avoid an unacceptable loss of precision.
- Variables whose value range is too wide to pack meaningfully with a *linear* scale/offset (for example **discharge**, which spans several orders of magnitude globally) are intentionally excluded from `OutputPacking`. Discharge has its own dedicated logarithmic packing option, described next.

### Logarithmic packing of discharge (`OutputPackingDischarge`)

Discharge cannot be packed with the linear `scale_factor`/`add_offset` scheme without unacceptable precision loss, because on a global domain it spans several orders of magnitude (from near-zero headwater flows to hundreds of thousands of m³/s at major river mouths). A single linear step size is either far too coarse for small flows or wastes the entire integer range on the largest ones.

`OutputPackingDischarge` solves this by packing discharge in **log10 space**, which gives near-constant *relative* precision (about 0.027% per step) across the whole range.

This option is **completely independent** of `OutputPacking`. You can enable either, both, or neither:

```xml
<textvar name="OutputPackingDischarge" value="True"/>
```

- `"False"` (default): discharge is written as raw floating-point values.
- `"True"`: the `DischargeMaps` output is packed into `int16` using the logarithmic encoding below.

**Encoding.** The resolved range is `0.01` to `500000` m³/s. The transform is:

```
stored = round( (log10(q) - packing_log_min) / packing_log_scale ) + packing_int_offset
q = 10 ** ( packing_log_min + packing_log_scale * (stored - packing_int_offset) )
```

**This is a non-CF encoding**, so standard netCDF readers will **not** auto-unpack it via `scale_factor`/`add_offset`. Instead, the writer tags the variable with explicit custom attributes describing the transform so a post-processing step can invert it:

| Attribute | Meaning |
|-----------|---------|
| `packing` | `"log10"` — identifies the scheme |
| `packing_log_min` | `log10(0.01) = -2.0` |
| `packing_log_scale` | log-space step size per integer level |
| `packing_int_offset` | integer offset (`-32766`) mapped to `packing_log_min` |
| `packing_valid_min` / `packing_valid_max` | resolved physical range (`0.01` / `500000` m³/s) |

**Value handling.**
- Discharge **below 0.01 m³/s** — including zero, negative, and missing values — is stored as `_FillValue` (read back as masked/NaN). Such values carry no useful information at this precision. **Please note that same `_FillValue` is also used in masked region, thus you will need mask map to check whether the value is outside the mask or is just below 0.01**
- Discharge **above 500000 m³/s** is clipped to the maximum.
- As with any lossy packing, a small relative quantization error (~0.027% per step, well under 0.02% round-trip) is expected.

**Interaction with temporal aggregation.** Monthly/yearly **mean** discharge keeps log packing (it stays within the resolved range). **Sum**-aggregated discharge disables log packing automatically (a period sum can exceed 500000 m³/s), and is written as floating-point — the same policy used for linear packing.

## Temporal aggregation of output maps

Instead of writing a value at every model time step, LISFLOOD can aggregate selected output variables to monthly or yearly means or sums. This dramatically reduces output volume for long simulations while retaining the climatologically relevant signal.
Instead of writing a value at every model time step, LISFLOOD can aggregate selected output variables to daily, monthly or yearly means or sums. This dramatically reduces output volume for long simulations while retaining the climatologically relevant signal.

Four settings control temporal aggregation, each taking a semicolon-separated list of output variable names:
Six settings control temporal aggregation, each taking a semicolon-separated list of output variable names:

```xml
<textvar name="OutputDailyMean" value="DischargeMaps"/>
<textvar name="OutputDailySum" value="ETActMaps"/>
<textvar name="OutputMonthlyMean" value="SnowCoverMaps;DischargeMaps;UZMaps;LZMaps"/>
<textvar name="OutputMonthlySum" value="ETActMaps;SurfaceRunoffMaps"/>
<textvar name="OutputYearlyMean" value="Theta1Maps;Theta2Maps"/>
Expand All @@ -127,10 +168,14 @@ Four settings control temporal aggregation, each taking a semicolon-separated li

- **Mean** aggregation writes the average of all time steps within each period.
- **Sum** aggregation writes the accumulated total over each period.
- A value is written only at the end of a completed period (month-end or year-end). Partial periods at the end of a simulation are not written.
- A value is written only at the end of a completed period (day-end, month-end or year-end). Partial periods at the end of a simulation are not written.
- When a variable is listed for aggregation, its normal per-time-step output is suppressed to avoid duplication.

The aggregation boundaries are detected directly from the calendar date and therefore work for daily as well as sub-daily time steps.
The aggregation boundaries are detected directly from the calendar date and therefore work for daily as well as sub-daily time steps. Daily aggregation is mainly useful with sub-daily steps (e.g. 6-hourly), where it writes one value per calendar day; with a daily time step it reproduces the per-step values.

**Requirement:** temporal aggregation is only available with netCDF output (`writeNetcdf` or `writeNetcdfStack`). Listing a variable in an aggregation setting while writing PCRaster maps stops the run at start-up with an error.

**Threaded output (`outputThreads`):** when enabled, map output is written by background threads. The output is identical to synchronous output, including aggregated values and file or time-step selection: the model clock state needed for a write (step, date, end-of-run flag, reporting checkpoint) is captured when the write is staged.

**Interaction with packing:**

Expand Down
Loading
Loading