Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
164 commits
Select commit Hold shift + click to select a range
b38854c
Update lisfloodSettings_reference.xml for TWS output
JensenLaura Mar 13, 2025
920e179
Update Lisflood_initial.py for TWS output
JensenLaura Mar 13, 2025
e8e031c
Update Lisflood_dynamic.py for TWS output
JensenLaura Mar 13, 2025
df9dd96
Update default_options.py for TWS output
JensenLaura Mar 13, 2025
f085ec9
Update routing.py for computing river flow momentum
JensenLaura Mar 13, 2025
3c4c82f
Update evapowater.py to deal with modified LakeMask
JensenLaura Mar 13, 2025
1e524b7
Add waterstorage.py for TWS computation
JensenLaura Mar 13, 2025
6c12856
Update waterstorage.py (soil storage computation)
JensenLaura Mar 27, 2025
2d91d8f
Merge pull request #203 from JensenLaura/GFZ/feature/tws
doc78 Jul 16, 2025
a9714d5
Fix to LakeMask in evapowater and to xml files for the TWS feature
doc78 Jul 18, 2025
94eb00e
Updated temporary version number
doc78 Jul 18, 2025
8259eef
Merge branch 'development' into feature/dev_tws
doc78 Jul 18, 2025
ead07d7
Merge pull request #215 from ec-jrc/development
doc78 Mar 11, 2026
c0e1df6
Delete src/lisflood/hydrological_modules/waterstorage.py
JensenLaura May 20, 2026
c62968e
Add files via upload
JensenLaura May 20, 2026
73e2d61
Add files via upload
JensenLaura May 20, 2026
09db0ce
Delete tests/test_results.py
JensenLaura May 20, 2026
9ac656f
Add files via upload
JensenLaura May 20, 2026
7f4edbb
Delete tests/data/LF_ETRS89_UseCase/settings/base.xml
JensenLaura May 20, 2026
7c909e9
Add files via upload
JensenLaura May 20, 2026
048c257
Delete tests/data/LF_ETRS89_UseCase/reference/output_reference_daily/…
JensenLaura May 20, 2026
540f782
Add files via upload
JensenLaura May 20, 2026
6c170cc
Delete tests/data/LF_ETRS89_UseCase/reference/output_reference_6h/tws.nc
JensenLaura May 20, 2026
17bac72
Add files via upload
JensenLaura May 20, 2026
60be2bb
Add files via upload
JensenLaura May 20, 2026
4df76a1
Add files via upload
JensenLaura May 20, 2026
5e65365
Add files via upload
JensenLaura May 20, 2026
91f0a81
Add files via upload
JensenLaura May 20, 2026
822dda7
Delete tests/test_utils.py
JensenLaura May 20, 2026
d5bfdb0
Add files via upload
JensenLaura May 20, 2026
11a5eb7
Merge pull request #220 from JensenLaura/feature/dev_tws
doc78 May 20, 2026
9936a0f
introducing packing with dummy values
fuchsiger Jul 8, 2026
a0fc652
Def Options Change
fuchsiger Jul 13, 2026
c34f0bd
Def Options Change
fuchsiger Jul 13, 2026
d2d6066
Def Options Change
fuchsiger Jul 13, 2026
757061b
Def Options Change
fuchsiger Jul 13, 2026
081ec5e
Def Options Change
fuchsiger Jul 13, 2026
33e8d55
Updates to default options and fill values
fuchsiger Jul 14, 2026
964a24a
Updates to default options and fill values
fuchsiger Jul 14, 2026
12e4b1b
Update fill value
fuchsiger Jul 17, 2026
9d71096
Update fill values
fuchsiger Jul 17, 2026
ac75a1a
adding Aggregation Functions
fuchsiger Jul 27, 2026
4f0751d
Merge branch 'feature/offset_scale_outputs' of https://github.com/ec-…
fuchsiger Jul 27, 2026
f88b4ce
Update settings.xml
fuchsiger Jul 28, 2026
8d9566d
Update settings.xml
fuchsiger Jul 28, 2026
5882f24
Update
fuchsiger Jul 28, 2026
9f916c4
Debug Statement
fuchsiger Jul 28, 2026
9dc1da9
Debug Statement
fuchsiger Jul 28, 2026
3580718
Debug Statement
fuchsiger Jul 28, 2026
0af2e47
Debug Statement
fuchsiger Jul 28, 2026
6a608ee
Debug Statement
fuchsiger Jul 28, 2026
b8d835b
Debug Statement
fuchsiger Jul 28, 2026
4969862
Debug Statement
fuchsiger Jul 28, 2026
ac359c3
Update Debug
fuchsiger Jul 28, 2026
f0e6971
Update Debug
fuchsiger Jul 28, 2026
3098d63
Remove discharge output packing due to precision
fuchsiger Jul 29, 2026
174a835
TWS merge
fuchsiger Jul 29, 2026
f31f67f
Fix non nan masking outside the basin mask when aggregating outputs
fuchsiger Jul 29, 2026
bd78ac8
Merge TWS computation into LISFLOODv5
fuchsiger Jul 29, 2026
97de718
Fix Typo
fuchsiger Jul 30, 2026
d505222
Load Lakes and Res Extent Maps only when active
fuchsiger Jul 30, 2026
4bf2c31
Update Doc
fuchsiger Jul 30, 2026
47e241a
Update Docs
fuchsiger Jul 30, 2026
75e37d0
print also TWS tss via repStateUpsGauges
fuchsiger Jul 30, 2026
c22201e
Implement Carlos Comments
fuchsiger Aug 4, 2026
cd2c9f2
Rounding some scale factors
fuchsiger Aug 4, 2026
cf98283
Added test for scale,offset and aggregation. Changed some scale value…
doc78 Aug 4, 2026
434d421
Modified Ta Maps scale and offset to fit the correct range. Rounded u…
doc78 Aug 5, 2026
82e2e7c
Removed temporary tracking files
doc78 Aug 5, 2026
2cbbbe9
transpiration scale offset update
fuchsiger Aug 5, 2026
20ed76d
Update ta ranges
fuchsiger Aug 6, 2026
bca2ed9
Update invalid res/lakes
fuchsiger Aug 6, 2026
cc5e212
Update neg. range tact
fuchsiger Aug 6, 2026
f85923f
Update outputs
fuchsiger Aug 18, 2026
3f76bc2
groundwater soil wording correction in docs
fuchsiger Aug 18, 2026
6357639
Merge branch 'feature/docs' of github.com:ec-jrc/lisflood-code into f…
fuchsiger Aug 18, 2026
06dfe39
2 typos
fuchsiger Aug 28, 2026
1c3afc0
tact range for scale offset update
fuchsiger Sep 1, 2026
a28ed71
Updated TWS output filename to TotalWS to avoid confusion with old on…
doc78 Sep 1, 2026
1b39572
Fix to segmentation fault in CI
doc78 Sep 2, 2026
fe56453
fix to waterstorage and result test
doc78 Sep 3, 2026
249c1a8
Added LakeExtent and ReservoirExtent in reference xml file
doc78 Sep 3, 2026
48136f7
Fix UT ReportedTSS
doc78 Sep 3, 2026
e3483f2
added computation of repTWS option from repStorage and repTWSMaps, to…
doc78 Sep 4, 2026
41bcd40
Merge remote-tracking branch 'origin/development' into feature/offset…
doc78 Sep 7, 2026
b54fcb6
Update scale offset documentation
fuchsiger Sep 7, 2026
3597f27
Merge pull request #230 from ec-jrc/feature/offset_scale_outputs
doc78 Sep 8, 2026
e1e4234
Add temporary markdown files to .gitignore
doc78 Sep 8, 2026
9d9103c
Delete implementation_guide_scale_offset_packing.md
doc78 Sep 8, 2026
59aef51
Delete lisflood_optimization_report.md
doc78 Sep 8, 2026
c7a5575
Delete lisflood_optimization_report_v2.md
doc78 Sep 8, 2026
e4ae90d
Merge pull request #231 from ec-jrc/feature/docs
doc78 Sep 8, 2026
377b348
Adding check for lakes and reservoirs to check method.
Nov 5, 2024
4ffb459
Adding extra step in checker to remove lakes/reservoirs that are not …
Mar 12, 2025
3ea7974
Updated installer using pyproject. Generate dynamic version number
doc78 Sep 8, 2026
9946cd1
removed tox, as it is not used anymore
doc78 Sep 8, 2026
b2dc46c
Removed leftover lines from old TWS implementation.
doc78 Sep 8, 2026
957d33c
Removed unused scalar import
doc78 Sep 8, 2026
f3aaf4e
Merge branch 'development' into feature/tws_merge
doc78 Sep 8, 2026
d706f35
Merge pull request #232 from ec-jrc/feature/tws_merge
doc78 Sep 8, 2026
b94cd6c
Removed TransLossMaps duplicated entry
doc78 Sep 9, 2026
959be07
Replaced np.bool8 with np.bool
doc78 Sep 9, 2026
1e1ee64
Added parallelization script to automatically select threading based …
doc78 Sep 15, 2026
85addfb
Fix to unit tests teardown issues
doc78 Sep 15, 2026
45074b5
Fixes to unit tests teardown issues
doc78 Sep 15, 2026
5b4d2eb
Optimize soilloop.dynamic_canopy with a numba parallel kernel
doc78 Sep 15, 2026
907d621
Update parallelization docstring to reflect numba is governed here
doc78 Sep 15, 2026
beca704
Added Soil initialization optimization via multiprocess pool
doc78 Sep 16, 2026
e62095b
Added doc for initial parallelization optimizations. Added info on ba…
doc78 Sep 16, 2026
8083b33
waterabstraction code in numba to speedup execution. Updated abs tole…
doc78 Sep 18, 2026
3e7afc1
Unit test output folder isolation via worker name. Enabled unit test …
doc78 Sep 18, 2026
46485a6
Update parallelization for the max numexpr limit and the correct cpu …
doc78 Sep 22, 2026
5191990
updated parallelization to get real number of numba threads
doc78 Sep 22, 2026
37a0027
Few cleanups and improvements in parallelization
doc78 Sep 22, 2026
d78c901
fix import path
doc78 Sep 23, 2026
8d98e08
restored few comments in soilloop. Removed evalutation of FlowMomentu…
doc78 Sep 29, 2026
711b26d
removed synchronous dask configuration as it slows down loading of ch…
doc78 Sep 30, 2026
2c4e089
added prefix tmpLocalVar to TotalCrossSectionArea in routing to avoid…
doc78 Sep 30, 2026
1529a0e
Optimizations in decompress function and tss production
doc78 Oct 5, 2026
30ecf2e
Improved initialization speed. Optimization in compress decompress in…
doc78 Oct 6, 2026
000ad11
fix to cache UT, as now Gauges map is not used multiple times anymore
doc78 Oct 8, 2026
00d6b2c
Set numCPU_soilInit default value to all cores
doc78 Oct 9, 2026
cd9e72e
Set numCPU_Numexpr and numCPU_BLAS default value to all cores or seri…
doc78 Oct 9, 2026
281d31d
Merge pull request #234 from ec-jrc/feature/performance
doc78 Oct 9, 2026
25d9d1f
Further optimizations in waterabtraction module
doc78 Oct 15, 2026
18703b4
vegSum optimizations and threading layer automatic selection to OpenM…
doc78 Oct 15, 2026
fd5c488
First optional float32 implementation in routing
doc78 Oct 16, 2026
6e2a82c
additional float32 model optional implementation
doc78 Oct 19, 2026
56c6490
small fix to force omp in numba layer
doc78 Oct 19, 2026
2f22beb
f32 meteomap loader optimization
doc78 Oct 20, 2026
f73296d
Added output write in threads with unit test
doc78 Oct 21, 2026
c2e9190
Global lock in output writing
doc78 Oct 22, 2026
63a4f1f
Moved HDF5 lock into strict read and write ops
doc78 Oct 23, 2026
46e9f01
Fix to NaN values in packed data writing
doc78 Oct 26, 2026
fc2679c
Optimized packing on compressed data
doc78 Oct 26, 2026
db2ca50
Add daily aggregation from subdaily
fuchsiger Oct 27, 2026
54b678b
Added prefetch of next chunk in meteo maps loader. Fix to not packed …
doc78 Oct 28, 2026
d3fb5d0
Merge pull request #235 from ec-jrc/feature/perf_day_agg
doc78 Oct 28, 2026
b9167b3
Added aggregate unit test for unpacked variables
doc78 Oct 28, 2026
036da16
Optional use of EarthKit package to replace PCRaster in tss reporting…
doc78 Oct 30, 2026
3ecd94a
Replace PCRaster downstream/ifthen in lakes
doc78 Nov 6, 2026
512eb2b
Replace PCRaster downstream/lddrepair in structures.initial with numpy
doc78 Nov 9, 2026
aac3c02
Removed unused ReservoirSitePcr map
doc78 Nov 9, 2026
a40e9b2
Fuse per-step UpsGauges into one batched earthkit upstream.sum
doc78 Nov 9, 2026
61c7564
Added option to set earthkit-hydro rayon threads number in xml numCPU…
doc78 Nov 10, 2026
b74a313
Log Packing of Q
fuchsiger Nov 11, 2026
0f61efa
Further threading optimizations and auto-tuning of routing-substep nu…
doc78 Nov 12, 2026
d18ee11
Removed numexpr to be replaced with numba, to simplify threading opti…
doc78 Nov 12, 2026
becbd65
Fix the Aggregation rm usage instead of dict key
fuchsiger Nov 12, 2026
ac734b7
Adding previos numexpr lines to numba
doc78 Nov 12, 2026
6c72a94
Correct uz2lz writing & daily aggregation packing allowance
Nov 13, 2026
4ead322
Fix aggregate time-axis overflow; add discharge log-packing tests
Nov 13, 2026
921813e
Set numba cores to auto in xml ref settings
doc78 Nov 16, 2026
6d6e7d2
Reverted previous fix for aggregate timestep issue with correct outpu…
doc78 Nov 16, 2026
40bad9b
Merge pull request #236 from ec-jrc/feature/packingQ
doc78 Nov 16, 2026
283f97e
Set useEarthkitHydro True by default. Updated docs
doc78 Nov 17, 2026
1db63a2
Added documentation for OutputDailyMean and OutputDailySum
doc78 Nov 17, 2026
c8cf083
Removed earthkit profiling
doc78 Nov 17, 2026
53d824e
Small change to reservoir channel check for safe NaN management
doc78 Nov 17, 2026
3f2566d
Inverted test config logic for useEarthKitHydro flag
doc78 Nov 17, 2026
0e53fe1
Updated unit test aggregation for daily mean and sum
doc78 Nov 17, 2026
41f46a2
Merge pull request #237 from ec-jrc/feature/replacePCR
doc78 Nov 17, 2026
e4666d8
Merge pull request #238 from ec-jrc/feature/performance
doc78 Nov 18, 2026
ffbfca8
Small fix to deprecated numpy tostring call
doc78 Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .github/workflows/ci_env.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
- name: Install lisflood-module
shell: bash -el {0}
run: |
pip install .
pip install .[test]
- name: Check installation
shell: bash -el {0}
run: |
Expand All @@ -52,5 +52,4 @@ jobs:
- name: Test with pytest
shell: bash -el {0}
run: |
pip install pytest pytest-cov
pytest -m "slow or not slow"
15 changes: 15 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
*.gem
*.sublime-project
*.sublime-workspace
.kiro
.glm53flash
.agents
.bundle
.DS_Store
.jekyll-metadata
Expand All @@ -27,3 +30,15 @@ lisflood_model.egg-info
.vscode/
*.ipynb
.ipynb_checkpoints/
tws_integration_master_diff.md
implementation_guide_scale_offset_packing.md
lisflood_optimization_report.md
lisflood_optimization_report_v2.md
# version file auto-generated by setuptools_scm (do not track)
src/lisflood/_version.py
# output folders generated by the test suite / example runs
tests/data/**/out/
# generated profiling artifacts
profiling/
# debug variants of source modules (not part of the package)
*_debug.py
6 changes: 3 additions & 3 deletions .zenodo.json
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,9 @@
"orcid": "0000-0002-2430-3634"
},
{
"name": "Jensen, Laura",
"affiliation": "GFZ Helmholtz Centre for Geosciences",
"orcid": "0000-0002-8318-791X"
"name": "Jensen, Laura",
"affiliation": "GFZ Helmholtz Centre for Geosciences",
"orcid": "0000-0002-8318-791X"
},
{
"name": "Morrison, Oisin",
Expand Down
1 change: 0 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ COPY src/lisflood/. /lisflood/
COPY src/lisf1.py /
COPY src/lisfloodSettings_reference.xml /
COPY LICENSE /
COPY VERSION /

# RUN Tests
COPY tests/. /tests/
Expand Down
1 change: 0 additions & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
include requirements.txt
include *.xml
include VERSION
1 change: 0 additions & 1 deletion VERSION

This file was deleted.

29 changes: 29 additions & 0 deletions docs/3_step2_preparing-setting-file/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -751,12 +751,41 @@ Within the 'lfoptions' element of the settings file, each option is defined usin

+ **NetCDFTimeChunks**: chunking size in the time dimension. This option was implemented to optimize the loading of the meteo maps. Recommended value is “auto" but chunking size can be specified manually (e.g. "10") or set to “-1" to load the whole time series into memory (very fast but expensive in terms of memory).
+ **MapsCaching** (True or False): option designed for lisflood calibration. If set to True, all the static maps and forcings will be stored in a cache so that they don't have to be loaded by each lisflood instance. This option sets the value of NetCDFTimeChunks to "-1", meaning that the whole time series in the NetCDF inputs is loaded into memory.
+ **NetCDFTimeChunksPrefetch** (1 or 0, default 0): when set to 1, the next forcing time chunk is loaded in a background thread while the model is still computing with the current one, so the chunk boundary no longer stalls the run. This only helps when forcing reads are slow compared to the computation (e.g. forcings on a network file system); on fast local disks the benefit is negligible. It requires `readNetcdfStack=1` and has no effect when `NetCDFTimeChunks` results in a single chunk (e.g. `-1` or `MapsCaching=1`), because there is no next chunk to load. Model outputs are identical with the option on or off.

### Options to manage output files

+ **OutputMapsChunks**: this option is used to write output maps every X steps (default 1).
+ **OutputMapsDataType**: this option sets the output data type and may take the following values: "float64" or "float32" (default float64). This option applies to all output maps.

### Option to select the reporting engine
+ **useEarthkitHydro** (1 or 0, default 1): selects the engine used for the river-network reductions of the reporting step, i.e. upstream accumulation (`accuflux`/`catchmenttotal`-type series) and the zonal means at gauge locations written to time series. When 1, these are computed with [earthkit-hydro](https://github.com/ecmwf/earthkit-hydro) directly on the model's compressed 1D arrays, which avoids the conversions to and from 2D PCRaster maps and the single-threaded PCRaster engine; it reduces run time compared to PCRaster. When 0, the original PCRaster operations are used. If the `earthkit-hydro` package is not installed, LISFLOOD falls back to the PCRaster path automatically. Install it with `pip install earthkit-hydro` (listed in `requirements.txt`). To force the PCRaster path (for example to reproduce results of earlier versions), add to the settings file:
```xml
<setoption name="useEarthkitHydro" choice="0"/>
```
The thread pool used by earthkit-hydro can be limited with `numCPUs_earthkit` (see below).


LISFLOOD uses several independent parallelization mechanisms. They all affect run time only, never results: model outputs are identical regardless of the values below. Every `numCPUs_*` option shares the same value legend: **absent or `"auto"`** selects the automatic policy (the default; each option's policy is described below), **`"all"` or `"0"`** forces all available CPU cores, and a **positive integer** sets a fixed thread/process count. Left at `auto`, LISFLOOD sizes each pool automatically from the host core count and the domain size, so most users do not need to set these at all.

These are `lfuser`/`lfbinding` variables (add them like any other `textvar`), for example:

```xml
<textvar name="numCPUs_parallelNumba" value="auto"/>
<textvar name="numCPUs_routingNumba" value="auto"/>
<textvar name="numCPUs_BLAS" value="auto"/>
<textvar name="numCPUs_soilInit" value="auto"/>
<textvar name="modelPrecision" value="float64"/>
```

+ **numCPUs_parallelNumba**: threads for the **land-phase** Numba kernels (the soil/vegetation loop, water abstraction, surface routing). `auto` is domain-size-aware: serial on small catchments (where thread fork/join overhead makes multi-threaded execution slower) and all cores on large/global domains. `0`/`all` forces all cores; a positive integer sets a fixed count.
+ **numCPUs_routingNumba**: threads for the **channel-routing substep** Numba kernels (kinematic-wave / Muskingum-Cunge), capped *independently* of the land-phase pool above. These kernels run many times per model step with a small per-call payload, so beyond a modest thread count the OpenMP fork/join cost dominates and more threads make routing *slower*. `auto` applies a host-relative cap (about half the cores, clamped to the 8–16 band, never above the land-phase count); `all`/`0` removes the routing-specific cap; a positive integer sets a fixed count. This is scheduling only — routing results are identical for any thread count.
+ **numCPUs_BLAS**: threads for the BLAS/OpenMP backend of NumPy/SciPy. In LISFLOOD, BLAS is used only by the cold-start soil-hydraulics least-squares solve (a scalar per-pixel fit, no matrix algebra) and is idle during the dynamic loop, so extra BLAS threads do not speed the model up and only risk oversubscribing against the soil-init process pool. `auto` is therefore a single thread (1); `0`/`all` forces all cores; a positive integer sets a fixed count.
+ **numCPUs_soilInit**: number of worker **processes** used to solve the soil layer-2 saturation during a cold start. This is a per-pixel SciPy solve that can dominate initialization time on large/global domains (turning it from serial minutes/hours into a few minutes with many cores). This is process-based parallelism, independent of the thread options above. Default `0` (use all available cores); the solve is bit-identical regardless of the worker count.
+ **numCPUs_earthkit**: thread count of the earthkit-hydro (rayon) pool used when `useEarthkitHydro` is 1. Default (absent or `auto`) is 1, which avoids oversubscribing the node against the routing kernel threads; a positive integer caps the pool to that many threads; `0` or `all` leaves it uncapped. Thread count affects scheduling only, not results.
- **Set `numCPUs_soilInit` to `1` when an outer tool already runs many LISFLOOD instances in parallel** — most importantly [lisflood-calibration](https://github.com/ec-jrc/lisflood-calibration), which runs one LISFLOOD per DEAP individual and expects each instance to use a single core. As a safeguard, this parallelization automatically disables itself (falls back to serial) when LISFLOOD detects it is running inside a forked/daemon worker, so an accidental high value cannot cause oversubscription under calibration.
+ **modelPrecision**: floating-point precision of the LISFLOOD model state and the kinematic-wave routing kernel. This is a performance/precision lever, separate from the thread options above. Default `float64` reproduces previous LISFLOOD output bit-for-bit (use it for calibration and any bitwise-reproducibility need). Setting it to `float32` makes the routing kernel faster by halving its memory traffic; results are **not** bit-identical but stay within the LISFLOOD reference tolerance (`atol=1e-4, rtol=1e-3`) on tested catchments, with discharge and channel storage differing only at the last reported digit. The Muskingum-Cunge-Todini (MCT) routing and all mass-balance reductions always stay `float64` regardless of this setting. An unknown value falls back to `float64` (precision is never silently degraded).

### Reference settings file
In order to facilitate the preparation of the settings file, a complete example is provided [here](https://github.com/ec-jrc/lisflood-code/tree/master/src/lisfloodSettings_reference.xml). The user is encouraged to update the paths, the names of the maps and of the tables in the provided template.

Expand Down
137 changes: 137 additions & 0 deletions docs/3_step6_model-output/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,140 @@ Time series file have *.tss* extension and they can be opened with any text edit
It is important to be aware of the spatial domain for which each time series is computed. For instance, all *rate variables* are reported as pixel-average values. Soil moisture and groundwater storage are reported for the permeable fraction of each pixel only. The reported snow cover is the average of the snow depths in snow zones A, B and C.

This [**Annex**](../5_annex_output-files/index.md) summarises most of the options to report additional output maps.

## Output data compression (int16 packing)

By default, output maps are written as floating-point values (`float32` or `float64`, controlled by the `OutputMapsDataType` setting). For large simulations (long time series, global domains, many variables) this can produce very large files.

LISFLOOD offers an optional lossy compression that stores output maps as signed 16-bit integers (`int16`) using the CF-convention `scale_factor`/`add_offset` packing scheme. This roughly halves the file size compared to `float32` (2 bytes per value instead of 4) and reduces it fourfold compared to `float64`.

### How to enable it

Add the following to the settings file:

```xml
<textvar name="OutputPacking" value="True"/>
```

- `"False"` (default): output maps are written as raw floating-point values.
- `"True"`: output maps are packed into `int16` using per-variable `scale_factor` and `add_offset`.

No further user action is required. The `scale_factor` and `add_offset` for each variable are defined internally (see `default_options.py`). Standard netCDF readers (xarray, CDO, NCO, ncview, Panoply) automatically unpack the values on read, so from a user perspective the data appears as normal floating-point.

### Packing and unpacking formulas

When writing, each floating-point value is encoded into an integer:

```
packed_int = round( (value - add_offset) / scale_factor )
```

When reading, the CF-compliant reader restores the physical value:

```
value = packed_int * scale_factor + add_offset
```

The packed integer is stored in the range `[-32766, +32767]`. The value `-32767` is reserved as the fill value (`_FillValue`) for masked / outside-domain pixels, so it is never used for real data.

### Precision and value range

The `scale_factor` directly determines both the resolution and the representable range of a packed variable:

- **Resolution (quantization step):** equal to `scale_factor`. A value read back can differ from the original by at most `scale_factor / 2`.
- **Representable range:**
- minimum = `add_offset + scale_factor * (-32766)`
- maximum = `add_offset + scale_factor * (+32767)`

For a target physical range `[min, max]`, the parameters are chosen as:

```
scale_factor = (max - min) / 65533
add_offset = (min + max) / 2
```

(65533 is the number of usable integer levels, i.e. `32767 - (-32766)`.)

**Example** — actual soil evaporation (`ESActMaps`) uses `scale_factor = 3.1e-4`, `add_offset = 10.0`:
- minimum = `10.0 + 3.1e-4 * (-32766)` ≈ `0.0 mm/day`
- maximum = `10.0 + 3.1e-4 * (+32767)` ≈ `+20.2 mm/day`
- resolution ≈ `3.1e-4 mm/day` (values accurate to about `±1.5e-4 mm/day`)

The `add_offset` is placed at the centre of the target range (here 10.0, the midpoint of roughly 0–20 mm/day), so the available integer levels are distributed symmetrically around it.

Values falling outside the representable range are clipped to the nearest bound, and a warning is issued once per variable during the run. Because packing is lossy, a tiny quantization error is expected: for instance an input of exactly `0.0` may be read back as a value on the order of `1e-4`. This is far below any hydrologically meaningful threshold and is normal for packed datasets (the same behaviour applies to ERA5, CMIP6, and other CF-packed products).

### Variables that are never packed

- **State and "end" maps** (used to initialise warm-start runs) are always written at full floating-point precision, regardless of the `OutputPacking` setting. This preserves the exact model state needed for a bit-reproducible restart.
- Variables whose value range is too wide to pack meaningfully with a *linear* scale/offset (for example **discharge**, which spans several orders of magnitude globally) are intentionally excluded from `OutputPacking`. Discharge has its own dedicated logarithmic packing option, described next.

### Logarithmic packing of discharge (`OutputPackingDischarge`)

Discharge cannot be packed with the linear `scale_factor`/`add_offset` scheme without unacceptable precision loss, because on a global domain it spans several orders of magnitude (from near-zero headwater flows to hundreds of thousands of m³/s at major river mouths). A single linear step size is either far too coarse for small flows or wastes the entire integer range on the largest ones.

`OutputPackingDischarge` solves this by packing discharge in **log10 space**, which gives near-constant *relative* precision (about 0.027% per step) across the whole range.

This option is **completely independent** of `OutputPacking`. You can enable either, both, or neither:

```xml
<textvar name="OutputPackingDischarge" value="True"/>
```

- `"False"` (default): discharge is written as raw floating-point values.
- `"True"`: the `DischargeMaps` output is packed into `int16` using the logarithmic encoding below.

**Encoding.** The resolved range is `0.01` to `500000` m³/s. The transform is:

```
stored = round( (log10(q) - packing_log_min) / packing_log_scale ) + packing_int_offset
q = 10 ** ( packing_log_min + packing_log_scale * (stored - packing_int_offset) )
```

**This is a non-CF encoding**, so standard netCDF readers will **not** auto-unpack it via `scale_factor`/`add_offset`. Instead, the writer tags the variable with explicit custom attributes describing the transform so a post-processing step can invert it:

| Attribute | Meaning |
|-----------|---------|
| `packing` | `"log10"` — identifies the scheme |
| `packing_log_min` | `log10(0.01) = -2.0` |
| `packing_log_scale` | log-space step size per integer level |
| `packing_int_offset` | integer offset (`-32766`) mapped to `packing_log_min` |
| `packing_valid_min` / `packing_valid_max` | resolved physical range (`0.01` / `500000` m³/s) |

**Value handling.**
- Discharge **below 0.01 m³/s** — including zero, negative, and missing values — is stored as `_FillValue` (read back as masked/NaN). Such values carry no useful information at this precision. **Please note that same `_FillValue` is also used in masked region, thus you will need mask map to check whether the value is outside the mask or is just below 0.01**
- Discharge **above 500000 m³/s** is clipped to the maximum.
- As with any lossy packing, a small relative quantization error (~0.027% per step, well under 0.02% round-trip) is expected.

**Interaction with temporal aggregation.** Monthly/yearly **mean** discharge keeps log packing (it stays within the resolved range). **Sum**-aggregated discharge disables log packing automatically (a period sum can exceed 500000 m³/s), and is written as floating-point — the same policy used for linear packing.

## Temporal aggregation of output maps

Instead of writing a value at every model time step, LISFLOOD can aggregate selected output variables to daily, monthly or yearly means or sums. This dramatically reduces output volume for long simulations while retaining the climatologically relevant signal.

Six settings control temporal aggregation, each taking a semicolon-separated list of output variable names:

```xml
<textvar name="OutputDailyMean" value="DischargeMaps"/>
<textvar name="OutputDailySum" value="ETActMaps"/>
<textvar name="OutputMonthlyMean" value="SnowCoverMaps;DischargeMaps;UZMaps;LZMaps"/>
<textvar name="OutputMonthlySum" value="ETActMaps;SurfaceRunoffMaps"/>
<textvar name="OutputYearlyMean" value="Theta1Maps;Theta2Maps"/>
<textvar name="OutputYearlySum" value="TotalRunoffMaps"/>
```

- **Mean** aggregation writes the average of all time steps within each period.
- **Sum** aggregation writes the accumulated total over each period.
- A value is written only at the end of a completed period (day-end, month-end or year-end). Partial periods at the end of a simulation are not written.
- When a variable is listed for aggregation, its normal per-time-step output is suppressed to avoid duplication.

The aggregation boundaries are detected directly from the calendar date and therefore work for daily as well as sub-daily time steps. Daily aggregation is mainly useful with sub-daily steps (e.g. 6-hourly), where it writes one value per calendar day; with a daily time step it reproduces the per-step values.

**Requirement:** temporal aggregation is only available with netCDF output (`writeNetcdf` or `writeNetcdfStack`). Listing a variable in an aggregation setting while writing PCRaster maps stops the run at start-up with an error.

**Threaded output (`outputThreads`):** when enabled, map output is written by background threads. The output is identical to synchronous output, including aggregated values and file or time-step selection: the model clock state needed for a write (step, date, end-of-run flag, reporting checkpoint) is captured when the write is staged.

**Interaction with packing:**

- **Mean** aggregates stay within the same value range as the daily values, so `int16` packing (if `OutputPacking = True`) is applied as usual.
- **Sum** aggregates can greatly exceed the daily value range (a monthly sum can be up to ~30× a daily value). To prevent overflow, packing is automatically disabled for sum-aggregated outputs, which are always written as floating-point.
Loading
Loading