From cb5f254bac6d4de3b5e671e1a70f14830982e3e2 Mon Sep 17 00:00:00 2001 From: Ola Skavhaug Date: Sun, 11 Oct 2026 22:23:21 +0200 Subject: [PATCH 1/8] 26a red: MapBiomas catalogue entry, per-year fetch, ClassWindow and class strips (@tester) Failing tests for increment 26a, "MapBiomas in the catalogue, and class windows" (docs/increments/26-land-cover.md, section 9 row 26a and section 10 "26a (Python)"), written before any production code. 48 tests, all red: .venv/bin/python -m pytest tests/python/test_sources_mapbiomas.py \ tests/python/test_io_class_window.py tests/python/test_fetch_per_year.py \ tests/python/test_cli_mapbiomas.py -o addopts="" --no-cov -> 35 failed, 13 errors What each file pins and what it fails on today: - tests/python/class_fixtures.py: a 40 x 48 uint8 class raster in EPSG:4326, PixelIsArea, 16 x 16 tiles, cell step 1/128 degree (so every cell corner is exact in binary and a box edge "on a cell edge" is on it), one grid per year, a 10 x 10 no-data (0) corner, empty tiles by cog_fixtures.with_zero_byte_count; the oracle `expected` zeroes the empty tiles' rectangles by hand from tifffile's page attributes. A per-year catalogue entry for the local range server. The pins are listed in its docstring. - test_sources_mapbiomas.py (16): SOURCES["mapbiomas-c11"] (KeyError today): kind one-cog-per-year, values "classes", years 1985..2025, EPSG:4326, nodata 0, the design's URL for 2025 and 1985 through url_for, the default year the last, a year outside the range refused naming 1985 and 2025, the credit with "Collection 11", {date} and {url}, CC BY-SA 4.0, the Souza et al. citation; credit_for renders the date as ISO; anadem-v1 and glo30 hold heights with no years; check_values refuses a class source as --dem and a heights source as --land-cover, naming source, values and option. - test_io_class_window.py (20): ClassWindow (AttributeError today): read-only uint8, corner(row, col), anything but 2-D uint8 refused; decode_class_window over five windows equals the written grid, empty tiles read as zeros, the PixelIsArea corner is the cell's corner (the DEM reader's meta is half a cell in), LZW with the horizontal predictor through imagecodecs (needs_codecs), one thread and eight the same bytes, a missing block NotCached before any read; CacheRepository.load_class_window and class_strips: strips of at most `rows` rows top to bottom covering the cells whose interior meets the box (open cells), a box on cell edges, a box past the raster clipped, a box meeting no cell a plain ValueError, a missing block NotCached. - test_fetch_per_year.py (9): RemoteSource refuses the per-year kind today. Default year names brazil_coverage-col11_2025; each year its own object in one manifest; a year outside the range or a year given for a one-cog source is a FetchError before any request; empty tiles never requested and written as empty files (report.empty 2); the cached class window reads them as zeros; CachedObject.fetched is the day the header was first fetched and is kept on a later run; NOTICE.txt carries each object's credit with its date and URL. Note: FetchRequest(year=...) is silently ignored by Pydantic today, so the one-cog refusal test is red as "did not raise" until `year` exists. - test_cli_mapbiomas.py (5): `fetch --year` ("No such option" today), the object named in the output, a year outside the range a usage error naming it and the range with no request made, --year on a source without years a usage error, and `mesh --dem mapbiomas-c11` refused naming the key, "classes" and --dem with the network closed (today the unknown key is read as a path). - test_cli_fetch.py: catalogue_with split into install(sources, monkeypatch, entry), so the new CLI suite installs a per-year entry the same way; the existing 15 tests pass unchanged. No throwaway implementation: section 10 says "no throwaway implementations", so the red reasons were checked per test (--tb=line) instead. No mutation targets: section 9 names only the ledger and the reduction suites. Co-Authored-By: Claude Fable 5.1 --- tests/python/class_fixtures.py | 183 +++++++++++++++++ tests/python/test_cli_fetch.py | 7 +- tests/python/test_cli_mapbiomas.py | 115 +++++++++++ tests/python/test_fetch_per_year.py | 214 ++++++++++++++++++++ tests/python/test_io_class_window.py | 269 +++++++++++++++++++++++++ tests/python/test_sources_mapbiomas.py | 118 +++++++++++ 6 files changed, 905 insertions(+), 1 deletion(-) create mode 100644 tests/python/class_fixtures.py create mode 100644 tests/python/test_cli_mapbiomas.py create mode 100644 tests/python/test_fetch_per_year.py create mode 100644 tests/python/test_io_class_window.py create mode 100644 tests/python/test_sources_mapbiomas.py diff --git a/tests/python/class_fixtures.py b/tests/python/class_fixtures.py new file mode 100644 index 00000000..eef3e0b8 --- /dev/null +++ b/tests/python/class_fixtures.py @@ -0,0 +1,183 @@ +"""Synthetic MapBiomas-like class rasters and a per-year catalogue entry (26a). + +`docs/increments/26-land-cover.md`, section 3.1 ("The catalogue entry") and +section 10's "26a (Python)". No real MapBiomas bytes: a class raster here is a +40 x 48 `uint8` grid of MapBiomas class numbers in EPSG:4326, `PixelIsArea` +(the tie point is the upper-left cell's outer corner), 16 x 16 tiles (3 x 3 +blocks), written by tifffile through `geotiff_fixtures.micro_tiff`. Cell +`(r, c)` of year `y` holds `CLASSES[(7 r + 3 c + y) % 16]`, so two years +differ cell by cell, and the lower-left 10 x 10 cells hold 0 (no data by +MapBiomas's convention; no `GDAL_NODATA` tag, as the real file). An empty +tile (byte count 0, outside Brazil in the real file) is made with +`cog_fixtures.with_zero_byte_count`; `expected` is the pixel oracle, the +written grid with each empty tile's rectangle zeroed, sharing nothing with +the reader. + +PINNED BY THESE SUITES (the design names the parts, not their spelling): +- `tin_engine.sources.RemoteSource`: `kind` gains `"one-cog-per-year"` + (`url_template` with `{year}`); `years: tuple[int, ...] = ()`, ascending, + non-empty for that kind; `values: Literal["heights", "classes"] = + "heights"`; `url_for(year=None) -> str`, the last year by default, a + `ValueError` naming the first and last year for one outside `years`; + `credit_for(*, url, date) -> str`, the credit with `{date}` (ISO + `YYYY-MM-DD`) and `{url}` filled in, unchanged for a credit without them. + `check_values(source, wanted, option) -> None`, a `ValueError` naming the + source, its `values` and `option` when they differ. The entry is + `SOURCES["mapbiomas-c11"]`. +- `tin_engine.fetch.plan.FetchRequest.year: int | None = None`; `fetch.run` + takes the object id from `url_for(year)` as for `one-cog` + (`brazil_coverage-col11_`), refuses a year outside `years` and a year + given for a source without years (`FetchError`, before any request), and + writes the rendered credit of every object fetched into `NOTICE.txt`. +- `tin_engine.io.repository.CachedObject.fetched: datetime.date | None = + None`: the day the header was first fetched, kept on later runs. +- `tin_engine.io.models.ClassWindow(x_min, y_max, delta_x, delta_y, array)`: + frozen; `array` a read-only 2-D `uint8` copy; `rows`, `cols`; + `corner(row, col) -> (x, y)`, the upper-left corner of that cell. +- `tin_engine.io.cog.decode_class_window(source, meta, window, *, + threads=None) -> ClassWindow`: `meta` as `read_page` gives it (node + corner shifted inward half a cell), the window's corner moved back out; + an empty block is zeros; a block the source lacks is `NotCached` before any + block is read. +- `tin_engine.io.repository.CacheRepository.load_class_window(name, + window) -> ClassWindow` and `class_strips(name, box, *, rows) -> + Iterator[ClassWindow]`: the cells whose interior meets `box` (open cells, + clipped to the raster; none is a `ValueError`), top to bottom in strips of + at most `rows` rows. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from typing import Any + +import numpy as np + +from cog_fixtures import block_rectangles, page_of, with_zero_byte_count +from fetch_fixtures import padded +from geotiff_fixtures import ( + EPSG_WGS84, + GEOGRAPHIC_TYPE, + GT_MODEL_TYPE, + GT_RASTER_TYPE, + PIXEL_IS_AREA, + micro_tiff, +) +from tin_engine.io.models import Bounds + +KEY = "test-classes" +REAL_KEY = "mapbiomas-c11" +YEARS = tuple(range(1985, 2026)) +LAST_YEAR = 2025 +#: The classes seen over the basin ("What was measured", "The MapBiomas file"). +CLASSES = (3, 4, 9, 11, 12, 15, 21, 24, 25, 30, 33, 39, 41, 48, 62, 75) +ROWS, COLS = 40, 48 +TILE = (16, 16) +#: 1/128 of a degree: every cell corner is exact in binary, so a box edge put +#: on a cell edge lies on it, not one ulp to either side. +LON0, LAT0, STEP = -45.0, -15.0, 0.0078125 +AREA_KEYS = {GT_MODEL_TYPE: 2, GT_RASTER_TYPE: PIXEL_IS_AREA, GEOGRAPHIC_TYPE: EPSG_WGS84} +#: The design's credit format (section 3.1), with the test's own words. +CREDIT = "Test classes - Collection 11, accessed on {date} through the link: {url}" +LICENCE = "CC BY-SA 4.0 (test)" +CITE = ("Test citation, 2020.",) + + +#: The served path's template, as the bucket's: the year in the file name. +PATH_TEMPLATE = "mb/brazil_coverage-col11_{year}.tif" + + +def object_id(year: int) -> str: + return f"brazil_coverage-col11_{year}" + + +def served_path(year: int) -> str: + return PATH_TEMPLATE.format(year=year) + + +def class_grid(year: int = LAST_YEAR) -> np.ndarray: + r, c = np.indices((ROWS, COLS)) + grid = np.asarray(CLASSES, dtype=np.uint8)[(7 * r + 3 * c + year) % len(CLASSES)] + grid[30:, :10] = 0 + return np.ascontiguousarray(grid) + + +def class_tiff( + year: int = LAST_YEAR, + *, + empty: Sequence[int] = (), + compression: str = "deflate", + predictor: int | None = None, +) -> bytes: + """One year's raster as GeoTIFF bytes, tiles `empty` with byte count 0.""" + extra: dict[str, Any] = {} if predictor is None else {"predictor": predictor} + stream = micro_tiff( + class_grid(year), + tiepoint=(0.0, 0.0, 0.0, LON0, LAT0, 0.0), + scale=(STEP, STEP, 0.0), + geokeys=AREA_KEYS, + tile=TILE, + compression=compression, + **extra, + ) + data = stream.getvalue() + for index in empty: + data = with_zero_byte_count(data, index) + return data + + +def expected(data: bytes, year: int = LAST_YEAR) -> np.ndarray: + """The pixel oracle: the written grid, each empty tile's rectangle zeroed.""" + page = page_of(data) + grid = class_grid(year) + for index, (r0, r1, c0, c1) in enumerate(block_rectangles(page)): + if page.databytecounts[index] == 0: + grid[r0:r1, c0:c1] = 0 + return grid + + +def per_year_entry(sources: Any, url_template: str, **changes: Any) -> Any: + """A `one-cog-per-year` entry of classes, as the design's MapBiomas one.""" + fields: dict[str, Any] = { + "id": KEY, + "kind": "one-cog-per-year", + "url_template": url_template, + "years": YEARS, + "values": "classes", + "crs": "EPSG:4326", + "nodata": 0.0, + "credit": CREDIT, + "licence_note": LICENCE, + "cite": CITE, + } + return sources.RemoteSource(**(fields | changes)) + + +def serve_years( + server: Any, years: Sequence[int], *, empty: Sequence[int] = () +) -> dict[int, bytes]: + """Each year's raster served at `served_path(year)`; the bytes by year.""" + files = {year: class_tiff(year, empty=empty) for year in years} + for year, data in files.items(): + server.put(served_path(year), padded(data)) + return files + + +def whole_box() -> Bounds: + """A `Bounds` a little inside the raster's cells, in degrees.""" + return Bounds( + x_min=LON0 + 0.3 * STEP, + y_min=LAT0 - (ROWS - 0.3) * STEP, + x_max=LON0 + (COLS - 0.3) * STEP, + y_max=LAT0 - 0.3 * STEP, + ) + + +def bbox_args() -> tuple[str, str, str, str]: + """`whole_box` as `--bbox` words.""" + return ( + str(LON0 + 0.3 * STEP), + str(LAT0 - (ROWS - 0.3) * STEP), + str(LON0 + (COLS - 0.3) * STEP), + str(LAT0 - 0.3 * STEP), + ) diff --git a/tests/python/test_cli_fetch.py b/tests/python/test_cli_fetch.py index 914f25d4..9aa0cad1 100644 --- a/tests/python/test_cli_fetch.py +++ b/tests/python/test_cli_fetch.py @@ -85,8 +85,13 @@ def catalogue_with( id=KEY, kind="one-cog", url=url, crs=crs, nodata=-9999.0, credit=CREDIT, licence_note=LICENCE, cite=cite, ) # fmt: skip + return install(sources, monkeypatch, entry) + + +def install(sources: ModuleType, monkeypatch: pytest.MonkeyPatch, entry: Any) -> Any: + """`SOURCES` plus `entry` under its id, wherever a module bound the name.""" original = sources.SOURCES - patched = {**original, KEY: entry} + patched = {**original, entry.id: entry} for module in list(sys.modules.values()): name = getattr(module, "__name__", "") or "" if name.startswith("tin_engine") and getattr(module, "SOURCES", None) is original: diff --git a/tests/python/test_cli_mapbiomas.py b/tests/python/test_cli_mapbiomas.py new file mode 100644 index 00000000..14366018 --- /dev/null +++ b/tests/python/test_cli_mapbiomas.py @@ -0,0 +1,115 @@ +"""`rasputin fetch --year` and the refusals by `values` through the CLI (26a, `cli.py`). + +`docs/increments/26-land-cover.md`, section 3.1 and section 10's "26a +(Python)": `--year` on a per-year source, a year outside the range refused, +`--year` on a source without years refused, and `mesh --dem` refusing a source +of classes. The server is `fetch_fixtures.RangeServer` on 127.0.0.1, its entry +put into `SOURCES` by `monkeypatch` under `class_fixtures.KEY`; the `mesh` +refusal uses the real `mapbiomas-c11` key with the network closed, since the +refusal must come from the catalogue entry before any read. + +HOW THIS FILE GOES RED: `fetch` has no `--year` ("No such option", exit 2, +asserted against), `RemoteSource` refuses the per-year kind, and `mesh --dem +mapbiomas-c11` does not name a class source. +""" + +from __future__ import annotations + +import importlib +import socket +from collections.abc import Iterator +from pathlib import Path +from types import ModuleType +from typing import Any + +import pytest + +from class_fixtures import ( + KEY, + PATH_TEMPLATE, + REAL_KEY, + bbox_args, + object_id, + per_year_entry, + serve_years, +) +from cli_driver import USAGE, invoke +from fetch_fixtures import RangeServer, projected +from test_cli_fetch import KEY as DEM_KEY +from test_cli_fetch import catalogue_with, install, no_network + + +@pytest.fixture(scope="module") +def sources() -> ModuleType: + return importlib.import_module("tin_engine.sources") + + +@pytest.fixture +def server() -> Iterator[RangeServer]: + with RangeServer() as running: + yield running + + +@pytest.fixture +def classes(sources: ModuleType, monkeypatch: pytest.MonkeyPatch, server: RangeServer) -> Any: + serve_years(server, [2025, 1985]) + return install(sources, monkeypatch, per_year_entry(sources, server.url(PATH_TEMPLATE))) + + +def fetched(cache: Path, *args: str) -> tuple[int, str]: + return invoke("fetch", KEY, *args, "--bbox", *bbox_args(), "--cache", str(cache)) + + +class TestFetchYear: + def test_year_names_the_object_and_writes_its_cache(self, classes: Any, tmp_path: Path) -> None: + code, output = fetched(tmp_path, "--year", "1985") + assert code == 0, output + assert object_id(1985) in output and object_id(2025) not in output, output + assert (tmp_path / KEY / object_id(1985) / "header.bin").is_file() + assert not (tmp_path / KEY / object_id(2025)).exists() + + def test_without_year_the_last_year_is_fetched(self, classes: Any, tmp_path: Path) -> None: + code, output = fetched(tmp_path) + assert code == 0, output + assert object_id(2025) in output, output + assert (tmp_path / KEY / object_id(2025) / "header.bin").is_file() + + def test_a_year_outside_the_range_is_a_usage_error_naming_the_range( + self, classes: Any, server: RangeServer, tmp_path: Path + ) -> None: + code, output = fetched(tmp_path, "--year", "1984") + assert code == USAGE, output + assert "No such option" not in output, output + assert "1984" in output and "1985" in output and "2025" in output, output + assert server.log == [] + assert not (tmp_path / KEY).exists() + + def test_year_with_a_source_without_years_is_a_usage_error( + self, sources: ModuleType, monkeypatch: pytest.MonkeyPatch, server: RangeServer, + tmp_path: Path, + ) -> None: # fmt: skip + catalogue_with(sources, monkeypatch, server.put("p.tif", projected())) + code, output = invoke( + "fetch", DEM_KEY, "--year", "2025", "--bbox", *bbox_args(), "--cache", str(tmp_path) + ) + assert code == USAGE, output + assert "No such option" not in output, output + assert "--year" in output and DEM_KEY in output, output + assert server.log == [] + + +class TestMeshRefusesClasses: + def test_dem_with_a_class_source_is_refused_naming_classes( + self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr(socket, "socket", no_network) + monkeypatch.setattr(socket, "create_connection", no_network) + out = tmp_path / "m.vtk" + code, output = invoke( + "mesh", "--dem", REAL_KEY, "--cache", str(tmp_path), "--out", str(out) + ) + assert code == USAGE, output + assert "Traceback" not in output, output + assert REAL_KEY in output and "classes" in output and "--dem" in output, output + assert "not in the cache" not in output, "refused for the wrong reason" + assert not out.exists() diff --git a/tests/python/test_fetch_per_year.py b/tests/python/test_fetch_per_year.py new file mode 100644 index 00000000..fe50c6e0 --- /dev/null +++ b/tests/python/test_fetch_per_year.py @@ -0,0 +1,214 @@ +"""Fetching a one-COG-per-year source of classes (26a, `tin_engine.fetch.run`). + +`docs/increments/26-land-cover.md`, section 3.1: the year, the object id per +year (`brazil_coverage-col11_`), empty tiles as no data, the fetch date +per object and the credit rendered with it. No network: the server is +`fetch_fixtures.RangeServer` on 127.0.0.1 serving `class_fixtures.class_tiff` +for each year, the cache is under `tmp_path`, and the oracles are the served +files' bytes and the server's request log. The pinned spellings are in +`class_fixtures`' docstring. + +HOW THIS FILE GOES RED: `RemoteSource` refuses `kind="one-cog-per-year"` +(`ValidationError` building the entry), `FetchRequest` has no `year`, and +`CachedObject` has no `fetched`; each test fails on its own and the rest of +`tests/python` collects. +""" + +from __future__ import annotations + +import datetime +import importlib +from collections.abc import Iterator +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import numpy as np +import pytest + +from class_fixtures import ( + COLS, + KEY, + PATH_TEMPLATE, + ROWS, + expected, + object_id, + per_year_entry, + serve_years, + served_path, + whole_box, +) +from cog_fixtures import same_bytes +from fetch_fixtures import PROJECTED_CRS, RangeServer, block_files, projected +from test_fetch_run import present_blocks, requested_blocks +from test_fetch_run import whole_box as dem_box +from tin_engine.io.models import IndexWindow + +OBJ = object_id(2025) +OLD = object_id(1985) +EMPTY = (4, 8) +EARLIER = datetime.date(2026, 1, 1) + + +@pytest.fixture(scope="module") +def api() -> SimpleNamespace: + return SimpleNamespace( + plan=importlib.import_module("tin_engine.fetch.plan"), + run=importlib.import_module("tin_engine.fetch.run"), + http=importlib.import_module("tin_engine.fetch.http"), + repository=importlib.import_module("tin_engine.io.repository"), + sources=importlib.import_module("tin_engine.sources"), + ) + + +@pytest.fixture +def server() -> Iterator[RangeServer]: + with RangeServer() as running: + yield running + + +def classes(api: SimpleNamespace, server: RangeServer) -> Any: + return per_year_entry(api.sources, server.url(PATH_TEMPLATE)) + + +async def fetch(api: SimpleNamespace, source: Any, root: Path, **request: Any) -> Any: + client = api.http.RangeClient(delays=(0, 0, 0)) + writer = api.repository.CacheWriter(root, source.id) + asked = api.plan.FetchRequest(source=source.id, box=whole_box(), margin=0, **request) + return await api.run.fetch(asked, source, client, writer) + + +def manifest(api: SimpleNamespace, root: Path) -> Any: + return api.repository.CacheManifest.model_validate_json( + (root / KEY / "manifest.json").read_bytes() + ) + + +def nothing_written(root: Path) -> bool: + directory = root / KEY + return not directory.exists() or all( + not block_files(p) for p in directory.iterdir() if p.is_dir() + ) + + +class TestThePerYearObject: + async def test_the_default_year_is_the_last_and_names_the_object( + self, api: SimpleNamespace, server: RangeServer, tmp_path: Path + ) -> None: + files = serve_years(server, [2025]) + source = classes(api, server) + report = await fetch(api, source, tmp_path) + assert report.objects == 1 + assert (tmp_path / KEY / OBJ).is_dir() + listed = manifest(api, tmp_path).objects + assert set(listed) == {OBJ} + assert listed[OBJ].url == source.url_for(2025) == server.url(served_path(2025)) + assert present_blocks(tmp_path / KEY / OBJ, files[2025]) == set(range(9)) + + async def test_each_year_is_its_own_object_in_one_manifest( + self, api: SimpleNamespace, server: RangeServer, tmp_path: Path + ) -> None: + files = serve_years(server, [2025, 1985]) + source = classes(api, server) + await fetch(api, source, tmp_path, year=2025) + await fetch(api, source, tmp_path, year=1985) + assert set(manifest(api, tmp_path).objects) == {OBJ, OLD} + assert present_blocks(tmp_path / KEY / OBJ, files[2025]) == set(range(9)) + assert present_blocks(tmp_path / KEY / OLD, files[1985]) == set(range(9)) + assert files[2025] != files[1985] + + async def test_a_year_outside_the_range_is_refused_before_any_request( + self, api: SimpleNamespace, server: RangeServer, tmp_path: Path + ) -> None: + serve_years(server, [2025]) + with pytest.raises(api.http.FetchError, match=r"1985.*2025") as caught: + await fetch(api, classes(api, server), tmp_path, year=1984) + assert "1984" in str(caught.value) + assert server.log == [] + assert nothing_written(tmp_path) + + async def test_a_year_given_for_a_source_without_years_is_refused( + self, api: SimpleNamespace, server: RangeServer, tmp_path: Path + ) -> None: + data = projected() + server.put("p.tif", data) + dem = api.sources.RemoteSource( + id=KEY, kind="one-cog", url=server.url("p.tif"), crs=PROJECTED_CRS, nodata=-9999.0, + credit="c", licence_note="l", + ) # fmt: skip + client = api.http.RangeClient(delays=(0, 0, 0)) + writer = api.repository.CacheWriter(tmp_path, KEY) + asked = api.plan.FetchRequest(source=KEY, box=dem_box(data), margin=0, year=2025) + with pytest.raises(api.http.FetchError, match="year"): + await api.run.fetch(asked, dem, client, writer) + assert server.log == [] + assert nothing_written(tmp_path) + + +class TestEmptyTiles: + async def test_empty_tiles_are_never_requested_and_are_empty_files( + self, api: SimpleNamespace, server: RangeServer, tmp_path: Path + ) -> None: + files = serve_years(server, [2025], empty=EMPTY) + report = await fetch(api, classes(api, server), tmp_path) + asked = requested_blocks(server, served_path(2025), files[2025]) + assert sorted(asked) == [0, 1, 2, 3, 5, 6, 7] + blocks = tmp_path / KEY / OBJ / "blocks" + for name in ("1/1.bin", "2/2.bin"): # indices 4 and 8, three across + assert (blocks / name).is_file() and (blocks / name).stat().st_size == 0, name + assert (report.needed, report.empty, report.fetched) == (9, 2, 7) + + async def test_the_cached_class_window_reads_empty_tiles_as_zeros( + self, api: SimpleNamespace, server: RangeServer, tmp_path: Path + ) -> None: + files = serve_years(server, [2025], empty=EMPTY) + await fetch(api, classes(api, server), tmp_path) + repo = api.repository.CacheRepository(tmp_path, KEY, nodata=0.0) + cw = repo.load_class_window(OBJ, IndexWindow(row0=0, col0=0, rows=ROWS, cols=COLS)) + assert same_bytes(np.asarray(cw.array), expected(files[2025])) + + +class TestTheFetchDate: + async def test_each_object_is_dated_the_day_its_header_was_fetched( + self, api: SimpleNamespace, server: RangeServer, tmp_path: Path + ) -> None: + serve_years(server, [2025, 1985]) + source = classes(api, server) + await fetch(api, source, tmp_path) + assert manifest(api, tmp_path).objects[OBJ].fetched == datetime.date.today() + await fetch(api, source, tmp_path, year=1985) + listed = manifest(api, tmp_path).objects + assert {o.fetched for o in listed.values()} == {datetime.date.today()} + + async def test_the_first_date_is_kept_on_a_later_run( + self, api: SimpleNamespace, server: RangeServer, tmp_path: Path + ) -> None: + files = serve_years(server, [2025]) + source = classes(api, server) + await fetch(api, source, tmp_path) + listed = manifest(api, tmp_path) + aged = listed.model_copy( + update={"objects": {OBJ: listed.objects[OBJ].model_copy(update={"fetched": EARLIER})}} + ) + (tmp_path / KEY / "manifest.json").write_text(aged.model_dump_json()) + server.reset() + await fetch(api, source, tmp_path) + assert manifest(api, tmp_path).objects[OBJ].fetched == EARLIER + assert server.block_ranges(served_path(2025)) == [] + assert present_blocks(tmp_path / KEY / OBJ, files[2025]) == set(range(9)) + + async def test_the_notice_carries_each_objects_credit_with_its_date( + self, api: SimpleNamespace, server: RangeServer, tmp_path: Path + ) -> None: + serve_years(server, [2025, 1985]) + source = classes(api, server) + today = datetime.date.today() + await fetch(api, source, tmp_path) + notice = (tmp_path / KEY / "NOTICE.txt").read_text() + assert source.credit_for(url=source.url_for(2025), date=today) in notice + assert "{date}" not in notice and "{url}" not in notice + assert source.licence_note in notice and source.cite[0] in notice + await fetch(api, source, tmp_path, year=1985) + notice = (tmp_path / KEY / "NOTICE.txt").read_text() + assert source.credit_for(url=source.url_for(2025), date=today) in notice + assert source.credit_for(url=source.url_for(1985), date=today) in notice diff --git a/tests/python/test_io_class_window.py b/tests/python/test_io_class_window.py new file mode 100644 index 00000000..9e6982ea --- /dev/null +++ b/tests/python/test_io_class_window.py @@ -0,0 +1,269 @@ +"""`ClassWindow`: `uint8` class blocks, empty tiles as zeros, `PixelIsArea` corners, +strips over a box (26a, `tin_engine.io.models`, `io.cog`, `io.repository`). + +`docs/increments/26-land-cover.md`, section 3.1 ("Reading") and section 10's +"26a (Python)". The pixel oracle is `class_fixtures.expected`: the grid as +written, each empty tile's rectangle zeroed by hand from tifffile's page +attributes, sharing nothing with the reader. The pinned spellings are in +`class_fixtures`' docstring. + +HOW THIS FILE GOES RED: `ClassWindow`, `decode_class_window`, +`load_class_window` and `class_strips` are reached through fixtures or +`getattr` at call time (`AttributeError`), so each test fails on its own and +the rest of `tests/python` collects. +""" + +from __future__ import annotations + +import importlib +import io +from pathlib import Path +from types import ModuleType +from typing import Any + +import numpy as np +import pytest + +from class_fixtures import ( + COLS, + KEY, + LAT0, + LON0, + ROWS, + STEP, + class_grid, + class_tiff, + expected, + object_id, +) +from cog_fixtures import AbsentBlocks, BytesBlocks, same_bytes, sliced, write_cache +from geotiff_fixtures import needs_codecs +from tin_engine.io.models import Bounds, IndexWindow + +OBJ = object_id(2025) +EMPTY = (4, 8) # the middle tile and the last (bottom-right) one +#: Degrees; the coordinates are of magnitude 45, checked at 45. +DEGREES = 1e-9 + + +@pytest.fixture(scope="module") +def models() -> ModuleType: + return importlib.import_module("tin_engine.io.models") + + +@pytest.fixture(scope="module") +def cog() -> ModuleType: + return importlib.import_module("tin_engine.io.cog") + + +@pytest.fixture(scope="module") +def repository() -> ModuleType: + return importlib.import_module("tin_engine.io.repository") + + +def meta_of(data: bytes) -> Any: + from tin_engine.io.geotiff import read_page + + return read_page(io.BytesIO(data), nodata=0.0)[0] + + +def window(row0: int, col0: int, rows: int, cols: int) -> IndexWindow: + return IndexWindow(row0=row0, col0=col0, rows=rows, cols=cols) + + +def decode(cog: ModuleType, source: Any, data: bytes, w: IndexWindow, **kw: Any) -> Any: + return cog.decode_class_window(source, meta_of(data), w, **kw) + + +class TestTheModel: + def test_a_class_window_is_a_read_only_uint8_grid_with_its_cells_corner( + self, models: ModuleType + ) -> None: + grid = class_grid() + cw = models.ClassWindow(x_min=LON0, y_max=LAT0, delta_x=STEP, delta_y=STEP, array=grid) + assert cw.array.dtype == np.uint8 and not cw.array.flags.writeable + assert (cw.rows, cw.cols) == (ROWS, COLS) + assert same_bytes(np.asarray(cw.array), grid) + assert cw.corner(0, 0) == (LON0, LAT0) + x, y = cw.corner(5, 7) + assert x == pytest.approx(LON0 + 7 * STEP, abs=DEGREES) + assert y == pytest.approx(LAT0 - 5 * STEP, abs=DEGREES) + grid[0, 0] = 255 # the caller's buffer is not shared + assert cw.array[0, 0] != 255 + + @pytest.mark.parametrize( + "array", + [class_grid().astype(np.float32), class_grid()[0], class_grid().astype(np.int16)], + ids=["float32", "one_dimension", "int16"], + ) + def test_anything_but_a_2d_uint8_array_is_refused( + self, models: ModuleType, array: np.ndarray + ) -> None: + with pytest.raises(ValueError, match="uint8"): + models.ClassWindow(x_min=LON0, y_max=LAT0, delta_x=STEP, delta_y=STEP, array=array) + + +class TestDecodeClassWindow: + @pytest.mark.parametrize( + "bounds", + [(0, 0, ROWS, COLS), (2, 3, 5, 6), (10, 12, 20, 20), (16, 32, 16, 16), (39, 47, 1, 1)], + ids=["whole", "inside_one_tile", "across_four", "on_tile_edges", "last_cell"], + ) + def test_a_window_is_the_grids_slice_in_uint8( + self, cog: ModuleType, bounds: tuple[int, int, int, int] + ) -> None: + data = class_tiff() + w = window(*bounds) + cw = decode(cog, BytesBlocks(data), data, w) + assert cw.array.dtype == np.uint8 + assert same_bytes(np.asarray(cw.array), sliced(expected(data), w)) + + def test_empty_tiles_read_as_zeros(self, cog: ModuleType) -> None: + data = class_tiff(empty=EMPTY) + oracle = expected(data) + assert not same_bytes(oracle, class_grid()), "the emptied tiles must change the oracle" + cw = decode(cog, BytesBlocks(data), data, window(0, 0, ROWS, COLS)) + assert same_bytes(np.asarray(cw.array), oracle) + assert not np.asarray(cw.array)[16:32, 16:32].any() + assert np.asarray(cw.array)[:16, :16].all() + + def test_pixel_is_area_the_corner_is_the_cells_corner_not_its_centre( + self, cog: ModuleType + ) -> None: + data = class_tiff() + meta = meta_of(data) + assert meta.pixel_is_area and meta.x_min == pytest.approx(LON0 + STEP / 2, abs=DEGREES) + cw = decode(cog, BytesBlocks(data), data, window(5, 7, 10, 9)) + assert cw.x_min == pytest.approx(LON0 + 7 * STEP, abs=DEGREES) + assert cw.y_max == pytest.approx(LAT0 - 5 * STEP, abs=DEGREES) + assert (cw.delta_x, cw.delta_y) == (STEP, STEP) + assert (cw.rows, cw.cols) == (10, 9) + whole = decode(cog, BytesBlocks(data), data, window(0, 0, ROWS, COLS)) + assert (whole.x_min, whole.y_max) == pytest.approx((LON0, LAT0), abs=DEGREES) + + @needs_codecs + def test_lzw_with_the_horizontal_predictor_decodes_through_imagecodecs( + self, cog: ModuleType + ) -> None: + data = class_tiff(empty=(4,), compression="lzw", predictor=2) + cw = decode(cog, BytesBlocks(data), data, window(0, 0, ROWS, COLS)) + assert same_bytes(np.asarray(cw.array), expected(data)) + + def test_one_thread_and_eight_give_the_same_bytes(self, cog: ModuleType) -> None: + data = class_tiff(empty=EMPTY) + w = window(3, 5, 30, 40) + one = decode(cog, BytesBlocks(data), data, w, threads=1) + eight = decode(cog, BytesBlocks(data), data, w, threads=8) + assert same_bytes(np.asarray(one.array), np.asarray(eight.array)) + assert same_bytes(np.asarray(one.array), sliced(expected(data), w)) + + def test_a_missing_block_is_not_cached_before_any_block_is_read(self, cog: ModuleType) -> None: + data = class_tiff() + source = AbsentBlocks(data, absent=(4,)) + with pytest.raises(cog.NotCached) as caught: + decode(cog, source, data, window(10, 12, 20, 20)) # tiles 0, 1, 4, 5 + assert (caught.value.missing, caught.value.needed) == (1, 4) + + +@pytest.fixture +def cache(tmp_path: Path) -> tuple[Path, bytes]: + data = class_tiff(empty=EMPTY) + write_cache(tmp_path / "cache", KEY, {OBJ: data}, crs="EPSG:4326") + return tmp_path / "cache", data + + +def strips(repo: Any, box: Bounds, rows: int) -> list[Any]: + return list(repo.class_strips(OBJ, box, rows=rows)) + + +def stacked(windows: list[Any]) -> np.ndarray: + return np.vstack([np.asarray(w.array) for w in windows]) + + +class TestTheRepository: + def test_load_class_window_is_the_decode_from_the_cache( + self, cog: ModuleType, repository: ModuleType, cache: tuple[Path, bytes] + ) -> None: + root, data = cache + w = window(10, 12, 20, 20) + cw = repository.CacheRepository(root, KEY, nodata=0.0).load_class_window(OBJ, w) + direct = decode(cog, BytesBlocks(data), data, w) + assert same_bytes(np.asarray(cw.array), np.asarray(direct.array)) + assert (cw.x_min, cw.y_max) == (direct.x_min, direct.y_max) + assert same_bytes(np.asarray(cw.array), sliced(expected(data), w)) + + def test_strips_over_a_box_cover_its_cells_top_to_bottom_and_no_more( + self, repository: ModuleType, cache: tuple[Path, bytes] + ) -> None: + """Cells 10..23 x 4..9 (the box's corners lie inside those cells), in + strips of at most 5 rows: 5, 5 and 4.""" + root, data = cache + repo = repository.CacheRepository(root, KEY, nodata=0.0) + box = Bounds( + x_min=LON0 + 4.5 * STEP, + y_min=LAT0 - 23.5 * STEP, + x_max=LON0 + 9.5 * STEP, + y_max=LAT0 - 10.5 * STEP, + ) + got = strips(repo, box, rows=5) + assert [s.rows for s in got] == [5, 5, 4] + assert all(s.cols == 6 for s in got) + for strip in got: + assert strip.x_min == pytest.approx(LON0 + 4 * STEP, abs=DEGREES) + y = LAT0 - 10 * STEP + for strip in got: + assert strip.y_max == pytest.approx(y, abs=DEGREES) + y -= strip.rows * STEP + assert same_bytes(stacked(got), expected(data)[10:24, 4:10]) + + def test_a_box_on_cell_edges_takes_the_cells_whose_interior_it_meets( + self, repository: ModuleType, cache: tuple[Path, bytes] + ) -> None: + root, data = cache + repo = repository.CacheRepository(root, KEY, nodata=0.0) + box = Bounds( + x_min=LON0 + 5 * STEP, + y_min=LAT0 - 20 * STEP, + x_max=LON0 + 10 * STEP, + y_max=LAT0 - 10 * STEP, + ) + (strip,) = strips(repo, box, rows=100) + assert (strip.rows, strip.cols) == (10, 5) + assert strip.corner(0, 0) == pytest.approx((LON0 + 5 * STEP, LAT0 - 10 * STEP), abs=DEGREES) + assert same_bytes(np.asarray(strip.array), expected(data)[10:20, 5:10]) + + def test_a_box_past_the_rasters_edge_is_clipped_to_it( + self, repository: ModuleType, cache: tuple[Path, bytes] + ) -> None: + root, data = cache + repo = repository.CacheRepository(root, KEY, nodata=0.0) + box = Bounds(x_min=LON0 - 1.0, y_min=LAT0 - 1.0, x_max=LON0 + 2.5 * STEP, y_max=LAT0 + 1.0) + got = strips(repo, box, rows=16) + assert [s.rows for s in got] == [16, 16, 8] + assert same_bytes(stacked(got), expected(data)[:, :3]) + + def test_a_box_meeting_no_cell_is_a_value_error( + self, repository: ModuleType, cache: tuple[Path, bytes] + ) -> None: + root, _ = cache + repo = repository.CacheRepository(root, KEY, nodata=0.0) + box = Bounds(x_min=LON0 - 2.0, y_min=LAT0 - 2.0, x_max=LON0 - 1.0, y_max=LAT0 - 1.0) + with pytest.raises(ValueError) as caught: + strips(repo, box, rows=16) + assert not isinstance(caught.value, repository.CacheError) + + def test_a_missing_block_is_not_cached(self, repository: ModuleType, tmp_path: Path) -> None: + data = class_tiff() + root = tmp_path / "cache" + write_cache(root, KEY, {OBJ: data}, skip={OBJ: (4,)}, crs="EPSG:4326") + repo = repository.CacheRepository(root, KEY, nodata=0.0) + with pytest.raises(repository.NotCached): + repo.load_class_window(OBJ, window(10, 12, 20, 20)) + box = Bounds( + x_min=LON0 + 20 * STEP, + y_min=LAT0 - 30 * STEP, + x_max=LON0 + 25 * STEP, + y_max=LAT0 - 20 * STEP, + ) + with pytest.raises(repository.NotCached): + strips(repo, box, rows=5) diff --git a/tests/python/test_sources_mapbiomas.py b/tests/python/test_sources_mapbiomas.py new file mode 100644 index 00000000..2e474872 --- /dev/null +++ b/tests/python/test_sources_mapbiomas.py @@ -0,0 +1,118 @@ +"""The MapBiomas catalogue entry and the refusals by `values` (26a, `tin_engine.sources`). + +`docs/increments/26-land-cover.md`, section 3.1 ("The catalogue entry"), +section 3.2 and Ola's rulings of 2026-10-04: Collection 11, year 2025 by +default (Q1); CC BY-SA 4.0 in the licence note (Q2 (a)). Pure: no file, no +network. The pinned spellings are in `class_fixtures`' docstring. + +HOW THIS FILE GOES RED: `SOURCES` has no `mapbiomas-c11` (`KeyError`), +`RemoteSource` has no `years`, `values`, `url_for` or `credit_for`, and +`sources` has no `check_values` (`ValidationError`, `AttributeError`); each +test fails on its own and the rest of `tests/python` collects. +""" + +from __future__ import annotations + +import datetime +import importlib +from types import ModuleType +from typing import Any + +import pytest + +from class_fixtures import REAL_KEY, per_year_entry + +URL = ( + "https://storage.googleapis.com/mapbiomas-public/initiatives/brasil/collection11/" + "lulc/coverage/brazil_coverage/brazil_coverage-col11_{year}.tif" +) +DOI = "10.3390/rs12172735" +DATE = datetime.date(2026, 10, 11) + + +@pytest.fixture(scope="module") +def sources() -> ModuleType: + return importlib.import_module("tin_engine.sources") + + +@pytest.fixture +def entry(sources: ModuleType) -> Any: + return sources.SOURCES[REAL_KEY] + + +class TestTheCatalogueEntry: + def test_one_cog_per_year_of_classes_in_wgs84_with_0_as_no_data(self, entry: Any) -> None: + assert entry.id == REAL_KEY + assert entry.kind == "one-cog-per-year" + assert entry.values == "classes" + assert entry.years == tuple(range(1985, 2026)) + assert entry.url is None and entry.url_template == URL + assert entry.crs == "EPSG:4326" + assert entry.nodata == 0 + + @pytest.mark.parametrize("year", [2025, 1985]) + def test_the_url_for_a_year(self, entry: Any, year: int) -> None: + assert entry.url_for(year) == URL.format(year=year) + + def test_the_default_year_is_the_last(self, entry: Any) -> None: + assert entry.url_for() == URL.format(year=2025) + + @pytest.mark.parametrize("year", [1984, 2026]) + def test_a_year_outside_the_range_is_refused_naming_the_range( + self, entry: Any, year: int + ) -> None: + with pytest.raises(ValueError, match=r"1985.*2025") as caught: + entry.url_for(year) + assert str(year) in str(caught.value) + + def test_the_credit_licence_and_citation(self, entry: Any) -> None: + assert "Collection 11" in entry.credit + assert "{date}" in entry.credit and "{url}" in entry.credit + assert "CC BY-SA 4.0" in entry.licence_note + assert len(entry.cite) == 1 + assert DOI in entry.cite[0] and "Souza" in entry.cite[0] + + def test_the_credit_rendered_with_the_fetch_date_and_url(self, entry: Any) -> None: + url = URL.format(year=2025) + rendered = entry.credit_for(url=url, date=DATE) + assert rendered == entry.credit.format(date="2026-10-11", url=url) + assert f"accessed on 2026-10-11 through the link: {url}" in rendered + assert "{" not in rendered + + @pytest.mark.parametrize("key", ["anadem-v1", "glo30"]) + def test_the_dem_sources_hold_heights_and_have_no_years( + self, sources: ModuleType, key: str + ) -> None: + dem = sources.SOURCES[key] + assert dem.values == "heights" + assert dem.years == () + assert dem.credit_for(url="https://example.invalid/x.tif", date=DATE) == dem.credit + + def test_a_per_year_entry_needs_years_and_a_year_in_its_template( + self, sources: ModuleType + ) -> None: + with pytest.raises(ValueError, match="years"): + per_year_entry(sources, URL, years=()) + with pytest.raises(ValueError, match=r"\{year\}"): + per_year_entry(sources, "https://example.invalid/one.tif") + + +class TestRefusalsByValues: + """`--dem` takes heights only; `--land-cover` and `water --source` (26b, + 26d) take classes only; one check for all of them, named by the option.""" + + def test_a_class_source_is_refused_as_heights(self, sources: ModuleType, entry: Any) -> None: + with pytest.raises(ValueError) as caught: + sources.check_values(entry, "heights", "--dem") + message = str(caught.value) + assert REAL_KEY in message and "classes" in message and "--dem" in message + + def test_a_heights_source_is_refused_as_classes(self, sources: ModuleType) -> None: + with pytest.raises(ValueError) as caught: + sources.check_values(sources.SOURCES["anadem-v1"], "classes", "--land-cover") + message = str(caught.value) + assert "anadem-v1" in message and "heights" in message and "--land-cover" in message + + def test_a_matching_source_passes(self, sources: ModuleType, entry: Any) -> None: + assert sources.check_values(entry, "classes", "--land-cover") is None + assert sources.check_values(sources.SOURCES["anadem-v1"], "heights", "--dem") is None From 8eec354ba97afce7e072ab81e6fc87d958ebfb34 Mon Sep 17 00:00:00 2001 From: Ola Skavhaug Date: Sun, 11 Oct 2026 22:44:07 +0200 Subject: [PATCH 2/8] 26a green: MapBiomas in the catalogue, per-year fetch, ClassWindow and class strips (@developer) The green step of increment 26a (docs/increments/26-land-cover.md, section 9 row 26a; section 3.1 "The catalogue entry"; Ola's rulings of 2026-10-04: Collection 11, year 2025 by default, CC BY-SA 4.0). The 48 tests of cb5f254b pass; no test file is touched. Python only, no new dependency; the C++ core sees no file and no CRS. sources.py: `kind` gains "one-cog-per-year" (`url_template` with `{year}`, validated), `years` (ascending, the default is the last) and `values` ("heights" | "classes"). `url_for(year=None)` picks the year's file and refuses a year outside the range naming the first and last, or any year for a source without years. `credit_for(url=, date=)` fills `{date}` (ISO) and `{url}` into the credit; a credit without them is unchanged. `check_values` is the one refusal by `values`, named by the option. The entry `mapbiomas-c11` carries the design's URL template, 1985..2025, EPSG:4326, nodata 0, MapBiomas's credit format, the CC BY-SA 4.0 note and the Souza et al. (2020) citation. `notice(source, credits=())` renders the given credits instead of the entry's, so `NOTICE.txt` carries each object's fetch date and URL; with none given it is unchanged (test_fetch_run pins the old text). The date type is a Protocol (`Dated`: `isoformat()`), because test_sources_imports_pydantic_only pins this module to Pydantic alone. fetch/plan.py: `FetchRequest.year: int | None = None`. fetch/run.py: every non-tiled source takes its file from `url_for(year)`, the object id from the file name as before (`brazil_coverage-col11_`); a `ValueError` there is a `FetchError` before any request, as is `--year` on a tiled source. A new object records `fetched=date.today()`; known objects keep theirs. The notice is built from every object's rendered credit, deduplicated. io/models.py: `ClassWindow(x_min, y_max, delta_x, delta_y, array)`, frozen, a read-only uint8 copy (anything else refused naming uint8), `rows`, `cols`, `corner(row, col)`. io/cog.py: `decode_class_window(source, meta, window, *, threads=None)`: the same window checks and NotCached-before-any-read as `decode_window`, the page's dtype must be uint8, empty blocks (byte count 0) stay 0 in a zeroed canvas, the corner is the node moved out half a cell (PixelIsArea). The block decoding loop is now `_decode_into`, shared with `decode_window`, whose behaviour is unchanged. io/repository.py: `CachedObject.fetched: date | None = None`; `CacheRepository.load_class_window(name, window)` and `class_strips(name, box, *, rows)`: the cells whose open interior meets the box (floor of the lower index, ceil of the upper, so a box edge on a cell edge takes neither neighbour), clipped to the raster, a box meeting none a plain ValueError, top to bottom in strips of at most `rows` rows. cli.py: `fetch --year`, refused as a usage error before any request when the source has no years or lacks the year; `mesh --dem ` refused by `check_values` in `_dem_sources`, before the cache is read. Checks run: the four 26a suites (48 passed), test_fetch_run, test_cli_fetch, test_cli_mesh_cache, test_refine_golden (.vtk unchanged) and test_layering (no new rows); mypy strict, ruff check and format; check_prohibited_deps, check_detria_boundary, check_citations. Whole Python suite: 7191 passed, 17 skipped, 1 failed on the datetime import, fixed as above and re-run. Co-Authored-By: Claude Fable 5.1 --- src_python/tin_engine/cli.py | 19 +++++- src_python/tin_engine/fetch/plan.py | 2 + src_python/tin_engine/fetch/run.py | 24 +++++-- src_python/tin_engine/io/cog.py | 62 +++++++++++++++-- src_python/tin_engine/io/models.py | 39 +++++++++++ src_python/tin_engine/io/repository.py | 49 ++++++++++++- src_python/tin_engine/sources.py | 95 +++++++++++++++++++++++--- 7 files changed, 264 insertions(+), 26 deletions(-) diff --git a/src_python/tin_engine/cli.py b/src_python/tin_engine/cli.py index 19db2920..0d314d96 100644 --- a/src_python/tin_engine/cli.py +++ b/src_python/tin_engine/cli.py @@ -138,7 +138,7 @@ stride_record, summary, ) -from tin_engine.sources import SOURCES, STATION_SOURCES +from tin_engine.sources import SOURCES, STATION_SOURCES, check_values from tin_engine.stats import PhaseClock, Report, Sizes, quality, render from tin_engine.target_grid import Block, TargetGrid from tin_engine.tolerance_field import ToleranceLines, ToleranceSlope, line_segments @@ -1391,6 +1391,12 @@ def fetch( bool, typer.Option("--refresh", help="Discard the source's cache and fetch anew.") ] = False, connections: Annotated[int, typer.Option("--connections", min=1)] = 8, + year: Annotated[ + int | None, + typer.Option( + "--year", help="For a source with one file per year: which. Default: the last." + ), + ] = None, ) -> None: """Copy what a mesh of the domain reads from a remote source into the tile cache (increment 23a-2), so ``rasputin mesh --dem SOURCE`` runs offline.""" @@ -1403,6 +1409,11 @@ def fetch( if source not in SOURCES: raise typer.BadParameter(f"{source} is not a catalogue source ({', '.join(SOURCES)})") + if year is not None: # 26a: refused here, before any request + try: + SOURCES[source].url_for(year) + except ValueError as exc: + raise typer.BadParameter(str(exc), param_hint="--year") from exc if (domain is None) == (bbox is None): raise typer.BadParameter("give exactly one of --domain and --bbox", param_hint="--bbox") root = cache_root(cache, os.environ) @@ -1418,7 +1429,7 @@ def fetch( box = None if bbox is None else Bounds(**dict(zip(corners, bbox, strict=True))) request = FetchRequest( source=source, domain=given, box=box, out_crs=out_crs, connections=connections, - dry_run=dry_run, refresh=refresh, + dry_run=dry_run, refresh=refresh, year=year, ) # fmt: skip except DomainError as exc: raise typer.BadParameter(str(exc), param_hint="--domain") from exc @@ -1505,6 +1516,10 @@ def _dem_sources(dem: list[str], root: Path | None) -> tuple[tuple[Path, ...], C f"{keys[0]} is a catalogue source; give it alone, not with other --dem", param_hint="--dem", ) + try: + check_values(SOURCES[keys[0]], "heights", "--dem") # 26a: a class source is not a DEM + except ValueError as exc: + raise typer.BadParameter(str(exc), param_hint="--dem") from exc if root is None: raise typer.BadParameter( "no cache: set RASPUTIN_DATA (the data root; the cache is $RASPUTIN_DATA/cache) " diff --git a/src_python/tin_engine/fetch/plan.py b/src_python/tin_engine/fetch/plan.py index d854d076..fc021837 100644 --- a/src_python/tin_engine/fetch/plan.py +++ b/src_python/tin_engine/fetch/plan.py @@ -47,6 +47,8 @@ class FetchRequest(BaseModel): connections: int = 8 dry_run: bool = False refresh: bool = False + #: The year of a one-cog-per-year source (26a); None is the source's last. + year: int | None = None @model_validator(mode="after") def _one_region(self) -> FetchRequest: diff --git a/src_python/tin_engine/fetch/run.py b/src_python/tin_engine/fetch/run.py index 8a6761a1..0d8f4e63 100644 --- a/src_python/tin_engine/fetch/run.py +++ b/src_python/tin_engine/fetch/run.py @@ -9,6 +9,10 @@ blocks, each request's only after its whole body arrived and every length matched (decided 4); then the request appended to the manifest. A dry run reads headers and the cache, and writes nothing. + +Increment 26a: a `one-cog-per-year` source is fetched as `one-cog`, the file +being the requested year's (`url_for`); each object records the day its header +was first fetched, and `NOTICE.txt` carries every object's credit with it. """ from __future__ import annotations @@ -95,12 +99,17 @@ async def header(object_id: str, url: str) -> Header: return await asyncio.to_thread(_header, client, source, object_id, url, entries) no_tile: tuple[str, ...] = () - if source.kind == "one-cog": - assert source.url is not None - object_id = source.url.rsplit("/", 1)[-1].split("?")[0].rsplit(".", 1)[0] - headers = {object_id: (source.url, await header(object_id, source.url))} + if source.kind != "cog-tiles": + try: + url = source.url_for(request.year) + except ValueError as exc: + raise FetchError(str(exc)) from exc + object_id = url.rsplit("/", 1)[-1].split("?")[0].rsplit(".", 1)[0] + headers = {object_id: (url, await header(object_id, url))} box = source_box(request, headers[object_id][1][3]) else: + if request.year is not None: + raise FetchError(f"{source.id} has no years; --year does not apply to it") box, listed, unlisted = await _tiles(request, source, client) no_tile = tuple(unlisted) assert source.url_template is not None @@ -145,6 +154,7 @@ async def header(object_id: str, url: str) -> Header: header_sha256=hashlib.sha256(prefix).hexdigest(), header_bytes=len(prefix), block_shape=tuple(int(n) for n in page.chunks[:2]), + fetched=datetime.date.today(), ) manifest = CacheManifest( source=source.id, @@ -155,7 +165,11 @@ async def header(object_id: str, url: str) -> Header: ) if new or known is None: writer.put_manifest(manifest) - writer.put_notice(notice(source)) + credits = dict.fromkeys( + source.credit_for(url=e.url, date=e.fetched or datetime.date.today()) + for e in entries.values() + ) + writer.put_notice(notice(source, tuple(credits))) for object_id, page, index in empty: writer.put_block(object_id, page, index, b"") diff --git a/src_python/tin_engine/io/cog.py b/src_python/tin_engine/io/cog.py index 8ba924cf..86ae5d16 100644 --- a/src_python/tin_engine/io/cog.py +++ b/src_python/tin_engine/io/cog.py @@ -11,6 +11,10 @@ refused before any block is decoded (decided 6), and a sparse block (byte count 0, which tifffile would fill with 0, a valid elevation) is refused (decided 5): a DEM's NoData must be stored, not implied. + +Increment 26a adds `decode_class_window` for a raster of class numbers +(`uint8`, `PixelIsArea`): the same blocks, but an empty block is a run of 0, +no data by MapBiomas's convention, and the window's corner is the cell's. """ from __future__ import annotations @@ -25,7 +29,7 @@ import tifffile from .geotiff import _stage -from .models import DemTile, GeoTiffError, IndexWindow, RasterMeta +from .models import ClassWindow, DemTile, GeoTiffError, IndexWindow, RasterMeta class BlockSource(Protocol): @@ -134,12 +138,61 @@ def decode_window( "a DEM's NoData must be stored, not implied" ) out: np.ndarray[Any, np.dtype[Any]] = np.empty((w.rows, w.cols), dtype=dtype) + _decode_into(source, w, indices, out, threads) + # The public constructor's copy, not the canvas's no-copy route: M15 + # (test_mosaic.py) pins that route's callers to `assemble` and `resample`. + # Dropping this copy is fix 5 of docs/research/basin-memory-options.md. + return DemTile(meta=meta.windowed(w), array=out) + + +def decode_class_window( + source: BlockSource, meta: RasterMeta, window: IndexWindow, *, threads: int | None = None +) -> ClassWindow: + """`window` of the class raster `source` holds, as `uint8` (26a). + + `meta` is the page's, as `read_page` gives it, with the upper-left node + half a cell inside the raster; the window's corner is moved back out to + the cell's. A block `source` lacks is `NotCached` before any block is + read; an empty block (byte count 0) is cells of 0, no data. + """ + w, page = window, source.page + ends = (meta.rows - w.row0 - w.rows, meta.cols - w.col0 - w.cols) + if min(w.row0, w.col0, *ends) < 0 or min(w.rows, w.cols) < 1: + raise ValueError(f"{w} is not a non-empty window of {meta.rows} x {meta.cols} cells") + if page.dtype != np.uint8: + raise GeoTiffError(f"{source.where}: classes must be uint8, the file holds {page.dtype}") + indices = blocks_meeting(page, w) + if absent := source.missing(indices): + raise NotCached(len(absent), len(indices), source.where) + out = np.zeros((w.rows, w.cols), dtype=np.uint8) + stored = [i for i in indices if page.databytecounts[i] > 0] + _decode_into(source, w, stored, out, threads) + x, y = meta.node_xy(w.row0, w.col0) + return ClassWindow( + x_min=x - meta.delta_x / 2, + y_max=y + meta.delta_y / 2, + delta_x=meta.delta_x, + delta_y=meta.delta_y, + array=out, + ) + + +def _decode_into( + source: BlockSource, + w: IndexWindow, + indices: Sequence[int], + out: np.ndarray[Any, np.dtype[Any]], + threads: int | None, +) -> None: + """Decode each of `indices` into its own slice of `out`, `w`'s canvas, on + `threads` workers (the machine's cores when None); none is sparse.""" + page = source.page def put(index: int) -> None: data = source.block(index) with _stage(f"{source.where}, block {index}"): segment, (_, _, r0, c0, _), _ = page.decode(data, index) - assert segment is not None # a sparse block was refused above + assert segment is not None # a sparse block was refused or skipped by the caller rows = slice(max(r0, w.row0), min(r0 + segment.shape[1], w.row0 + w.rows)) cols = slice(max(c0, w.col0), min(c0 + segment.shape[2], w.col0 + w.cols)) block = segment[0, rows.start - r0 : rows.stop - r0, cols.start - c0 : cols.stop - c0, 0] @@ -149,10 +202,6 @@ def put(index: int) -> None: with ThreadPoolExecutor(threads if threads is not None else os.cpu_count() or 1) as pool: list(pool.map(put, indices)) - # The public constructor's copy, not the canvas's no-copy route: M15 - # (test_mosaic.py) pins that route's callers to `assemble` and `resample`. - # Dropping this copy is fix 5 of docs/research/basin-memory-options.md. - return DemTile(meta=meta.windowed(w), array=out) __all__ = [ @@ -162,5 +211,6 @@ def put(index: int) -> None: "NotCached", "block_grid", "blocks_meeting", + "decode_class_window", "decode_window", ] diff --git a/src_python/tin_engine/io/models.py b/src_python/tin_engine/io/models.py index ae9c5680..eff16af6 100644 --- a/src_python/tin_engine/io/models.py +++ b/src_python/tin_engine/io/models.py @@ -5,6 +5,10 @@ contract: `epsg`, `nodata_source`, `pixel_is_area` and `vertical_unit_assumed` are provenance and must not be forwarded to `_core` (increment 12's adapter owns that). This module imports nothing first-party and never `_core`. + +Increment 26a adds `ClassWindow`: a window of land-cover class numbers +(`uint8`) with the corner of its upper-left *cell*, not a node: a class raster +is `PixelIsArea`, and what matters is which cell a point falls in. """ from __future__ import annotations @@ -136,6 +140,41 @@ def valid_mask(values: npt.NDArray[Any], nodata: float | None) -> npt.NDArray[np return valid if nodata is None else valid & (values != nodata) +class ClassWindow(BaseModel): + """A window of class numbers: `array[r, c]` is the class of the cell whose + upper-left corner is `corner(r, c)`; the window's own is `(x_min, y_max)`. + Frozen; `array` is a read-only `uint8` copy of what was given.""" + + model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) + + x_min: float + y_max: float + delta_x: float + delta_y: float + array: npt.NDArray[np.uint8] + + @field_validator("array") + @classmethod + def _read_only_uint8(cls, value: npt.NDArray[Any]) -> npt.NDArray[np.uint8]: + if value.ndim != 2 or value.dtype != np.uint8: + raise ValueError(f"need a 2-D uint8 array, got {value.dtype} {value.shape}") + copy = np.array(value, order="C", copy=True) + copy.flags.writeable = False + return copy.view() + + @property + def rows(self) -> int: + return int(self.array.shape[0]) + + @property + def cols(self) -> int: + return int(self.array.shape[1]) + + def corner(self, row: int, col: int) -> tuple[float, float]: + """The upper-left corner of cell `(row, col)`.""" + return self.x_min + col * self.delta_x, self.y_max - row * self.delta_y + + class TileFootprint(BaseModel): """One tile as its header describes it: the file's name, its node grid, and the dtype it decodes to (float32 unless given; S2).""" diff --git a/src_python/tin_engine/io/repository.py b/src_python/tin_engine/io/repository.py index d9eee675..b00ce2f4 100644 --- a/src_python/tin_engine/io/repository.py +++ b/src_python/tin_engine/io/repository.py @@ -19,6 +19,10 @@ Increment 29 adds :func:`read_json`, for the station and river readers (`io/station_set.py`, `io/rivers.py`), which take a path but open nothing. Audit PR C adds :func:`read_text`, for `io/domain_file.py`'s WKT domain. + +Increment 26a adds the class raster's reads from the cache +(`CacheRepository.load_class_window`, `class_strips`: the cells a box meets, +in strips) and the day each object was first fetched (`CachedObject.fetched`). """ from __future__ import annotations @@ -28,19 +32,36 @@ import hashlib import io import json +import math import os import shutil import sqlite3 -from collections.abc import Callable, Iterable, Sequence +from collections.abc import Callable, Iterable, Iterator, Sequence from pathlib import Path from typing import TYPE_CHECKING, Any, Protocol import numpy as np from pydantic import BaseModel, ConfigDict -from .cog import CacheError, LocalTiffBlocks, NotCached, block_grid, blocks_meeting, decode_window +from .cog import ( + CacheError, + LocalTiffBlocks, + NotCached, + block_grid, + blocks_meeting, + decode_class_window, + decode_window, +) from .geotiff import decode_dem, read_header, read_page -from .models import DemTile, GeoTiffError, IndexWindow, RasterMeta, TileFootprint +from .models import ( + Bounds, + ClassWindow, + DemTile, + GeoTiffError, + IndexWindow, + RasterMeta, + TileFootprint, +) if TYPE_CHECKING: import tifffile @@ -161,6 +182,8 @@ class CachedObject(BaseModel): header_sha256: str header_bytes: int block_shape: tuple[int, int] + #: The day the header was first fetched (26a); kept over later runs. + fetched: datetime.date | None = None class CacheManifest(BaseModel): @@ -248,6 +271,26 @@ def load_window(self, name: str, window: IndexWindow) -> DemTile: meta, dtype, _ = self._page(name) return decode_window(self._blocks(name), meta, dtype, window) + def load_class_window(self, name: str, window: IndexWindow) -> ClassWindow: + """`window` of the class raster `name` (26a), empty blocks as 0.""" + return decode_class_window(self._blocks(name), self._page(name)[0], window) + + def class_strips(self, name: str, box: Bounds, *, rows: int) -> Iterator[ClassWindow]: + """The cells of `name` whose interior meets `box`, clipped to the + raster, top to bottom in strips of at most `rows` rows (26a). A box + meeting no cell is a `ValueError`.""" + meta = self._page(name)[0] + x0, y0 = meta.x_min - meta.delta_x / 2, meta.y_max + meta.delta_y / 2 # the raster's corner + c0 = max(math.floor((box.x_min - x0) / meta.delta_x), 0) + c1 = min(math.ceil((box.x_max - x0) / meta.delta_x), meta.cols) + r0 = max(math.floor((y0 - box.y_max) / meta.delta_y), 0) + r1 = min(math.ceil((y0 - box.y_min) / meta.delta_y), meta.rows) + if c0 >= c1 or r0 >= r1: + raise ValueError(f"{self._source}/{name}: the box {box} meets none of its cells") + for start in range(r0, r1, rows): + window = IndexWindow(row0=start, col0=c0, rows=min(rows, r1 - start), cols=c1 - c0) + yield self.load_class_window(name, window) + def check(self, plan: MosaicPlan) -> None: """One `NotCached` for every block the plan needs and the cache lacks, before any is decoded (decided 6).""" diff --git a/src_python/tin_engine/sources.py b/src_python/tin_engine/sources.py index 58022405..84642a09 100644 --- a/src_python/tin_engine/sources.py +++ b/src_python/tin_engine/sources.py @@ -8,29 +8,47 @@ Increment 29 adds the catalogue of station lists `rasputin fetch-stations` copies (`STATION_SOURCES`): NVE's HRD stations, their catchment polygons and the river lines near them. + +Increment 26a adds a source of land-cover classes, MapBiomas, as one COG per +year (`one-cog-per-year`, `years`, `values`; `docs/increments/26-land-cover.md`, +section 3.1): `url_for` picks the year's file, `credit_for` fills the fetch +date and URL into the credit, and `check_values` keeps classes out of `--dem` +and heights out of `--land-cover`. """ from __future__ import annotations -from collections.abc import Mapping +from collections.abc import Mapping, Sequence from types import MappingProxyType -from typing import Literal +from typing import Literal, Protocol, Self + +from pydantic import BaseModel, ConfigDict, model_validator + -from pydantic import BaseModel, ConfigDict +class Dated(Protocol): + """A `datetime.date`, by its one method used here: this module imports + only Pydantic (23a-1, decided 2; `test_sources_imports_pydantic_only`).""" + + def isoformat(self) -> str: ... class RemoteSource(BaseModel): - """One source: one COG (`url`), or one COG per tile (`url_template` and - the `tile_list_url` saying which tiles exist). `crs` is the one expected, + """One source: one COG (`url`), one COG per tile (`url_template` and the + `tile_list_url` saying which tiles exist), or one COG per year + (`url_template` with `{year}`, `years`). `crs` is the one expected, checked against each header; `credit` goes into every mesh made from it.""" model_config = ConfigDict(frozen=True) id: str - kind: Literal["one-cog", "cog-tiles"] + kind: Literal["one-cog", "cog-tiles", "one-cog-per-year"] url: str | None = None url_template: str | None = None tile_list_url: str | None = None + #: The years a `one-cog-per-year` source has, ascending; the default is the last. + years: tuple[int, ...] = () + #: What the cells hold: heights for a DEM, land-cover class numbers for MapBiomas. + values: Literal["heights", "classes"] = "heights" crs: str nodata: float | None credit: str @@ -38,6 +56,42 @@ class RemoteSource(BaseModel): #: The works the distributor asks to cite (23a-2); `notice` and the mesh file carry them. cite: tuple[str, ...] = () + @model_validator(mode="after") + def _per_year_has_years(self) -> Self: + if self.kind == "one-cog-per-year": + if not self.years: + raise ValueError(f"{self.id}: a one-cog-per-year source needs its years") + if "{year}" not in (self.url_template or ""): + raise ValueError(f"{self.id}: the url_template needs a {{year}} field") + return self + + def url_for(self, year: int | None = None) -> str: + """The file to read: `url`, or the year's (the last year when None). + A year outside `years`, or any year for a source without them, is a + `ValueError` naming the range.""" + if not self.years: + if year is not None: + raise ValueError(f"{self.id} has no years; --year does not apply to it") + assert self.url is not None, f"{self.id} is not a one-file source" + return self.url + year = self.years[-1] if year is None else year + if year not in self.years: + raise ValueError( + f"{self.id} has no year {year}; it covers {self.years[0]} to {self.years[-1]}" + ) + assert self.url_template is not None + return self.url_template.format(year=year) + + def credit_for(self, *, url: str, date: Dated) -> str: + """The credit with `{date}` (ISO) and `{url}` filled in; unchanged without them.""" + return self.credit.format(date=date.isoformat(), url=url) + + +def check_values(source: RemoteSource, wanted: Literal["heights", "classes"], option: str) -> None: + """Refuse `source` for `option` unless its cells hold `wanted`.""" + if source.values != wanted: + raise ValueError(f"{option} takes a source of {wanted}; {source.id} holds {source.values}") + _GLO30 = "https://copernicus-dem-30m.s3.amazonaws.com" @@ -76,6 +130,25 @@ class RemoteSource(BaseModel): '"The organisations in charge of the Copernicus programme by law or by delegation ' 'do not incur any liability for any use of the Copernicus WorldDEM-30"', ), + "mapbiomas-c11": RemoteSource( + id="mapbiomas-c11", + kind="one-cog-per-year", + url_template="https://storage.googleapis.com/mapbiomas-public/initiatives/brasil/" + "collection11/lulc/coverage/brazil_coverage/brazil_coverage-col11_{year}.tif", + years=tuple(range(1985, 2026)), + values="classes", + crs="EPSG:4326", + nodata=0.0, + # MapBiomas's own format (terms of use); the date is the day the file was fetched. + credit="MapBiomas Project - Collection 11 of the Annual Series of Land Use and Land " + "Cover Maps of Brazil, accessed on {date} through the link: {url}", + licence_note="CC BY-SA 4.0 (Creative Commons Attribution-ShareAlike 4.0 International)", + cite=( + "Souza, C. M., et al. (2020). Reconstructing Three Decades of Land Use and Land " + "Cover Changes in Brazilian Biomes with Landsat Archive and Earth Engine. " + "Remote Sensing, 12(17), 2735. https://doi.org/10.3390/rs12172735", + ), + ), } ) @@ -114,13 +187,15 @@ class StationSource(BaseModel): ) -def notice(source: RemoteSource | StationSource) -> str: +def notice(source: RemoteSource | StationSource, credits: Sequence[str] = ()) -> str: """`//NOTICE.txt` (23a-2, decided 8): a rendering of the - catalogue entry, which stays the one place these words are kept.""" - lines = [f"{source.id}", "", "Credit:", source.credit, "", "Licence:", source.licence_note] + catalogue entry, which stays the one place these words are kept. `credits` + are the rendered credits of the objects fetched (26a), else the entry's.""" + credit = list(credits) or [source.credit] + lines = [f"{source.id}", "", "Credit:", *credit, "", "Licence:", source.licence_note] if source.cite: lines += ["", "Cite:", *source.cite] return "\n".join(lines) + "\n" -__all__ = ["SOURCES", "STATION_SOURCES", "RemoteSource", "StationSource", "notice"] +__all__ = ["SOURCES", "STATION_SOURCES", "RemoteSource", "StationSource", "check_values", "notice"] From 54cd575dbe736728aec014069a236a356b2df5c7 Mon Sep 17 00:00:00 2001 From: Ola Skavhaug Date: Sun, 11 Oct 2026 22:55:09 +0200 Subject: [PATCH 3/8] 26a as built: red and green recorded, with the Status line and ROADMAP row, by the main session Co-Authored-By: Claude Fable 5.1 --- docs/increments/26-land-cover.md | 52 +++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/docs/increments/26-land-cover.md b/docs/increments/26-land-cover.md index cd2c3e82..9097ad0c 100644 --- a/docs/increments/26-land-cover.md +++ b/docs/increments/26-land-cover.md @@ -1,6 +1,6 @@ # Increment 26: land cover for the São Francisco basin — water bodies as constraints, class fractions per triangle -Status: **designed by `@architect`, 2026-10-03, at Ola's request; ruled by +Status: **26a built** on branch `worktree-mapbiomas-26a` (red `cb5f254b`, `@tester`, 48 tests; green `8eec354b`, `@developer`, 163 net production lines by `python3 tools/count_loc.py de9b6c03 HEAD` against the estimate of 170; no mutation round, section 9; section 13 is the as-built record; next `@perf`'s quick check, then code review). 26b onward not built. The design: **by `@architect`, 2026-10-03, at Ola's request; ruled by Ola, 2026-10-04** (all eleven questions; "Ruled by Ola, 2026-10-04", below, and each question marked). Q8 was ruled against the recommendation: triangles inside a lake carry the raster's own mix, nothing is forced to @@ -1479,6 +1479,56 @@ Each with the recommendation, which is the default if Ola does not rule. - A cap on classes per triangle (measured harmful). +## 13. As built: 26a (recorded by the main session from the handbacks) + +Ola picked 26a for the night of 2026-10-11 ("note down that we should finish +the MapBiomas as well"; "go ahead on 2 and 3"). `@tester` checked the design +of 2026-10-03 against today's code first and found no staleness. + +**Red** `cb5f254b` (`@tester`): 48 tests, all red, no network and no real +MapBiomas bytes (a synthetic uint8 class raster in +`tests/python/class_fixtures.py`, so no fixture licence line): +`test_sources_mapbiomas.py` (16: the catalogue entry, the URL for 2025 and +1985, the year range, the credit with date, refusals by `values`), +`test_io_class_window.py` (20: `ClassWindow`, empty tiles as zeros, the +PixelIsArea corner, LZW plus predictor via `needs_codecs`, strips over a +box), `test_fetch_per_year.py` (9: one object per year, empty tiles, the fetch +date, the NOTICE credit), `test_cli_mapbiomas.py` (5: `fetch --year`, +`mesh --dem mapbiomas-c11` refused). Red command: `.venv/bin/python -m pytest +tests/python/test_sources_mapbiomas.py tests/python/test_io_class_window.py +tests/python/test_fetch_per_year.py tests/python/test_cli_mapbiomas.py -o +addopts="" --no-cov`, 35 failed and 13 errors. Pins beyond the design, in +`class_fixtures.py`'s docstring: `RemoteSource.years: tuple[int, ...]`, +`values: Literal["heights", "classes"]`, `url_for(year=None)`, +`credit_for(*, url, date)`, `check_values(source, wanted, option)`; +`FetchRequest.year`, year refusals as `FetchError` before any request, +`--year` on a source without years a usage error; `CachedObject.fetched: date +| None` kept on later runs, NOTICE.txt carrying each object's rendered credit; +`ClassWindow(x_min, y_max, delta_x, delta_y, array)` with `rows`, `cols`, +`corner(row, col)`; `io.cog.decode_class_window(source, meta, window, *, +threads)`; `CacheRepository.load_class_window(name, window)` and +`class_strips(name, box, *, rows)` (cells whose interior meets the box, +clipped to the raster; none is a plain `ValueError`); `mesh --dem ` refused before any cache read. A `STEP` of 1/128 degree keeps the +on-the-cell-edge cases exact. + +**Green** `8eec354b` (`@developer`): `sources.py` 49, `io/cog.py` 33, +`io/repository.py` 32, `io/models.py` 23, `cli.py` 15, `fetch/run.py` 10, +`fetch/plan.py` 1; 163 net counted lines. Whole Python suite 7192 passed, 17 +skipped; mypy strict, ruff, the governance gates green; `test_refine_golden.py` +passes (`.vtk` untouched); `test_layering.py` passes, no new rows. Pins beyond +the design: `sources.py` imports Pydantic alone (`test_cli_mesh_cache` pins +it), so `credit_for`'s date is a `Dated` protocol with `isoformat()`; +`notice(source, credits=())` lists each object's rendered credit, +deduplicated, and leaves NOTICE.txt unchanged with no credits; +`decode_class_window` refuses a page whose dtype is not uint8 +(`GeoTiffError`); strip cells are floor of the lower index and ceil of the +upper, so a box edge on a cell edge takes neither neighbour; the +block-decoding loop of `decode_window` moved into a shared `_decode_into`, +behaviour unchanged, which is why `@perf`'s quick check runs before review. +Lesson: running only the increment's suites missed an import-boundary test +elsewhere; the full suite caught it. + ## Review **Design review, round 1, 2026-10-03.** Range `origin/master...` 77f35b4, 9df3822. Verdict: CHANGES REQUESTED. LOC: 0 production lines (design and throwaway probes). The rulings of 2026-10-01 are kept; the fractions-and-ledger figures, the fetch sizes, the CRS figures, the sources and the licence conflict (terms of use CC BY-SA 4.0, factsheet CC BY) all re-checked and hold. Blocking, doc fixes only: (1) unit 769's water row omits its largest body (Três Marias joins the channel, and the probe assigns a body to a unit by its first cell, which lies outside the outline), so the basin-wide lake counts are low; (2) the "Sobradinho" ring is the main São Francisco river across unit 761 (670 × 442 km, 3,159 holes), and "346 km" is half the extent, not a length; on the whole basin the channel likely joins the reservoirs into one body; (3) the ledger measurement excludes lakes forced to 100 % water, which @perf's acceptance must cover; (4) the licence question must also name the water polygons and meshes constrained by them. Not pushed; no CI. From f51f0a187bc0319ecd0e4d4ddce523677771edbf Mon Sep 17 00:00:00 2001 From: Ola Skavhaug Date: Sun, 11 Oct 2026 22:56:16 +0200 Subject: [PATCH 4/8] ROADMAP row 26: 26a built, the status the previous commit's message promised, by the main session Co-Authored-By: Claude Fable 5.1 --- ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index 356964d8..d4966cfc 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -50,7 +50,7 @@ increment that most needs a picture to check against | 27 | The sampler reads a vertex on a DEM node from that node: `_core.sample` refused a cell with any NoData corner even at zero weight, so on the path without `--tolerance` a valid node next to a NoData node lost its height (increment 12's accepted one-cell trim, `12-dem-to-mesh.md` R2). Found again by increment 25's green step. A point that is a node bit for bit (`RasterGeometry::node_at`) now reads `value_at`; every other point keeps the four-corner rule; the `--tolerance` path is unchanged | **merged (#160, 2026-10-04)** (designed and approved 2026-10-03; Ola ruled nodes only, the rule in the C++ sampler; red `6605dfe`, pins `80990e0`, green `6fd4076`/`a0817ba`; 13 net production lines; no `@perf` run needed) | `docs/increments/27-node-sampling.md` | | 23 | Basin scale: the domain cut into pieces along lattice lines, Nx × Ny ≈ `--pieces` from the domain's bounding box, cut only when the estimated memory at the tolerance exceeds `--memory-budget` (16 GB; BHO dropped as geometry; sub-catchments later from the DEM, never seams), each seam given its heights by a one-dimensional seam pass and then frozen, so pieces refine independently and in parallel; windowed DEM reads with no dense canvas; `rasputin fetch`, a resumable block cache of remote COGs (ANADEM, GLO-30), so meshing never touches the network; pieces plus an index, stitched into one file by default, with the seams removed and locally thinned. Absorbs 15d's window decoding; supersedes Q9's dense canvas at basin scale | **designed and ruled** (`@architect`, 2026-10-01; Ola ruled B1-B14 the same day, B15 (the refusals at half of physical memory: delete both) and B16 (licence notes in the mesh file) on 2026-10-02; 23a-1 merged as #136, 23a-2 merged as #138; 23b (frozen edges and the seam pass) merged as #162 (merge commit 4f56551), green 3c464ec with N18 in, 249 net lines, `@perf`'s acceptance recorded at 91c7cb5 and again at the merged head 4541e38, ACCEPTED; 23c split in two, 23c-1 (the partition rule, the piece vocabulary, the index model, a start mesh from arrays) built at 156 lines and merged as #164 (merge commit e56a2bf), 23c-2 (pieces, the run, the CLI) in progress on `worktree-23c`, not pushed, its state in that branch's copy of `23-basin-scale.md`): eight PRs, 23a-1, 23a-2, 23b, 23c-1, 23c-2, 23d, 23f, 23g (about 225, 373, 285, 215, 265, 235, 300 and 345 lines estimated); measured for it: the whole basin is 3,061 ANADEM blocks, 1.71 GiB, cut into 23 pieces at 1 m at the default budget | `docs/increments/23-basin-scale.md` | | 24 | Release hardening, switched on: the bounds-checking standard library (libc++ FAST, libstdc++ assertions) on by default in every build, with a build option to turn it off for heavy runs (Ola, 2026-10-03: "So normal release, turn it on. For real heavy lifting, option to turn it off."); the loaded mode reported by `_core.hardening`, `rasputin version` and `--stats`; a guard test that fails when a toolchain ignores the macro; `bench.py --hardening on|off`, with hardened the baseline from here on | **designed and ruled** (`@architect`, 2026-10-03; Ola ruled Q1-Q4 the same day): the CMake option `RASPUTIN_HARDENING`, default ON, off by rebuilding into a second venv; an Ubuntu CI leg builds it OFF; one PR; **implemented on `worktree-agent-a08216a68d5534b18`** (red `21d5168`, green `23d4dad`, 53 net production lines; T6 reworked `57e1264`); `@reviewer` APPROVED (round 3); **`@perf` acceptance ACCEPTED** (2026-10-03, AC: built OFF it matches master `6cdc8cc`, meshes byte-identical; hardening costs refine +4.2 to +5.9 % at 1 thread and +3.2 to +11.1 % over 2 to 20 threads; first hardened baseline `docs/benchmarks/2026-10-03/24-on-r2/`, evidence `docs/benchmarks/2026-10-03/24-acceptance.md`). Two extensions in one wheel (option (b)) rejected. Measurement: `docs/benchmarks/2026-10-03/release-hardening/README.md` (#149) | `docs/increments/24-release-hardening.md` | -| 26 | Land cover for the São Francisco basin, MapBiomas (the land-cover half of "the basin's own inputs"): water bodies of at least a minimum area traced on cell edges and reduced with their area kept, as constraints (`rasputin water`); every other class as fractions per triangle from exact cell areas, small entries (under 5 % of the triangle and under 1 ha) dropped and their area carried along a Hilbert curve to the next triangles holding the class (Ola's "local out-of-balance ledger"); `rasputin fetch mapbiomas-c11 --year` | **designed and ruled** (`@architect`, 2026-10-03; Ola ruled all 11 questions on 2026-10-04: Collection 11, year 2025 by default; CC BY-SA 4.0 on the outputs; triangles inside a lake carry the raster's own mix, nothing forced to water; 26a-26c now alongside the basin work, the water PRs after the basin PRs, 26e with 23g). 26a ready for `@tester`. Measured for it on the Rio Corrente meshes at 20 to 2 m: renormalising inside triangles loses 8 to 45 % of the water, the ledger keeps every class within 0.05 %. Six PRs, 26a to 26e (about 170, 420, 300, 425, 355 and 210 lines) | `docs/increments/26-land-cover.md` | +| 26 | Land cover for the São Francisco basin, MapBiomas (the land-cover half of "the basin's own inputs"): water bodies of at least a minimum area traced on cell edges and reduced with their area kept, as constraints (`rasputin water`); every other class as fractions per triangle from exact cell areas, small entries (under 5 % of the triangle and under 1 ha) dropped and their area carried along a Hilbert curve to the next triangles holding the class (Ola's "local out-of-balance ledger"); `rasputin fetch mapbiomas-c11 --year` | **26a built** (branch `worktree-mapbiomas-26a`: red `cb5f254b`, green `8eec354b`, 163 net production lines; next `@perf`'s quick check and code review); 26b onward (`@architect`, 2026-10-03; Ola ruled all 11 questions on 2026-10-04: Collection 11, year 2025 by default; CC BY-SA 4.0 on the outputs; triangles inside a lake carry the raster's own mix, nothing forced to water; 26a-26c now alongside the basin work, the water PRs after the basin PRs, 26e with 23g). 26a ready for `@tester`. Measured for it on the Rio Corrente meshes at 20 to 2 m: renormalising inside triangles loses 8 to 45 % of the water, the ledger keeps every class within 0.05 %. Six PRs, 26a to 26e (about 170, 420, 300, 425, 355 and 210 lines) | `docs/increments/26-land-cover.md` | | 29 | NVE reference catchments: a catchment from the DEM for each of the 140 gauging stations in NVE's reference list of active, unregulated stations (Ola, 2026-10-04: "NVE should have a list of unregulated catchments, where they also record the discharge"), each compared with NVE's own catchment polygon (share of NVE's polygon in ours and of ours in NVE's, area ratio, mean distance between the divides). Adds flow accumulation from increment 22's flood and places each gauge where it physically is: on NVE's mapped river line, then on the DEM's flow path along it (burnt where they disagree), never by an area or flow-count objective (Ola's direction, 2026-10-04); a per-station sensitivity of the area along the river marks ill-posed stations `uncertain`. `rasputin fetch-stations nve-hrd` fetches the stations, rivers and NVE's polygons (NLOD), `rasputin station-catchments` runs them all. Residual inflow between gauges is later; the interfaces leave room for it. Discharge out of scope; station and series numbers kept for a later join. 138 of the 140 polygons lie inside DTM10's data; two cross into Finland and are expected to be refused | **PR 1, 2, 3 and 4 merged (#173, #179, #178, #182); the acceptance run over all 140 stations is in, at `docs/benchmarks/2026-10-05/nve-hrd/` (as fixed in `cd8b5be`): 74 match, 5 close, 6 miss, 39 uncertain, 16 refused (14 of them expected), 33 minutes and at most 7.7 GB; evidence review round 2 approved, the push waits for Ola; PR 5 (the nearest-stream fallback) and acceptance step 5 (the comparison with nearest stream) not started; questions 6 and 10 go back to Ola with the run's figures** | `docs/increments/29-nve-reference-catchments.md` | | 30 | The bottlenecks of `rasputin mesh` on real catchments, in three pull requests, with every output byte-identical: 30a the `land cover` phase (point-in-polygon against prepared polygons, one sort fewer, no sort to pick each area's point), 30b the CORINE clip (point tests before the segment test, a feature whose lines miss the search region skipped whole, no clip of a line lying in the domain's interior), 30c reading the DEM (decode threads from the machine's cores instead of a fixed 4, the seam report's test against the domain's region run only on the nodes it can count). Measured by `@perf` on Numedalslågen and Skiensvassdraget (14.4 s and 21.2 s on battery; `docs/benchmarks/2026-10-06/bottlenecks/README.md`), projected to about 7.8 s and 11.7 s with all three. Approved by Ola 2026-10-06, in that order ("yes, go ahead in that order") | **30a merged (#199)** (`worktree-landcover-speed`: red `6cde0eb`, green `3066d60`, +12 net production lines; `@perf` ACCEPTED on AC power, median of 3: land cover 2.85 → 0.67 s on Numedalslågen and 6.54 → 1.74 s on Skiensvassdraget, whole run 13.66 → 11.41 s and 20.46 → 15.68 s, every `.vtk` byte-identical to the base; `docs/benchmarks/2026-10-06/30a-landcover/README.md`); **30b merged (#200)** (`worktree-clip-speed`: red `08269e77`, green `c5340459`, +23 net production lines; `@perf` ACCEPTED on AC power, median of 3: `features clip` 5.83 → 1.83 s on Numedalslågen and 5.66 → 1.35 s on Skiensvassdraget, whole run 11.39 → 7.52 s and 15.69 → 11.41 s, every `.vtk` byte-identical to the base and every base line of the probe unchanged; `docs/benchmarks/2026-10-06/30b-clip/README.md`); **30c approved by code review round 2; waits for Ola's yes to push** (`worktree-dem-read-speed`, 2026-10-06; Ola said "yes, build 30c"; red `bdf7b57d`, green `0663efc9`, +3 net production lines; `@perf` ACCEPTED on AC power, median of 3: `decode` 1.788 → 1.051 s on Numedalslågen and 1.323 → 0.842 s on Skiensvassdraget, whole run 7.48 → 6.72 s and 11.46 → 10.97 s, every `.vtk` byte-identical to the base and every base line of the probe unchanged; `docs/benchmarks/2026-10-06/30c-dem-read/README.md`; master merged in as `26a5d839`, where every base line of the probe is unchanged and the 16 new lines are all from the red suite) | `docs/increments/30a-landcover-speed.md`, `docs/increments/30b-clip-speed.md`, `docs/increments/30c-dem-read-speed.md` | | 30d | Growing the catchment outline, made faster, with every mesh byte-identical: the same mitred buffer of the domain computed once instead of twice on the reprojected path; the CORINE search region grown from the domain's convex hull instead of taking the hull of the grown domain; and a long staircase outline (SMHI's raster-traced ones) grown in pieces of 1,000 edges and united, behind a gate that keeps smooth outlines on GEOS's single buffer. Measured by `@perf` on Lagan with the European CORINE GeoPackage: 103.4 s on master, about 17 s with all three (`docs/benchmarks/2026-10-07/30d-buffer/README.md`); from the performance review's finding F1 (PR #208). Ola, 2026-10-07: grow the outline in pieces; the gate is an open question, default yes | **merged (#215, 2026-10-07)** (branch `worktree-buffer-speed-2`; 29 net production lines; Python only, no refine or mesh code; Lagan 103.38 s → 16.97 s, meshes byte-identical; push waits on Ola) | `docs/increments/30d-outline-buffer-speed.md` | From 38f869303212e6adfd3084113f62371bd5bb35f3 Mon Sep 17 00:00:00 2001 From: Ola Skavhaug Date: Sun, 11 Oct 2026 23:05:09 +0200 Subject: [PATCH 5/8] Increment 26a quick check: NO CHANGE, every mesh unchanged, decode within band; ACCEPTED (@perf) tools/bench_quick.py run at f51f0a18 (worktree-mapbiomas-26a, base master de9b6c03) against docs/benchmarks/quick/baseline-ac.json (357808b1, AC, numpy 2.5.4, saved by #229). Verdict NO CHANGE, exit 0: no MESH CHANGED, BROKEN, SLOWER or FASTER line on any of the seven cases, so every mesh SHA-256 equals the baseline's, every total is within its band, and decode (judged on every case, 42 % of geilo-al-ramp) is within its band. 26a touches no refine or mesh code; the one speed-relevant change is the block-decoding loop moved into _decode_into in io/cog.py (8eec354b). Hotspot lines: the same eight as the baseline run, six at the same share, tile refine 46 -> 44 % and quarter refine 48 -> 47 %. Lagan features clip (ruled 40.7 %) did not print. AC power, battery 100 %, caffeinate on, nothing else running. Evidence: docs/benchmarks/2026-10-11/mapbiomas-26a/ (README.md, run.log, power-state.txt). The run's per-case medians are not recorded: the tool saves a record only as the baseline, and a second run with --save-baseline (to copy the record here and restore the baseline) was refused by the session's permission classifier. baseline-ac.json is untouched. Co-Authored-By: Claude Fable 5.1 --- .../2026-10-11/mapbiomas-26a/README.md | 86 +++++++++++++++++++ .../2026-10-11/mapbiomas-26a/power-state.txt | 2 + .../2026-10-11/mapbiomas-26a/run.log | 10 +++ 3 files changed, 98 insertions(+) create mode 100644 docs/benchmarks/2026-10-11/mapbiomas-26a/README.md create mode 100644 docs/benchmarks/2026-10-11/mapbiomas-26a/power-state.txt create mode 100644 docs/benchmarks/2026-10-11/mapbiomas-26a/run.log diff --git a/docs/benchmarks/2026-10-11/mapbiomas-26a/README.md b/docs/benchmarks/2026-10-11/mapbiomas-26a/README.md new file mode 100644 index 00000000..c4833fa1 --- /dev/null +++ b/docs/benchmarks/2026-10-11/mapbiomas-26a/README.md @@ -0,0 +1,86 @@ +# Increment 26a quick check: MapBiomas in the catalogue, and class windows (@perf, 2026-10-11) + +Branch `worktree-mapbiomas-26a`, head `f51f0a18`, base master `de9b6c03` +(the merge of increment 35). The rule: `docs/increments/perf-quick-check.md`, +section 3, the quick check only. 26a touches no refinement or mesh code, so +no `tools/bench.py` 1 m run and no thread sweep. It does touch the DEM +decode path: `8eec354b` moved the block-decoding loop of `decode_window` +(`src_python/tin_engine/io/cog.py`) into a shared `_decode_into`, which the +new `decode_class_window` also calls; `decode` is the largest phase of the +`geilo-al-ramp` case. So the check is: every case's mesh byte-identical to +the baseline's, every total within its band, and `decode` within its band +where the tool judges it (a phase at 5 % or more of the case's total). + +## Verdict: ACCEPTED + +`tools/bench_quick.py run` at `f51f0a18`: **`NO CHANGE`, exit 0** +(`run.log`). The tool prints a `MESH CHANGED:` line for any case whose mesh +SHA-256 differs from the baseline's, a `BROKEN:` line for a `max_error` over +the tolerance, and a `SLOWER:`/`FASTER:` line for any measure outside its +band (the case's own spread, at least 5 %, and more than 0.05 s), on the +total and on every phase at 5 % or more of the total; `decode` is judged on +every case. It printed none of them, so: + +| gate | result | +|---|---| +| mesh SHA-256 unchanged, all 7 cases | met (no `MESH CHANGED` line) | +| `max_error` within tolerance, all 7 cases | met (no `BROKEN` line) | +| every case's total within its band | met (no `SLOWER`/`FASTER` line) | +| `decode` within its band, `geilo-al-ramp` and the tile cases | met (no `SLOWER`/`FASTER` line) | + +Baseline: `docs/benchmarks/quick/baseline-ac.json`, commit `357808b1`, AC +power, Apple M1 Max (8 P + 2 E cores, 32 GB), macOS 27.0, Python 3.14.7, +numpy 2.5.4, shapely 2.2.0 / GEOS 3.14.1, pyproj 3.8.0 / PROJ 9.8.1; saved +by increment 35's acceptance (#229). Its per-case medians are the "head" +column of `docs/benchmarks/2026-10-11/cleanup-speed/README.md`: tile 0.416 s, +tile t=1 0.699 s, quarter 0.355 s, numedalslagen 8.829 s, lagan 14.542 s, +geilo-al-ramp 0.329 s (decode 0.138 s), romsdal-slope 3.578 s. The tool +compared this run's medians with those and found each inside its band; the +run's own medians are not on disk, see "Limits" below. + +## Hotspots, as printed (`docs/benchmarks/quick/hotspots.toml` rules only Lagan `features clip`) + +``` +HOTSPOT: tile refine 44 % of the run (not in hotspots.toml) +HOTSPOT: tile t=1 refine 67 % of the run (not in hotspots.toml) +HOTSPOT: tile t=1 refine: scan (parallel) 50 % of the run (not in hotspots.toml) +HOTSPOT: quarter refine 47 % of the run (not in hotspots.toml) +HOTSPOT: numedalslagen features clip 68 % of the run (not in hotspots.toml) +HOTSPOT: numedalslagen features clip: clean-up 67 % of the run (not in hotspots.toml) +HOTSPOT: geilo-al-ramp decode 42 % of the run (not in hotspots.toml) +HOTSPOT: romsdal-slope refine 56 % of the run (not in hotspots.toml) +``` + +The same eight lines as the baseline run printed +(`docs/benchmarks/2026-10-11/cleanup-speed/quick/head-run.log`), six at the +same share; tile `refine` 46 -> 44 % and quarter `refine` 48 -> 47 % (both +within those cases' run-to-run noise on a 0.4 s case). geilo-al-ramp `decode` +is 42 % on both, 0.138 of 0.329 s on the baseline. None is new to 26a. Lagan +`features clip` (ruled at 40.7 %) did not print, so it is under 50.7 % of +Lagan's run, as after increment 35. + +## Method + +* Tree `/Users/skavhaug/projects/rasputin/.claude/worktrees/mapbiomas-26a` + at `f51f0a18`, clean. `tools/bench_quick.py run` builds Release into + `build-bench/` and runs the seven cases of `docs/benchmarks/quick/cases.toml` + from there (3 to 5 timed runs per case, the median judged), so no stale + extension is measured. `RASPUTIN_DATA=/Users/skavhaug/projects/rasputin_data`. +* Power: AC, battery 100 %, `caffeinate` running, before and after + (`power-state.txt`). Nothing else ran beside the timing run + (`pgrep` for cmake, clang, pytest, bench and `rasputin mesh` found nothing). +* Full tool output: `run.log` (Python's two `sys.prefix` RuntimeWarnings from + the Homebrew interpreter are noise, not from the tool). +* Meshes: written by the tool to its own temporary directories and removed; + regenerate by re-running the command above. + +## Limits + +* `bench_quick.py` writes a run's record (per-case medians, spreads and mesh + hashes) only with `--save-baseline`, which replaces + `docs/benchmarks/quick/baseline-ac.json`. A second run with that flag, + meant to be copied here and the baseline restored, was refused by the + session's permission classifier, so this directory holds the tool's + verdict and hotspot lines, not the medians. A `--record ` option that + saves the record without touching the baseline would close this gap; it + is tooling (`tools/`) and goes through the TDD loop. diff --git a/docs/benchmarks/2026-10-11/mapbiomas-26a/power-state.txt b/docs/benchmarks/2026-10-11/mapbiomas-26a/power-state.txt new file mode 100644 index 00000000..3bce19f6 --- /dev/null +++ b/docs/benchmarks/2026-10-11/mapbiomas-26a/power-state.txt @@ -0,0 +1,2 @@ +Now drawing from 'AC Power' +Sun Oct 11 22:58:56 CEST 2026 diff --git a/docs/benchmarks/2026-10-11/mapbiomas-26a/run.log b/docs/benchmarks/2026-10-11/mapbiomas-26a/run.log new file mode 100644 index 00000000..a101aebf --- /dev/null +++ b/docs/benchmarks/2026-10-11/mapbiomas-26a/run.log @@ -0,0 +1,10 @@ +HOTSPOT: tile refine 44 % of the run (not in hotspots.toml) +HOTSPOT: tile t=1 refine 67 % of the run (not in hotspots.toml) +HOTSPOT: tile t=1 refine: scan (parallel) 50 % of the run (not in hotspots.toml) +HOTSPOT: quarter refine 47 % of the run (not in hotspots.toml) +HOTSPOT: numedalslagen features clip 68 % of the run (not in hotspots.toml) +HOTSPOT: numedalslagen features clip: clean-up 67 % of the run (not in hotspots.toml) +HOTSPOT: geilo-al-ramp decode 42 % of the run (not in hotspots.toml) +HOTSPOT: romsdal-slope refine 56 % of the run (not in hotspots.toml) +NO CHANGE +EXIT 0 From 15a51ed609dd46a42fbd773c9df3f3f08601efe8 Mon Sep 17 00:00:00 2001 From: Ola Skavhaug Date: Sun, 11 Oct 2026 23:06:08 +0200 Subject: [PATCH 6/8] 26a quick check ACCEPTED recorded, with the Status line and ROADMAP row, by the main session Co-Authored-By: Claude Fable 5.1 --- ROADMAP.md | 2 +- docs/increments/26-land-cover.md | 12 +++++++++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index d4966cfc..b33dce5a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -50,7 +50,7 @@ increment that most needs a picture to check against | 27 | The sampler reads a vertex on a DEM node from that node: `_core.sample` refused a cell with any NoData corner even at zero weight, so on the path without `--tolerance` a valid node next to a NoData node lost its height (increment 12's accepted one-cell trim, `12-dem-to-mesh.md` R2). Found again by increment 25's green step. A point that is a node bit for bit (`RasterGeometry::node_at`) now reads `value_at`; every other point keeps the four-corner rule; the `--tolerance` path is unchanged | **merged (#160, 2026-10-04)** (designed and approved 2026-10-03; Ola ruled nodes only, the rule in the C++ sampler; red `6605dfe`, pins `80990e0`, green `6fd4076`/`a0817ba`; 13 net production lines; no `@perf` run needed) | `docs/increments/27-node-sampling.md` | | 23 | Basin scale: the domain cut into pieces along lattice lines, Nx × Ny ≈ `--pieces` from the domain's bounding box, cut only when the estimated memory at the tolerance exceeds `--memory-budget` (16 GB; BHO dropped as geometry; sub-catchments later from the DEM, never seams), each seam given its heights by a one-dimensional seam pass and then frozen, so pieces refine independently and in parallel; windowed DEM reads with no dense canvas; `rasputin fetch`, a resumable block cache of remote COGs (ANADEM, GLO-30), so meshing never touches the network; pieces plus an index, stitched into one file by default, with the seams removed and locally thinned. Absorbs 15d's window decoding; supersedes Q9's dense canvas at basin scale | **designed and ruled** (`@architect`, 2026-10-01; Ola ruled B1-B14 the same day, B15 (the refusals at half of physical memory: delete both) and B16 (licence notes in the mesh file) on 2026-10-02; 23a-1 merged as #136, 23a-2 merged as #138; 23b (frozen edges and the seam pass) merged as #162 (merge commit 4f56551), green 3c464ec with N18 in, 249 net lines, `@perf`'s acceptance recorded at 91c7cb5 and again at the merged head 4541e38, ACCEPTED; 23c split in two, 23c-1 (the partition rule, the piece vocabulary, the index model, a start mesh from arrays) built at 156 lines and merged as #164 (merge commit e56a2bf), 23c-2 (pieces, the run, the CLI) in progress on `worktree-23c`, not pushed, its state in that branch's copy of `23-basin-scale.md`): eight PRs, 23a-1, 23a-2, 23b, 23c-1, 23c-2, 23d, 23f, 23g (about 225, 373, 285, 215, 265, 235, 300 and 345 lines estimated); measured for it: the whole basin is 3,061 ANADEM blocks, 1.71 GiB, cut into 23 pieces at 1 m at the default budget | `docs/increments/23-basin-scale.md` | | 24 | Release hardening, switched on: the bounds-checking standard library (libc++ FAST, libstdc++ assertions) on by default in every build, with a build option to turn it off for heavy runs (Ola, 2026-10-03: "So normal release, turn it on. For real heavy lifting, option to turn it off."); the loaded mode reported by `_core.hardening`, `rasputin version` and `--stats`; a guard test that fails when a toolchain ignores the macro; `bench.py --hardening on|off`, with hardened the baseline from here on | **designed and ruled** (`@architect`, 2026-10-03; Ola ruled Q1-Q4 the same day): the CMake option `RASPUTIN_HARDENING`, default ON, off by rebuilding into a second venv; an Ubuntu CI leg builds it OFF; one PR; **implemented on `worktree-agent-a08216a68d5534b18`** (red `21d5168`, green `23d4dad`, 53 net production lines; T6 reworked `57e1264`); `@reviewer` APPROVED (round 3); **`@perf` acceptance ACCEPTED** (2026-10-03, AC: built OFF it matches master `6cdc8cc`, meshes byte-identical; hardening costs refine +4.2 to +5.9 % at 1 thread and +3.2 to +11.1 % over 2 to 20 threads; first hardened baseline `docs/benchmarks/2026-10-03/24-on-r2/`, evidence `docs/benchmarks/2026-10-03/24-acceptance.md`). Two extensions in one wheel (option (b)) rejected. Measurement: `docs/benchmarks/2026-10-03/release-hardening/README.md` (#149) | `docs/increments/24-release-hardening.md` | -| 26 | Land cover for the São Francisco basin, MapBiomas (the land-cover half of "the basin's own inputs"): water bodies of at least a minimum area traced on cell edges and reduced with their area kept, as constraints (`rasputin water`); every other class as fractions per triangle from exact cell areas, small entries (under 5 % of the triangle and under 1 ha) dropped and their area carried along a Hilbert curve to the next triangles holding the class (Ola's "local out-of-balance ledger"); `rasputin fetch mapbiomas-c11 --year` | **26a built** (branch `worktree-mapbiomas-26a`: red `cb5f254b`, green `8eec354b`, 163 net production lines; next `@perf`'s quick check and code review); 26b onward (`@architect`, 2026-10-03; Ola ruled all 11 questions on 2026-10-04: Collection 11, year 2025 by default; CC BY-SA 4.0 on the outputs; triangles inside a lake carry the raster's own mix, nothing forced to water; 26a-26c now alongside the basin work, the water PRs after the basin PRs, 26e with 23g). 26a ready for `@tester`. Measured for it on the Rio Corrente meshes at 20 to 2 m: renormalising inside triangles loses 8 to 45 % of the water, the ledger keeps every class within 0.05 %. Six PRs, 26a to 26e (about 170, 420, 300, 425, 355 and 210 lines) | `docs/increments/26-land-cover.md` | +| 26 | Land cover for the São Francisco basin, MapBiomas (the land-cover half of "the basin's own inputs"): water bodies of at least a minimum area traced on cell edges and reduced with their area kept, as constraints (`rasputin water`); every other class as fractions per triangle from exact cell areas, small entries (under 5 % of the triangle and under 1 ha) dropped and their area carried along a Hilbert curve to the next triangles holding the class (Ola's "local out-of-balance ledger"); `rasputin fetch mapbiomas-c11 --year` | **26a built** (branch `worktree-mapbiomas-26a`: red `cb5f254b`, green `8eec354b`, 163 net production lines; `@perf`'s quick check `38f86930` NO CHANGE, every mesh byte-identical; next code review); 26b onward designed and ruled (`@architect`, 2026-10-03; Ola ruled all 11 questions on 2026-10-04: Collection 11, year 2025 by default; CC BY-SA 4.0 on the outputs; triangles inside a lake carry the raster's own mix, nothing forced to water; 26a-26c now alongside the basin work, the water PRs after the basin PRs, 26e with 23g). 26a ready for `@tester`. Measured for it on the Rio Corrente meshes at 20 to 2 m: renormalising inside triangles loses 8 to 45 % of the water, the ledger keeps every class within 0.05 %. Six PRs, 26a to 26e (about 170, 420, 300, 425, 355 and 210 lines) | `docs/increments/26-land-cover.md` | | 29 | NVE reference catchments: a catchment from the DEM for each of the 140 gauging stations in NVE's reference list of active, unregulated stations (Ola, 2026-10-04: "NVE should have a list of unregulated catchments, where they also record the discharge"), each compared with NVE's own catchment polygon (share of NVE's polygon in ours and of ours in NVE's, area ratio, mean distance between the divides). Adds flow accumulation from increment 22's flood and places each gauge where it physically is: on NVE's mapped river line, then on the DEM's flow path along it (burnt where they disagree), never by an area or flow-count objective (Ola's direction, 2026-10-04); a per-station sensitivity of the area along the river marks ill-posed stations `uncertain`. `rasputin fetch-stations nve-hrd` fetches the stations, rivers and NVE's polygons (NLOD), `rasputin station-catchments` runs them all. Residual inflow between gauges is later; the interfaces leave room for it. Discharge out of scope; station and series numbers kept for a later join. 138 of the 140 polygons lie inside DTM10's data; two cross into Finland and are expected to be refused | **PR 1, 2, 3 and 4 merged (#173, #179, #178, #182); the acceptance run over all 140 stations is in, at `docs/benchmarks/2026-10-05/nve-hrd/` (as fixed in `cd8b5be`): 74 match, 5 close, 6 miss, 39 uncertain, 16 refused (14 of them expected), 33 minutes and at most 7.7 GB; evidence review round 2 approved, the push waits for Ola; PR 5 (the nearest-stream fallback) and acceptance step 5 (the comparison with nearest stream) not started; questions 6 and 10 go back to Ola with the run's figures** | `docs/increments/29-nve-reference-catchments.md` | | 30 | The bottlenecks of `rasputin mesh` on real catchments, in three pull requests, with every output byte-identical: 30a the `land cover` phase (point-in-polygon against prepared polygons, one sort fewer, no sort to pick each area's point), 30b the CORINE clip (point tests before the segment test, a feature whose lines miss the search region skipped whole, no clip of a line lying in the domain's interior), 30c reading the DEM (decode threads from the machine's cores instead of a fixed 4, the seam report's test against the domain's region run only on the nodes it can count). Measured by `@perf` on Numedalslågen and Skiensvassdraget (14.4 s and 21.2 s on battery; `docs/benchmarks/2026-10-06/bottlenecks/README.md`), projected to about 7.8 s and 11.7 s with all three. Approved by Ola 2026-10-06, in that order ("yes, go ahead in that order") | **30a merged (#199)** (`worktree-landcover-speed`: red `6cde0eb`, green `3066d60`, +12 net production lines; `@perf` ACCEPTED on AC power, median of 3: land cover 2.85 → 0.67 s on Numedalslågen and 6.54 → 1.74 s on Skiensvassdraget, whole run 13.66 → 11.41 s and 20.46 → 15.68 s, every `.vtk` byte-identical to the base; `docs/benchmarks/2026-10-06/30a-landcover/README.md`); **30b merged (#200)** (`worktree-clip-speed`: red `08269e77`, green `c5340459`, +23 net production lines; `@perf` ACCEPTED on AC power, median of 3: `features clip` 5.83 → 1.83 s on Numedalslågen and 5.66 → 1.35 s on Skiensvassdraget, whole run 11.39 → 7.52 s and 15.69 → 11.41 s, every `.vtk` byte-identical to the base and every base line of the probe unchanged; `docs/benchmarks/2026-10-06/30b-clip/README.md`); **30c approved by code review round 2; waits for Ola's yes to push** (`worktree-dem-read-speed`, 2026-10-06; Ola said "yes, build 30c"; red `bdf7b57d`, green `0663efc9`, +3 net production lines; `@perf` ACCEPTED on AC power, median of 3: `decode` 1.788 → 1.051 s on Numedalslågen and 1.323 → 0.842 s on Skiensvassdraget, whole run 7.48 → 6.72 s and 11.46 → 10.97 s, every `.vtk` byte-identical to the base and every base line of the probe unchanged; `docs/benchmarks/2026-10-06/30c-dem-read/README.md`; master merged in as `26a5d839`, where every base line of the probe is unchanged and the 16 new lines are all from the red suite) | `docs/increments/30a-landcover-speed.md`, `docs/increments/30b-clip-speed.md`, `docs/increments/30c-dem-read-speed.md` | | 30d | Growing the catchment outline, made faster, with every mesh byte-identical: the same mitred buffer of the domain computed once instead of twice on the reprojected path; the CORINE search region grown from the domain's convex hull instead of taking the hull of the grown domain; and a long staircase outline (SMHI's raster-traced ones) grown in pieces of 1,000 edges and united, behind a gate that keeps smooth outlines on GEOS's single buffer. Measured by `@perf` on Lagan with the European CORINE GeoPackage: 103.4 s on master, about 17 s with all three (`docs/benchmarks/2026-10-07/30d-buffer/README.md`); from the performance review's finding F1 (PR #208). Ola, 2026-10-07: grow the outline in pieces; the gate is an open question, default yes | **merged (#215, 2026-10-07)** (branch `worktree-buffer-speed-2`; 29 net production lines; Python only, no refine or mesh code; Lagan 103.38 s → 16.97 s, meshes byte-identical; push waits on Ola) | `docs/increments/30d-outline-buffer-speed.md` | diff --git a/docs/increments/26-land-cover.md b/docs/increments/26-land-cover.md index 9097ad0c..efa7e456 100644 --- a/docs/increments/26-land-cover.md +++ b/docs/increments/26-land-cover.md @@ -1,6 +1,6 @@ # Increment 26: land cover for the São Francisco basin — water bodies as constraints, class fractions per triangle -Status: **26a built** on branch `worktree-mapbiomas-26a` (red `cb5f254b`, `@tester`, 48 tests; green `8eec354b`, `@developer`, 163 net production lines by `python3 tools/count_loc.py de9b6c03 HEAD` against the estimate of 170; no mutation round, section 9; section 13 is the as-built record; next `@perf`'s quick check, then code review). 26b onward not built. The design: **by `@architect`, 2026-10-03, at Ola's request; ruled by +Status: **26a built** on branch `worktree-mapbiomas-26a` (red `cb5f254b`, `@tester`, 48 tests; green `8eec354b`, `@developer`, 163 net production lines by `python3 tools/count_loc.py de9b6c03 HEAD` against the estimate of 170; no mutation round, section 9; section 13 is the as-built record; `@perf`'s quick check `38f86930` ACCEPTED; next code review). 26b onward not built. The design: **by `@architect`, 2026-10-03, at Ola's request; ruled by Ola, 2026-10-04** (all eleven questions; "Ruled by Ola, 2026-10-04", below, and each question marked). Q8 was ruled against the recommendation: triangles inside a lake carry the raster's own mix, nothing is forced to @@ -1529,6 +1529,16 @@ behaviour unchanged, which is why `@perf`'s quick check runs before review. Lesson: running only the increment's suites missed an import-boundary test elsewhere; the full suite caught it. +**Quick check** `38f86930` (`@perf`, ACCEPTED; +`docs/benchmarks/2026-10-11/mapbiomas-26a/README.md`): `tools/bench_quick.py +run` at `f51f0a18` against the stored baseline (`357808b1`, AC, numpy 2.5.4), +AC power, nothing else running: `NO CHANGE`, exit 0; every mesh SHA-256 equal, +every total and every `decode` phase within its band, the same eight hotspot +lines as the baseline run (geilo-al-ramp decode 42 % on both). The run's own +per-case medians are not on disk: `bench_quick.py` writes a record only with +`--save-baseline`, so a passing quick check leaves the tool's silence as the +evidence. Idea for the tooling: `bench_quick.py run --record `. + ## Review **Design review, round 1, 2026-10-03.** Range `origin/master...` 77f35b4, 9df3822. Verdict: CHANGES REQUESTED. LOC: 0 production lines (design and throwaway probes). The rulings of 2026-10-01 are kept; the fractions-and-ledger figures, the fetch sizes, the CRS figures, the sources and the licence conflict (terms of use CC BY-SA 4.0, factsheet CC BY) all re-checked and hold. Blocking, doc fixes only: (1) unit 769's water row omits its largest body (Três Marias joins the channel, and the probe assigns a body to a unit by its first cell, which lies outside the outline), so the basin-wide lake counts are low; (2) the "Sobradinho" ring is the main São Francisco river across unit 761 (670 × 442 km, 3,159 holes), and "346 km" is half the extent, not a length; on the whole basin the channel likely joins the reservoirs into one body; (3) the ledger measurement excludes lakes forced to 100 % water, which @perf's acceptance must cover; (4) the licence question must also name the water polygons and meshes constrained by them. Not pushed; no CI. From 7053d795411991e8f8cb7418a8a55b4c6e131f01 Mon Sep 17 00:00:00 2001 From: Ola Skavhaug Date: Sun, 11 Oct 2026 23:12:00 +0200 Subject: [PATCH 7/8] 26a code review round 1 CHANGES REQUESTED recorded, the two stale 'ready for @tester' phrases fixed, by the main session Co-Authored-By: Claude Fable 5.1 --- ROADMAP.md | 2 +- docs/increments/26-land-cover.md | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index b33dce5a..8fc18f25 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -50,7 +50,7 @@ increment that most needs a picture to check against | 27 | The sampler reads a vertex on a DEM node from that node: `_core.sample` refused a cell with any NoData corner even at zero weight, so on the path without `--tolerance` a valid node next to a NoData node lost its height (increment 12's accepted one-cell trim, `12-dem-to-mesh.md` R2). Found again by increment 25's green step. A point that is a node bit for bit (`RasterGeometry::node_at`) now reads `value_at`; every other point keeps the four-corner rule; the `--tolerance` path is unchanged | **merged (#160, 2026-10-04)** (designed and approved 2026-10-03; Ola ruled nodes only, the rule in the C++ sampler; red `6605dfe`, pins `80990e0`, green `6fd4076`/`a0817ba`; 13 net production lines; no `@perf` run needed) | `docs/increments/27-node-sampling.md` | | 23 | Basin scale: the domain cut into pieces along lattice lines, Nx × Ny ≈ `--pieces` from the domain's bounding box, cut only when the estimated memory at the tolerance exceeds `--memory-budget` (16 GB; BHO dropped as geometry; sub-catchments later from the DEM, never seams), each seam given its heights by a one-dimensional seam pass and then frozen, so pieces refine independently and in parallel; windowed DEM reads with no dense canvas; `rasputin fetch`, a resumable block cache of remote COGs (ANADEM, GLO-30), so meshing never touches the network; pieces plus an index, stitched into one file by default, with the seams removed and locally thinned. Absorbs 15d's window decoding; supersedes Q9's dense canvas at basin scale | **designed and ruled** (`@architect`, 2026-10-01; Ola ruled B1-B14 the same day, B15 (the refusals at half of physical memory: delete both) and B16 (licence notes in the mesh file) on 2026-10-02; 23a-1 merged as #136, 23a-2 merged as #138; 23b (frozen edges and the seam pass) merged as #162 (merge commit 4f56551), green 3c464ec with N18 in, 249 net lines, `@perf`'s acceptance recorded at 91c7cb5 and again at the merged head 4541e38, ACCEPTED; 23c split in two, 23c-1 (the partition rule, the piece vocabulary, the index model, a start mesh from arrays) built at 156 lines and merged as #164 (merge commit e56a2bf), 23c-2 (pieces, the run, the CLI) in progress on `worktree-23c`, not pushed, its state in that branch's copy of `23-basin-scale.md`): eight PRs, 23a-1, 23a-2, 23b, 23c-1, 23c-2, 23d, 23f, 23g (about 225, 373, 285, 215, 265, 235, 300 and 345 lines estimated); measured for it: the whole basin is 3,061 ANADEM blocks, 1.71 GiB, cut into 23 pieces at 1 m at the default budget | `docs/increments/23-basin-scale.md` | | 24 | Release hardening, switched on: the bounds-checking standard library (libc++ FAST, libstdc++ assertions) on by default in every build, with a build option to turn it off for heavy runs (Ola, 2026-10-03: "So normal release, turn it on. For real heavy lifting, option to turn it off."); the loaded mode reported by `_core.hardening`, `rasputin version` and `--stats`; a guard test that fails when a toolchain ignores the macro; `bench.py --hardening on|off`, with hardened the baseline from here on | **designed and ruled** (`@architect`, 2026-10-03; Ola ruled Q1-Q4 the same day): the CMake option `RASPUTIN_HARDENING`, default ON, off by rebuilding into a second venv; an Ubuntu CI leg builds it OFF; one PR; **implemented on `worktree-agent-a08216a68d5534b18`** (red `21d5168`, green `23d4dad`, 53 net production lines; T6 reworked `57e1264`); `@reviewer` APPROVED (round 3); **`@perf` acceptance ACCEPTED** (2026-10-03, AC: built OFF it matches master `6cdc8cc`, meshes byte-identical; hardening costs refine +4.2 to +5.9 % at 1 thread and +3.2 to +11.1 % over 2 to 20 threads; first hardened baseline `docs/benchmarks/2026-10-03/24-on-r2/`, evidence `docs/benchmarks/2026-10-03/24-acceptance.md`). Two extensions in one wheel (option (b)) rejected. Measurement: `docs/benchmarks/2026-10-03/release-hardening/README.md` (#149) | `docs/increments/24-release-hardening.md` | -| 26 | Land cover for the São Francisco basin, MapBiomas (the land-cover half of "the basin's own inputs"): water bodies of at least a minimum area traced on cell edges and reduced with their area kept, as constraints (`rasputin water`); every other class as fractions per triangle from exact cell areas, small entries (under 5 % of the triangle and under 1 ha) dropped and their area carried along a Hilbert curve to the next triangles holding the class (Ola's "local out-of-balance ledger"); `rasputin fetch mapbiomas-c11 --year` | **26a built** (branch `worktree-mapbiomas-26a`: red `cb5f254b`, green `8eec354b`, 163 net production lines; `@perf`'s quick check `38f86930` NO CHANGE, every mesh byte-identical; next code review); 26b onward designed and ruled (`@architect`, 2026-10-03; Ola ruled all 11 questions on 2026-10-04: Collection 11, year 2025 by default; CC BY-SA 4.0 on the outputs; triangles inside a lake carry the raster's own mix, nothing forced to water; 26a-26c now alongside the basin work, the water PRs after the basin PRs, 26e with 23g). 26a ready for `@tester`. Measured for it on the Rio Corrente meshes at 20 to 2 m: renormalising inside triangles loses 8 to 45 % of the water, the ledger keeps every class within 0.05 %. Six PRs, 26a to 26e (about 170, 420, 300, 425, 355 and 210 lines) | `docs/increments/26-land-cover.md` | +| 26 | Land cover for the São Francisco basin, MapBiomas (the land-cover half of "the basin's own inputs"): water bodies of at least a minimum area traced on cell edges and reduced with their area kept, as constraints (`rasputin water`); every other class as fractions per triangle from exact cell areas, small entries (under 5 % of the triangle and under 1 ha) dropped and their area carried along a Hilbert curve to the next triangles holding the class (Ola's "local out-of-balance ledger"); `rasputin fetch mapbiomas-c11 --year` | **26a built** (branch `worktree-mapbiomas-26a`: red `cb5f254b`, green `8eec354b`, 163 net production lines; `@perf`'s quick check `38f86930` NO CHANGE, every mesh byte-identical; next code review); 26b onward designed and ruled (`@architect`, 2026-10-03; Ola ruled all 11 questions on 2026-10-04: Collection 11, year 2025 by default; CC BY-SA 4.0 on the outputs; triangles inside a lake carry the raster's own mix, nothing forced to water; 26a-26c now alongside the basin work, the water PRs after the basin PRs, 26e with 23g). Measured for it on the Rio Corrente meshes at 20 to 2 m: renormalising inside triangles loses 8 to 45 % of the water, the ledger keeps every class within 0.05 %. Six PRs, 26a to 26e (about 170, 420, 300, 425, 355 and 210 lines) | `docs/increments/26-land-cover.md` | | 29 | NVE reference catchments: a catchment from the DEM for each of the 140 gauging stations in NVE's reference list of active, unregulated stations (Ola, 2026-10-04: "NVE should have a list of unregulated catchments, where they also record the discharge"), each compared with NVE's own catchment polygon (share of NVE's polygon in ours and of ours in NVE's, area ratio, mean distance between the divides). Adds flow accumulation from increment 22's flood and places each gauge where it physically is: on NVE's mapped river line, then on the DEM's flow path along it (burnt where they disagree), never by an area or flow-count objective (Ola's direction, 2026-10-04); a per-station sensitivity of the area along the river marks ill-posed stations `uncertain`. `rasputin fetch-stations nve-hrd` fetches the stations, rivers and NVE's polygons (NLOD), `rasputin station-catchments` runs them all. Residual inflow between gauges is later; the interfaces leave room for it. Discharge out of scope; station and series numbers kept for a later join. 138 of the 140 polygons lie inside DTM10's data; two cross into Finland and are expected to be refused | **PR 1, 2, 3 and 4 merged (#173, #179, #178, #182); the acceptance run over all 140 stations is in, at `docs/benchmarks/2026-10-05/nve-hrd/` (as fixed in `cd8b5be`): 74 match, 5 close, 6 miss, 39 uncertain, 16 refused (14 of them expected), 33 minutes and at most 7.7 GB; evidence review round 2 approved, the push waits for Ola; PR 5 (the nearest-stream fallback) and acceptance step 5 (the comparison with nearest stream) not started; questions 6 and 10 go back to Ola with the run's figures** | `docs/increments/29-nve-reference-catchments.md` | | 30 | The bottlenecks of `rasputin mesh` on real catchments, in three pull requests, with every output byte-identical: 30a the `land cover` phase (point-in-polygon against prepared polygons, one sort fewer, no sort to pick each area's point), 30b the CORINE clip (point tests before the segment test, a feature whose lines miss the search region skipped whole, no clip of a line lying in the domain's interior), 30c reading the DEM (decode threads from the machine's cores instead of a fixed 4, the seam report's test against the domain's region run only on the nodes it can count). Measured by `@perf` on Numedalslågen and Skiensvassdraget (14.4 s and 21.2 s on battery; `docs/benchmarks/2026-10-06/bottlenecks/README.md`), projected to about 7.8 s and 11.7 s with all three. Approved by Ola 2026-10-06, in that order ("yes, go ahead in that order") | **30a merged (#199)** (`worktree-landcover-speed`: red `6cde0eb`, green `3066d60`, +12 net production lines; `@perf` ACCEPTED on AC power, median of 3: land cover 2.85 → 0.67 s on Numedalslågen and 6.54 → 1.74 s on Skiensvassdraget, whole run 13.66 → 11.41 s and 20.46 → 15.68 s, every `.vtk` byte-identical to the base; `docs/benchmarks/2026-10-06/30a-landcover/README.md`); **30b merged (#200)** (`worktree-clip-speed`: red `08269e77`, green `c5340459`, +23 net production lines; `@perf` ACCEPTED on AC power, median of 3: `features clip` 5.83 → 1.83 s on Numedalslågen and 5.66 → 1.35 s on Skiensvassdraget, whole run 11.39 → 7.52 s and 15.69 → 11.41 s, every `.vtk` byte-identical to the base and every base line of the probe unchanged; `docs/benchmarks/2026-10-06/30b-clip/README.md`); **30c approved by code review round 2; waits for Ola's yes to push** (`worktree-dem-read-speed`, 2026-10-06; Ola said "yes, build 30c"; red `bdf7b57d`, green `0663efc9`, +3 net production lines; `@perf` ACCEPTED on AC power, median of 3: `decode` 1.788 → 1.051 s on Numedalslågen and 1.323 → 0.842 s on Skiensvassdraget, whole run 7.48 → 6.72 s and 11.46 → 10.97 s, every `.vtk` byte-identical to the base and every base line of the probe unchanged; `docs/benchmarks/2026-10-06/30c-dem-read/README.md`; master merged in as `26a5d839`, where every base line of the probe is unchanged and the 16 new lines are all from the red suite) | `docs/increments/30a-landcover-speed.md`, `docs/increments/30b-clip-speed.md`, `docs/increments/30c-dem-read-speed.md` | | 30d | Growing the catchment outline, made faster, with every mesh byte-identical: the same mitred buffer of the domain computed once instead of twice on the reprojected path; the CORINE search region grown from the domain's convex hull instead of taking the hull of the grown domain; and a long staircase outline (SMHI's raster-traced ones) grown in pieces of 1,000 edges and united, behind a gate that keeps smooth outlines on GEOS's single buffer. Measured by `@perf` on Lagan with the European CORINE GeoPackage: 103.4 s on master, about 17 s with all three (`docs/benchmarks/2026-10-07/30d-buffer/README.md`); from the performance review's finding F1 (PR #208). Ola, 2026-10-07: grow the outline in pieces; the gate is an open question, default yes | **merged (#215, 2026-10-07)** (branch `worktree-buffer-speed-2`; 29 net production lines; Python only, no refine or mesh code; Lagan 103.38 s → 16.97 s, meshes byte-identical; push waits on Ola) | `docs/increments/30d-outline-buffer-speed.md` | diff --git a/docs/increments/26-land-cover.md b/docs/increments/26-land-cover.md index efa7e456..cba719af 100644 --- a/docs/increments/26-land-cover.md +++ b/docs/increments/26-land-cover.md @@ -5,8 +5,8 @@ Ola, 2026-10-04** (all eleven questions; "Ruled by Ola, 2026-10-04", below, and each question marked). Q8 was ruled against the recommendation: triangles inside a lake carry the raster's own mix, nothing is forced to water, and the design is changed to match. Written before `@tester` per -`docs/increments/README.md` step 1. Nothing is built. **26a is ready for -`@tester`.** Six pull requests, listed under "The PR split". +`docs/increments/README.md` step 1. 26a is built (section 13); 26b onward +are not. Six pull requests, listed under "The PR split". **Why "26".** 25 is plain output (`docs/increments/25-plain-output.md`, on its own branch); 26 is the first free number on every branch @@ -1544,3 +1544,5 @@ evidence. Idea for the tooling: `bench_quick.py run --record `. **Design review, round 1, 2026-10-03.** Range `origin/master...` 77f35b4, 9df3822. Verdict: CHANGES REQUESTED. LOC: 0 production lines (design and throwaway probes). The rulings of 2026-10-01 are kept; the fractions-and-ledger figures, the fetch sizes, the CRS figures, the sources and the licence conflict (terms of use CC BY-SA 4.0, factsheet CC BY) all re-checked and hold. Blocking, doc fixes only: (1) unit 769's water row omits its largest body (Três Marias joins the channel, and the probe assigns a body to a unit by its first cell, which lies outside the outline), so the basin-wide lake counts are low; (2) the "Sobradinho" ring is the main São Francisco river across unit 761 (670 × 442 km, 3,159 holes), and "346 km" is half the extent, not a length; on the whole basin the channel likely joins the reservoirs into one body; (3) the ledger measurement excludes lakes forced to 100 % water, which @perf's acceptance must cover; (4) the licence question must also name the water polygons and meshes constrained by them. Not pushed; no CI. **Design review, round 2, 2026-10-03.** Range `3fd0237..097b26b`. Verdict: APPROVED. LOC: 0 production lines. All four round-1 blockers fixed, checked against the corrected probe's saved outputs (769: 2,836 bodies of at least 1.44 ha holding 94.8 %, Três Marias with the upper channel 1,290 km²; basin estimate about 11,600 bodies and 130,000 vertices at 60 m). Nit: 769's largest body is 455 × 279 km, not 270. Not pushed; no CI. + +**Code review, round 1, 2026-10-11 (26a).** Range `de9b6c03..15a51ed6`, 163 net production lines, @reviewer: CHANGES REQUESTED, doc only, the code clean. Two stale phrases this branch made false: /Users/skavhaug/projects/rasputin/.claude/worktrees/mapbiomas-26a/docs/increments/26-land-cover.md@15a51ed6:8-9 ("Nothing is built. 26a is ready for @tester") and /Users/skavhaug/projects/rasputin/.claude/worktrees/mapbiomas-26a/ROADMAP.md@15a51ed6:53 ("26a ready for @tester"); both fixed in this recording commit. Sound: `_decode_into` (/Users/skavhaug/projects/rasputin/.claude/worktrees/mapbiomas-26a/src_python/tin_engine/io/cog.py@15a51ed6:185-203) is the original `put` closure verbatim, sparse-block refusal and edge clipping unchanged, so partial blocks and NoData behave as before (the quick check's byte-identical meshes agree); `decode_class_window` undoes the half-cell PixelIsArea shift correctly; `class_strips` floor/ceil gives cells whose interior meets the box; `credit_for` leaves ANADEM and GLO-30 unchanged; `fetched` kept on later runs; credit wording and CC BY-SA match the terms page and Ola's Q2 ruling; no prohibited dependency; 48 tests pass; no red scaffolding; check_citations at-risk lines drifted at base. Lesson: a Status paragraph with two statements of readiness drifts; keep one. Idea for 26b: `decode_class_window` does not check `meta.pixel_is_area`; a PixelIsPoint class raster would shift half a cell silently. From 1203bbfcd86d85fb90d43147d1b994b7500b779e Mon Sep 17 00:00:00 2001 From: Ola Skavhaug Date: Sun, 11 Oct 2026 23:15:44 +0200 Subject: [PATCH 8/8] 26a code review round 2 APPROVED recorded, with the Status line and ROADMAP row, by the main session Co-Authored-By: Claude Fable 5.1 --- ROADMAP.md | 2 +- docs/increments/26-land-cover.md | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 8fc18f25..40f97935 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -50,7 +50,7 @@ increment that most needs a picture to check against | 27 | The sampler reads a vertex on a DEM node from that node: `_core.sample` refused a cell with any NoData corner even at zero weight, so on the path without `--tolerance` a valid node next to a NoData node lost its height (increment 12's accepted one-cell trim, `12-dem-to-mesh.md` R2). Found again by increment 25's green step. A point that is a node bit for bit (`RasterGeometry::node_at`) now reads `value_at`; every other point keeps the four-corner rule; the `--tolerance` path is unchanged | **merged (#160, 2026-10-04)** (designed and approved 2026-10-03; Ola ruled nodes only, the rule in the C++ sampler; red `6605dfe`, pins `80990e0`, green `6fd4076`/`a0817ba`; 13 net production lines; no `@perf` run needed) | `docs/increments/27-node-sampling.md` | | 23 | Basin scale: the domain cut into pieces along lattice lines, Nx × Ny ≈ `--pieces` from the domain's bounding box, cut only when the estimated memory at the tolerance exceeds `--memory-budget` (16 GB; BHO dropped as geometry; sub-catchments later from the DEM, never seams), each seam given its heights by a one-dimensional seam pass and then frozen, so pieces refine independently and in parallel; windowed DEM reads with no dense canvas; `rasputin fetch`, a resumable block cache of remote COGs (ANADEM, GLO-30), so meshing never touches the network; pieces plus an index, stitched into one file by default, with the seams removed and locally thinned. Absorbs 15d's window decoding; supersedes Q9's dense canvas at basin scale | **designed and ruled** (`@architect`, 2026-10-01; Ola ruled B1-B14 the same day, B15 (the refusals at half of physical memory: delete both) and B16 (licence notes in the mesh file) on 2026-10-02; 23a-1 merged as #136, 23a-2 merged as #138; 23b (frozen edges and the seam pass) merged as #162 (merge commit 4f56551), green 3c464ec with N18 in, 249 net lines, `@perf`'s acceptance recorded at 91c7cb5 and again at the merged head 4541e38, ACCEPTED; 23c split in two, 23c-1 (the partition rule, the piece vocabulary, the index model, a start mesh from arrays) built at 156 lines and merged as #164 (merge commit e56a2bf), 23c-2 (pieces, the run, the CLI) in progress on `worktree-23c`, not pushed, its state in that branch's copy of `23-basin-scale.md`): eight PRs, 23a-1, 23a-2, 23b, 23c-1, 23c-2, 23d, 23f, 23g (about 225, 373, 285, 215, 265, 235, 300 and 345 lines estimated); measured for it: the whole basin is 3,061 ANADEM blocks, 1.71 GiB, cut into 23 pieces at 1 m at the default budget | `docs/increments/23-basin-scale.md` | | 24 | Release hardening, switched on: the bounds-checking standard library (libc++ FAST, libstdc++ assertions) on by default in every build, with a build option to turn it off for heavy runs (Ola, 2026-10-03: "So normal release, turn it on. For real heavy lifting, option to turn it off."); the loaded mode reported by `_core.hardening`, `rasputin version` and `--stats`; a guard test that fails when a toolchain ignores the macro; `bench.py --hardening on|off`, with hardened the baseline from here on | **designed and ruled** (`@architect`, 2026-10-03; Ola ruled Q1-Q4 the same day): the CMake option `RASPUTIN_HARDENING`, default ON, off by rebuilding into a second venv; an Ubuntu CI leg builds it OFF; one PR; **implemented on `worktree-agent-a08216a68d5534b18`** (red `21d5168`, green `23d4dad`, 53 net production lines; T6 reworked `57e1264`); `@reviewer` APPROVED (round 3); **`@perf` acceptance ACCEPTED** (2026-10-03, AC: built OFF it matches master `6cdc8cc`, meshes byte-identical; hardening costs refine +4.2 to +5.9 % at 1 thread and +3.2 to +11.1 % over 2 to 20 threads; first hardened baseline `docs/benchmarks/2026-10-03/24-on-r2/`, evidence `docs/benchmarks/2026-10-03/24-acceptance.md`). Two extensions in one wheel (option (b)) rejected. Measurement: `docs/benchmarks/2026-10-03/release-hardening/README.md` (#149) | `docs/increments/24-release-hardening.md` | -| 26 | Land cover for the São Francisco basin, MapBiomas (the land-cover half of "the basin's own inputs"): water bodies of at least a minimum area traced on cell edges and reduced with their area kept, as constraints (`rasputin water`); every other class as fractions per triangle from exact cell areas, small entries (under 5 % of the triangle and under 1 ha) dropped and their area carried along a Hilbert curve to the next triangles holding the class (Ola's "local out-of-balance ledger"); `rasputin fetch mapbiomas-c11 --year` | **26a built** (branch `worktree-mapbiomas-26a`: red `cb5f254b`, green `8eec354b`, 163 net production lines; `@perf`'s quick check `38f86930` NO CHANGE, every mesh byte-identical; next code review); 26b onward designed and ruled (`@architect`, 2026-10-03; Ola ruled all 11 questions on 2026-10-04: Collection 11, year 2025 by default; CC BY-SA 4.0 on the outputs; triangles inside a lake carry the raster's own mix, nothing forced to water; 26a-26c now alongside the basin work, the water PRs after the basin PRs, 26e with 23g). Measured for it on the Rio Corrente meshes at 20 to 2 m: renormalising inside triangles loses 8 to 45 % of the water, the ledger keeps every class within 0.05 %. Six PRs, 26a to 26e (about 170, 420, 300, 425, 355 and 210 lines) | `docs/increments/26-land-cover.md` | +| 26 | Land cover for the São Francisco basin, MapBiomas (the land-cover half of "the basin's own inputs"): water bodies of at least a minimum area traced on cell edges and reduced with their area kept, as constraints (`rasputin water`); every other class as fractions per triangle from exact cell areas, small entries (under 5 % of the triangle and under 1 ha) dropped and their area carried along a Hilbert curve to the next triangles holding the class (Ola's "local out-of-balance ledger"); `rasputin fetch mapbiomas-c11 --year` | **26a built** (branch `worktree-mapbiomas-26a`: red `cb5f254b`, green `8eec354b`, 163 net production lines; `@perf`'s quick check `38f86930` NO CHANGE, every mesh byte-identical; code review APPROVED in two rounds; next the push on Ola's yes); 26b onward designed and ruled (`@architect`, 2026-10-03; Ola ruled all 11 questions on 2026-10-04: Collection 11, year 2025 by default; CC BY-SA 4.0 on the outputs; triangles inside a lake carry the raster's own mix, nothing forced to water; 26a-26c now alongside the basin work, the water PRs after the basin PRs, 26e with 23g). Measured for it on the Rio Corrente meshes at 20 to 2 m: renormalising inside triangles loses 8 to 45 % of the water, the ledger keeps every class within 0.05 %. Six PRs, 26a to 26e (about 170, 420, 300, 425, 355 and 210 lines) | `docs/increments/26-land-cover.md` | | 29 | NVE reference catchments: a catchment from the DEM for each of the 140 gauging stations in NVE's reference list of active, unregulated stations (Ola, 2026-10-04: "NVE should have a list of unregulated catchments, where they also record the discharge"), each compared with NVE's own catchment polygon (share of NVE's polygon in ours and of ours in NVE's, area ratio, mean distance between the divides). Adds flow accumulation from increment 22's flood and places each gauge where it physically is: on NVE's mapped river line, then on the DEM's flow path along it (burnt where they disagree), never by an area or flow-count objective (Ola's direction, 2026-10-04); a per-station sensitivity of the area along the river marks ill-posed stations `uncertain`. `rasputin fetch-stations nve-hrd` fetches the stations, rivers and NVE's polygons (NLOD), `rasputin station-catchments` runs them all. Residual inflow between gauges is later; the interfaces leave room for it. Discharge out of scope; station and series numbers kept for a later join. 138 of the 140 polygons lie inside DTM10's data; two cross into Finland and are expected to be refused | **PR 1, 2, 3 and 4 merged (#173, #179, #178, #182); the acceptance run over all 140 stations is in, at `docs/benchmarks/2026-10-05/nve-hrd/` (as fixed in `cd8b5be`): 74 match, 5 close, 6 miss, 39 uncertain, 16 refused (14 of them expected), 33 minutes and at most 7.7 GB; evidence review round 2 approved, the push waits for Ola; PR 5 (the nearest-stream fallback) and acceptance step 5 (the comparison with nearest stream) not started; questions 6 and 10 go back to Ola with the run's figures** | `docs/increments/29-nve-reference-catchments.md` | | 30 | The bottlenecks of `rasputin mesh` on real catchments, in three pull requests, with every output byte-identical: 30a the `land cover` phase (point-in-polygon against prepared polygons, one sort fewer, no sort to pick each area's point), 30b the CORINE clip (point tests before the segment test, a feature whose lines miss the search region skipped whole, no clip of a line lying in the domain's interior), 30c reading the DEM (decode threads from the machine's cores instead of a fixed 4, the seam report's test against the domain's region run only on the nodes it can count). Measured by `@perf` on Numedalslågen and Skiensvassdraget (14.4 s and 21.2 s on battery; `docs/benchmarks/2026-10-06/bottlenecks/README.md`), projected to about 7.8 s and 11.7 s with all three. Approved by Ola 2026-10-06, in that order ("yes, go ahead in that order") | **30a merged (#199)** (`worktree-landcover-speed`: red `6cde0eb`, green `3066d60`, +12 net production lines; `@perf` ACCEPTED on AC power, median of 3: land cover 2.85 → 0.67 s on Numedalslågen and 6.54 → 1.74 s on Skiensvassdraget, whole run 13.66 → 11.41 s and 20.46 → 15.68 s, every `.vtk` byte-identical to the base; `docs/benchmarks/2026-10-06/30a-landcover/README.md`); **30b merged (#200)** (`worktree-clip-speed`: red `08269e77`, green `c5340459`, +23 net production lines; `@perf` ACCEPTED on AC power, median of 3: `features clip` 5.83 → 1.83 s on Numedalslågen and 5.66 → 1.35 s on Skiensvassdraget, whole run 11.39 → 7.52 s and 15.69 → 11.41 s, every `.vtk` byte-identical to the base and every base line of the probe unchanged; `docs/benchmarks/2026-10-06/30b-clip/README.md`); **30c approved by code review round 2; waits for Ola's yes to push** (`worktree-dem-read-speed`, 2026-10-06; Ola said "yes, build 30c"; red `bdf7b57d`, green `0663efc9`, +3 net production lines; `@perf` ACCEPTED on AC power, median of 3: `decode` 1.788 → 1.051 s on Numedalslågen and 1.323 → 0.842 s on Skiensvassdraget, whole run 7.48 → 6.72 s and 11.46 → 10.97 s, every `.vtk` byte-identical to the base and every base line of the probe unchanged; `docs/benchmarks/2026-10-06/30c-dem-read/README.md`; master merged in as `26a5d839`, where every base line of the probe is unchanged and the 16 new lines are all from the red suite) | `docs/increments/30a-landcover-speed.md`, `docs/increments/30b-clip-speed.md`, `docs/increments/30c-dem-read-speed.md` | | 30d | Growing the catchment outline, made faster, with every mesh byte-identical: the same mitred buffer of the domain computed once instead of twice on the reprojected path; the CORINE search region grown from the domain's convex hull instead of taking the hull of the grown domain; and a long staircase outline (SMHI's raster-traced ones) grown in pieces of 1,000 edges and united, behind a gate that keeps smooth outlines on GEOS's single buffer. Measured by `@perf` on Lagan with the European CORINE GeoPackage: 103.4 s on master, about 17 s with all three (`docs/benchmarks/2026-10-07/30d-buffer/README.md`); from the performance review's finding F1 (PR #208). Ola, 2026-10-07: grow the outline in pieces; the gate is an open question, default yes | **merged (#215, 2026-10-07)** (branch `worktree-buffer-speed-2`; 29 net production lines; Python only, no refine or mesh code; Lagan 103.38 s → 16.97 s, meshes byte-identical; push waits on Ola) | `docs/increments/30d-outline-buffer-speed.md` | diff --git a/docs/increments/26-land-cover.md b/docs/increments/26-land-cover.md index cba719af..4a102c5c 100644 --- a/docs/increments/26-land-cover.md +++ b/docs/increments/26-land-cover.md @@ -1,6 +1,6 @@ # Increment 26: land cover for the São Francisco basin — water bodies as constraints, class fractions per triangle -Status: **26a built** on branch `worktree-mapbiomas-26a` (red `cb5f254b`, `@tester`, 48 tests; green `8eec354b`, `@developer`, 163 net production lines by `python3 tools/count_loc.py de9b6c03 HEAD` against the estimate of 170; no mutation round, section 9; section 13 is the as-built record; `@perf`'s quick check `38f86930` ACCEPTED; next code review). 26b onward not built. The design: **by `@architect`, 2026-10-03, at Ola's request; ruled by +Status: **26a built** on branch `worktree-mapbiomas-26a` (red `cb5f254b`, `@tester`, 48 tests; green `8eec354b`, `@developer`, 163 net production lines by `python3 tools/count_loc.py de9b6c03 HEAD` against the estimate of 170; no mutation round, section 9; section 13 is the as-built record; `@perf`'s quick check `38f86930` ACCEPTED; code review rounds 1 and 2, APPROVED; next the push on Ola's yes). 26b onward not built. The design: **by `@architect`, 2026-10-03, at Ola's request; ruled by Ola, 2026-10-04** (all eleven questions; "Ruled by Ola, 2026-10-04", below, and each question marked). Q8 was ruled against the recommendation: triangles inside a lake carry the raster's own mix, nothing is forced to @@ -1546,3 +1546,5 @@ evidence. Idea for the tooling: `bench_quick.py run --record `. **Design review, round 2, 2026-10-03.** Range `3fd0237..097b26b`. Verdict: APPROVED. LOC: 0 production lines. All four round-1 blockers fixed, checked against the corrected probe's saved outputs (769: 2,836 bodies of at least 1.44 ha holding 94.8 %, Três Marias with the upper channel 1,290 km²; basin estimate about 11,600 bodies and 130,000 vertices at 60 m). Nit: 769's largest body is 455 × 279 km, not 270. Not pushed; no CI. **Code review, round 1, 2026-10-11 (26a).** Range `de9b6c03..15a51ed6`, 163 net production lines, @reviewer: CHANGES REQUESTED, doc only, the code clean. Two stale phrases this branch made false: /Users/skavhaug/projects/rasputin/.claude/worktrees/mapbiomas-26a/docs/increments/26-land-cover.md@15a51ed6:8-9 ("Nothing is built. 26a is ready for @tester") and /Users/skavhaug/projects/rasputin/.claude/worktrees/mapbiomas-26a/ROADMAP.md@15a51ed6:53 ("26a ready for @tester"); both fixed in this recording commit. Sound: `_decode_into` (/Users/skavhaug/projects/rasputin/.claude/worktrees/mapbiomas-26a/src_python/tin_engine/io/cog.py@15a51ed6:185-203) is the original `put` closure verbatim, sparse-block refusal and edge clipping unchanged, so partial blocks and NoData behave as before (the quick check's byte-identical meshes agree); `decode_class_window` undoes the half-cell PixelIsArea shift correctly; `class_strips` floor/ceil gives cells whose interior meets the box; `credit_for` leaves ANADEM and GLO-30 unchanged; `fetched` kept on later runs; credit wording and CC BY-SA match the terms page and Ola's Q2 ruling; no prohibited dependency; 48 tests pass; no red scaffolding; check_citations at-risk lines drifted at base. Lesson: a Status paragraph with two statements of readiness drifts; keep one. Idea for 26b: `decode_class_window` does not check `meta.pixel_is_area`; a PixelIsPoint class raster would shift half a cell silently. + +**Code review, round 2, 2026-10-11 (26a).** Range `15a51ed6..7053d795`, 0 production lines, @reviewer: APPROVED. Both stale phrases gone; the replacement at /Users/skavhaug/projects/rasputin/.claude/worktrees/mapbiomas-26a/docs/increments/26-land-cover.md@7053d795:8-9 agrees with the Status line and section 13 (@7053d795:1482); ROADMAP row 26 (/Users/skavhaug/projects/rasputin/.claude/worktrees/mapbiomas-26a/ROADMAP.md@7053d795:53) reads consistently; the round-1 record matches; check_citations exit 0. Not pushed, no CI yet: CI must be green after the push.