Repository navigation
Increment 38: --threads and --memory-budget as settings, no core count hardcoded - #239
Merged
Merged
Conversation
The first fix from the generality audit (rows D1, D2, D3, D5, D6, D7; section 6 item 1), as docs/increments/38-threads-and-memory.md plus the ROADMAP row. Design only; no production code. What it settles: - `--threads N` on mesh, catchment and station-catchments; 0 = every core; RASPUTIN_THREADS is the default when the option is absent (0 there means every core too); a count above the cores is honoured, not clamped (OpenMP and Rayon honour it; the benchmark sweeps to 20 on 10 cores). - One `Resources` model (threads, memory_budget), built once in cli.py from the options and os.environ, carried to open_dem and delineate; the leaf functions take the one integer they use. The C++ keeps a plain integer and its 0 default; no C++ file changes. - Pools default to one worker; resources.py is the only module that reads the machine (a test greps for it). The reprojector's cap of 8 goes; the reprojector inside the target grid's workers runs serial instead of opening a pool inside a pool. - `--memory-budget 8G` caps the mosaic canvas and the flood window; default half of min(physical, cgroup limit) with memory.max (v2) and memory/memory.limit_in_bytes (v1) read from a cgroup root the tests can point at a synthetic directory; cpu.max likewise for the core count. - Stats row "Threads used: N." for every DEM run; run record entry `threads`. - decompose.py and its test deleted; project_structure.md row swapped for resources.py. Measured on 5d2e706: five _core entry points take threads and five call sites reach them, four passing nothing; three pools and four core-count reads plus one memory read (the greps and their output are in section 3). Estimate 140 added, 78 removed, net about 60 (the audit said 130 before counting call sites); worst case +39 % about 120 net. Prior art read on 2026-10-12: OpenMP 5.0 section 6.2, GNU nproc, Rayon's ThreadPoolBuilder, threadpoolctl, Python 3.13's os.process_cpu_count, the JDK's UseContainerSupport (JDK-8146115), the kernel's cgroup-v2 guide. Legacy greps against legacy-archive: nothing on threads; one `#include <memory.h>` on memory. Nothing carried. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… rulings recorded (@architect) Design review round 1 (recorded at 3ff4843) asked for five fixes and three wordings; all are in docs/increments/38-threads-and-memory.md, cites pinned at 5d2e706 with directory paths. python3 tools/check_citations.py: exit 0, 46 at-risk lines, all ROADMAP.md rows, none in this file. (1) T10: the figures now come from a run, not the test's comment. Probe: delineate on tests/python/test_catchment.py:296-304's fixture with physical_memory faked to twice each budget. The run floods three windows, 49 x 49, 113 x 95 and 224 x 175 nodes; after each flood the growth check plans a probe window at the base margin (catchment.py:279), judged by the mosaic cap. Measured: 8192 refused by the mosaic cap ("49 x 49, 9604 bytes at float32"); 9604, 13824 and 14405 refused by the flood cap ("14406 bytes to flood"); 14406 floods the first window and is refused on the probe window ("73 x 55, 16060 bytes"); 235199 refused on the last window ("224 x 175, 235200 bytes to flood"); 235200 floods. The tests pin 12000 (flood cap), 14406 (names 73 x 55, so the first window passed exactly at its figure), 235199 and 235200 (exactly at the budget, three windows); the 8K CLI case is the mosaic refusal naming 49 x 49, 9604 and 8192. The reviewer's "14406 for exactly at" would still raise, on the second window; hence the extra pair at 235200. (2) T11: tests/python/test_layering.py:44's "decompose" row goes and a "resources" row (L1, no first-party import) comes in the red commit; :153 fails on the stale row, :152 on the missing one. (3) T8: one spy per large reprojector site: feature_input.reprojector on test_cli_mesh_features.py:328-338 (the EPSG:4326 GeoJSON over the EPSG:25833 DEM, the one fixture whose feature CRS differs), asserting threads=3 on the call whose source CRS is the file's; land_cover_fractions.reprojector on the fractions fixture (built on every run, the class raster being EPSG:4326). Five tests plus two defaults. (4) Reprojector callers: 15, the grep pasted with its file list. Four sites move large arrays at three places; target_grid.py:239 moves a block's rim and then at most 512 x 512 = 262 144 valid nodes (:32, :238, :252), under the 500 000 threshold (crs.py:70-73), so only :181 nests a pool today. The other eleven listed by line with what each moves. (5) cpu.max: the memory rule. A quota counts only when the file holds two words, both positive whole numbers (max is no quota; a zero or missing period is no quota, never a division); section 6 and T3 list the malformed files, each giving the plain count, never an exception. Wordings: the two refusals read "(set with --memory-budget; half the machine's memory when it is not set)", true with or without a budget; section 4.5 says bench.py's child prints "Threads used: 10." whatever the sweep's count, since its monkeypatch overrides refine only; T9 names DerivedCommand.argv(model) (mcp/schema.py:108). Section 10 is now "Ola's rulings": "Yes to all", 2026-10-12, taking each default (mesh gets --memory-budget; a count above the cores is honoured; a container's CPU quota counts toward every core; the row reads "Threads used: 3."), with sections 4.1-4.4 and 7 pointing at it. Status: designed, review round 1 fixes in; next round 2. Estimate unmoved at about 60 net. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The failing suite of docs/increments/38-threads-and-memory.md, section 8,
before any production code. 110 tests are red, 85 of them in two new files;
every one fails on a missing module, option, keyword or wording, none on a
typo (the red run's distinct reasons: 65 "No module named
tin_engine.resources", 19 "No such option: --threads" or "--memory-budget",
7 "plan_mosaic() got an unexpected keyword argument 'memory_budget'", 2
"reprojector() got an unexpected keyword argument 'threads'", the rest the
old "Threads: 10 (hardware concurrency)" wording, the cap of 8 workers, the
block decoder's default of 10 workers, crs.WORKERS still present, the
layering table's missing row, decompose still importable, and the five
modules that still read the machine). The 606 other tests in the touched
files pass; the full Python suite passes apart from the red tests.
New files:
- tests/python/test_resources.py (T1 to T5, T11): the grep-equivalent that
only resources.py holds cpu_count, sched_getaffinity, sysconf or
/sys/fs/cgroup; the Resources model (frozen, never 0; fallback() is one
thread and half the memory); the threads rule of section 4.2, one case
each, with 1.5 and 0x10 added to abc and -1 as non-whole numbers;
machine_cores on synthetic cpu.max files (2 CPUs, 0.5 rounds up to 1,
1.5 to 2, and every malformed case of section 6 plus an unreadable file,
skipped as root); machine_memory on synthetic memory.max and v1
memory.limit_in_bytes (1 GiB and 512 MiB, under the machine as asserted;
v1's 2^63 loses to the physical figure; v2 wins over v1; max, 0, -1, text
and an empty file are no limit); parse_bytes and the default budget; and
find_spec("tin_engine.decompose") is None. Every machine figure is
computed from the module's own readers, never a literal.
- tests/python/test_cli_threads.py (T6 to T8, T10's command line): the
stats row "Threads used: N." from --threads, from RASPUTIN_THREADS, from
both, from 0 and from nothing (every core, through machine_cores()), on a
stride run too; the --record entry threads and the Inputs row "Threads
used"; --threads -1 and RASPUTIN_THREADS=abc or -1 refused in the CLI's
words; the help of mesh, catchment and station-catchments lists both
options; a catchment run with --threads 3. One spy per _core site
(cli.refine, edge_strip.refine_strip, final_check.refine_points on the
Velhas --out-crs run, cli.SlopeTolerance on a --tolerance-slope run,
land_cover_fractions._core.class_areas on the fractions fixture), each
asserting the mesh is still written. One spy per pool (io/cog.py on a
cached source and on a tile, target_grid.py's two pools on the Velhas
run) and per large reprojector site (target_grid.reprojector with
threads=1 inside the workers, feature_input.reprojector on the EPSG:4326
gallery over the EPSG:25833 DEM, picking the call from the file's CRS to
the DEM's, land_cover_fractions.reprojector on the fractions fixture).
--memory-budget 8K on the catchment fixture refused by the mosaic cap
naming 49 x 49, 9604 and 8192; 8KB names the text; mesh --memory-budget 1
refused naming 17 x 21, 1428 and the option, 8G passes. The 8K and 12000
figures were re-measured through the CLI on test_cli_catchment's dem_dir
with the cap faked to twice the budget before being written down.
Changed:
- test_catchment.py: the monkeypatched cap test becomes TestTheMemoryBudget
through delineate(..., resources=Resources(threads=1, memory_budget=)):
12000 names 49 x 49, 14406 bytes to flood and 12000; 14406 names 73 x 55,
16060; 235199 names 224 x 175, 235200; 235200 floods three windows of the
measured shapes; resources=None still floods. Resources is imported inside
the helper so only these tests fail on its absence.
- test_mosaic.py: M5 on plan_mosaic(..., memory_budget=) (the plan fixture
passes the keyword only when given); the refusal names --memory-budget;
physical_memory's test moves to test_resources.py as T4's plain case.
- test_dem_input.py: the cap through open_dem(..., resources=).
- test_io_cog.py: the default thread count is one worker (was the cores).
- test_crs.py: crs.WORKERS is gone; the pool takes 9 workers for 9 chunks
with threads=9 and 2 for 2 chunks; without a count no pool opens.
- test_stats.py, test_cli_mesh_stats.py: "Threads used: 10.", printed on
every DEM run and never on a fixture run (T12).
- test_mcp_schema.py: T9, threads and memory_budget in the three tools'
schemas and in argv.
- test_layering.py: the decompose row goes, resources joins L1.
- test_decompose_partition.py deleted (T11).
Left out, by rule: T11's assertion on project_structure.md, which is a
prose file under tools/ci_changes.py and so may not be read by a test
(conftest.py, h11 T5); @Reviewer reads its two rows instead.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ompose test pinned at 5d2e706, by the main session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The minimal code for docs/increments/38-threads-and-memory.md, sections 4 to 6 and Ola's four rulings (section 10), against @tester's red commit 4b9c0df. No test file is touched. New: src_python/tin_engine/resources.py, the one module that reads the machine. Resources (frozen; threads >= 1, memory_budget >= 1; fallback() is one thread and half the machine's memory); machine_cores() (the process's count, capped by a cgroup v2 cpu.max quota of two positive whole numbers, ceil(MAX / PERIOD); anything else, a negative quota included, is no quota); machine_memory() (physical memory capped by memory.max, else v1 memory/memory.limit_in_bytes; "max", 0, -1, text, an empty or unreadable file are no limit); parse_bytes() (a whole number with an optional K, M, G or T, 1024-based; 0, "8GB", "1.5G", "G", "" refused naming the text); resources() (section 4.2's table: --threads, else RASPUTIN_THREADS stripped and whole, 0 = every core; --memory-budget, else half the memory). Where the value travels: cli.py gets two Annotated aliases THREADS (min=0, metavar N: the default "<int range>" widened Rich's metavar column and truncated the longest option names in --help) and MEMORY_BUDGET on mesh, catchment and station-catchments, resolves them once per command (_resources; a refusal is a usage error naming RASPUTIN_THREADS or --memory-budget), and passes the count to the five _core sites (refine, SlopeTolerance, edge_strip.run -> refine_strip, final_check.run -> refine_points, assign -> class_areas) and the three pools: repository_for and both repositories keep `threads` for the block decoder; open_dem(request, resources) plans every mosaic under the budget and resamples and checks on the count; the reprojector inside target_grid's workers takes threads=1; FeatureRequest.threads reaches _Tally's reprojector; index_space's reprojector takes assign's count. The library defaults are one worker (reprojector, decode_window, decode_class_window, assign) and the C++'s 0 for the two _core wrappers. crs.WORKERS and mosaic.physical_memory are gone; the chunked reprojector takes min(threads, chunks) workers and opens no pool for threads=1. The budget: plan_mosaic(..., memory_budget=None) caps the canvas (None is Resources.fallback()'s); delineate(request, repository, resources=None) passes it to _grow and _plan; both refusals read "over the memory budget of B bytes (set with --memory-budget; half the machine's memory when it is not set)". run_batch(..., resources=) hands it to each delineate. dem_input._domain_plan's budget defaults to None (test_cli_fetch calls it with two arguments). The report: stats.render prints "Threads used: N."; the record gains the Inputs entry threads ("Threads used"), set on every DEM run; a fixture run has neither. decompose.py is deleted; mesh_index's docstring and project_structure.md's rows follow; README and INSTALL name RASPUTIN_THREADS and --memory-budget. Beyond the design, because a test pins it: mcp/schema.py keeps a constraint of 0 (`minimum: 0` for --threads; `if v` dropped it), one line, where section 4.5 said no mcp/ change. Not green, for @tester (the full suite: 7402 passed, 21 failed before the help and _domain_plan fixes above, which clear 4; the 17 left are tests that pin the old surface or stand in for a function with a narrower signature than the design gives it): test_layering's rows for cli, dem_input, catchment, catchment_batch and mosaic lack `resources`, which section 4.4 has each import (5 tests); test_cli_mesh_domain_crs's as_16_opened_it(request) stands in for open_dem(request, resources) with one parameter (2); test_cli_mesh_plain_output wraps cli.refine with `real(*a, **k, threads=1)`, which the CLI's own threads= keyword now collides with (--threads 1 is the way since 38), and asserts "threads" is not a record key, against T6 and section 4.4 (2); test_mosaic's test_the_plan_never_loads pins plan_mosaic's parameters without memory_budget (1); test_feature_input's fake(src, dst) reprojectors and single_call_reprojector(src, dst) (3) and test_target_grid's _SpiedReprojector.__call__(source, target) (4) do not take the threads= keyword the design's call sites pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e-38 surface (@tester) After @developer's green b252a70, every red test of 4b9c0df passes but the five layering rows, and 17 older tests failed on the test side. Each fixed in tests/ only, as docs/increments/38-threads-and-memory.md intends: - test_layering.py: `resources` added to the rows for cli, dem_input, catchment, catchment_batch and mosaic (5 tests). Checked with importscan against b252a70: each module's first-party import set grew by exactly tin_engine.resources, and resources is L1, under all five. - test_cli_mesh_domain_crs.py: `as_16_opened_it(request, resources=None)` now stands in for `open_dem(request, resources)` and passes the second argument through (2 tests). - test_cli_mesh_plain_output.py: the bit-identity test runs `--threads 1` and `--threads 2` to the same paths instead of wrapping cli.refine with a colliding keyword; the mesh bytes are equal, the records differ only in the `threads` entry (1 and 2) and the echoed command line. The "threads is not a record key" assertion goes, as T6 pins the entry (2 tests). Two imports left unused by the rewrite removed. - test_mosaic.py: plan_mosaic's parameters gain memory_budget (1 test). - test_feature_input.py: the two `fake(src, dst)` reprojectors and `single_call_reprojector` accept `threads=1` (the fakes pass it to the real one; the pre-35 verbatim copy ignores it) (3 tests). - test_target_grid.py: `_SpiedReprojector.__call__` accepts `threads` and passes it through (4 tests). - test_mcp_schema.py: T9's docstring no longer says "no change to mcp/"; it names the one line kept (`minimum: 0` is a bound). No red scaffolding of 38's remained (the `importorskip("mcp")` and the test_mosaic docstring predate it). Full `pytest tests/python/` with the coverage gate: 7422 passed, 1 failed, 18 skipped, 98.44% coverage (floor 85%). The one failure, test_core_cdt.py::TestGil::test_the_ticker_distinguishes_a_held_gil_from_a_released_one ("a GIL-holding call let 7 ticks through", bound 3), ran beside the three detached `rasputin mesh` processes; alone it passed 5 of 5 runs. Untouched by 38; a load-sensitive bound, reported in the handback. `ruff check tests/` and `ruff format --check tests/`: clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… Status line and ROADMAP row, by the main session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…three no-mcp-change sentences corrected, by the main session Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ADMAP row, by the main session Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
skavhaug
enabled auto-merge
October 12, 2026 20:46
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Oct 12, 2026
skavhaug
enabled auto-merge
October 13, 2026 05:29
…P rows 38 and 39 both kept, by the main session Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Oct 13, 2026
…tree-threads (@developer) Conflicts only in src_python/tin_engine/cli.py, resolved by keeping both sides: the feature sources use 41's loaded maps; _open_features takes the vocabulary and then the thread count (threads: int = 1 kept last, after 41's required vocabulary); FeatureRequest gets sources, vocabulary and threads; _dem_mesh takes threads: int = 0 then vocabulary, and its call passes res.threads then vocabulary. feature_input.py, run_record.py and the docs merged without conflict. Gates: ruff check, ruff format --check, mypy clean; pytest 7554 passed, 18 skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Increment 38,
docs/increments/38-threads-and-memory.md(section 12 is the as-built record), the first fix from the generality audit. 96 net production lines bypython3 tools/count_loc.py 5d2e706d HEAD(237 added, 141 removed).tin_engine/resources.py: oneResourcesmodel, the only place that reads the machine. Threads from--threads(0 = all cores),RASPUTIN_THREADSas the default, else the machine's cores with a container'scpu.maxquota counted; a memory budget from--memory-budget, with a container'smemory.maxread when present.--threadsand--memory-budgetonmesh,catchmentandstation-catchments. The count reaches all five C++ entry points and the three Python pools; the reprojector's fixed cap of 8 threads is gone. The flood and mosaic memory caps are overridable, and refusals name--memory-budgetand the figure.threadsentry. The unuseddecompose.pyis deleted.mcp/schema.pykeeps a zero lower bound (one line).Evidence: design approved after two review rounds; red
4b9c0dfe(110 tests), greenb252a70d, test fixes9de3212c; full suite 7,422 passed.@perf's quick check against a baseline refreshed on an idle machine: NO CHANGE, meshes identical on every case. Code review rounds 1 and 2, APPROVED.🤖 Generated with Claude Code