Skip to content

perf(ghost): read the boundary geometry at its level in the outer ghost update - #504

Open
gouarin wants to merge 2 commits into
hpc-maths:mainfrom
gouarin:set-algebra-granularity
Open

gouarin wants to merge 2 commits into
hpc-maths:mainfrom
gouarin:set-algebra-granularity

Conversation

@gouarin

@gouarin gouarin commented Sep 7, 2026

Copy link
Copy Markdown
Contributor
  • I have installed pre-commit locally and use it to validate my commits.
  • The PR title follows the conventional commits convention.
    Available tags: 'build', 'chore', 'ci', 'docs', 'feat', 'fix', 'perf', 'refactor', 'revert', 'style', 'test'
  • This new PR is documented.
  • This new PR is tested.

Description

Half to 60 % of ghost update (advection 2D and 3D, scalar Burgers 2D) is the filling of the outer ghosts by the boundary conditions, and 85 to 90 % of that phase is pure set algebra visiting 20 to 40 rows per interval emitted. Two causes, both in the way the expressions are written:

  • the domain, the corners and the B.C. regions are stored at max_level and coarsened on the fly with self(x).on(level), whose traverser materialises 2^(max_level - level) fine rows per coarse row;
  • difference(cells, translate(domain, -d)) visits every row of cells even when the boundary in direction d lies on a single row.

The geometry that depends on the domain only is now built once per domain at every level and inherited by the meshes constructed from a reference mesh (level_pyramid.hpp):

  • mesh.corner(direction, level): corner pyramids. This also stops construct_corners from recomputing the domain corners at every mesh construction (2.1 s of a 17.7 s 3D run);
  • mesh.boundary_inner_layer(level, d) = domain \ translate(domain, -d) and mesh.boundary_outer_layer(level, d, k) = translate(domain, k d) \ translate(domain, (k - 1) d);
  • bc.region(d, level): the B.C. regions at every level.

The outer ghost expressions read those sets at their level and intersect with the thin layers instead of subtracting the domain: domain_boundary is intersection(cells, inner layer); project_bc intersects the translated union with the outer layer (the union lies inside the domain, so the sets are the same); predict_bc, on a box domain, predicts each layer straight from the expression with the order-0 prediction kernel per interval instead of materialising a level cell array and copying cell by cell. The non-box paths keep their former expressions.

The first commit adds the measurement build used to find this (-DSAMURAI_MEASURE_SET_ALGEBRA=ON, off by default, expands to nothing otherwise): per-phase timers in update_ghost_mr and, per apply(), the pure set algebra time, the rows visited, the intervals emitted, the row blocks and the empty row segments. It can be reviewed or dropped independently of the second commit.

Measurements

Sandbox Linux aarch64, gcc 15, Release, 3 runs, median, --timers, --nfiles 1:

advection 2D (levels 4-10) scalar Burgers 2D (levels 4-10) advection 3D (levels 4-8)
ghost update (adaptation) 0.430 s -> 0.241 s (x1.78) 2.583 s -> 1.819 s (x1.42) 4.225 s -> 2.219 s (x1.90)
ghost update (time loop) 0.213 s -> 0.121 s (x1.76) 1.281 s -> 0.895 s (x1.43) 2.047 s -> 1.064 s (x1.92)
mesh construction 0.243 s -> 0.227 s 1.548 s -> 1.486 s 3.951 s -> 1.889 s (x2.09)
total runtime 2.302 s -> 1.991 s (x1.16) 12.427 s -> 11.205 s (x1.11) 17.710 s -> 12.651 s (x1.40)

bench_set (3 repetitions, medians): no entry moves by more than 2 %, the codegen canary BM_RefineFootprint_adapted<3> is unchanged; subset/ is not touched. Not yet measured with clang on macOS nor with MPI.

Related issue

None.

How has this been tested?

  • h5diff on the 16 HDF5 files written by finite-volume-advection-2d (both prediction stencils), finite-volume-scalar-burgers-2d and finite-volume-advection-3d: bit-identical to main.
  • New tests/test_boundary_geometry.cpp (2D and 3D, adapted mesh whose refinement reaches the boundary and the corners, levels 2 to 6): every precomputed set equals its .on(level) definition cell by cell, every rewritten expression equals its former form, and the predict_bc layers are disjoint.
  • test_samurai_lib: 442 tests pass (440 existing + 2 new).
  • All targets (demos, tests, benchmarks) compile.

Code of Conduct

By submitting this PR, you agree to follow our Code of Conduct

  • I agree to follow this project's Code of Conduct

@codacy-production

codacy-production Bot commented Sep 7, 2026

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 17 complexity · 0 duplication

Metric Results
Complexity 17
Duplication 0

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

Add the CMake option SAMURAI_MEASURE_SET_ALGEBRA (OFF by default). When it
is on and the timers are enabled, every apply() traverses its set
expression three times: once with a sink callback (timer "set algebra
(pure)", count column = rows visited), once for real (timer "set algebra +
callback", count column = intervals emitted) and once, untimed, to count
the row blocks (consecutive rows emitting the same x intervals) and the
empty row segments. SAMURAI_MEASURE_APPLY=0 in the environment keeps only
the phase timers.

SAMURAI_MEASURE_PHASE(name) splits a hot function into timed phases and
publishes the phase name, so that finalize() can print the counters
(applies, rows, intervals, cells, blocks, empty segments) per phase.
update_ghost_mr is split into projection, prediction, outer (B.C.) and
periodic ghosts.

Everything expands to nothing without the option.
@gouarin
gouarin force-pushed the set-algebra-granularity branch from c3916cb to 681edbc Compare September 7, 2026 07:42
…st update

Half to 60 % of the ghost update (advection 2D and 3D, scalar Burgers 2D)
was the filling of the outer ghosts by the boundary conditions, and 85 to
90 % of that phase was pure set algebra visiting 20 to 40 rows per
interval emitted. Two causes, both in the way the expressions were written:

- the domain, the corners and the B.C. regions are stored at max_level and
  were coarsened on the fly with self(x).on(level), whose traverser
  materialises 2^(max_level - level) fine rows per coarse row;
- difference(cells, translate(domain, -d)) visits every row of cells even
  when the boundary in direction d lies on a single row.

The geometry that depends on the domain only is now built once per domain
at every level and inherited by the meshes constructed from a reference
mesh (level_pyramid.hpp):

- mesh.corner(direction, level): corner pyramids, which also stops
  construct_corners from recomputing the domain corners at every mesh
  construction (2.1 s of a 17.7 s 3D run);
- mesh.boundary_inner_layer(level, d) = domain \ translate(domain, -d) and
  mesh.boundary_outer_layer(level, d, k) = translate(domain, k d) \
  translate(domain, (k - 1) d);
- bc.region(d, level): the B.C. regions at every level.

The outer ghost expressions read those sets at their level and intersect
with the thin layers instead of subtracting the domain: domain_boundary is
intersection(cells, inner layer); project_bc intersects the translated
union with the outer layer (inner is inside the domain, so the sets are
the same); predict_bc, on a box domain, predicts each layer straight from
the expression with the order-0 prediction kernel per interval instead of
materialising a level cell array and copying cell by cell. The non-box
paths keep their former expressions.

Measured in the sandbox (gcc 15, Release, 3 runs, median), ghost update
inside the adaptation: advection 2D 0.430 s -> 0.241 s (x1.78), scalar
Burgers 2D 2.583 s -> 1.819 s (x1.42), advection 3D 4.225 s -> 2.219 s
(x1.90); total runtime x1.16, x1.11 and x1.40. The HDF5 outputs of the
three demos are bit-identical to main. bench_set: no entry moves by more
than 2 %, the codegen canary BM_RefineFootprint_adapted<3> is unchanged
(subset/ is not touched).

tests/test_boundary_geometry.cpp checks, cell by cell on an adapted 2D
and 3D mesh whose refinement reaches the boundary, that every precomputed
set equals its .on(level) definition, that the rewritten expressions
equal the former ones, and that the predict_bc layers are disjoint.

The geometry is a pure function of the domain: it is copied to the
neighbour meshes together with the domain, and the accessors build it on
first use when a mesh received its domain without going through
finalize_mesh (a deserialised neighbour mesh, on which
petsc::compute_cell_ownership calls domain_boundary_outer_layer).
@gouarin
gouarin force-pushed the set-algebra-granularity branch from 681edbc to 41d6788 Compare September 7, 2026 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant