Skip to content

refactor: consolidate grid arrays into type(grid_axis) :: x, y, z#1430

Closed
sbryngelson wants to merge 7 commits into
masterfrom
refactor/grid-axis-type
Closed

refactor: consolidate grid arrays into type(grid_axis) :: x, y, z#1430
sbryngelson wants to merge 7 commits into
masterfrom
refactor/grid-axis-type

Conversation

@sbryngelson
Copy link
Copy Markdown
Member

Summary

Implements item 1 of #1427: consolidates the nine flat allocatable grid arrays into a type(grid_axis) derived type across all three executables (47 files).

Before:

real(wp), target, allocatable, dimension(:) :: x_cb, y_cb, z_cb
real(wp), target, allocatable, dimension(:) :: x_cc, y_cc, z_cc
real(wp), target, allocatable, dimension(:) :: dx, dy, dz

After:

type grid_axis
    real(wp), allocatable, dimension(:) :: cb, cc, spacing
end type grid_axis

type(grid_axis), target :: x, y, z
! usage: x%cc(i), x%cb(i), x%spacing(i)

Design decisions

  • pre_process retains scalar dx/dy/dz as minimum cell-width values (not per-cell arrays); only x_cb/x_cc are folded into x%cb/x%cc
  • OpenMP GPU target declares the whole struct (x, y, z) since component-level declare target is invalid in OpenMP; OpenACC uses component-level declarations
  • 2dHardcodedIC.fpp uses #ifdef MFC_PRE_PROCESS to handle dx*dy (scalar min widths) vs x%spacing(i)*y%spacing(j) (per-cell) contexts

Variable collisions fixed

  • m_chemistry.fpp: local loop integers x, y, zcx, cy, cz
  • m_weno.fpp: local scratch y(1:4)ys
  • m_viscous.fpp: local scratch dx(1:3)ds
  • m_ibm.fpp: local cell-size scalars dx, dy, dzdx_loc, dy_loc, dz_loc
  • m_cbc.fpp: Fypp template d${XYZ}$${XYZ}$%spacing

Test plan

  • All 6 CI precheck gates pass (./mfc.sh precheck -j 8)
  • All three targets build with gfortran (./mfc.sh build -j 8 --no-mpi)
  • 160/160 non-convergence 1D tests pass
  • 222/222 non-convergence 2D tests pass
  • CI full matrix (gfortran, nvfortran, Cray ftn, Intel ifx)

Replace flat allocatable arrays x_cb/y_cb/z_cb, x_cc/y_cc/z_cc,
and dx/dy/dz with a derived type having .cb, .cc, and .spacing
components. All three executables (pre_process, simulation,
post_process) updated across 47 files.

Key design decisions:
- pre_process keeps scalar dx/dy/dz as minimum cell-width scalars;
  only x_cb and x_cc are folded into x%cb and x%cc
- OpenMP GPU target uses whole-struct declare target (x, y, z) since
  component-level declare target is invalid; OpenACC uses component-level
- 2dHardcodedIC.fpp wraps dx*dy in #ifdef MFC_PRE_PROCESS for the
  scalar vs per-cell context difference

Special variable collisions fixed:
- m_chemistry.fpp: local integer x/y/z -> cx/cy/cz
- m_weno.fpp: local real y(1:4) scratch -> ys
- m_viscous.fpp: local real dx(1:3) scratch -> ds
- m_ibm.fpp: local scalar dx/dy/dz -> dx_loc/dy_loc/dz_loc
- m_cbc.fpp: Fypp template d${XYZ}$ -> ${XYZ}$%spacing
@github-actions
Copy link
Copy Markdown

Claude Code Review

Head SHA: 64ca703

Files changed:

  • 47 files
  • src/common/include/1dHardcodedIC.fpp
  • src/common/include/2dHardcodedIC.fpp
  • src/common/m_derived_types.fpp
  • src/simulation/m_global_parameters.fpp
  • src/simulation/m_derived_variables.fpp
  • src/simulation/m_hypoelastic.fpp
  • src/simulation/m_riemann_solvers.fpp
  • src/simulation/m_weno.fpp
  • src/pre_process/m_global_parameters.fpp
  • src/post_process/m_global_parameters.fpp

Findings

[Correctness] Mathematical derivative notation corrupted by mechanical substitution

src/simulation/m_derived_variables.fpp (diff lines ~1372–1396), src/simulation/m_hypoelastic.fpp (diff line ~3803), src/simulation/m_riemann_solvers.fpp (diff lines ~6663, ~6673):

The rename pass substituted dxx%spacing, dyy%spacing, dzz%spacing inside comments where those tokens represent standard calculus notation, not Fortran variable references:

-  ! d()/dx
+  ! d()/x%spacing

-  ! S_thetatheta += rho * ( -(tau_thetatheta + 2/3*G)*(du/dx + dv/dr + v/r) + ...
+  ! S_thetatheta += rho * ( -(tau_thetatheta + 2/3*G)*(du/x%spacing + dv/dr + v/r) + ...

-  ! 1D: d/dx flux only & Bx = Bx0 = const.
+  ! 1D: d/x%spacing flux only & Bx = Bx0 = const.

d()/dx, du/dx, d/dx are mathematical derivative operators, not references to the spacing array. These comments should be restored to their original form.


[Correctness] Non-uniform grid IC inconsistency in RMI case (2dHardcodedIC.fpp, hcid=208)

src/common/include/2dHardcodedIC.fpp (diff lines ~196–200):

The PR introduces an #ifdef MFC_PRE_PROCESS split for the interface smoothing function:

+#ifdef MFC_PRE_PROCESS
             fsm = 0.5_wp*(1.0_wp + erf(d/(ei*sqrt(dx*dy))))
+#else
+            fsm = 0.5_wp*(1.0_wp + erf(d/(ei*sqrt(x%spacing(i)*y%spacing(j)))))
+#endif

In pre_process, dx and dy are scalar minimum cell-widths (real(wp) :: dx, dy in src/pre_process/m_global_parameters.fpp). In simulation, per-cell spacing x%spacing(i)*y%spacing(j) is now used. For non-uniform grids these differ at every cell that is not the minimum-width cell: pre_process smooths the interface with the global minimum width, while simulation smooths with the local cell width. On non-uniform grids this produces a different IC in pre_process than simulation would reconstruct from the same hcid. The pre_process branch should document this approximation explicitly, or be updated to use per-cell spacing if x%spacing is made available in pre_process.


[GPU Correctness] OpenMP GPU_DECLARE on derived type with allocatable components

src/simulation/m_global_parameters.fpp (diff lines ~3664–3668):

-    $:GPU_DECLARE(create='[x_cb, y_cb, z_cb, x_cc, y_cc, z_cc, dx, dy, dz, dt, m, n, p]')
+#if defined(MFC_OpenACC)
+    $:GPU_DECLARE(create='[x%cb, y%cb, z%cb, x%cc, y%cc, z%cc, x%spacing, y%spacing, z%spacing, dt, m, n, p]')
+#elif defined(MFC_OpenMP)
+    $:GPU_DECLARE(create='[x, y, z, dt, m, n, p]')
+#endif

The new grid_axis type (real(wp), allocatable, dimension(:) :: cb, cc, spacing) has allocatable array components. The OpenACC path is correct: it names each allocatable component individually. The OpenMP path maps the three struct variables as a whole.

OpenMP target declare create on a non-allocatable host variable that contains allocatable array members does not recursively allocate those allocatable components on the device. The device-side struct holds array descriptors, but the device memory they point to is not automatically created. Without explicit device allocation of x%cb, x%cc, x%spacing (and same for y, z), GPU kernel accesses to these arrays will use uninitialized device pointers, producing incorrect results or crashes on Cray and AMD flang GPU builds. The OpenMP path should list the individual allocatable array components explicitly, matching the OpenACC path.

Implements items 7 and 5 from issue #1427:
- x_a/x_b/y_a/y_b/z_a/z_b -> type(bounds_info) :: x_stretch, y_stretch, z_stretch
- bf_x/bf_y/bf_z + k_x/w_x/p_x/g_x (and y/z) -> type(body_force_axis) :: bf_x, bf_y, bf_z
@sbryngelson sbryngelson deleted the refactor/grid-axis-type branch May 12, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant