Multilayer-grating design workflow: survey + energy scan, with a web tab

.gitignore

@@ -6,6 +6,7 @@ examples/reference_baselines/**/results/latest.csv
 .codex
 uv.lock
 examples/simulation/multilayer_theta_search/results/theta_scans/*
+examples/simulation/multilayer_grating_design/results/
 reticolo
 *.log
 .token

CHANGELOG.md

@@ -2,7 +2,19 @@
 
 ## Unreleased
 
-- Added a three-stage multilayer-grating design workflow: `grax.run_d_spacing_study`, `grax.run_gamma_study` and `grax.run_blaze_study`, driven by one frozen `grax.MultilayerOptimizationConfig`. Stage 0 derives a bilayer d-spacing from the grating geometry (grazing angle at the configured CFF, then the first-order Bragg law) and scans practical candidates with planar-multilayer reflectivity; stage 1 scans the bilayer thickness ratio `gamma`; stage 2 builds the multilayer-coated blazed grating and scans the blaze angle through `run_multilayer_theta_search_sweep`. Stages hand values forward only through `optimization_state.json` and only when a config value is the string `"auto"` -- a numeric value always wins, and no stage rewrites the config. Reflectivity for stages 0-1 comes from the new `grax.MultilayerReflectivity`, a thin wrapper over XRT's dynamical-diffraction engine; `xrt` (already a hard dependency) is imported lazily, so `import grax` still pulls in neither `xrt` nor `matplotlib.pyplot` (now covered by a test). A runnable Ru/B4C second-order example lives at `examples/simulation/multilayer_optimization_rub4c/`.
+- Added a multilayer-grating design workflow: `grax.MultilayerGratingDesigner`, driven by one frozen `grax.MultilayerDesignConfig`. The rough/fine/final theta-search scan settings are a separate frozen `grax.ThetaSearchScanSettings`, held on two independent config fields -- `survey_scan_settings` and `energy_scan_settings` -- so the survey (many single-energy searches, one per grid cell) and the energy scan (fewer designs, often many energies each) can be tuned for speed or accuracy independently. `run_survey` scans a 2-D grid of bilayer d-spacing against blaze angle and, for every pair, builds the multilayer-coated blazed grating and runs a single-energy multilayer theta search through `run_multilayer_theta_search_sweep` (using `survey_scan_settings`) with `output_dir` set to that pair's own folder; the incident angle is a result of that search, not an input parameter. Each pair therefore keeps the full standard theta-search artifact bundle under `survey/runs/d<d>nm/blaze<b>deg/` (`multilayer_theta_search_summary.csv`, `*_all_orders.csv`, `theta_scans/`, profile/stack plots, `checkpoints/`), plus a `search_parameters.json` recording the search settings for review and tuning; each `d<d>nm/` folder also gets an `overlay.png` of its blaze angles. From the selected results the survey reports, per d-spacing, the blaze angle with the highest efficiency (`optimal_blaze_deg = argmax_blaze efficiency`), and the headline artifacts are three plots, all under `MultilayerDesignConfig.plot_dir` (`<output_dir>/plots/`): optimal blaze angle versus d-spacing (labelled with efficiency), max efficiency versus d-spacing (labelled with the optimal blaze angle), and a `(d, blaze) -> efficiency` heatmap of the full `efficiency_map` with the optimal-blaze ridge overlaid. `survey/survey.csv` holds the combined per-cell table. `run_energy_scan` then sweeps chosen `(d, blaze)` designs over an explicit energy range with `run_multilayer_theta_search_sweep`, and additionally writes each design's `EnergyScanResult.titled_plot_path` into that same shared `plot_dir` -- efficiency versus energy titled with the coating (`MultilayerDesignConfig.coating_label`, defaulting to `"<material_a name>/<material_b name>"`), the diffraction order, and that design's d-spacing and blaze angle, so plot titles show the real compound (e.g. "Ru/B4C") even when a material is modelled with a stand-in optical-constants table; since every design's plot shares one folder, the filename itself carries all four (`efficiency_vs_energy_<materials>_order<n>_d<d>nm_blaze<b>deg.png`). `evaluate_survey` / `evaluate_energy_scan` re-derive the aggregates (rebuilt `survey.csv`, regenerated plots, collected results) from the per-run artifacts already on disk without launching any solver -- the example's `0_run_survey.py --eval` and `1_run_energy_scan.py --eval`. `run_survey` / `run_energy_scan` take optional `progress_callback` (called with a `grax.StageProgress` per item) and `run_survey` also takes `should_continue` (stops between cells, optimal blaze computed from the completed subset, `aborted=True` on the result). `import grax` still pulls in neither `xrt` nor `matplotlib.pyplot` (covered by a test). A runnable Ru/B4C second-order example lives at `examples/simulation/multilayer_grating_design/`. This replaces the earlier unreleased three-stage `run_d_spacing_study` / `run_gamma_study` / `run_blaze_study` workflow, its `grax.MultilayerOptimizationConfig`, and the `grax-web` "Multilayer study" page, all of which are removed.
+- `grax.MultilayerGratingDesigner` gained `plot_energy_scan_overlay(results)`, which overlays several designs' efficiency-versus-energy curves on one axis and writes `plots/efficiency_vs_energy_comparison_<materials>_order<n>.png`; `run_energy_scan` and `evaluate_energy_scan` call it automatically whenever they produce two or more designs. `run_energy_scan` also gained a keyword-only `should_continue`, mirroring `run_survey`, so a long scan can be stopped cooperatively between designs (it returns the designs completed so far).
+- Added a **Multilayer design** tab to the grax web app, driving the two-step `grax.MultilayerGratingDesigner` workflow end to end. The creation form exposes every `grax.MultilayerDesignConfig` field grouped into the dataclass's three sections (shared / survey-only / energy-scan-only), with the two `grax.ThetaSearchScanSettings` blocks and the runtime knobs behind a collapsed *Advanced*, and a live readout of how many theta searches the survey grid implies (warning past 200). Running the survey shows a live progress bar and then its three headline plots; the energy scan is then chosen with three buttons -- *Scan best* (the survey's global argmax), *Scan all d* (the optimal blaze at every d-spacing), or *Choose manually* to build a list of `(d, blaze)` cases from dropdowns that show each cell's surveyed efficiency. Ticking *scan the best design over energy automatically* on the form chains step 2 straight off the survey, so the plots render while the scan runs underneath them. Two or more scanned designs also get an overlay comparison plot. Either stage runs in a background thread against the shared run registry (so it appears in `/system/resource-status`), can be aborted with a keep-or-discard choice, and can be reset; re-running the survey marks a finished energy scan *stale*. Each study lives in `<data_dir>/multilayer_designs/<id>/`, holding `study.json` beside the library's own `survey/`, `plots/` and `energy_scan/` trees.
+- The **Multilayer design** tab's three step-1 plots are now interactive Plotly charts instead of static PNGs: hover reads d-spacing, blaze angle and efficiency off any point, and the two line plots sit side by side at equal width rather than the old main-plus-sidebar split. **Clicking a cell of the `(d, blaze)` heatmap selects that design for step 2** -- it opens the manual picker, adds the case, and marks the cell; clicking it again removes it. The dropdown picker is unchanged and stays in sync with the map. The library still writes the matplotlib PNGs, which remain the CLI's output and the page's fallback when Plotly is not installed.
+- The multilayer design survey's plots now fill in while the survey runs, instead of appearing only at the end. `MultilayerGratingDesigner.run_survey` rewrites `survey/survey.csv` after every cell -- atomically, via a temp file and `os.replace`, so a reader never catches a half-written table -- and the web page polls a new `survey-options` endpoint every few seconds to redraw the three charts from the table as it grows. The page reloads once when the stage finishes, which is what stops the polling.
+- The multilayer design energy scan draws a live efficiency-versus-energy overlay while it runs, one curve per design, fed from each design's checkpoint file (which already gains a record per solved energy) through a new `energy-scan-points` endpoint. The finished stage still renders the saved plots. Its subtitle names the coating and order but not the survey energy, which is a property of the survey rather than of a scan across energies.
+- A finished multilayer design stage now shows its results without a manual refresh. The stage monitor stopped polling on `completed`/`failed`/`aborted` but left the progress card on screen, so the plots -- which are rendered server-side -- never appeared on a page that had been open while the stage ran; the monitor now reloads once when it reaches a terminal state. The energy-scan result plots also share the survey's equal-width grid, which gives a single scanned design the full page width instead of half of it.
+- **Aborting a multilayer design stage now kills the running solves** instead of waiting for the item in flight to finish -- a 200-energy scan used to be uninterruptible for its whole multi-hour run. `run_multilayer_theta_search_sweep` gained keyword-only `stop_event` and `on_worker_pids_changed`: once the event is set, queued energies are not submitted, the live worker processes are terminated, and the sweep returns `stopped_early=True` with the energies solved so far (`MultilayerThetaSearchSweepResult` also now reports `resolved_max_workers`). Because an in-process solve cannot be interrupted, passing a `stop_event` routes execution through worker processes even at `max_workers=1`; callers that pass none keep the in-process path. `MultilayerGratingDesigner.run_survey` and `run_energy_scan` forward both arguments -- a killed survey cell is discarded (its half-written summary CSV would break later `evaluate_survey` calls) and a half-scanned design is not reported as finished, though its checkpoint keeps every solved energy so re-running resumes from there. In the web app a stage killed on purpose is recorded as *aborted* rather than *failed*, and the run monitor reports the real worker-pool size instead of a hardcoded 1.
+- A multilayer design study's parameters can be changed after it is created. *Edit parameters* on the study page reopens the creation form with the stored values filled in, so a survey grid, an energy range or a theta-search setting can be adjusted without starting a new study and losing the results. Saving a change that a finished stage depended on marks that stage *stale* rather than deleting anything -- the artifacts stay on disk until the stage is re-run or reset -- while purely cosmetic edits (the coating label, which plots get saved, checkpointing, worker count) invalidate nothing, and an edit to only the energy-scan settings leaves a finished survey alone. Editing is refused while a stage is running.
+- A new multilayer design study's form starts from the most recent study's settings rather than the dataclass defaults, so the advanced theta-search parameters someone tuned for a quick exploratory run carry over to the next one. The form says which study it copied and links to it; with no studies yet it falls back to the defaults as before.
+- The multilayer-design tutorial now walks through the bundled Ru/B4C example rather than just pointing at it, and opens with a *Quick start* section showing that example's own output: the code to run each step immediately followed by the real plots it produces, pulled from a full 50 x 28-cell survey and a 1000-point energy scan and checked into `docs/tutorials/images/simulation/` (the `(d, blaze)` heatmap and both headline curves for the survey, the efficiency-versus-energy curve for the best design) -- so the picture comes before the explanation rather than after it. The walkthrough below it covers what each of the example's four files is for, how to run it (and how to shrink the 1400-cell grid for a first pass), how to read the survey tree, and why the rough theta-search window has to be wide -- with the measured numbers from that real run, where the Bragg seed sits 0.79 deg above the angle the search actually settles on for the best cell. It also documents the `stop_event` stop path alongside `should_continue`.
+- Every "there is no CFF input" mention is gone from the multilayer-design workflow's documentation and source -- the web app's design pages, the tutorial, the API reference page, the library's own module docstring, and the bundled example's parameter file -- in favor of stating what the workflow does (every cell gets its own theta search) rather than dwelling on an input a removed workflow used to need.
+- A multilayer design study can be downloaded as a standalone script to run offline. *Download script* on the study page writes one self-contained Python file: the study's parameters as named constants under the same SHARED / SURVEY / ENERGY SCAN banners `MultilayerDesignConfig` groups them by, the `MultilayerDesignConfig` built from them, and both stages behind `--survey` / `--energy-scan` flags -- with `--best`, `--pairs "d,blaze; ..."` and `--eval` as in the bundled example, and both stages in order when no stage flag is given. It imports only `grax` and pandas, so it runs anywhere grax is installed, with no sibling parameters module and nothing from the web app.
 - Fixed a native crash (segmentation fault, no Python traceback) during a serial (`max_workers=1`) Nevière theta-search sweep on Linux/OpenBLAS. The differential method issues thousands of tiny dense `zgesv`/`zgemm` calls per photon-energy point, and a threaded BLAS both wastes its time on dispatch and, on some OpenBLAS builds, crashes under that pattern. `BatchSimulationRunner` already pinned `OPENBLAS_NUM_THREADS` and friends to `1` in its spawned workers, but a serial run or a direct `grax.run_simulation` call executes in the current process where those environment variables can no longer take effect. `run_simulation` now wraps the Nevière solve in `threadpoolctl.threadpool_limits(1, "blas")` (a no-op where no controllable native library is loaded, e.g. NumPy on Apple Accelerate, and redundant inside a spawned worker); the RCWA path, whose single large eigensolve benefits from threads, is unchanged. `threadpoolctl` (already an indirect dependency via SciPy) is now a direct one. Additionally, the interface-response cascade in the Nevière/RCWA shared code now raises a `ValueError` when a slab transfer matrix or a cascaded block goes non-finite, instead of passing it to `np.linalg.solve` where some LAPACK builds crash rather than raising. New `examples/simulation/neviere_grazing_stability/` sweeps a coated Mo/B4C grating from a Bragg angle down to 0.01 deg in p-polarization and confirms every point stays finite.
 - Internal cleanup pass; no public API or numerical behaviour changes.
 - Progress bars now follow the terminal width while running. The `tqdm` bars in `run_parameter_study`, `BatchSimulationRunner`, and `run_multilayer_theta_search_sweep` pass `dynamic_ncols=True`, so resizing the terminal mid-run re-fits the bar instead of keeping the width measured at start-up. The `DummyProgress` test doubles were widened to accept the extra keyword.

docs/api/simulation.md

@@ -12,6 +12,6 @@ simulation/fixed-angle-sweep
 simulation/monochromator-sweep
 simulation/energy-angle-sweep
 simulation/multilayer-theta-search
-simulation/multilayer-optimization
+simulation/multilayer-design
 simulation/parameter-study
 ```

docs/api/simulation/multilayer-design.md

@@ -0,0 +1,44 @@
+# Multilayer grating design
+
+APIs for the multilayer-grating design workflow: a 2-D
+d-spacing / blaze-angle survey at one optimization energy, followed by per-design
+energy scans. For every `(d_spacing, blaze angle)` pair the survey builds the
+multilayer-coated blazed grating and runs graxPy's single-energy multilayer theta
+search, which scans the incident angle and returns the angle that maximizes the
+selected-order efficiency for that pair.
+
+```{eval-rst}
+.. autoclass:: grax.MultilayerDesignConfig
+```
+
+```{eval-rst}
+.. autoclass:: grax.ThetaSearchScanSettings
+```
+
+```{eval-rst}
+.. autoclass:: grax.MultilayerGratingDesigner
+   :members: run_survey, evaluate_survey, run_energy_scan, evaluate_energy_scan, plot_energy_scan_overlay
+```
+
+```{eval-rst}
+.. autoclass:: grax.SurveyResult
+```
+
+```{eval-rst}
+.. autoclass:: grax.EnergyScanResult
+```
+
+```{eval-rst}
+.. autoclass:: grax.StageProgress
+```
+
+## Planar-multilayer reflectivity
+
+The design workflow itself never touches XRT, but the planar-multilayer
+reflectivity wrapper remains available for standalone Bragg-reflectivity work.
+`xrt` is imported lazily, so importing `grax` does not pull it in.
+
+```{eval-rst}
+.. autoclass:: grax.MultilayerReflectivity
+   :members: reflectivity_vs_energy
+```

docs/developer/module-guide.md

@@ -24,7 +24,7 @@ This guide summarizes the source layout for contributors.
   parameter studies
 - `multilayer_reflectivity.py`: planar-multilayer peak Bragg reflectivity versus
   energy, wrapping the XRT dynamical-diffraction engine (imported lazily)
-- `multilayer_optimization.py`: three-stage d-spacing / gamma / blaze design
+- `multilayer_design.py`: d-spacing / blaze-angle survey plus per-design energy scans
   workflow built on the public API, with a JSON state file for the
   `"auto"` hand-off between stages
 - `solvers/`: the one-dimensional electromagnetic solvers

docs/tutorials/choosing-a-solver.md

@@ -159,7 +159,7 @@ Solver selection reaches every workflow, not just one-point solves:
 | {class}`grax.BatchSimulationRunner` | `solver=`, `solver_options=`, or a per-case `"solver"` key |
 | {func}`grax.run_multilayer_theta_search` | `solver=` — used for all three scan stages |
 | {func}`grax.run_multilayer_theta_search_sweep` | `solver=` |
-| {func}`grax.run_blaze_study` | `MultilayerOptimizationConfig.solver` (stage 2 only) |
+| {class}`grax.MultilayerGratingDesigner` | `MultilayerDesignConfig.solver` |
 | {func}`grax.run_parameter_study` | `solver=` |
 | `grax_opt` measurement fits | `solver` / `solver_options` on the config |
 | Web UI | Solver dropdown in the run form |