From e67d5aabba40cfd699979d9e3b10029c63679f3b Mon Sep 17 00:00:00 2001 From: Simone Vadilonga Date: Fri, 4 Sep 2026 14:58:40 +0200 Subject: [PATCH 01/15] Add a Multilayer study page to the grax web app Exposes the three-stage d-spacing / gamma / blaze workflow in grax-web as its own page. A study is a self-contained directory under `/multilayer_studies//`; its stages write straight into it through the library's own layout (0_d_spacing/, 1_gamma/, 2_blaze/, plot/, optimization_state.json) and a study.json manifest on top tracks the shared config plus each stage's status, inputs and suggestions. Library: - run_d_spacing_study / run_gamma_study / run_blaze_study gain optional progress_callback (called with a new grax.StageProgress per scanned item) and should_continue (checked per item; returning False stops the scan, computes the suggestion from the completed subset, and sets aborted=True on the result). Both default to None -- no change for existing callers/CLI/tests. - The d-spacing scan now evaluates the geometry candidate first so an aborted run still yields its suggestion. Web: - New grax/web/multilayer_studies.py: MultilayerStudyStore (mirrors RunStore), the editable-field spec list, and config parse/build helpers. - New routes in create_app: /multilayer (list + create + bulk delete), /multilayer/new, /multilayer/, per-stage run / status / abort / reset, and /multilayer//delete. Each stage runs in a daemon thread reusing ActiveRunState + resource_manager; the status endpoint returns the shape initRunMonitor already consumes, and web.js now binds every [data-live-run-monitor] on a page. - Re-running a stage flags later completed stages "stale" (results kept); a per-stage Reset deletes one stage's outputs and clears its state keys; Delete study removes the directory. Slug-guarded study ids reject path traversal. - New templates (_multilayer_macros, multilayer_index / study_form / study_detail / stage_abort), .stage-card / .status-pill CSS, a nav entry, and a web-docs section. Tests: tests/unit/test_web_multilayer.py (stage runners faked) covers create, run, stale marking, reset, abort+discard, delete and the traversal guard; tests/unit/test_multilayer_optimization.py gains progress_callback / should_continue / partial-abort coverage. Full unit + smoke suites pass. Co-Authored-By: Claude Sonnet 5 --- CHANGELOG.md | 1 + docs/tutorials/multilayer-optimization.md | 12 + src/grax/__init__.py | 2 + src/grax/multilayer_optimization.py | 172 ++++++- src/grax/web/app.py | 468 ++++++++++++++++++ src/grax/web/multilayer_studies.py | 371 ++++++++++++++ src/grax/web/static/web.css | 48 ++ src/grax/web/static/web.js | 5 +- .../web/templates/_multilayer_macros.html | 48 ++ src/grax/web/templates/base.html | 1 + src/grax/web/templates/index.html | 1 + src/grax/web/templates/multilayer_index.html | 47 ++ .../web/templates/multilayer_stage_abort.html | 33 ++ .../templates/multilayer_study_detail.html | 117 +++++ .../web/templates/multilayer_study_form.html | 42 ++ src/grax/web/templates/web_docs.html | 29 ++ tests/unit/test_multilayer_optimization.py | 43 ++ tests/unit/test_web_multilayer.py | 304 ++++++++++++ 18 files changed, 1737 insertions(+), 7 deletions(-) create mode 100644 src/grax/web/multilayer_studies.py create mode 100644 src/grax/web/templates/_multilayer_macros.html create mode 100644 src/grax/web/templates/multilayer_index.html create mode 100644 src/grax/web/templates/multilayer_stage_abort.html create mode 100644 src/grax/web/templates/multilayer_study_detail.html create mode 100644 src/grax/web/templates/multilayer_study_form.html create mode 100644 tests/unit/test_web_multilayer.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 496fa17..930753c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## Unreleased +- The `grax-web` app has a dedicated **Multilayer study** page for the three-stage d-spacing / gamma / blaze workflow. A study is a self-contained directory under `multilayer_studies//`; each stage runs in a background thread with a live progress bar, can be aborted between scan items, and stores its plot, CSV and `optimization_state.json` keys. Re-running an earlier stage flags the later ones *stale* (results kept); a per-stage "Reset" deletes one stage's outputs and clears its state keys, and "Delete study" removes the whole directory. To support this, `run_d_spacing_study` / `run_gamma_study` / `run_blaze_study` gained optional `progress_callback` (called with a new `grax.StageProgress` per scanned item) and `should_continue` (checked per item; returning `False` stops the scan and computes the suggestion from the completed subset, with `aborted=True` on the result). Both default to `None`, so existing callers are unaffected; the d-spacing scan now evaluates the geometry candidate first so an aborted run still yields its suggestion. - Added a three-stage multilayer-grating design workflow: `grax.run_d_spacing_study`, `grax.run_gamma_study` and `grax.run_blaze_study`, driven by one frozen `grax.MultilayerOptimizationConfig`. Stage 0 derives a bilayer d-spacing from the grating geometry (grazing angle at the configured CFF, then the first-order Bragg law) and scans practical candidates with planar-multilayer reflectivity; stage 1 scans the bilayer thickness ratio `gamma`; stage 2 builds the multilayer-coated blazed grating and scans the blaze angle through `run_multilayer_theta_search_sweep`. Stages hand values forward only through `optimization_state.json` and only when a config value is the string `"auto"` -- a numeric value always wins, and no stage rewrites the config. Reflectivity for stages 0-1 comes from the new `grax.MultilayerReflectivity`, a thin wrapper over XRT's dynamical-diffraction engine; `xrt` (already a hard dependency) is imported lazily, so `import grax` still pulls in neither `xrt` nor `matplotlib.pyplot` (now covered by a test). A runnable Ru/B4C second-order example lives at `examples/simulation/multilayer_optimization_rub4c/`. - Fixed a native crash (segmentation fault, no Python traceback) during a serial (`max_workers=1`) Nevière theta-search sweep on Linux/OpenBLAS. The differential method issues thousands of tiny dense `zgesv`/`zgemm` calls per photon-energy point, and a threaded BLAS both wastes its time on dispatch and, on some OpenBLAS builds, crashes under that pattern. `BatchSimulationRunner` already pinned `OPENBLAS_NUM_THREADS` and friends to `1` in its spawned workers, but a serial run or a direct `grax.run_simulation` call executes in the current process where those environment variables can no longer take effect. `run_simulation` now wraps the Nevière solve in `threadpoolctl.threadpool_limits(1, "blas")` (a no-op where no controllable native library is loaded, e.g. NumPy on Apple Accelerate, and redundant inside a spawned worker); the RCWA path, whose single large eigensolve benefits from threads, is unchanged. `threadpoolctl` (already an indirect dependency via SciPy) is now a direct one. Additionally, the interface-response cascade in the Nevière/RCWA shared code now raises a `ValueError` when a slab transfer matrix or a cascaded block goes non-finite, instead of passing it to `np.linalg.solve` where some LAPACK builds crash rather than raising. New `examples/simulation/neviere_grazing_stability/` sweeps a coated Mo/B4C grating from a Bragg angle down to 0.01 deg in p-polarization and confirms every point stays finite. - Internal cleanup pass; no public API or numerical behaviour changes. diff --git a/docs/tutorials/multilayer-optimization.md b/docs/tutorials/multilayer-optimization.md index 67c4c87..76371e8 100644 --- a/docs/tutorials/multilayer-optimization.md +++ b/docs/tutorials/multilayer-optimization.md @@ -93,6 +93,18 @@ See `examples/simulation/multilayer_optimization_rub4c/` for the full runnable workflow: `ru_b4c_parameters.py` builds the config and the three numbered scripts run the stages. `run_all.sh` runs them in order. +## In the web app + +`grax-web` exposes the same workflow on a **Multilayer study** page. Create a +study, then run each stage from its own card: results are stored under +`multilayer_studies//`, each stage shows a live progress bar and can be +aborted between scan items, and re-running an earlier stage marks the later ones +*stale* rather than discarding them. A per-stage "Reset" deletes one stage's +outputs (and clears its keys from `optimization_state.json`); "Delete study" +removes the whole directory. `run_d_spacing_study`, `run_gamma_study` and +`run_blaze_study` accept the optional `progress_callback` and `should_continue` +arguments the page relies on. + ## Solver selection Stage 2 takes `solver` (`rcwa` or `neviere`) from the config; the example's diff --git a/src/grax/__init__.py b/src/grax/__init__.py index 87d4eaf..76a27c6 100644 --- a/src/grax/__init__.py +++ b/src/grax/__init__.py @@ -67,6 +67,7 @@ DSpacingStudyResult, GammaStudyResult, MultilayerOptimizationConfig, + StageProgress, run_blaze_study, run_d_spacing_study, run_gamma_study, @@ -101,6 +102,7 @@ "SingleLayerStack", "SingleSimulationResult", "SlagConfig", + "StageProgress", "ThetaSearchDiagnostics", "assemble_custom_stack", "available_material_symbols", diff --git a/src/grax/multilayer_optimization.py b/src/grax/multilayer_optimization.py index 34a054a..12e6376 100644 --- a/src/grax/multilayer_optimization.py +++ b/src/grax/multilayer_optimization.py @@ -22,6 +22,12 @@ ``config.gamma`` directly -- the gamma suggestion from stage 1 is recorded for traceability but is not auto-applied. +Each ``run_*_study`` accepts an optional ``progress_callback`` (called with a +:class:`StageProgress` before every scanned item) and ``should_continue`` (checked +before every item; returning ``False`` stops the scan and computes the suggestion +from the completed subset). Both default to ``None`` and change nothing for +callers that do not pass them. + Two numerical conventions are inherited from the original workflow and kept deliberately: the geometry d-spacing derivation uses ``HC_EV_NM = 1239.841984`` while :func:`grax.monochromator_grazing_angles_deg` uses ``1239.8`` internally @@ -34,7 +40,7 @@ from __future__ import annotations import json -from collections.abc import Iterable, Mapping +from collections.abc import Callable, Iterable, Mapping from dataclasses import dataclass, field from pathlib import Path from typing import Any, Literal @@ -54,6 +60,7 @@ "DSpacingStudyResult", "GammaStudyResult", "MultilayerOptimizationConfig", + "StageProgress", "d_spacing_bounds_from_bragg_angles", "energy_to_wavelength_nm", "ensure_target_energy", @@ -69,6 +76,24 @@ HC_EV_NM = 1239.841984 +@dataclass(frozen=True) +class StageProgress: + """Progress report emitted before each scanned item by a study stage. + + Attributes: + stage: ``"d_spacing"``, ``"gamma"`` or ``"blaze"``. + completed: Items finished so far. + total: Total items in the scan. + current_label: Human-readable label of the item about to run, or + ``"done"`` on the final call once the scan has finished. + """ + + stage: str + completed: int + total: int + current_label: str + + @dataclass(frozen=True) class MultilayerOptimizationConfig: """Every knob for the three multilayer-optimization stages. @@ -267,6 +292,8 @@ class DSpacingStudyResult: combined_csv_path: Combined per-d reflectivity table. plot_path: Reflectivity-versus-energy summary plot. state_path: The updated state file. + aborted: Whether the scan stopped early on a ``should_continue`` signal + (the suggestion is then computed from the completed subset). results: The combined reflectivity table. """ @@ -281,6 +308,7 @@ class DSpacingStudyResult: combined_csv_path: Path plot_path: Path state_path: Path + aborted: bool results: pd.DataFrame = field(repr=False) @@ -295,6 +323,7 @@ class GammaStudyResult: combined_csv_path: Combined per-gamma reflectivity table. plot_path: Reflectivity-versus-energy summary plot. state_path: The updated state file. + aborted: Whether the scan stopped early on a ``should_continue`` signal. results: The combined reflectivity table. """ @@ -304,6 +333,7 @@ class GammaStudyResult: combined_csv_path: Path plot_path: Path state_path: Path + aborted: bool results: pd.DataFrame = field(repr=False) @@ -320,6 +350,7 @@ class BlazeStudyResult: combined_csv_path: Combined per-blaze theta-search summary table. plot_path: Efficiency-versus-energy summary plot. state_path: The updated state file. + aborted: Whether the scan stopped early on a ``should_continue`` signal. results: The combined theta-search summary table. """ @@ -330,6 +361,7 @@ class BlazeStudyResult: combined_csv_path: Path plot_path: Path state_path: Path + aborted: bool results: pd.DataFrame = field(repr=False) @@ -890,7 +922,30 @@ def _run_blaze_case( return pd.read_csv(sweep.summary_csv_path) -def run_d_spacing_study(config: MultilayerOptimizationConfig) -> DSpacingStudyResult: +def _emit_stage_progress( + callback: Callable[[StageProgress], None] | None, + *, + stage: str, + completed: int, + total: int, + current_label: str, +) -> None: + """Report progress through ``callback`` when one was supplied.""" + + if callback is not None: + callback( + StageProgress( + stage=stage, completed=completed, total=total, current_label=current_label + ) + ) + + +def run_d_spacing_study( + config: MultilayerOptimizationConfig, + *, + progress_callback: Callable[[StageProgress], None] | None = None, + should_continue: Callable[[], bool] | None = None, +) -> DSpacingStudyResult: """Run stage 0: derive and scan the bilayer d-spacing. Derives the grazing angle at the target energy and CFF, converts it to a @@ -901,10 +956,19 @@ def run_d_spacing_study(config: MultilayerOptimizationConfig) -> DSpacingStudyRe Args: config: The workflow configuration. + progress_callback: Optional callable invoked with a :class:`StageProgress` + before each d-spacing candidate and once more when the scan finishes. + should_continue: Optional callable checked before each candidate; when it + returns ``False`` the scan stops early and the suggestion is computed + from the completed subset (the result's ``aborted`` flag is set). Returns: A :class:`DSpacingStudyResult` with the suggestion, diagnostics and artifact paths. + + Raises: + RuntimeError: If ``should_continue`` stops the scan before any candidate + has been evaluated. """ target_energy = float(config.target_energy_ev) @@ -937,11 +1001,28 @@ def run_d_spacing_study(config: MultilayerOptimizationConfig) -> DSpacingStudyRe ) d_suggested = round(d_geometry, 1) d_values = _rounded_d_grid(lower, upper, int(config.d_spacing_points), d_suggested) + # Scan the geometry candidate first so an aborted run still yields the + # geometry suggestion. Order does not affect the plot or the selection. + d_values = np.concatenate( + ([d_suggested], d_values[~np.isclose(d_values, d_suggested, rtol=0.0, atol=1.0e-9)]) + ) energies = _stage_energy_grid(config, "d_spacing") results_dir = config.d_spacing_results_dir + progress_total = len(d_values) curves = [] + aborted = False for d_spacing in d_values: + if should_continue is not None and not should_continue(): + aborted = True + break + _emit_stage_progress( + progress_callback, + stage="d_spacing", + completed=len(curves), + total=progress_total, + current_label=f"d = {d_spacing:.1f} nm", + ) print(f"Calculating multilayer reflectivity, d = {d_spacing:.1f} nm") curve = _reflectivity_curve( config, @@ -952,6 +1033,15 @@ def run_d_spacing_study(config: MultilayerOptimizationConfig) -> DSpacingStudyRe ) curve.insert(0, "d_spacing_nm", float(d_spacing)) curves.append(curve) + if not curves: + raise RuntimeError("multilayer d-spacing study aborted before any result") + _emit_stage_progress( + progress_callback, + stage="d_spacing", + completed=len(curves), + total=progress_total, + current_label="done", + ) combined = pd.concat(curves, ignore_index=True) metric = _reflectivity_metric(config) @@ -1022,11 +1112,17 @@ def run_d_spacing_study(config: MultilayerOptimizationConfig) -> DSpacingStudyRe combined_csv_path=csv_path, plot_path=plot_path, state_path=config.state_path, + aborted=aborted, results=combined, ) -def run_gamma_study(config: MultilayerOptimizationConfig) -> GammaStudyResult: +def run_gamma_study( + config: MultilayerOptimizationConfig, + *, + progress_callback: Callable[[StageProgress], None] | None = None, + should_continue: Callable[[], bool] | None = None, +) -> GammaStudyResult: """Run stage 1: scan the bilayer thickness ratio at the selected d-spacing. Resolves ``config.d_spacing_nm`` (numeric, or ``"auto"`` from the state @@ -1036,9 +1132,18 @@ def run_gamma_study(config: MultilayerOptimizationConfig) -> GammaStudyResult: Args: config: The workflow configuration. + progress_callback: Optional callable invoked with a :class:`StageProgress` + before each gamma value and once more when the scan finishes. + should_continue: Optional callable checked before each gamma value; when + it returns ``False`` the scan stops early and the suggestion is + computed from the completed subset (``aborted`` is set on the result). Returns: A :class:`GammaStudyResult` with the suggested gamma and artifact paths. + + Raises: + RuntimeError: If ``should_continue`` stops the scan before any gamma + value has been evaluated. """ target_energy = float(config.target_energy_ev) @@ -1059,14 +1164,35 @@ def run_gamma_study(config: MultilayerOptimizationConfig) -> GammaStudyResult: energies = _stage_energy_grid(config, "gamma") results_dir = config.gamma_results_dir + progress_total = len(gamma_values) curves = [] + aborted = False for gamma in gamma_values: + if should_continue is not None and not should_continue(): + aborted = True + break + _emit_stage_progress( + progress_callback, + stage="gamma", + completed=len(curves), + total=progress_total, + current_label=f"gamma = {gamma:.3f}", + ) print(f"Calculating multilayer reflectivity, gamma = {gamma:.3f}") curve = _reflectivity_curve( config, d_spacing, float(gamma), results_dir / f"gamma_{gamma:.3f}", energies ) curve.insert(0, "gamma", float(gamma)) curves.append(curve) + if not curves: + raise RuntimeError("multilayer gamma study aborted before any result") + _emit_stage_progress( + progress_callback, + stage="gamma", + completed=len(curves), + total=progress_total, + current_label="done", + ) combined = pd.concat(curves, ignore_index=True) metric = _reflectivity_metric(config) @@ -1109,11 +1235,17 @@ def run_gamma_study(config: MultilayerOptimizationConfig) -> GammaStudyResult: combined_csv_path=csv_path, plot_path=plot_path, state_path=config.state_path, + aborted=aborted, results=combined, ) -def run_blaze_study(config: MultilayerOptimizationConfig) -> BlazeStudyResult: +def run_blaze_study( + config: MultilayerOptimizationConfig, + *, + progress_callback: Callable[[StageProgress], None] | None = None, + should_continue: Callable[[], bool] | None = None, +) -> BlazeStudyResult: """Run stage 2: scan the blaze angle with graxPy's theta search. Resolves ``config.d_spacing_nm`` (numeric, or ``"auto"`` from the state @@ -1125,10 +1257,20 @@ def run_blaze_study(config: MultilayerOptimizationConfig) -> BlazeStudyResult: Args: config: The workflow configuration. + progress_callback: Optional callable invoked with a :class:`StageProgress` + before each blaze angle and once more when the scan finishes. + should_continue: Optional callable checked before each blaze angle; when + it returns ``False`` the scan stops early (between blaze angles, not + mid theta-search) and the suggestion is computed from the completed + subset (``aborted`` is set on the result). Returns: A :class:`BlazeStudyResult` with the suggested blaze angle and artifact paths. + + Raises: + RuntimeError: If ``should_continue`` stops the scan before any blaze + angle has been evaluated. """ target_energy = float(config.target_energy_ev) @@ -1150,14 +1292,35 @@ def run_blaze_study(config: MultilayerOptimizationConfig) -> BlazeStudyResult: energies = _stage_energy_grid(config, "blaze") results_dir = config.blaze_results_dir + progress_total = len(blaze_values) curves = [] + aborted = False for blaze in blaze_values: + if should_continue is not None and not should_continue(): + aborted = True + break + _emit_stage_progress( + progress_callback, + stage="blaze", + completed=len(curves), + total=progress_total, + current_label=f"blaze = {blaze:.4f} deg", + ) print(f"Running theta search, blaze = {blaze:.4f} deg") curve = _run_blaze_case( config, d_spacing, gamma, float(blaze), energies, results_dir / f"blaze_{blaze:.4f}deg" ) curve.insert(0, "blaze_angle_deg", float(blaze)) curves.append(curve) + if not curves: + raise RuntimeError("multilayer blaze study aborted before any result") + _emit_stage_progress( + progress_callback, + stage="blaze", + completed=len(curves), + total=progress_total, + current_label="done", + ) combined = pd.concat(curves, ignore_index=True) suggested_blaze, suggested_efficiency = select_target_energy_optimum( @@ -1203,5 +1366,6 @@ def run_blaze_study(config: MultilayerOptimizationConfig) -> BlazeStudyResult: combined_csv_path=csv_path, plot_path=plot_path, state_path=config.state_path, + aborted=aborted, results=combined, ) diff --git a/src/grax/web/app.py b/src/grax/web/app.py index b77656b..7004566 100644 --- a/src/grax/web/app.py +++ b/src/grax/web/app.py @@ -36,6 +36,17 @@ from grax.materials import available_material_symbols, material_density_catalog, material_density_g_cm3 from grax.simulation.core import normalize_polarization +from .multilayer_studies import ( + STAGE_LABELS, + STAGES, + MultilayerStudyStore, + build_optimization_config, + downstream_stages, + parse_study_config, + stage_form_fields, + study_config_defaults, + study_form_sections, +) from .persistence import GratingStore, build_grating_from_spec from .runs import RunStore @@ -605,6 +616,167 @@ def plot_delete(plot_id: str): return redirect(url_for("plot_index")) return render_template("plot_delete.html", plot=manifest) + # ------------------------------------------------------------------ # + # Multilayer-optimization studies # + # ------------------------------------------------------------------ # + def _study_store() -> MultilayerStudyStore: + return MultilayerStudyStore(app.config["GRAx_DATA_DIR"] / "multilayer_studies") + + def _study_or_404(study_id: str) -> dict[str, Any]: + try: + return _study_store().load(study_id) + except (ValueError, FileNotFoundError, OSError): + abort(404) + + def _study_view_model(manifest: dict[str, Any]) -> dict[str, Any]: + stages = [] + for stage in STAGES: + state = dict(manifest["stages"][stage]) + state["stage"] = stage + state["label"] = STAGE_LABELS[stage] + state["fields"] = stage_form_fields(stage) + state["field_values"] = state.get("config_snapshot") or manifest["config"] + stages.append(state) + return {"study": manifest, "stages": stages} + + @app.get("/multilayer") + def multilayer_index(): + return render_template( + "multilayer_index.html", + studies=_study_store().list(), + stage_labels=STAGE_LABELS, + stages=STAGES, + ) + + @app.post("/multilayer") + def multilayer_create(): + if request.form.get("action") == "delete": + _study_store().delete_many(request.form.getlist("delete_study_id")) + return redirect(url_for("multilayer_index")) + display_name = request.form.get("display_name", "").strip() or "Multilayer study" + try: + config = parse_study_config(request.form) + build_optimization_config( + app.config["GRAx_DATA_DIR"] / "multilayer_studies" / "_validate", config + ) + except (TypeError, ValueError) as error: + abort(400, str(error)) + study = _study_store().create(display_name=display_name, config=config) + return redirect(url_for("multilayer_detail", study_id=study["id"])) + + @app.get("/multilayer/new") + def multilayer_new(): + return render_template( + "multilayer_study_form.html", + defaults=study_config_defaults(), + basic_sections=study_form_sections(advanced=False), + advanced_sections=study_form_sections(advanced=True), + materials=available_material_symbols(), + material_density_map=dict(material_density_catalog()), + ) + + @app.get("/multilayer/") + def multilayer_detail(study_id: str): + manifest = _study_or_404(study_id) + return render_template( + "multilayer_study_detail.html", + materials=available_material_symbols(), + material_density_map=dict(material_density_catalog()), + **_study_view_model(manifest), + ) + + @app.post("/multilayer//delete") + def multilayer_delete(study_id: str): + _study_store().delete_many([study_id]) + return redirect(url_for("multilayer_index")) + + @app.post("/multilayer//stages//run") + def multilayer_run_stage(study_id: str, stage: str): + if stage not in STAGES: + abort(404) + store = _study_store() + manifest = _study_or_404(study_id) + data_dir = app.config["GRAx_DATA_DIR"] + if any( + _is_run_active(app, f"multilayer:{study_id}:{other}") for other in STAGES + ): + abort(409, "Another stage of this study is still running.") + try: + snapshot = parse_study_config(request.form, base=dict(manifest["config"])) + build_optimization_config(store.study_dir(study_id), snapshot) # validate + except (TypeError, ValueError) as error: + abort(400, str(error)) + manifest["config"] = snapshot + manifest["stages"][stage]["config_snapshot"] = snapshot + manifest["stages"][stage]["status"] = "queued" + manifest["stages"][stage]["error_text"] = "" + for later in downstream_stages(stage): + if manifest["stages"][later]["status"] in {"completed", "aborted"}: + manifest["stages"][later]["status"] = "stale" + store.save(manifest) + _start_multilayer_stage_worker( + app=app, data_dir=data_dir, study_id=study_id, stage=stage + ) + return redirect(url_for("multilayer_detail", study_id=study_id)) + + @app.get("/multilayer//stages//status") + def multilayer_stage_status(study_id: str, stage: str): + if stage not in STAGES: + abort(404) + _study_or_404(study_id) + return jsonify( + _multilayer_stage_status_payload( + app=app, + data_dir=app.config["GRAx_DATA_DIR"], + study_id=study_id, + stage=stage, + ) + ) + + @app.get("/multilayer//stages//abort") + def multilayer_stage_abort_dialog(study_id: str, stage: str): + if stage not in STAGES: + abort(404) + manifest = _study_or_404(study_id) + return render_template( + "multilayer_stage_abort.html", + study=manifest, + stage=stage, + stage_label=STAGE_LABELS[stage], + ) + + @app.post("/multilayer//stages//abort") + def multilayer_stage_abort(study_id: str, stage: str): + if stage not in STAGES: + abort(404) + _study_or_404(study_id) + _abort_multilayer_stage( + app=app, + data_dir=app.config["GRAx_DATA_DIR"], + study_id=study_id, + stage=stage, + discard=request.form.get("disposition") == "discard", + ) + return redirect(url_for("multilayer_detail", study_id=study_id)) + + @app.post("/multilayer//stages//reset") + def multilayer_reset_stage(study_id: str, stage: str): + if stage not in STAGES: + abort(404) + _study_or_404(study_id) + if _is_run_active(app, f"multilayer:{study_id}:{stage}"): + abort(409, "This stage is still running.") + from .multilayer_studies import STATE_KEYS_BY_STAGE + + _reset_multilayer_stage( + data_dir=app.config["GRAx_DATA_DIR"], + store=_study_store(), + state_keys=STATE_KEYS_BY_STAGE, + study_id=study_id, + stage=stage, + ) + return redirect(url_for("multilayer_detail", study_id=study_id)) + @app.get("/_data/") def data_file(filename: str): return send_from_directory(app.config["GRAx_DATA_DIR"], filename) @@ -1225,6 +1397,302 @@ def _execute_run_job( release_workers(run_id) +# --------------------------------------------------------------------------- # +# Multilayer-optimization studies # +# --------------------------------------------------------------------------- # +def _multilayer_job_key(study_id: str, stage: str) -> str: + """Registry key for one running study stage.""" + + return f"multilayer:{study_id}:{stage}" + + +def _multilayer_stage_total(config: Any, stage: str) -> int: + """Best-effort count of scan items for one stage, for the progress bar.""" + + import numpy as _np + + if stage == "d_spacing": + return int(config.d_spacing_points) + if stage == "gamma": + step = float(config.gamma_step) + return int( + len(_np.arange(float(config.gamma_min), float(config.gamma_max) + 0.5 * step, step)) + ) + return int(config.blaze_angle_points) + + +def _start_multilayer_stage_worker( + *, app: Any, data_dir: Path, study_id: str, stage: str +) -> None: + """Register an active-run entry for a study stage and start its worker thread.""" + + from .multilayer_studies import MultilayerStudyStore, build_optimization_config + + store = MultilayerStudyStore(data_dir / "multilayer_studies") + manifest = store.load(study_id) + stage_state = manifest["stages"][stage] + config_dict = stage_state.get("config_snapshot") or manifest["config"] + config = build_optimization_config(store.study_dir(study_id), config_dict) + + key = _multilayer_job_key(study_id, stage) + active_state = ActiveRunState( + run_id=key, + workflow=f"multilayer_{stage}", + total_points=_multilayer_stage_total(config, stage), + worker_mode="auto", + requested_workers=None, + resolved_workers=None, + ) + with _active_runs_lock(app): + _active_runs(app)[key] = active_state + + worker = threading.Thread( + target=_execute_multilayer_stage_job, + kwargs={"app": app, "data_dir": data_dir, "study_id": study_id, "stage": stage}, + daemon=True, + name=f"grax-multilayer-{study_id}-{stage}", + ) + active_state.worker_thread = worker + worker.start() + + +def _execute_multilayer_stage_job( + *, app: Any, data_dir: Path, study_id: str, stage: str +) -> None: + """Run one study stage in a background thread and record the outcome.""" + + from grax import multilayer_optimization as mlo + + from .multilayer_studies import ( + STAGE_CSVS, + STAGE_PLOTS, + MultilayerStudyStore, + build_optimization_config, + downstream_stages, + ) + from .resource_manager import allocate_workers, release_workers + + key = _multilayer_job_key(study_id, stage) + store = MultilayerStudyStore(data_dir / "multilayer_studies") + stage_runner = { + "d_spacing": mlo.run_d_spacing_study, + "gamma": mlo.run_gamma_study, + "blaze": mlo.run_blaze_study, + }[stage] + + def _set_stage(**updates: Any) -> None: + manifest = store.load(study_id) + manifest["stages"][stage].update(updates) + store.save(manifest) + + allocate_workers(key) + _update_active_run(app, key, state="running", started=True) + _set_stage(status="running", error_text="", aborted=False) + try: + manifest = store.load(study_id) + config_dict = manifest["stages"][stage].get("config_snapshot") or manifest["config"] + config = build_optimization_config(store.study_dir(study_id), config_dict) + + def _progress(report: Any) -> None: + _update_active_run( + app, key, completed_points=report.completed, resolved_workers=1 + ) + with _active_runs_lock(app): + entry = _active_runs(app).get(key) + if entry is not None and report.total: + entry.total_points = report.total + + def _keep_going() -> bool: + with _active_runs_lock(app): + entry = _active_runs(app).get(key) + return entry is None or not entry.stop_event.is_set() + + result = stage_runner(config, progress_callback=_progress, should_continue=_keep_going) + suggested = { + name: _json_safe_scalar(getattr(result, name)) + for name in _MULTILAYER_STAGE_SUGGESTIONS[stage] + } + _set_stage( + status="aborted" if getattr(result, "aborted", False) else "completed", + ran_at=datetime.now().isoformat(timespec="seconds"), + aborted=bool(getattr(result, "aborted", False)), + suggested=suggested, + error_text="", + artifacts={"plot": STAGE_PLOTS[stage], "csv": STAGE_CSVS[stage]}, + ) + _finish_active_run(app, key, state="completed") + except RuntimeError as error: + _set_stage(status="aborted", aborted=True, error_text=str(error)) + _finish_active_run(app, key, state="aborted", error_text=str(error)) + except Exception as error: # pragma: no cover - surfaced through the status payload + _set_stage(status="failed", error_text=str(error)) + _finish_active_run(app, key, state="failed", error_text=str(error)) + finally: + _mark_downstream_stale(store, study_id, stage, downstream_stages) + release_workers(key) + + +_MULTILAYER_STAGE_SUGGESTIONS: dict[str, tuple[str, ...]] = { + "d_spacing": ( + "geometry_grazing_angle_deg", + "geometry_d_nm", + "d_suggested_nm", + "d_suggested_peak_rp", + "d_reflectivity_best_nm", + "d_reflectivity_best_peak_rp", + ), + "gamma": ("d_spacing_nm", "gamma_suggested", "gamma_suggested_peak_rp"), + "blaze": ("d_spacing_nm", "gamma", "blaze_suggested_deg", "blaze_suggested_efficiency"), +} + + +def _json_safe_scalar(value: Any) -> Any: + """Return a JSON-safe copy of a scalar result field.""" + + if isinstance(value, np.generic): + return value.item() + return value + + +def _mark_downstream_stale(store: Any, study_id: str, stage: str, downstream_fn: Any) -> None: + """Flag every completed downstream stage as ``stale`` after ``stage`` ran.""" + + manifest = store.load(study_id) + changed = False + for later in downstream_fn(stage): + if manifest["stages"][later]["status"] in {"completed", "aborted"}: + manifest["stages"][later]["status"] = "stale" + changed = True + if changed: + store.save(manifest) + + +def _multilayer_stage_status_payload( + *, app: Any, data_dir: Path, study_id: str, stage: str +) -> dict[str, Any]: + """Return a status payload for one study stage in the shape ``initRunMonitor`` reads.""" + + from .multilayer_studies import STAGE_PLOTS, MultilayerStudyStore + + _cleanup_finished_runs(app) + key = _multilayer_job_key(study_id, stage) + manifest = MultilayerStudyStore(data_dir / "multilayer_studies").load(study_id) + stage_state = manifest["stages"][stage] + + with _active_runs_lock(app): + active = _active_runs(app).get(key) + live = active is not None and _is_active_run_entry_live(active) + if live: + elapsed = _elapsed_seconds(active) + eta = _eta_seconds(active) + payload = { + "state": active.state, + "completed_points": active.completed_points, + "total_points": active.total_points, + "remaining_points": max(active.total_points - active.completed_points, 0), + "elapsed_seconds": elapsed, + "eta_seconds": eta, + "worker_mode": "auto", + "requested_workers": None, + "resolved_workers": active.resolved_workers, + "plot_url": None, + "plot_token": "", + "error_text": active.error_text, + "can_abort": active.state in {"queued", "running"} and not active.abort_requested, + } + return payload + + status = stage_state.get("status", "not_run") + normalized = {"queued": "running", "aborting": "running"}.get(status, status) + if normalized not in {"completed", "failed", "aborted", "stale", "not_run", "running"}: + normalized = "not_run" + plot_rel = STAGE_PLOTS[stage] + plot_path = data_dir / "multilayer_studies" / study_id / plot_rel + plot_url = None + if plot_path.exists(): + plot_url = f"/_data/multilayer_studies/{study_id}/{plot_rel}?v={_file_token(plot_path)}" + return { + "state": normalized if normalized != "stale" else "completed", + "completed_points": 0, + "total_points": 0, + "remaining_points": 0, + "elapsed_seconds": None, + "eta_seconds": None, + "worker_mode": "auto", + "requested_workers": None, + "resolved_workers": None, + "plot_url": plot_url, + "plot_token": _file_token(plot_path) if plot_path.exists() else "", + "error_text": stage_state.get("error_text", ""), + "can_abort": False, + } + + +def _abort_multilayer_stage( + *, app: Any, data_dir: Path, study_id: str, stage: str, discard: bool +) -> None: + """Request a cooperative stop for a running stage and wait for it to finish.""" + + from .multilayer_studies import ( + STATE_KEYS_BY_STAGE, + MultilayerStudyStore, + ) + + key = _multilayer_job_key(study_id, stage) + with _active_runs_lock(app): + active = _active_runs(app).get(key) + if active is not None and active.state in {"queued", "running"}: + active.abort_requested = True + active.stop_event.set() + active.state = "aborting" + _wait_for_run_shutdown(app, key) + if discard: + _reset_multilayer_stage( + data_dir=data_dir, + store=MultilayerStudyStore(data_dir / "multilayer_studies"), + state_keys=STATE_KEYS_BY_STAGE, + study_id=study_id, + stage=stage, + ) + + +def _reset_multilayer_stage( + *, data_dir: Path, store: Any, state_keys: dict[str, tuple[str, ...]], study_id: str, stage: str +) -> None: + """Delete one stage's outputs and clear its state keys.""" + + from .multilayer_studies import STAGE_DIRNAMES, STAGE_PLOTS, downstream_stages + + study_dir = store.study_dir(study_id) + stage_dir = study_dir / STAGE_DIRNAMES[stage] + if stage_dir.exists(): + shutil.rmtree(stage_dir) + plot_path = study_dir / STAGE_PLOTS[stage] + if plot_path.exists(): + plot_path.unlink() + state_path = study_dir / "optimization_state.json" + if state_path.exists(): + state = json.loads(state_path.read_text(encoding="utf-8")) + for state_key in state_keys[stage]: + state.pop(state_key, None) + state_path.write_text(json.dumps(state, indent=2, sort_keys=True) + "\n", encoding="utf-8") + + manifest = store.load(study_id) + manifest["stages"][stage] = { + "status": "not_run", + "ran_at": None, + "error_text": "", + "aborted": False, + "config_snapshot": manifest["stages"][stage].get("config_snapshot"), + "suggested": {}, + "artifacts": {}, + } + for later in downstream_stages(stage): + if manifest["stages"][later]["status"] in {"completed", "aborted"}: + manifest["stages"][later]["status"] = "stale" + store.save(manifest) + + def _worker_settings_from_form(form: Any) -> tuple[str, str | int, int | None]: """Return worker-mode metadata and the runner max_workers setting.""" diff --git a/src/grax/web/multilayer_studies.py b/src/grax/web/multilayer_studies.py new file mode 100644 index 0000000..54fec67 --- /dev/null +++ b/src/grax/web/multilayer_studies.py @@ -0,0 +1,371 @@ +"""File-based persistence for multilayer-optimization studies in the local web app. + +A *study* is one directory under ``/multilayer_studies/``. Its three +stages write straight into it through the library's own layout +(``MultilayerOptimizationConfig(output_dir=)`` derives +``0_d_spacing/``, ``1_gamma/``, ``2_blaze/``, ``plot/`` and +``optimization_state.json``). This module adds one ``study.json`` manifest on top +tracking the shared config and each stage's status, inputs and suggestions. +""" + +from __future__ import annotations + +import json +import shutil +from collections.abc import Sequence +from dataclasses import dataclass, fields +from datetime import datetime +from pathlib import Path +from typing import Any + +from grax.multilayer_optimization import MultilayerOptimizationConfig + +from .persistence import _slugify + +STAGES: tuple[str, ...] = ("d_spacing", "gamma", "blaze") +STAGE_LABELS: dict[str, str] = { + "d_spacing": "1. D-spacing study", + "gamma": "2. Gamma study", + "blaze": "3. Blaze study", +} +STAGE_DIRNAMES: dict[str, str] = { + "d_spacing": "0_d_spacing", + "gamma": "1_gamma", + "blaze": "2_blaze", +} +STAGE_PLOTS: dict[str, str] = { + "d_spacing": "plot/0_d_spacing_study.png", + "gamma": "plot/1_gamma_study.png", + "blaze": "plot/2_blaze_study.png", +} +STAGE_CSVS: dict[str, str] = { + "d_spacing": "0_d_spacing/d_spacing_study.csv", + "gamma": "1_gamma/gamma_study.csv", + "blaze": "2_blaze/blaze_study.csv", +} +# optimization_state.json keys each stage owns; cleared when a stage is reset. +STATE_KEYS_BY_STAGE: dict[str, tuple[str, ...]] = { + "d_spacing": ( + "target_energy_eV", + "wavelength_nm", + "grating_grazing_angle_deg", + "d_geometry_estimate_nm", + "d_geometry_search_min_nm", + "d_geometry_search_max_nm", + "d_search_min_nm", + "d_search_max_nm", + "d_suggested_nm", + "d_suggested_peak_rp", + "d_reflectivity_best_nm", + "d_reflectivity_best_peak_rp", + ), + "gamma": ("gamma_suggested", "gamma_suggested_peak_rp"), + "blaze": ("blaze_suggested_deg", "blaze_suggested_efficiency"), +} +_TERMINAL_STAGE_STATES = {"completed", "aborted"} + + +def downstream_stages(stage: str) -> tuple[str, ...]: + """Return the stages that run after ``stage``.""" + + return STAGES[STAGES.index(stage) + 1 :] + + +@dataclass(frozen=True) +class FieldSpec: + """One editable config field on the study forms. + + Attributes: + name: ``MultilayerOptimizationConfig`` field name. + kind: ``number`` / ``int`` / ``text`` / ``select`` / ``checkbox`` / + ``material`` (a name + density pair). + label: Human-readable label. + section: Fieldset heading it belongs to. + advanced: Rendered inside the collapsible "Advanced" section. + choices: Options for ``select`` fields. + stages: Which stage forms show this field (empty = the new-study form + only). + """ + + name: str + kind: str + label: str + section: str + advanced: bool = False + choices: tuple[str, ...] = () + stages: tuple[str, ...] = () + + +STUDY_FIELDS: tuple[FieldSpec, ...] = ( + FieldSpec("target_energy_ev", "number", "Target energy, eV", "Target & geometry"), + FieldSpec("grating_density_lpermm", "number", "Line density, l/mm", "Target & geometry"), + FieldSpec("diffraction_order", "int", "Diffraction order", "Target & geometry"), + FieldSpec("cff", "number", "CFF", "Target & geometry"), + FieldSpec("multilayer_bragg_order", "int", "Multilayer Bragg order", "Target & geometry"), + FieldSpec("material_a", "material", "Material A (top)", "Materials"), + FieldSpec("material_b", "material", "Material B", "Materials"), + FieldSpec("substrate_material", "material", "Substrate", "Materials"), + FieldSpec("n_bilayers", "int", "Bilayers", "Materials"), + FieldSpec("solver", "select", "Solver", "Numerics", choices=("neviere", "rcwa")), + FieldSpec("polarization", "select", "Polarization", "Numerics", choices=("p", "s")), + FieldSpec("d_spacing_energy_min_ev", "number", "d-spacing energy min, eV", "Energy grids"), + FieldSpec("d_spacing_energy_max_ev", "number", "d-spacing energy max, eV", "Energy grids"), + FieldSpec("d_spacing_energy_step_ev", "number", "d-spacing energy step, eV", "Energy grids"), + FieldSpec("gamma_energy_min_ev", "number", "gamma energy min, eV", "Energy grids"), + FieldSpec("gamma_energy_max_ev", "number", "gamma energy max, eV", "Energy grids"), + FieldSpec("gamma_energy_step_ev", "number", "gamma energy step, eV", "Energy grids"), + FieldSpec("blaze_energy_min_ev", "number", "blaze energy min, eV", "Energy grids"), + FieldSpec("blaze_energy_max_ev", "number", "blaze energy max, eV", "Energy grids"), + FieldSpec("blaze_energy_points", "int", "blaze energy points", "Energy grids"), + FieldSpec("bragg_angle_min_deg", "number", "Bragg angle min, deg", "Scan ranges"), + FieldSpec("bragg_angle_max_deg", "number", "Bragg angle max, deg", "Scan ranges"), + FieldSpec("d_spacing_relative_range", "number", "d relative range", "Scan ranges"), + FieldSpec("d_spacing_min_practical_nm", "number", "d practical min, nm", "Scan ranges"), + FieldSpec("d_spacing_max_practical_nm", "number", "d practical max, nm", "Scan ranges"), + FieldSpec("d_spacing_points", "int", "d candidates", "Scan ranges"), + FieldSpec("gamma_min", "number", "gamma min", "Scan ranges"), + FieldSpec("gamma_max", "number", "gamma max", "Scan ranges"), + FieldSpec("gamma_step", "number", "gamma step", "Scan ranges"), + FieldSpec("blaze_angle_deg", "number", "Blaze center, deg", "Scan ranges"), + FieldSpec("blaze_angle_half_range_deg", "number", "Blaze half-range, deg", "Scan ranges"), + FieldSpec("blaze_angle_points", "int", "Blaze points", "Scan ranges"), + FieldSpec("anti_blaze_angle_deg", "number", "Anti-blaze, deg (0 = sawtooth)", "Scan ranges"), + FieldSpec("d_spacing_nm", "text", "d-spacing, nm (or 'auto')", "Selected values"), + FieldSpec("gamma", "number", "gamma", "Selected values"), + *( + FieldSpec(name, kind, label, "Advanced", advanced=True) + for name, kind, label in ( + ("rough_fourier_orders", "int", "Rough Fourier orders"), + ("fine_fourier_orders", "int", "Fine Fourier orders"), + ("final_fourier_orders", "int", "Final Fourier orders"), + ("rough_scan_points", "int", "Rough scan points"), + ("fine_scan_points", "int", "Fine scan points"), + ("grax_x_resolution_nm", "number", "Grating x resolution, nm"), + ("grax_z_resolution_nm", "number", "Grating z resolution, nm"), + ("final_x_resolution_nm", "number", "Final x resolution, nm"), + ("final_z_resolution_nm", "number", "Final z resolution, nm"), + ("xrt_window_deg", "number", "XRT window, deg"), + ("xrt_angle_points", "int", "XRT angle points"), + ("roughness_sigma_nm", "number", "Roughness sigma, nm (blank = none)"), + ) + ), + FieldSpec("quick", "checkbox", "Quick mode (coarser grids)", "Advanced", advanced=True), +) + +# Which fields each stage's inline form shows (the rest come from the study config). +_STAGE_FORM_FIELDS: dict[str, tuple[str, ...]] = { + "d_spacing": ( + "target_energy_ev", "grating_density_lpermm", "diffraction_order", "cff", + "multilayer_bragg_order", "material_a", "material_b", "substrate_material", + "n_bilayers", "bragg_angle_min_deg", "bragg_angle_max_deg", "d_spacing_relative_range", + "d_spacing_min_practical_nm", "d_spacing_max_practical_nm", "d_spacing_points", + "gamma", "d_spacing_energy_min_ev", "d_spacing_energy_max_ev", "d_spacing_energy_step_ev", + ), + "gamma": ( + "d_spacing_nm", "gamma_min", "gamma_max", "gamma_step", + "gamma_energy_min_ev", "gamma_energy_max_ev", "gamma_energy_step_ev", + "solver", "polarization", + ), + "blaze": ( + "d_spacing_nm", "gamma", "blaze_angle_deg", "blaze_angle_half_range_deg", + "blaze_angle_points", "anti_blaze_angle_deg", "blaze_energy_min_ev", + "blaze_energy_max_ev", "blaze_energy_points", "solver", "polarization", + ), +} + + +def _field_by_name() -> dict[str, FieldSpec]: + return {spec.name: spec for spec in STUDY_FIELDS} + + +def stage_form_fields(stage: str) -> list[FieldSpec]: + """Return the field specs shown on one stage's inline form.""" + + lookup = _field_by_name() + return [lookup[name] for name in _STAGE_FORM_FIELDS[stage]] + + +def study_form_sections(advanced: bool) -> list[tuple[str, list[FieldSpec]]]: + """Return ``(section, fields)`` groups for the new-study form. + + Args: + advanced: ``True`` for the advanced (collapsible) fields, ``False`` for + the always-visible ones. + + Returns: + Section groups preserving :data:`STUDY_FIELDS` order. + """ + + sections: dict[str, list[FieldSpec]] = {} + for spec in STUDY_FIELDS: + if bool(spec.advanced) != advanced: + continue + sections.setdefault(spec.section, []).append(spec) + return list(sections.items()) + + +def study_config_defaults() -> dict[str, Any]: + """Return the JSON-safe config dict from ``MultilayerOptimizationConfig`` defaults.""" + + dataclass_defaults = {f.name: f.default for f in fields(MultilayerOptimizationConfig)} + config: dict[str, Any] = {} + for spec in STUDY_FIELDS: + default = dataclass_defaults.get(spec.name) + if spec.kind == "material": + name, density = default if isinstance(default, (tuple, list)) else ("", None) + config[spec.name] = [str(name), float(density)] + elif spec.kind == "checkbox": + config[spec.name] = bool(default) + elif spec.name == "roughness_sigma_nm": + config[spec.name] = None if default is None else float(default) + else: + config[spec.name] = default + return config + + +def _coerce_field(spec: FieldSpec, raw: str) -> Any: + """Coerce one raw form value for ``spec`` into its JSON-safe type.""" + + text = raw.strip() + if spec.kind == "int": + return int(float(text)) + if spec.kind == "number": + if spec.name == "roughness_sigma_nm" and text == "": + return None + return float(text) + if spec.name == "d_spacing_nm": + return "auto" if text.lower() == "auto" else float(text) + return text + + +def parse_study_config(form: Any, base: dict[str, Any] | None = None) -> dict[str, Any]: + """Overlay a form's values onto ``base`` (or the defaults) and return a config dict.""" + + config = dict(base or study_config_defaults()) + for spec in STUDY_FIELDS: + if spec.kind == "material": + name = form.get(f"{spec.name}_name") + density = form.get(f"{spec.name}_density") + if name is not None and density not in (None, ""): + config[spec.name] = [str(name).strip(), float(density)] + continue + if spec.kind == "checkbox": + if any(key == spec.name for key in form): + config[spec.name] = form.get(spec.name) not in (None, "", "0", "false") + elif base is None: + config[spec.name] = False + continue + if spec.name in form and str(form.get(spec.name)).strip() != "": + config[spec.name] = _coerce_field(spec, str(form.get(spec.name))) + return config + + +def build_optimization_config( + study_dir: Path, config: dict[str, Any] +) -> MultilayerOptimizationConfig: + """Build a ``MultilayerOptimizationConfig`` for ``study_dir`` from a config dict.""" + + kwargs: dict[str, Any] = {} + for key, value in config.items(): + if key in {"material_a", "material_b", "substrate_material"} and isinstance( + value, (list, tuple) + ): + kwargs[key] = (str(value[0]), float(value[1])) + else: + kwargs[key] = value + return MultilayerOptimizationConfig(output_dir=study_dir, **kwargs) + + +class MultilayerStudyStore: + """Store multilayer-optimization study manifests in a filesystem directory.""" + + def __init__(self, directory: str | Path) -> None: + """Initialise the store rooted at ``directory``.""" + + self.directory = Path(directory) + + def list(self) -> list[dict[str, Any]]: + """Return study manifests, newest first.""" + + if not self.directory.exists(): + return [] + studies = [self.load(path.parent.name) for path in self.directory.glob("*/study.json")] + return sorted( + studies, + key=lambda study: (str(study.get("created_at", "")), str(study.get("id", ""))), + reverse=True, + ) + + def load(self, study_id: str) -> dict[str, Any]: + """Load one study manifest by id.""" + + path = self._study_dir(study_id) / "study.json" + with path.open("r", encoding="utf-8") as handle: + payload = json.load(handle) + payload.setdefault("id", study_id) + return payload + + def save(self, manifest: dict[str, Any]) -> dict[str, Any]: + """Persist a study manifest atomically and return it.""" + + payload = dict(manifest) + payload["updated_at"] = datetime.now().isoformat(timespec="seconds") + study_dir = self._study_dir(str(payload["id"])) + study_dir.mkdir(parents=True, exist_ok=True) + path = study_dir / "study.json" + temp_path = path.with_name(f"study.json.{datetime.now().timestamp():.9f}.tmp") + with temp_path.open("w", encoding="utf-8") as handle: + json.dump(payload, handle, indent=2, sort_keys=True) + handle.write("\n") + temp_path.replace(path) + return payload + + def create(self, *, display_name: str, config: dict[str, Any]) -> dict[str, Any]: + """Create a new study directory + manifest and return it.""" + + slug = _slugify(display_name) or "study" + study_id = f"{datetime.now():%Y%m%d-%H%M%S}-{slug}" + candidate = study_id + suffix = 2 + while (self._study_dir(candidate) / "study.json").exists(): + candidate = f"{study_id}-{suffix}" + suffix += 1 + manifest = { + "id": candidate, + "created_at": datetime.now().isoformat(timespec="seconds"), + "display_name": display_name.strip() or candidate, + "comment": "", + "config": config, + "stages": {stage: _blank_stage() for stage in STAGES}, + } + return self.save(manifest) + + def delete_many(self, study_ids: Sequence[str]) -> None: + """Delete several study directories.""" + + for study_id in study_ids: + study_dir = self._study_dir(study_id) + if study_dir.exists(): + shutil.rmtree(study_dir) + + def study_dir(self, study_id: str) -> Path: + """Return the directory for one study id (validated).""" + + return self._study_dir(study_id) + + def _study_dir(self, study_id: str) -> Path: + if _slugify(study_id) != study_id: + raise ValueError("Invalid study id.") + return self.directory / study_id + + +def _blank_stage() -> dict[str, Any]: + return { + "status": "not_run", + "ran_at": None, + "error_text": "", + "aborted": False, + "config_snapshot": None, + "suggested": {}, + "artifacts": {}, + } diff --git a/src/grax/web/static/web.css b/src/grax/web/static/web.css index 16dfabe..e0760a6 100644 --- a/src/grax/web/static/web.css +++ b/src/grax/web/static/web.css @@ -422,6 +422,54 @@ select { font-size: 0.85rem; } +.stage-card { + margin-bottom: 24px; + padding: 18px; + border: 1px solid var(--line); + border-radius: 8px; + background: #fcfcfc; +} + +.stage-card .form { + margin-top: 12px; +} + +.status-pill { + display: inline-flex; + align-items: center; + min-height: 24px; + padding: 2px 10px; + border: 1px solid var(--line); + border-radius: 999px; + font-size: 0.82rem; + color: var(--muted); + background: #fff; +} + +.status-pill.is-completed { + border-color: var(--accent); + color: var(--accent-dark); +} + +.status-pill.is-running, +.status-pill.is-queued, +.status-pill.is-aborting { + border-color: #b5860b; + color: #8a6508; +} + +.status-pill.is-stale { + border-color: #b5860b; + color: #8a6508; + background: #fff8e6; +} + +.status-pill.is-failed, +.status-pill.is-aborted { + border-color: #b42318; + color: #b42318; +} + .export-dialog { width: min(820px, 92vw); border: 1px solid var(--line); diff --git a/src/grax/web/static/web.js b/src/grax/web/static/web.js index 525a12e..2031891 100644 --- a/src/grax/web/static/web.js +++ b/src/grax/web/static/web.js @@ -484,8 +484,7 @@ document.addEventListener("DOMContentLoaded", () => { initSavedPlotFigure(container); }); - const runMonitor = document.querySelector("[data-live-run-monitor]"); - if (runMonitor) { + document.querySelectorAll("[data-live-run-monitor]").forEach((runMonitor) => { initRunMonitor(runMonitor); - } + }); }); diff --git a/src/grax/web/templates/_multilayer_macros.html b/src/grax/web/templates/_multilayer_macros.html new file mode 100644 index 0000000..568a93c --- /dev/null +++ b/src/grax/web/templates/_multilayer_macros.html @@ -0,0 +1,48 @@ +{% macro render_field(spec, values) %} + {% set value = values.get(spec.name) %} + {% if spec.kind == "material" %} + {% set pair = value if value is iterable and value is not string else ["", ""] %} + + + {% elif spec.kind == "select" %} + + {% elif spec.kind == "checkbox" %} + + {% else %} + + {% endif %} +{% endmacro %} + +{% macro material_datalist(materials, density_map) %} + + {% for material in materials %} + + {% endfor %} + +{% endmacro %} diff --git a/src/grax/web/templates/base.html b/src/grax/web/templates/base.html index 8d80831..2eb5471 100644 --- a/src/grax/web/templates/base.html +++ b/src/grax/web/templates/base.html @@ -15,6 +15,7 @@ Grax Web