Release v0.4.9: develop -> main - #47
Merged
Merged
Conversation
These functions (_resolve_max_workers, _available_memory_bytes, _multiprocessing_start_method, _worker_initializer, _current_process_memory_bytes, _calibrate_auto_max_workers_from_result, _parallel_worker_execute) were verbatim copies of the canonical versions in batch.py. Nothing imported them from this module, and they referenced names never imported here (sys, ctypes, os, MaxWorkers, _run_payload, SolverProfiler), so they would have raised NameError if called. One copy also carried the macOS fork start-method that batch.py had already fixed to "spawn". No behaviour change: the record-conversion helpers that other modules actually import are untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
solver_benchmark.py is a developer tool, not a public API: it pulled matplotlib and tqdm into `import grax` and had a main() with no console-script entry and no tests. Move it to tools/solver_benchmark/ (alongside the other dev tools), drop its names from grax.__all__, and delete the lazy __getattr__ shim that only existed to import it. Imports switched from package-relative to absolute `grax.*`. Fixes carried in the move: - Type BenchmarkCase.grating_factory as Callable[[BenchmarkPreset], BaseGrating] and drop the `# type: ignore`; annotate run_solver_benchmark's energies_ev. - Serial path: run the timed repeats profiler-free and take a single extra untimed profiling pass, so profiler overhead no longer contaminates the samples and the kept profile is deterministic (was rebuilt every repeat and overwritten). - benchmark_energies() clamps/truncates to 100 points instead of raising. - Lift the "10 serial / 100 multiprocessing" default-count branch out of the call argument into a named local; memoize default_cases() with lru_cache. - export_benchmark(): 4-space indentation, grouped filtering, no over-long lines. - README documents that the serial (fixed-angle) and multiprocessing (cff monochromator sweep) modes measure deliberately different things. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_refractive_index_row, _build_material_code_grid, and _build_refractive_index_grid each carried their own copy of the same non-multilayer layer walk: seed the bottom interface, then for every layer accumulate thickness, evaluate the rough upper interface, build a z-mask and write a payload, advance. Only the payload (refractive index vs material code) and the 1-D/2-D masking differed. Add BaseGrating._iter_rough_layer_interfaces(), a generator yielding (material_name, lower_interface, upper_interface) bottom-up with a final (None, top_interface, None) sentinel for the incident medium, and have the three methods consume it. The _rough_interface call sequence and interface indices are unchanged, so output is bit-identical: a before/after capture of the three grids, _refractive_index_row over 15 z-levels, and end-to-end RCWA/Nevière efficiencies for laminar / blazed / blazed-multilayer gratings (smooth and random-interface rough) matches exactly across all 72 arrays. Full unit suite and smoke tests pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Six shipped modules did `import matplotlib.pyplot as plt` at module scope, so every `import grax` (including the re-import each spawned batch worker performs) paid the full pyplot + backend cost -- ~160 ms, about a third of import time -- even for headless library and batch use. afm_preprocessing.py already imports pyplot lazily inside its plotting functions; this brings the rest in line. Each plotting function/method now imports pyplot locally; the four modules that use `plt` only in type annotations also carry a `TYPE_CHECKING` import so those annotations still resolve for type checkers and linters. matplotlib.ticker in parameter_sweep.py gets the same treatment. `import grax` no longer pulls in matplotlib.pyplot (verified via sys.modules); plotting behaviour is unchanged. One white-box test that reached into `grax.simulation.batch.plt` now patches `matplotlib.pyplot` directly (same singleton the lazy import resolves to). Full unit + smoke suites pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Internal cleanup: dedup, de-package solver benchmark, lazy matplotlib
Pass dynamic_ncols=True to the three tqdm bars so a mid-run terminal resize re-fits the bar instead of keeping the width measured at start-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The progress-bar change in afb1c79 passes dynamic_ncols=True to the tqdm constructor. The DummyProgress fakes in the batch-runner and theta-search progress tests took a fixed (total, desc, unit) signature and raised TypeError. Accept and ignore extra keyword arguments. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ascades A serial (max_workers=1) p-polarised Mo/B4C second-order theta-search sweep segfaulted on Linux/OpenBLAS with no Python traceback, during a run whose tracked-theta logic had wandered down to ~0.29 deg grazing (well below the Bragg estimate) at ~1e-4 efficiency. Root cause is the differential method's access pattern under a threaded BLAS: it issues thousands of tiny dense zgesv/zgemm calls per photon-energy point (one interface-response block per z-slice, plus the sub-block cascade), and a many-threaded OpenBLAS both wastes its time on dispatch and, on some builds, crashes under that churn. BatchSimulationRunner already sets OPENBLAS_NUM_THREADS=1 in its spawned workers, but a serial run or a direct run_simulation call executes in the current process, where the env var was read once at BLAS import and no longer takes effect. - New grax._threads.single_threaded_blas() context manager (threadpoolctl.threadpool_limits(1, "blas")); no-op when no controllable native library is loaded (e.g. NumPy on Apple Accelerate). - run_simulation wraps only the Neviere solve in it. RCWA, whose single large eigensolve does benefit from threads, is untouched. - threadpoolctl promoted from indirect (via SciPy) to a direct dependency. - _boundary_block_from_transfer and _cascade_boundary_pair now raise ValueError on non-finite input/output instead of feeding it to np.linalg.solve, which crashes rather than raising on some LAPACK builds. The cascade's existing post-check is escalated from a warning to that same error. - New examples/simulation/neviere_grazing_stability/ sweeps a coated Mo/B4C grating from a Bragg angle down to 0.01 deg in p and asserts finiteness. - Regression tests in tests/unit/test_neviere.py. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix native crash in serial Nevière theta-search sweeps
Ports the d-spacing / gamma / blaze design workflow into graxPy as first-class package code, tests, one example and docs. New source: - multilayer_reflectivity.py: MultilayerReflectivity, a thin wrapper over XRT's dynamical-diffraction engine for planar-multilayer peak Bragg reflectivity vs energy (prescan, peak selection, FWHM in deg and eV). xrt (already a hard dependency) is imported lazily inside _xrt_materials, so `import grax` stays free of xrt and matplotlib.pyplot. - multilayer_optimization.py: frozen MultilayerOptimizationConfig plus run_d_spacing_study / run_gamma_study / run_blaze_study, each returning a typed result. Stages hand values forward only through optimization_state.json and only when a config value is "auto"; numeric values always win and no stage rewrites the config. Stage 0 derives the d-spacing from the grating geometry (grazing angle at the configured CFF, then the first-order Bragg law); stage 2 builds the multilayer-coated blazed grating and drives run_multilayer_theta_search_sweep per blaze angle. Both are re-exported from grax. Example: examples/simulation/multilayer_optimization_rub4c/ (ru_b4c_parameters.py builds the config; three numbered scripts run the stages; run_all.sh chains them), registered in examples/simulation/run_all.sh and EXAMPLE_SCRIPT_PATHS. Docs: new API page (docs/api/simulation/multilayer-optimization.md) and tutorial (docs/tutorials/multilayer-optimization.md), wired into the simulation API and sweep-recipes toctrees; choosing-a-solver table row; module guide entry; build_docs.sh image copy; CHANGELOG entry. Tests: test_multilayer_reflectivity.py (XRT stubbed), test_multilayer_optimization.py (reflectivity + sweep faked, modelled on test_grax_opt_resume.py), test_import_side_effects.py (import grax pulls in neither xrt nor pyplot), a small real stage-0 smoke run and an assets-exist check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…flow Add three-stage multilayer-grating optimization workflow
Exposes the three-stage d-spacing / gamma / blaze workflow in grax-web as its own page. A study is a self-contained directory under `<data_dir>/multilayer_studies/<id>/`; its stages write straight into it through the library's own layout (0_d_spacing/, 1_gamma/, 2_blaze/, plot/, optimization_state.json) and a study.json manifest on top tracks the shared config plus each stage's status, inputs and suggestions. Library: - run_d_spacing_study / run_gamma_study / run_blaze_study gain optional progress_callback (called with a new grax.StageProgress per scanned item) and should_continue (checked per item; returning False stops the scan, computes the suggestion from the completed subset, and sets aborted=True on the result). Both default to None -- no change for existing callers/CLI/tests. - The d-spacing scan now evaluates the geometry candidate first so an aborted run still yields its suggestion. Web: - New grax/web/multilayer_studies.py: MultilayerStudyStore (mirrors RunStore), the editable-field spec list, and config parse/build helpers. - New routes in create_app: /multilayer (list + create + bulk delete), /multilayer/new, /multilayer/<id>, per-stage run / status / abort / reset, and /multilayer/<id>/delete. Each stage runs in a daemon thread reusing ActiveRunState + resource_manager; the status endpoint returns the shape initRunMonitor already consumes, and web.js now binds every [data-live-run-monitor] on a page. - Re-running a stage flags later completed stages "stale" (results kept); a per-stage Reset deletes one stage's outputs and clears its state keys; Delete study removes the directory. Slug-guarded study ids reject path traversal. - New templates (_multilayer_macros, multilayer_index / study_form / study_detail / stage_abort), .stage-card / .status-pill CSS, a nav entry, and a web-docs section. Tests: tests/unit/test_web_multilayer.py (stage runners faked) covers create, run, stale marking, reset, abort+discard, delete and the traversal guard; tests/unit/test_multilayer_optimization.py gains progress_callback / should_continue / partial-abort coverage. Full unit + smoke suites pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…nergy scan Drops the unreleased three-stage run_d_spacing_study/run_gamma_study/ run_blaze_study workflow (and its web UI page) in favor of grax.MultilayerGratingDesigner: a 2-D d-spacing x blaze-angle survey that seeds every grating's incident angle from the multilayer theta search itself (no CFF input), followed by per-design energy scans. Each solver run keeps its full artifact bundle on disk, survey/energy-scan results can be re-aggregated and re-plotted without re-solving (--eval), scan settings are independent per step, and plot titles show the real material compound. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
MultilayerDesignConfig.plot_dir is now "<output_dir>/plots" (was "plot"), and EnergyScanResult.titled_plot_path is written there alongside the survey's headline plots instead of inside each design's own energy_scan/ folder. Since every design now shares one folder, the filename itself carries the coating, diffraction order, d-spacing and blaze angle: efficiency_vs_energy_<materials>_order<n>_d<d>nm_blaze<b>deg.png (e.g. efficiency_vs_energy_Ru-B4C_order2_d3.102nm_blaze0.859deg.png). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gives grax.MultilayerGratingDesigner a web UI, replacing the /multilayer page removed with the old three-stage API. The form exposes every MultilayerDesignConfig field in the dataclass's three sections, with the nested scan settings behind an Advanced toggle and a live warning once the survey grid gets large. After the survey its three headline plots render and the energy scan is chosen with three buttons -- best only, the optimal blaze at every d-spacing, or a manually built list of (d, blaze) cases; the best-only choice can also be made up front so step 2 chains straight off the survey. Supporting library additions: plot_energy_scan_overlay for comparing several scanned designs on one axis, and should_continue on run_energy_scan so a long scan can be aborted between designs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fifteen scan settings previously flowed as an undifferentiated block in the fieldset grid. FieldSpec gains a `row`, and study_form_sections returns (row_label, fields) pairs, so each theta-search pass claims its own labelled line: rough, fine, final solve, then peak selection and roughness. With the pass named in the row heading the field labels drop their prefix, leaving "Half-width, deg" / "Points" / "Fourier orders" / "x resolution, nm". Presentation only -- every input keeps its dotted config name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_energy_scan reports progress once per design, so a single-design scan sat at "running 0 / 1" for its entire multi-hour run -- visually indistinguishable from a job that never started. The web monitor now counts solved energies from each design's checkpoint file instead, and names the design currently being scanned. Because scans resume from checkpoints, the ETA divides the elapsed time by the energies solved by *this* run only (ActiveRunState.resumed_points), not by the resumed ones, which cost it no time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two changes to the Multilayer design tab. The three step-1 plots become interactive Plotly charts, built client-side from the survey grid the page already embeds for the design picker. The two line plots now sit at equal width (.grid.halves) instead of the main-plus- sidebar .grid.two, and clicking a heatmap cell selects that (d, blaze) for step 2 -- opening the manual picker, adding the case and marking the cell, with a second click removing it. The matplotlib PNGs are still written and still serve as the fallback when plotly is not installed. Aborting a stage now terminates the work in flight. should_continue was only consulted between survey cells and between energy-scan designs, so a 200-energy scan could not be interrupted at all until the whole design finished. run_multilayer_theta_search_sweep takes a stop_event and an on_worker_pids_changed callback; setting the event stops submissions, terminates the pool and returns stopped_early with the energies already solved. Since an in-process solve cannot be interrupted, supplying a stop_event forces worker-process execution even at max_workers=1 -- callers that pass none keep the cheaper in-process path untouched. A killed survey cell is discarded, because its header-only summary CSV would make every later evaluate_survey() raise on .iloc[0]. A half-scanned design is not returned, but its checkpoint keeps every solved energy, so re-running resumes there. The web app records a deliberately killed stage as aborted rather than failed, and reports the real pool size instead of a hardcoded 1 worker. Verified against a live 99-cell study: abort dropped 12 processes to 4 in under a second, the stage read "aborted" with no error text, and all 147 checkpointed energies survived. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The stage monitor cleared its polling timers on completed/failed/aborted but left the progress card on screen. Stage results are rendered server-side, so a page opened while the stage was running could never show them -- a finished energy scan looked like it had produced no plot until the user reloaded by hand. Monitors now reload once on a terminal state, opted into with data-run-reload-on-finish so the shared run-detail monitor is unchanged. The reloaded page renders the results in place of the monitor, so there is no loop. The energy-scan result plots also move to the survey's equal-width grid, now auto-fit so a lone scanned design takes the full width rather than half of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Parameters could only be set when the study was created, so adjusting a survey grid or an energy range meant starting over and abandoning the results. "Edit parameters" on the study page now reopens the same form, prefilled from the stored config, and posts to a new edit route. Changing something a finished stage depended on marks that stage stale instead of deleting anything, so the artifacts stay on disk until the user re-runs or resets. stages_invalidated_by() decides which stages those are from each field's section: shared and survey fields invalidate the survey and everything downstream, energy-scan fields only the scan, and a small cosmetic set (coating label, plot saving, checkpointing, worker count) invalidates nothing. Editing is refused while a stage is running. Checkboxes now render a hidden "0" companion, because an unchecked box submits nothing at all -- which on an edit, where the parser overlays the form onto the stored config, would have silently kept the old value. Verified against a live study: submitting the form unchanged leaves the config byte-identical, all four checkbox fields included. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The survey plots only existed once the stage finished, so a long survey showed a progress bar and nothing else. run_survey now rewrites survey/survey.csv after every cell and the design page polls a new survey-options endpoint to redraw the three charts from the growing table; the reload-on-finish already in place is what ends the polling. That rewrite has to be atomic. A plain to_csv truncates before writing, and the page reading the file in that window got a partial or empty table -- a 500 on the detail page, reproducible within seconds of starting a survey. It now writes a temp file and os.replace()s it, and survey_design_options answers an unreadable read with "nothing yet" rather than raising. The new-study form also starts from the newest study's config instead of the dataclass defaults, so tuned advanced scan settings carry over; the form names the study it copied. With no studies it falls back to defaults. Verified live: 20 polls of the detail page and the endpoint during a running survey all returned 200 with the cell count climbing 0 to 9, and the charts grew in the browser without a reload. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each design's sweep already checkpoints one JSON record per solved energy, so the live curve needs no new bookkeeping -- only a reader. A new energy-scan-points endpoint turns those records into per-design (energy, efficiency) series, sorted by energy because workers finish out of order, skipping failed cases and the partial last line. The running stage card polls it every few seconds and draws one curve per design; the finished stage renders its saved plots as before. The scan's subtitle names the coating and order but not the survey energy -- that belongs to the survey, not to a scan across energies -- and its legend sits below the axes, where a rising curve cannot hide it. _energy_scan_checkpoint_progress now shares the path helper rather than rebuilding the design directory name itself. Verified against a live 4-design scan: the overlay drew the finished 200-point curve plus the design in progress, and the checkpoint grew from 3 to 11 points in 45 s across 8 workers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Download script" on the study page writes the study out as one self-contained Python file: the parameters as named constants under the same SHARED / SURVEY / ENERGY SCAN banners MultilayerDesignConfig groups them by, the config built from them, then both stages behind --survey / --energy-scan, with --best, --pairs and --eval matching the bundled example. No stage flag runs both in order. It imports only grax and pandas -- no sibling parameters module, nothing from the web app -- so it can be copied to a cluster and run there. The scan-settings blocks are emitted in dataclass order (rough, fine, final, peak) rather than whatever order the stored JSON holds. Verified by generating from a real study and running the result: --survey, --energy-scan --best, --energy-scan --pairs and both --eval paths all produced their artifacts, and the generated CONFIG compares equal to the stored one field by field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Unreleased entries for this branch were newest-first, so a reader met the offline script download before learning the design tab existed. They now read foundation (library workflow) to UI (the tab, its plots) to refinements (live updates, abort, editing, export). Also drops a claim the abort work invalidated: the tab entry still said a stage "can be aborted between items". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tutorial covered the API but gave the bundled example four lines of pointer. It now walks through it: what each of the four files is for, why the parameters file is split into three banner sections, how to run it and how to shrink the shipped 1400-cell grid for a first pass, how to read the survey tree, how to pick designs, and how --eval and checkpointing let you iterate without re-solving. The part worth having written down is why the rough theta-search window has to be wide. The search is seeded from the multilayer Bragg estimate, which overshoots the true grating optimum for shallow inside orders: in the example's best cell the seed is 1.356 deg and the search settles at 0.578 deg, so a 0.2 deg half-width would never reach the peak and every cell would report an edge artefact. The numbers come from a real 11x9 run, as does the efficiency-versus-d profile showing how sharp the resonance is. The abort section now covers stop_event beside should_continue. Also removes "There is no CFF input" from the design form and web docs. It answered a question only someone who had used the removed three-stage workflow would think to ask; the docs page now states the positive fact instead. Docs build clean (2 pre-existing unrelated toctree warnings); every grax cross-reference on the page resolves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… mentions
The tutorial explained the workflow before showing a single result. It now
opens with a Quick start: the two commands that run the bundled Ru/B4C
example, immediately followed by the real plots those commands produce --
the (d, blaze) efficiency heatmap for the survey, then the efficiency-
versus-energy curve for the best design. The four images are the actual
output of a real 50x28-cell survey and 1000-point energy scan, checked
into docs/tutorials/images/simulation/ (the project's existing convention
for checked-in result plots, matching e.g. fixed_angle_roughness's). The
two survey headline curves are placed lower, next to the Step 1 prose that
already describes them in detail, so nothing needed to be shown twice.
The in-depth walkthrough below it is otherwise the prior content, with its
example numbers corrected against the real full-grid run rather than a
smaller partial one: the Bragg seed sits 0.79 deg above the angle the
search settles on for the actual best cell (d = 3.102 nm, blaze = 0.859
deg), not the earlier partial-run estimate.
Also removes every "there is no CFF input" mention from the multilayer-
design workflow's documentation and source: the tutorial, the API
reference page, the library's own module docstring, and the bundled
example's parameter file. It was explaining the absence of an input a
now-removed workflow used to need, which reads as a non sequitur to
anyone who never saw that workflow -- the text now just states what the
search does.
Docs build clean (2 pre-existing unrelated toctree warnings only); all ten
grax cross-references and both {doc} links on the page resolve. Verified
by rendering the built HTML in a browser: both new plots and their
captions land exactly where the quick-start code block expects them, and
the two headline curves render inline where Step 1 discusses them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rkflow Multilayer-grating design workflow: survey + energy scan, with a web tab
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release promotion
developtomain.v0.4.9