From fe12a5bcdf4d6795d8412e7bc213709747a3a13a Mon Sep 17 00:00:00 2001 From: geoffreyweal Date: Thu, 12 Nov 2026 16:45:03 +1300 Subject: [PATCH 1/3] ORCA: Genoa/Milan examples, NUMA settings, benchmarking tool link - Example batch scripts for Genoa and Milan, with the NUMA settings from the NUMA Domains page, --account, module purge and the version macro - New "NUMA settings" section that links to the NUMA Domains page - New "Benchmark your ORCA jobs" section that links to the ORCA benchmarking tool - Fix the placeholder, a typo in the checkpointing warning and "file name", and simplify "Requesting a parallel run" Co-Authored-By: Claude Fable 5.2 --- docs/Software/Available_Applications/ORCA.md | 136 ++++++++++++++++--- 1 file changed, 114 insertions(+), 22 deletions(-) diff --git a/docs/Software/Available_Applications/ORCA.md b/docs/Software/Available_Applications/ORCA.md index d5322986b..927848818 100644 --- a/docs/Software/Available_Applications/ORCA.md +++ b/docs/Software/Available_Applications/ORCA.md @@ -5,7 +5,7 @@ tags: - chemistry vote_count: 6 vote_sum: 4 -description: General purpose quantum chemistry and spectroscopic properties with ORCA +description: ORCA quantum chemistry on Mahuika, with example batch scripts, NUMA settings, benchmarking and checkpointing --- @@ -35,47 +35,139 @@ research group is expected to register with the ORCA developers. If you have any questions regarding your eligibility to access ORCA or any particular version of it, please {% include "partials/support_request.html" %}. -## Example script +## Example batch scripts -``` sl -#!/bin/bash -e -#SBATCH --job-name ORCA_job -#SBATCH --time 01:00:00 -#SBATCH --ntasks 16 -#SBATCH --mem-per-cpu 1G +{% set genoa = slurm_limits.cpu_nodes.genoa %} +{% set milan = slurm_limits.cpu_nodes.milan %} -module load ORCA/5.0.4-OpenMPI-4.1.5 +These batch scripts run ORCA on 16 cores. +Use the one for your partition, as `--cores-per-socket` differs (see [NUMA settings](#numa-settings)). -# ORCA under MPI requires that it be called via its full absolute path -orca_exe=$(which orca) +=== "Genoa" -# Don't use "srun" as ORCA does that itself when launching its MPI process. -${orca_exe} MyInput.inp -``` + ```sl + #!/bin/bash -e + + #SBATCH --job-name ORCA_job + #SBATCH --account nesi99991 + #SBATCH --time 01:00:00 + #SBATCH --partition genoa + #SBATCH --ntasks 16 + #SBATCH --mem-per-cpu 1G + #SBATCH --cores-per-socket {{ genoa.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local + + module purge + module load ORCA/{{ applications.ORCA.default }} + + export OMP_PROC_BIND=true + export OMP_PLACES=cores + + srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\"" + + # ORCA must be called by its full path + orca_exe=$(which orca) + + # Don't use srun here: ORCA starts its own MPI processes + ${orca_exe} MyInput.inp + ``` + +=== "Milan" + + ```sl + #!/bin/bash -e + + #SBATCH --job-name ORCA_job + #SBATCH --account nesi99991 + #SBATCH --time 01:00:00 + #SBATCH --partition milan + #SBATCH --ntasks 16 + #SBATCH --mem-per-cpu 1G + #SBATCH --cores-per-socket {{ milan.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local + + module purge + module load ORCA/{{ applications.ORCA.default }} + + export OMP_PROC_BIND=true + export OMP_PLACES=cores + + srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\"" + + # ORCA must be called by its full path + orca_exe=$(which orca) + + # Don't use srun here: ORCA starts its own MPI processes + ${orca_exe} MyInput.inp + ``` + +Replace the following: + +- `nesi99991` with your project code. +- `MyInput.inp` with your ORCA input file. + +Your input file must ask for the same number of cores as `--ntasks` +(see [Requesting a parallel run](#requesting-a-parallel-run)). +To choose that number, see [Benchmark your ORCA jobs](#benchmark-your-orca-jobs). + +## NUMA settings + +Each Mahuika node is split into [NUMA domains](../Parallel_Computing/NUMA_Domains.md): +groups of cores with the memory closest to them. +If your job's cores are spread over several NUMA domains, ORCA can run slower, and its runtime can vary between runs +(see [When NUMA domains matter](../Parallel_Computing/NUMA_Domains.md#when-numa-domains-matter)). +The example batch scripts have settings to avoid this: + +- The last four `#SBATCH` options ask Slurm for cores next to each other, filling one NUMA domain before the next. +- The two `export` lines stop threads moving between cores. + +[What the options mean](../Parallel_Computing/NUMA_Domains.md#what-the-options-mean) explains each one. +Only `--cores-per-socket` depends on the partition. +It's the number of cores in one NUMA domain, which Slurm treats as a socket: +{{ genoa.cores_per_numa_domain }} on Genoa and {{ milan.cores_per_numa_domain }} on Milan. + +Before starting ORCA, each batch script runs a `print_binding_stats` step, +which prints the cores each task was given and the node's NUMA domains. +[Check a job's NUMA domains](../Parallel_Computing/NUMA_Domains.md#check-a-jobs-numa-domains) explains its output. +If your runtime varies a lot between runs with the same input, +check this output for cores spread over several NUMA domains. + +## Benchmark your ORCA jobs + +Past a certain number of cores, more cores add little speed, but still use up your project's +[allocation](../../Getting_Started/Allocations/What_is_an_allocation.md). +To find that number for your calculation, use the +[ORCA benchmarking tool](https://github.com/geoffreyweal/orca-benchmarking). +It runs one calculation on a range of core counts, then reports how its speed and CPU efficiency change. +Its GitHub page explains how to install and use it. + +You only need to benchmark one calculation that is typical of those you plan to run. ## Further notes ### Requesting a parallel run -ORCA requires a parallel run to be requested in its input as well as -from the batch scheduler. To request a parallel run, you need to add a -line to the input file like the following: +ORCA must be told how many cores to use in its input file, as well as in the batch script. +Add a line like this to the input file: ``` sh -%pal nprocs end +%pal nprocs 16 end ``` -where `` represents the total number of processors (cores) you have -requested from the scheduler. +Replace `16` with the number you gave `--ntasks`. ### Checkpointing !!! warning "Checkpointing" - We strongly the use of [checkpointing](../../Batch_Computing/Job_Checkpointing.md) for any job running for more than a day. + We strongly recommend the use of [checkpointing](../../Batch_Computing/Job_Checkpointing.md) for any job running for more than a day. ORCA provides for saving of checkpoint data, especially molecular orbital information, in a file with extension ".gbw" (short for -Geometry-Basis-Wavefunction). Given an input file name of "foo.inp" or +Geometry-Basis-Wavefunction). Given an input filename of "foo.inp" or some equivalent, the GBW file will be named "foo.gbw". The GBW file, like temporary and other output files, will be written in the same directory from which the ORCA executable is invoked. From 2b0f476a2a0dc96c5a7cf4bcf78742a4bf05e28d Mon Sep 17 00:00:00 2001 From: geoffreyweal Date: Thu, 12 Nov 2026 18:56:56 +1300 Subject: [PATCH 2/3] ORCA: memory settings, remove repeated intro, tidy checkpointing - Add %maxcore to the input file settings, at about 80% of --mem-per-cpu. ORCA's default (4096 MB per core in ORCA 6.1) is more than the examples ask for, and ORCA can use more than %maxcore - Move the %pal line up next to the examples, as "Input file settings" - Remove the intro paragraph and homepage link, which the app header already shows - Shorten the licensing text and remove its licence link, which returns 404 - Make Checkpointing a top-level section and shorten it, put filenames in backticks (fixes the 'inp' spelling warning), and link the ORCA manual's restart section Co-Authored-By: Claude Opus 5.5 --- docs/Software/Available_Applications/ORCA.md | 88 +++++++------------- 1 file changed, 32 insertions(+), 56 deletions(-) diff --git a/docs/Software/Available_Applications/ORCA.md b/docs/Software/Available_Applications/ORCA.md index 927848818..36846fc38 100644 --- a/docs/Software/Available_Applications/ORCA.md +++ b/docs/Software/Available_Applications/ORCA.md @@ -5,7 +5,7 @@ tags: - chemistry vote_count: 6 vote_sum: 4 -description: ORCA quantum chemistry on Mahuika, with example batch scripts, NUMA settings, benchmarking and checkpointing +description: Run ORCA quantum chemistry on Mahuika, with example batch scripts, input file settings for cores and memory, NUMA settings, benchmarking and checkpointing --- @@ -15,25 +15,12 @@ description: ORCA quantum chemistry on Mahuika, with example batch scripts, NUMA {% include "partials/app_header.html" %} [//]: <> (APPS PAGE BOILERPLATE END) -ORCA is a flexible, efficient and easy-to-use general purpose tool for -quantum chemistry with specific emphasis on spectroscopic properties of -open-shell molecules. It features a wide variety of standard quantum -chemical methods ranging from semiempirical methods to DFT to single- -and multireference correlated *ab initio* methods. It can also treat -environmental and relativistic effects. - -The ORCA home page is -at [https://orcaforum.kofo.mpg.de](https://orcaforum.kofo.mpg.de) - ## Licensing requirements -ORCA is released as precompiled binaries at no cost, pursuant to a -closed-source licence.  Users are advised that the terms of [the ORCA -licence](https://orcaforum.kofo.mpg.de/app.php/dlext/?view=detail&df_id=41) -allow its use in the course of academic research only, and that each -research group is expected to register with the ORCA developers. If you -have any questions regarding your eligibility to access ORCA or any -particular version of it, please {% include "partials/support_request.html" %}. +ORCA is free, closed-source software, released as precompiled binaries. +Its licence allows it to be used for academic research only, +and each research group is expected to register with the ORCA developers. +If you're unsure whether you can use ORCA, or a particular version of it, {% include "partials/support_request.html" %}. ## Example batch scripts @@ -110,9 +97,19 @@ Replace the following: - `nesi99991` with your project code. - `MyInput.inp` with your ORCA input file. -Your input file must ask for the same number of cores as `--ntasks` -(see [Requesting a parallel run](#requesting-a-parallel-run)). -To choose that number, see [Benchmark your ORCA jobs](#benchmark-your-orca-jobs). +## Input file settings + +Your ORCA input file must set the number of cores, and the memory for each core in MB: + +```txt +%pal nprocs 16 end +%maxcore 800 +``` + +- Set `nprocs` to the value of `--ntasks`. + To choose it, see [Benchmark your ORCA jobs](#benchmark-your-orca-jobs). +- Set `%maxcore` to about 80% of `--mem-per-cpu`, as ORCA can use more memory than `%maxcore` + (see ORCA's [Memory Settings](https://www.faccts.de/docs/orca/6.1/tutorials/first_steps/memory.html)). ## NUMA settings @@ -147,48 +144,27 @@ Its GitHub page explains how to install and use it. You only need to benchmark one calculation that is typical of those you plan to run. -## Further notes - -### Requesting a parallel run - -ORCA must be told how many cores to use in its input file, as well as in the batch script. -Add a line like this to the input file: - -``` sh -%pal nprocs 16 end -``` - -Replace `16` with the number you gave `--ntasks`. - -### Checkpointing +## Checkpointing -!!! warning "Checkpointing" - We strongly recommend the use of [checkpointing](../../Batch_Computing/Job_Checkpointing.md) for any job running for more than a day. +!!! warning + Use [checkpointing](../../Batch_Computing/Job_Checkpointing.md) for any job that runs for more than a day. -ORCA provides for saving of checkpoint data, especially molecular -orbital information, in a file with extension ".gbw" (short for -Geometry-Basis-Wavefunction). Given an input filename of "foo.inp" or -some equivalent, the GBW file will be named "foo.gbw". The GBW file, -like temporary and other output files, will be written in the same -directory from which the ORCA executable is invoked. +ORCA saves checkpoint data, mainly the molecular orbitals, in a GBW (geometry, basis, wavefunction) file. +For the input file `MyInput.inp`, it's `MyInput.gbw`, in the directory ORCA runs in. -To restart from an existing GBW file, you should do the following: +To restart from a GBW file: -1. Ensure that the GBW file you want to start from is renamed so that - it does not have the same base name as your intended input file. - Otherwise, it will be overwritten and destroyed as soon as ORCA - starts running. -2. In your input file, specify the following lines, replacing - "checkpoint.gbw" with the name of the GBW file you intend to read - from: +1. Rename the GBW file so its name differs from your input file's, for example to `checkpoint.gbw`. + Otherwise ORCA overwrites it as soon as it starts. +2. Add these lines to your input file: - ``` sh + ```txt ! moread - % moinp "checkpoint.gbw" + %moinp "checkpoint.gbw" ``` 3. Run the calculation. -For more information about restarting from an older GBW file, including -how to restart from GBW files produced using earlier versions of ORCA, -please consult the ORCA manual. +To restart from a GBW file made by an older version of ORCA, see +[restarting calculations](https://orca-manual.mpi-muelheim.mpg.de/contents/essentialelements/initialguess.html#restarting-scf-calculations) +in the ORCA manual. From 507b4e7728be3751db3e381d121bcf778fb29bf8 Mon Sep 17 00:00:00 2001 From: geoffreyweal Date: Mon, 16 Nov 2026 16:07:55 +1300 Subject: [PATCH 3/3] ORCA: show the NUMA settings and print_binding_stats command - Add Genoa and Milan tabs to "NUMA settings" with the four #SBATCH options and the two export lines from the example batch scripts - Show the print_binding_stats command under its explanation Co-Authored-By: Claude Opus 5.5 --- docs/Software/Available_Applications/ORCA.md | 40 +++++++++++++++++--- 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/docs/Software/Available_Applications/ORCA.md b/docs/Software/Available_Applications/ORCA.md index 36846fc38..b7042fd15 100644 --- a/docs/Software/Available_Applications/ORCA.md +++ b/docs/Software/Available_Applications/ORCA.md @@ -117,18 +117,46 @@ Each Mahuika node is split into [NUMA domains](../Parallel_Computing/NUMA_Domain groups of cores with the memory closest to them. If your job's cores are spread over several NUMA domains, ORCA can run slower, and its runtime can vary between runs (see [When NUMA domains matter](../Parallel_Computing/NUMA_Domains.md#when-numa-domains-matter)). -The example batch scripts have settings to avoid this: +The example batch scripts have these settings to avoid this: -- The last four `#SBATCH` options ask Slurm for cores next to each other, filling one NUMA domain before the next. +=== "Genoa" + + ```sh + #SBATCH --cores-per-socket {{ genoa.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local + + export OMP_PROC_BIND=true + export OMP_PLACES=cores + ``` + +=== "Milan" + + ```sh + #SBATCH --cores-per-socket {{ milan.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local + + export OMP_PROC_BIND=true + export OMP_PLACES=cores + ``` + +- The four `#SBATCH` options ask Slurm for cores next to each other, filling one NUMA domain before the next. - The two `export` lines stop threads moving between cores. [What the options mean](../Parallel_Computing/NUMA_Domains.md#what-the-options-mean) explains each one. -Only `--cores-per-socket` depends on the partition. -It's the number of cores in one NUMA domain, which Slurm treats as a socket: -{{ genoa.cores_per_numa_domain }} on Genoa and {{ milan.cores_per_numa_domain }} on Milan. +Only `--cores-per-socket` differs between Genoa and Milan. +It's the number of cores in one NUMA domain, which Slurm treats as a socket. Before starting ORCA, each batch script runs a `print_binding_stats` step, -which prints the cores each task was given and the node's NUMA domains. +which prints the cores each task was given and the node's NUMA domains: + +```sh +srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\"" +``` + [Check a job's NUMA domains](../Parallel_Computing/NUMA_Domains.md#check-a-jobs-numa-domains) explains its output. If your runtime varies a lot between runs with the same input, check this output for cores spread over several NUMA domains.