diff --git a/docs/Software/Available_Applications/VASP.md b/docs/Software/Available_Applications/VASP.md index a81612538..ad17f17d1 100644 --- a/docs/Software/Available_Applications/VASP.md +++ b/docs/Software/Available_Applications/VASP.md @@ -20,202 +20,330 @@ For more information on what you can do with VASP see the [official documentatio ## Examples +{% set genoa = slurm_limits.cpu_nodes.genoa %} +{% set milan = slurm_limits.cpu_nodes.milan %} + +!!! warning + Our Genoa and Milan partitions need different values of `--cores-per-socket`: {{ genoa.cores_per_numa_domain }} on Genoa and {{ milan.cores_per_numa_domain }} on Milan, the number of cores in one of their NUMA domains. See [How do we decrease the latency (and increase speed) of calculations?](#how-do-we-decrease-the-latency-and-increase-speed-of-calculations) for more information. + === "VASP6" VASP6 parallelises using both MPI (`ntasks`) and OpenMP (`cpus-per-task`). The `NCORE` flag is ignored. + The GPU node batch scripts use a GPU build of VASP6 and one MPI task per GPU (see [GPU versions of VASP6](#gpu-versions-of-vasp6)). - ``` sl - #!/bin/bash -e + === "Genoa" - #SBATCH --job-name=my_VASP6_job - #SBATCH --account=nesi12345 - #SBATCH --time=01:00:00 - #SBATCH --ntasks=8 - #SBATCH --cpus-per-task=4 - #SBATCH --mem-per-cpu=1G - #SBATCH --extra-node-info=1:*:1 # Restrict node selection to nodes with at least 1 completely free socket and turn off simultaneous multithreading (Hyperthreading). - #SBATCH --distribution=*:block:* # Bind tasks to CPUs on the same socket, and fill that socket before moving to the next consecutive socket. - #SBATCH --mem-bind=local - #SBATCH --profile=task + === "CPU nodes" - module purge 2> /dev/null - module load VASP/6.4.2-foss-2023a + ``` sl + #!/bin/bash -e + + #SBATCH --job-name my_VASP6_job + #SBATCH --account nesi12345 + #SBATCH --time 01:00:00 + #SBATCH --partition genoa + #SBATCH --ntasks 8 + #SBATCH --cpus-per-task 4 + #SBATCH --mem-per-cpu 1G + #SBATCH --cores-per-socket {{ genoa.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local + #SBATCH --profile task + + module purge 2> /dev/null + module load VASP/6.5.1-foss-2023a + + export OMP_PROC_BIND=true + export OMP_PLACES=cores - # update a VASP job log once the job starts running. - echo "Job ${SLURM_JOB_ID} was submitted on $(date) from directory $(pwd)" >> ~/VASP_job_log.txt + srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\"" - # Start two job steps, one that prints MPI process CPU binding and another that starts VASP. - srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname). \n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\"" - echo -e "\n====== Finished printing CPU binding information, now launching VASP ======\n" + srun vasp_std + ``` - srun vasp_std - ``` + === "GPU nodes" + + ``` sl + #!/bin/bash -e + + #SBATCH --job-name my_VASP6_job + #SBATCH --account nesi12345 + #SBATCH --time 01:00:00 + #SBATCH --gpus-per-node h100:1 + #SBATCH --ntasks 1 + #SBATCH --cpus-per-task 1 + #SBATCH --mem-per-cpu 8G + #SBATCH --cores-per-socket {{ genoa.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local + #SBATCH --gres-flags enforce-binding + #SBATCH --gpu-bind closest + #SBATCH --profile task + + module purge 2> /dev/null + module load VASP/6.5.1-nvofbf-2023b + + export OMP_PROC_BIND=true + export OMP_PLACES=cores + + srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\"" + + srun vasp_std + ``` + + === "Milan" + + === "CPU nodes" + + ``` sl + #!/bin/bash -e + + #SBATCH --job-name my_VASP6_job + #SBATCH --account nesi12345 + #SBATCH --time 01:00:00 + #SBATCH --partition milan + #SBATCH --ntasks 8 + #SBATCH --cpus-per-task 4 + #SBATCH --mem-per-cpu 1G + #SBATCH --cores-per-socket {{ milan.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local + #SBATCH --profile task + + module purge 2> /dev/null + module load VASP/6.5.1-foss-2023a + + export OMP_PROC_BIND=true + export OMP_PLACES=cores + + srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\"" + + srun vasp_std + ``` + + === "GPU nodes" + + ``` sl + #!/bin/bash -e + + #SBATCH --job-name my_VASP6_job + #SBATCH --account nesi12345 + #SBATCH --time 01:00:00 + #SBATCH --gpus-per-node a100:1 + #SBATCH --constraint a100_80gb + #SBATCH --ntasks 1 + #SBATCH --cpus-per-task 1 + #SBATCH --mem-per-cpu 8G + #SBATCH --cores-per-socket {{ milan.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local + #SBATCH --gres-flags enforce-binding + #SBATCH --gpu-bind closest + #SBATCH --profile task + + module purge 2> /dev/null + module load VASP/6.5.1-nvofbf-2023b + + export OMP_PROC_BIND=true + export OMP_PLACES=cores + + srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\"" + + srun vasp_std + ``` === "VASP5" VASP5 parallelises using MPI only (`ntasks`), so `cpus-per-task` is left at 1. The work of individual Kohn-Sham orbitals is shared across MPI processes by setting the `NCORE` flag in the `INCAR` (for example `NCORE = 4`). - ``` sl - #!/bin/bash -e + === "Genoa" - #SBATCH --job-name=my_VASP5_job - #SBATCH --account=nesi12345 - #SBATCH --time=01:00:00 - #SBATCH --ntasks=32 - #SBATCH --cpus-per-task=1 - #SBATCH --mem-per-cpu=1G - #SBATCH --extra-node-info=1:*:1 # Restrict node selection to nodes with at least 1 completely free socket and turn off simultaneous multithreading (Hyperthreading). - #SBATCH --distribution=*:block:* # Bind tasks to CPUs on the same socket, and fill that socket before moving to the next consecutive socket. - #SBATCH --mem-bind=local - #SBATCH --profile=task + ``` sl + #!/bin/bash -e - module purge 2> /dev/null - module load VASP/5.4.4-intel-2020a + #SBATCH --job-name my_VASP5_job + #SBATCH --account nesi12345 + #SBATCH --time 01:00:00 + #SBATCH --partition genoa + #SBATCH --ntasks 32 + #SBATCH --cpus-per-task 1 + #SBATCH --mem-per-cpu 1G + #SBATCH --cores-per-socket {{ genoa.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local + #SBATCH --profile task - # update a VASP job log once the job starts running. - echo "Job ${SLURM_JOB_ID} was submitted on $(date) from directory $(pwd)" >> ~/VASP_job_log.txt + module purge 2> /dev/null + module load VASP/5.4.4-intel-2020a - # Start two job steps, one that prints MPI process CPU binding and another that starts VASP. - srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname). \n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\"" - echo -e "\n====== Finished printing CPU binding information, now launching VASP ======\n" + export OMP_PROC_BIND=true + export OMP_PLACES=cores - srun vasp_std - ``` + srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\"" -## Licences + srun vasp_std + ``` -A VASP licence is managed at the research group level. Which versions you have access to depends on what version your research group leader has purchased. You will have access to either VASP4, VASP5, or VASP6. Minor releases are included under the major release licence, for example VASP6.?.? is available to those who have a VASP6 licence. + === "Milan" -If your research group has a valid licence, please {% include "partials/support_request.html" %} and CC the group leader. The Support Team will grant you permission to using VASP on Mahuika. You may be asked to provide proof of your licence if you are not from a known group or if the licence is new. + ``` sl + #!/bin/bash -e -## Assessing VASP Efficiency + #SBATCH --job-name my_VASP5_job + #SBATCH --account nesi12345 + #SBATCH --time 01:00:00 + #SBATCH --partition milan + #SBATCH --ntasks 32 + #SBATCH --cpus-per-task 1 + #SBATCH --mem-per-cpu 1G + #SBATCH --cores-per-socket {{ milan.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local + #SBATCH --profile task -VASP is a very CPU-hungry piece of software. Therefore, being able to assess your VASP calculation's efficiency is critical and can allow you to perform more calculations at a lower cost. In general, the [VASP manual](https://www.vasp.at/wiki/index.php/The_VASP_Manual) is the best place to go for information on how to begin using VASP. + module purge 2> /dev/null + module load VASP/5.4.4-intel-2020a -The steps below describe how to best optimise the efficiency of your VASP calculations. + export OMP_PROC_BIND=true + export OMP_PLACES=cores -!!! note - You only need to perform these steps on a representative system. You can assume all other similar systems will have the same optimisation parameters. + srun --job-name=print_binding_stats bash -c "echo -e \"Task #\${SLURM_PROCID} is running on node \$(hostname).\n\$(hostname) has the following NUMA configuration:\n\$(lscpu | grep -i --color=none numa)\nTask #\${SLURM_PROCID} has \$(nproc) CPUs, their core IDs are \$(taskset -c -p \$\$ | awk '{print \$NF}')\n===========================================\"" -### Step 1: INCAR & KPOINTS Benchmarking + srun vasp_std + ``` -The first step to optimising your system in VASP is to test various parameters in your `INCAR` and `KPOINTS` files. Here you want to **choose your parameters such that the energy of the system converges across your parameter space**. This means that the energy of your system does not decrease dramatically by increasing/decreasing a parameter of your `INCAR`/`KPOINTS` file. The purpose of doing this is that: +Before starting VASP, each batch script runs a `print_binding_stats` job step, which prints the CPUs and NUMA domains each task was given. [Check a job's NUMA domains](../Parallel_Computing/NUMA_Domains.md#check-a-jobs-numa-domains) explains its output. -* You do not want to underestimate parameters such that you get the wrong energy for your system, but -* You do not want to overestimate parameters such that your calculation takes longer to run (and thus requires more computational resources to complete). +## Licences -To help you find the best `INCAR` and `KPOINTS` parameters, we have created the **`vasp-parameter-benchmarking` tool to help you easily choose the ideal parameters for your `INCAR` and `KPOINTS`**. This can include setting up your `ENCUT` value, determining whether to set `LREAL` to true or false, etc. +A VASP licence is managed at the research group level. Which versions you have access to depends on what version your research group leader has purchased. You will have access to either VASP4, VASP5, or VASP6. Minor releases are included under the major release licence, for example VASP6.?.? is available to those who have a VASP6 licence. -* See the [`vasp-parameter-benchmarking` GitHub page](https://github.com/geoffreyweal/vasp-parameter-benchmarking) for a guide on how to use the `vasp-parameter-benchmarking` tool to optimise your `INCAR` and `KPOINTS` files for your system. +If your research group has a valid licence, please {% include "partials/support_request.html" %} and CC the group leader. The Support Team will grant you permission to using VASP on Mahuika. You may be asked to provide proof of your licence if you are not from a known group or if the licence is new. -### Step 2: Core Benchmarking +## Assessing VASP Efficiency -VASP6 uses two technologies to parallelise calculations across CPUs. These are: +VASP is CPU-hungry, so assessing the efficiency of your calculations is critical: it lets you perform more calculations at a lower cost. For how to begin using VASP itself, the [VASP manual](https://www.vasp.at/wiki/index.php/The_VASP_Manual) is the best place to go. -* MPI (`ntasks` in your slurm script): This is the main way that tasks are parallelised over CPUs or groups of CPUs. -* OpenMP (`cpus-per-task` in your slurm script): This is a secondary method of parallelising such that a group of CPUs work together on a single task. +!!! note + You only need to perform these steps on a representative system. You can assume all other similar systems will have the same optimisation parameters. -VASP6 uses MPI and OpenMP in different ways to speed up your calculation: +### Step 1: INCAR & KPOINTS Benchmarking -* MPI (`ntasks`) is used to calculate each band/orbital in your chemical system. -* OpenMP (`cpus-per-task`) is used to allow a group of CPUs to increase the speed of calculating a single band/orbital in your chemical system. +Test various parameters in your `INCAR` and `KPOINTS` files, and **choose them such that the energy of the system converges across your parameter space**: the energy does not decrease dramatically when you increase or decrease a parameter. -By changing the ratio of MPI/OpenMP used, you can increase the speed of your calculation using the same number of CPUs. +* If you underestimate parameters, you get the wrong energy for your system. +* If you overestimate parameters, your calculation takes longer to run, and so requires more computational resources to complete. -Determining the ideal ratio of MPI/OpenMP requires testing. To make it easier to perform all these tests, we have created the `vasp-core-benchmarking` tool. **The `vasp-core-benchmarking` tool is designed to sweep across a region of `ntasks` and `cpus-per-task` values with minimal effort on your part**. This tool then helps you determine which values of `ntasks` and `cpus-per-task` minimise the time required to perform an electronic step. +We have created the **[`vasp-parameter-benchmarking` tool](https://github.com/geoffreyweal/vasp-parameter-benchmarking) to help you easily choose the ideal parameters for your `INCAR` and `KPOINTS`**, such as your `ENCUT` value or whether to set `LREAL` to true or false. Its GitHub page has a guide on how to use it. -* See the [`vasp-core-benchmarking` GitHub page](https://github.com/geoffreyweal/vasp-core-benchmarking) for a guide on how to use the `vasp-core-benchmarking` tool to optimise your values of `ntasks` and `cpus-per-task` in your slurm script. +### Step 2: Core Benchmarking -## The theory behind VASP efficiency +VASP6 uses two technologies to parallelise calculations across CPUs: -In this section, we will cover in detail how MPI and OpenMP are used to increase the speed of calculations in VASP. YOU do not need to read this to measure VASP efficiency, this is reading just for interest. +* MPI (`ntasks` in your batch script) is the main one. It calculates the bands/orbitals in your chemical system across CPUs or groups of CPUs. +* OpenMP (`cpus-per-task` in your batch script) is secondary. It lets a group of CPUs work together to calculate a single band/orbital faster. -### Useful Nomenclature +By changing the ratio of MPI to OpenMP, you can increase the speed of your calculation using the same number of CPUs. Determining the ideal ratio requires testing, so we have created the **[`vasp-core-benchmarking` tool](https://github.com/geoffreyweal/vasp-core-benchmarking), which sweeps across a range of `ntasks` and `cpus-per-task` values with minimal effort on your part**. It then helps you determine which values minimise the time required to perform an electronic step. Its GitHub page has a guide on how to use it, and [The theory behind VASP efficiency](#the-theory-behind-vasp-efficiency) explains how MPI and OpenMP work together. -* *MPI (Message Passing Interface)*: The technology that lets VASP spread a calculation across multiple CPUs. Crucially, *memory is not shared between CPUs* — each CPU holds its own private copy, and the CPUs coordinate by passing messages to one another. -* *OpenMP*: A technology that lets a group of CPUs work on the same piece of a calculation simultaneously, where the *memory is shared between them*. -* In MPI you will come across the words *rank*, *task*, and *process* — these all mean the same thing. A rank/task/process is a single, self-contained piece of work that your program is carrying out. -* A rank can be broken down further so that parts of it are worked on at the same time. When it is, we call the rank *multithreaded*: it contains several OpenMP *threads* that work on it simultaneously. - * Think of a rank as building a wall. Rather than one builder laying every brick in sequence, several builders (threads) can work along the wall at once, so the job finishes sooner. +## GPU versions of VASP6 -* In VASP, the terms *wavefunction*, *Kohn-Sham orbital*, and *band* all refer to the same thing, since each Kohn-Sham orbital is a single-electron wavefunction. Their total number is given by `NBANDS` in the `OUTCAR`. +VASP modules containing *\*-nvofbf-\** in the name have been built with GPU support (see the [OpenACC GPU port of VASP](https://www.vasp.at/wiki/index.php/OpenACC_GPU_port_of_VASP)). The [examples](#examples) include GPU batch scripts for Genoa and Milan. -### How parallelisation works in VASP +VASP can run really well on GPUs, but how much you benefit depends on the specific calculation you are running. Before running larger production simulations, run some smaller benchmarks with different GPU and CPU-only configurations, and take both performance and compute unit cost into account when choosing a configuration. -In Density Functional Theory (DFT), Kohn-Sham orbitals represent the orbitals that electrons can reside in. VASP uses multiple CPUs to perform the calculations across all the Kohn-Sham orbitals in a chemical system. Two main methods are used concurrently to calculate the Kohn-Sham orbitals across CPUs: +See [Using GPUs](../../Batch_Computing/Using_GPUs.md) for further instructions, and [Hardware](../../Batch_Computing/Hardware.md#gpus) for full GPU specifications. -1. Assign CPUs to different orbitals. This allows VASP to calculate multiple orbitals simultaneously. -2. Assign multiple CPUs to the same orbital. This allows VASP to calculate a single orbital faster than just using 1 CPU. +* There is no longer a `vasp_gpu` executable. The VASP executables (`vasp_std`, `vasp_gam`, `vasp_ncl`) are all built with OpenACC GPU support. +* Always select one MPI process (Slurm task) per GPU. +* If you see memory errors like `call to cuMemAlloc returned error 2: Out of memory`, you probably ran out of GPU memory. Try requesting more GPUs (so the total amount of available memory is higher), GPUs with more memory, or both. GPU memory is distinct from the memory you request for your job with `#SBATCH --mem` or similar: when you are allocated a GPU, you get access to all the GPU memory on that device. -In the various versions of VASP: +## Useful VASP Information -* In **VASP5**: MPI (Message Passing Interface) is used for both 1 and 2, where `NCORE` determines the number of CPUs assigned to an individual orbital. -* In **VASP6**: MPI is used for 1, while OpenMP is used for 2. +* Higher levels of theory and more complicated exchange correlation functionals often require more FFTs, so they may benefit more from increasing `cpus-per-task` than from increasing `ntasks`. Test this out using the [`vasp-core-benchmarking`](https://github.com/geoffreyweal/vasp-core-benchmarking) tool. +* If you include a number of k-points in your `KPOINTS` file, try increasing `KPAR`. It sets the number of k-points calculated in parallel upon the same band/orbital, and must divide evenly into the total number of MPI processes. +* For VASP, turn [hyperthreading](../Parallel_Computing/Simultaneous_Multithreading.md) off with `--threads-per-core 1` (`2` turns it on). This is the default on Mahuika, and the example batch scripts set it. +* For visualising VASP structure files like the `POSCAR` and `CONTCAR`, use the [Atomic Simulation Environment (ASE)](https://docs.ase-lib.org/). See [ASE GUI basics](https://docs.ase-lib.org/ase/gui/basics.html) to learn more about visualising your chemical systems in ASE. -### What happens when calculating an orbital in VASP +## VASP extensions -VASP uses a plane-wave basis set to represent the wavefunction. These plane-waves are naturally defined in reciprocal space. Some terms of the Hamiltonian are computed in this reciprocal space (also known as k-space) and other terms are computed in real-space. Converting the plane waves between real and reciprocal space requires the [Fast Fourier Transform (FFT) algorithm](https://en.wikipedia.org/wiki/Fast_Fourier_transform). +If you are wondering what extensions our VASP modules have been built with, please email the {% include "partials/support_request.html" %}. You can also check what precompiler options (used to activate/deactivate certain code features at the time of compilation) were included when the module was built by loading a module and running `vasp_std --cpp-options`. -CPUs work together to convert the Hamiltonian between real and reciprocal space. Because of this, they need to be very good at passing messages (or information) between each other, as well as reading memory from the RAM. The measure of "very good" is low latency, i.e. pass messages and read memory in a short amount of time. +## The theory behind VASP efficiency -### How do we decrease the latency (and increase speed) of calculations? +This section covers in detail how MPI and OpenMP are used to increase the speed of calculations in VASP. You do not need to read it to measure VASP efficiency: it is just for interest. -The best way to decrease the latency of a calculation is to: +### Useful Nomenclature -1. Decrease the distance between CPUs that constantly communicate with each other, and -2. Decrease the distance between the CPU and the memory it is constantly reading from. +* *MPI (Message Passing Interface)* lets VASP spread a calculation across multiple CPUs. Crucially, *memory is not shared between CPUs*: each CPU holds its own private copy, and the CPUs coordinate by passing messages to one another. +* *OpenMP* lets a group of CPUs work on the same piece of a calculation simultaneously, where the *memory is shared between them*. +* *Rank*, *task* and *process* all mean the same thing in MPI: a single, self-contained piece of work that your program is carrying out. +* A rank is *multithreaded* when several OpenMP *threads* work on parts of it at the same time. Think of a rank as building a wall: several builders (threads) working along the wall at once finish sooner than one builder laying every brick in sequence. +* *Wavefunction*, *Kohn-Sham orbital* and *band* all refer to the same thing in VASP, since each Kohn-Sham orbital is a single-electron wavefunction. Their total number is given by `NBANDS` in the `OUTCAR`. -To do this, we can assign or *pin* orbitals to CPUs that are close to each other and the memory they read from on the physical die. +### How parallelisation works in VASP -On Mahuika, each node is split into [NUMA domains](../Parallel_Computing/NUMA_Domains.md). A NUMA domain is a group of CPUs together with the part of the node's memory closest to them, and those CPUs can read their own domain's memory faster than memory in other domains. +In Density Functional Theory (DFT), Kohn-Sham orbitals represent the orbitals that electrons can reside in. VASP spreads the calculation of these orbitals across CPUs using two methods at once: -In slurm, we can make sure that our calculations run with low latency by pinning orbitals to a NUMA domain or several nearby NUMA domains. This is done by Slurm using the following commands: +1. Assign CPUs to different orbitals, so that VASP calculates multiple orbitals simultaneously. +2. Assign multiple CPUs to the same orbital, so that VASP calculates a single orbital faster than with 1 CPU. -```sl -#SBATCH --extra-node-info=1:*:1 # Restrict node selection to nodes with at least 1 completely free socket and turn off simultaneous multithreading (Hyperthreading). -#SBATCH --distribution=*:block:* # Bind tasks to CPUs on the same socket, and fill that socket before moving to the next consecutive socket. -``` +In the various versions of VASP: -Here, `distribution=*:block:*` crucially makes slurm assign CPUs to VASP that are as close as possible. We note that other jobs are likely to also be running on the die at the same time on Mahuika, so Slurm does this as best as possible considering the circumstances. +* **VASP5** uses MPI for both 1 and 2, where `NCORE` sets the number of CPUs assigned to an individual orbital. +* **VASP6** uses MPI for 1 and OpenMP for 2. -### Some CPUs do not need to lie as close to each other as others +### What happens when calculating an orbital in VASP -It is critical that those CPUs that are performing FFTs together *on the same band/orbital* be close to each other. These are those CPUs involved in `cpus-per-task`. +To represent the wavefunction, VASP uses a plane-wave basis set. Plane waves are naturally defined in reciprocal space (also known as k-space), where some terms of the Hamiltonian are computed, while other terms are computed in real space. Converting the plane waves between real and reciprocal space requires the [Fast Fourier Transform (FFT) algorithm](https://en.wikipedia.org/wiki/Fast_Fourier_transform). -Those CPUs that are performing calculations on different bands do not necessarily need to be close together. It's a nice to have, but not critical to performance. This is because the only information that needs to be passed between bands/orbitals is the total charge density, which only needs to be exchanged per electronic step. +CPUs work together on these conversions, so they need low latency: passing messages (or information) between each other, and reading memory from the RAM, in a short amount of time. -### My job is spread across different nodes — is this a problem? +### How do we decrease the latency (and increase speed) of calculations? -By using the `cpus-per-task` tag in slurm, those CPUs that need to be in constant communication with each other will be located on the same node (and using the `extra-node-info=1:*:1` and `distribution=*:block:*` tags will hopefully be located on the same NUMA domain). The groups of CPUs (given by `ntasks`) can (usually) be safely spread across nodes if needed. +Latency is lowest when CPUs that constantly communicate are close to each other, and close to the memory they are constantly reading from. -* **You do not need to do anything**: Slurm will determine what nodes to use based on your value of `ntasks` and the availability of CPUs on Mahuika. +On Mahuika, each node is split into [NUMA domains](../Parallel_Computing/NUMA_Domains.md): groups of CPUs together with the part of the node's memory closest to them, which they can read faster than memory in other domains. A Genoa node has {{ genoa.numa_domains }} NUMA domains of {{ genoa.cores_per_numa_domain }} cores, and a Milan CPU node has {{ milan.numa_domains }} of {{ milan.cores_per_numa_domain }} cores ([NUMA domains on Mahuika](../Parallel_Computing/NUMA_Domains.md#numa-domains-on-mahuika) lists every type of node). -## GPU versions of VASP6 +VASP loses time when its CPUs are spread over several NUMA domains, or when a thread moves to another NUMA domain and leaves its data behind (see [When NUMA domains matter](../Parallel_Computing/NUMA_Domains.md#when-numa-domains-matter)). To avoid both, the example batch scripts *pin* the tasks and threads working on each orbital to CPUs in one NUMA domain, along with the memory they read from: -VASP modules containing *\*-NVHPC-\** in the name have been built with GPU support (see the [OpenACC GPU port of VASP](https://www.vasp.at/wiki/index.php/OpenACC_GPU_port_of_VASP)). +=== "Genoa" -VASP can run really well on GPUs, although how much you will benefit from GPUs largely depends on the specific simulation/calculation that you are running. As usual, it could be useful to run some smaller benchmarks with different GPU and CPU-only configurations in your slurm scripts, before moving on to run larger production simulations. When considering which configuration to use for production you should take into account performance and compute unit cost. + ```sh + #SBATCH --cores-per-socket {{ genoa.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local -See [Using GPUs](../../Batch_Computing/Using_GPUs.md) for further instructions, and [Hardware](../../Batch_Computing/Hardware.md#gpus) for full GPU specifications. + export OMP_PROC_BIND=true + export OMP_PLACES=cores + ``` -Some additional notes specific to running VASP on GPUs: +=== "Milan" -* There is no longer a `vasp_gpu` executable. The VASP executables (`vasp_std`, `vasp_gam`, `vasp_ncl`) are all built with OpenACC GPU support. -* Always select one MPI process (slurm task) per GPU. -* If you see memory errors like `call to cuMemAlloc returned error 2: Out of memory` you probably ran out of GPU memory. You could try requesting more GPUs (so the total amount of available memory is higher) and/or moving to GPUs with more memory (note: GPU memory is distinct from the usual memory you have to request for your job via `#SBATCH --mem` or similar; when you are allocated a GPU you get access to all the GPU memory on that device). + ```sh + #SBATCH --cores-per-socket {{ milan.cores_per_numa_domain }} + #SBATCH --threads-per-core 1 + #SBATCH --distribution *:block:* + #SBATCH --mem-bind local -## Useful VASP Information + export OMP_PROC_BIND=true + export OMP_PLACES=cores + ``` -* Higher levels of theory and more complicated exchange correlation functionals often require more FFTs. For this reason, you may find that these functionals benefit from increasing `cpus-per-task` (compared with increasing `ntasks`). Again, test this out using the [`vasp-core-benchmarking`](https://github.com/geoffreyweal/vasp-core-benchmarking) tool. -* If you include a number of k-points in your `KPOINTS` file, you could try increasing the number of these k-point calculations that are done in parallel. This is controlled by `KPAR`, where increasing `KPAR` increases the number of kpoints being performed in parallel upon the same band/orbital. +Slurm treats each NUMA domain as a socket, so `--cores-per-socket` only lets your job run on nodes whose NUMA domains have at least this many cores. `--distribution *:block:*` is the crucial option: it gives VASP CPUs that are next to each other, filling one NUMA domain before starting on the next. [What the options mean](../Parallel_Computing/NUMA_Domains.md#what-the-options-mean) explains the others. -!!! note - KPAR must divide evenly into the total number of MPI processes. +If your VASP runtime changes a lot between runs with the same input, check where your tasks were placed in the output of the `print_binding_stats` step in the [example batch scripts](#examples). -* If you come across `threads-per-core=1` or `threads-per-core=2`, this indicates whether you want to use [hyperthreading or not](../Parallel_Computing/Simultaneous_Multithreading.md). For VASP, turn hyperthreading off (i.e. set `threads-per-core=1`; this is the default on Mahuika). -* For visualising VASP structure files like the `POSCAR` and `CONTCAR`, use the [Atomic Simulation Environment (ASE)](https://docs.ase-lib.org/). See [ASE GUI basics](https://docs.ase-lib.org/ase/gui/basics.html) to learn more about visualising your chemical systems in ASE. +### My job is spread across different nodes — is this a problem? -## VASP extensions +Usually not, and you do not need to do anything: Slurm determines what nodes to use based on your value of `ntasks` and the availability of CPUs on Mahuika. -If you are wondering what extensions our VASP modules have been built with, please email the {% include "partials/support_request.html" %}. You can also check what precompiler options (used to activate/deactivate certain code features at the time of compilation) were included when the module was built by loading a module and running `vasp_std --cpp-options`. +* The CPUs that perform FFTs together *on the same band/orbital* (those in `cpus-per-task`) must be close to each other. Slurm places them on the same node, and the options above aim to keep them in the same NUMA domain. +* The CPUs working on different bands (the groups of CPUs given by `ntasks`) do not need to be close together. It is nice to have, but not critical to performance, because the only information passed between bands/orbitals is the total charge density, which is only exchanged per electronic step. These groups can usually be safely spread across nodes. ## References