Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 63 additions & 35 deletions docs/Software/Available_Applications/ANSYS.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@

- **ANSYS Teaching License** (`aa_t`)

This is the default license type, it can be used on up to 6 CPUs on

Check warning on line 35 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.licence

'Write 'licence' for the noun ('license' is the verb), not 'license type'.'
models with less than 512k nodes

- **ANSYS Research license** (`aa_r`)
Expand All @@ -43,15 +43,15 @@

- **ANSYS HPC License** (`aa_r_hpc`)
One of these is required for each CPU over 16 when using
a research license.

Check warning on line 46 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.licence

'Write 'licence' for the noun ('license' is the verb), not 'research license'.'

### License Order

Whether to use a teaching or research license **must be set manually**.

Check warning on line 50 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.licence

'Write 'licence' for the noun ('license' is the verb), not 'research license'.'
If your job is greater than the node limit, not switching to the
research license before submitting a job will **cause the job to fail**.

Check warning on line 52 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.licence

'Write 'licence' for the noun ('license' is the verb), not 'research license'.'

The license order can be changed in workbench under tools > license

Check warning on line 54 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.licence

'Write 'licence' for the noun ('license' is the verb), not 'The license'.'
preferences (provided you have X11 forwarding set up), or by running
either of the following (ANSYS module must be loaded first using
`module load ANSYS`).
Expand Down Expand Up @@ -167,12 +167,15 @@
#SBATCH --cpus-per-task 1 #
#SBATCH --mem 512MB # Total memory (per node)

module load ANSYS/{{ applications.ANSYS.default }}
module purge
module load ANSYS/{{app.default}}

JOURNAL_FILE=/share/test/ansys/fluent/wing.in
JOURNAL_FILE=my_journal.jou
fluent 3ddp -g -i ${JOURNAL_FILE}
```

Replace `my_journal.jou` with the path to your journal file.

=== "Distributed Memory Job"
Multiple processes each with a single thread.
Not limited to one node.
Expand All @@ -195,11 +198,26 @@
#SBATCH --cpus-per-task 1 #
#SBATCH --mem-per-cpu 1500 # Fine for small jobs; increase if needed

module load ANSYS/{{ applications.ANSYS.default }}
JOURNAL_FILE=/share/test/ansys/fluent/wing.in
fluent 3ddp -g -t ${SLURM_NTASKS} -i ${JOURNAL_FILE}
module purge
module load ANSYS/{{app.default}}

JOURNAL_FILE=my_journal.jou
HOSTFILE=${PWD}/hosts.${SLURM_JOB_ID} # must be a full path
srun hostname -s | sort > ${HOSTFILE}
fluent 3ddp -g -t ${SLURM_NTASKS} -cnf=${HOSTFILE} -i ${JOURNAL_FILE}
```

Replace `my_journal.jou` with the path to your journal file.

!!! warning "Give Fluent a host file"
Fluent does not read the list of nodes Slurm has given your job.
Without `-cnf`, versions 2023R1 and later fail with "Requested node configuration is not available", even on one node.
The host file path must be a full path: some versions reject a relative one.

- For 2021R2, use `-mpi=openmpi` instead of `-cnf`.
- 19.2 needs neither.
- If you use `-mpi=openmpi` with 2025R2, run `unset FLUENT_MPIRUN_FLAGS` first.

### Interactive

While it will always be more time and resource efficient using a slurm
Expand All @@ -222,33 +240,29 @@
salloc: job 10270935 has been allocated resources
salloc: Granted job allocation 10270935
salloc: Waiting for resource configuration
salloc: Nodes wbn[053-056] are ready for job
salloc: Nodes c[001-004] are ready for job
```

!!! tip
Include all the commands you would usually use in your Slurm header
here.

Once you have your allocation, run the command
Once you have your allocation, load the module, write a host file and start Fluent with it.
Without `-cnf`, every Fluent process runs on the login node.

``` sh
fluent
module load ANSYS/{{app.default}}
HOSTFILE=${PWD}/hosts.${SLURM_JOB_ID}
srun hostname -s | sort > ${HOSTFILE}
fluent 3ddp -t ${SLURM_NTASKS} -cnf=${HOSTFILE}
```

You will then be presented with the launcher, make any necessary changes
then click launch.

If everything has set up correctly you should see a printout of the
hostnames with the resources requested. Note: 'host' should be
login[1-2].

``` sh
n24-31 wbn056 8/72 Linux-64 71521-71528 Intel(R) Xeon(R) E5-2695 v4
n16-23 wbn055 8/72 Linux-64 52264-52271 Intel(R) Xeon(R) E5-2695 v4
n8-15 wbn054 8/72 Linux-64 177090-177097 Intel(R) Xeon(R) E5-2695 v4
n0-7 wbn053 8/72 Linux-64 48376-48384 Intel(R) Xeon(R) E5-2695 v4
host mahuika01 Linux-64 185962 Intel(R) Xeon(R) E5-2695 v4
```
If everything has set up correctly, Fluent prints a table of the
hostnames with the resources requested. The compute processes should be
on the compute nodes of your allocation, and only 'host' on the login node.

!!! warning
Closing the fluent GUI will not end the SLURM interactive session. Use
Expand All @@ -260,7 +274,7 @@
!!! warning "Checkpointing"
We recommend [checkpointing](../../Batch_Computing/Job_Checkpointing.md) for any job running for more than a day.

It is best practice when running long jobs to enable autosaves.

Check warning on line 277 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'autosaves' is misspelled.

```fluent
/file/autosave/data-frequency 500
Expand Down Expand Up @@ -291,7 +305,7 @@

When setting up the case file on your local machine, make sure you
select 'Compiled UDF', and select the \`.c\` source file. You can also
specify the name of the library, the default being 'libudf', if possible

Check warning on line 308 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'libudf' is misspelled.
you should stick with the default name.

Make sure all names follows unix naming conventions (no spaces or
Expand All @@ -306,10 +320,10 @@
When calling a function, make sure you select the compiled NOT the
interpreted version.

\`udf funcName\` is funcName as being interpreted directly from your

Check warning on line 323 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'funcName' is misspelled.

Check warning on line 323 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'funcName' is misspelled.
\`.c\` source file.

\`udf funcName::libudf\` is funcName as compiled in library \`libudf\`

Check warning on line 326 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'libudf' is misspelled.

Check warning on line 326 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'funcName' is misspelled.

Check warning on line 326 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'libudf' is misspelled.

Check warning on line 326 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'funcName' is misspelled.

#### Compilation

Expand Down Expand Up @@ -358,12 +372,12 @@
Error: chip-exec: function
```

might be using interpreted func

Check warning on line 375 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'func' is misspelled.

solution specify as relative path, or unload compiled lib before saving
`*.cas` file.

## CFX

Check warning on line 380 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Spelling

spelling

Word 'CFX' is misspelled.

`cfx5solve -help` for a list of commands.

Expand All @@ -379,17 +393,19 @@
#SBATCH --time 00:05:00 # Walltime
#SBATCH --mem 512MB # total mem

module load ANSYS/{{ applications.ANSYS.default }}
module load ANSYS//{{app.default}}

input="/share/test/ansys/cfx/pump.def"
input=my_model.def
cfx5solve -batch -def ${input}
```

Replace `my_model.def` with the path to your definition file.

=== "Distributed Memory"
Multiple *processes* each with a single *thread*.
Not limited to one node.
Model will be segmented into `-np` pieces which should be equal to `--ntasks`.
Each task could be running on a different node leading to increased communication overhead. Jobs can be limited to a single node by adding `--nodes=1` however this may increase you time in the queue as contiguous cpu's are harder to schedule.
Model will be segmented into `-part` pieces which should be equal to `--ntasks`.
Each task could be running on a different node leading to increased communication overhead. Jobs can be limited to a single node by adding `--nodes=1` however this may increase your time in the queue as contiguous cpu's are harder to schedule.

```sl
#!/bin/bash -e
Expand All @@ -402,12 +418,15 @@
#SBATCH --mem-per-cpu 512MB # Memory per cpu

module purge
module load ANSYS/{{ applications.ANSYS.default }}
input="/share/test/ansys/mechanical/structural.dat"
module load ANSYS/{{app.default}}
input=my_model.def

cfx5solve -batch -def "${input}" -part ${SLURM_NTASKS}
```

Replace `my_model.def` with the path to your definition file.
The ANSYS module sets the parallel start method for CFX, so you do not need `-start-method`.

!!! tip
Initial values path specified in '.def' file can be overridden using
the `-ini ` flag.
Expand Down Expand Up @@ -444,7 +463,7 @@
#SBATCH --mem 1500M # total mem

module purge
module load ANSYS/{{ applications.ANSYS.default }}
module load ANSYS/{{app.default}}

input=${ANSYS_ROOT}/ansys/data/verif/vm263.dat
mapdl -b -i "${input}"
Expand All @@ -467,7 +486,7 @@
#SBATCH --cpus-per-task 8 #
#SBATCH --mem 12G # 8 threads at 1500 MB per thread

module load ANSYS/{{ applications.ANSYS.default }}
module load ANSYS/{{app.default}}
input=${ANSYS_ROOT}/ansys/data/verif/vm263.dat
mapdl -b -np ${SLURM_CPUS_PER_TASK} -i ${input}
```
Expand All @@ -479,7 +498,6 @@
Model will be segmented into `-np` pieces which should be equal to `--ntasks`.
Each task could be running on a different node leading to increased communication overhead.
Jobs can be limited to a single node by adding `--nodes=1` however this will increase your time in the queue as contiguous cpu's are harder to schedule.
Distributed Memory Parallel is currently not supported on Māui.

```sl
#!/bin/bash -e
Expand All @@ -491,7 +509,7 @@
#SBATCH --ntasks 16 # Number processes
#SBATCH --mem-per-cpu 1500

module load ANSYS/{{ applications.ANSYS.default }}
module load ANSYS/{{app.default}}
input=${ANSYS_ROOT}/ansys/data/verif/vm263.dat
mapdl -b -dis -np ${SLURM_NTASKS} -i "${input}"
```
Expand Down Expand Up @@ -544,7 +562,7 @@
#SBATCH --mem-per-cpu 1G # Memory per cpu

module purge
module load ANSYS/{{ applications.ANSYS.default }}
module load ANSYS/{{app.default}}
lsdyna i=myinput.k NCPUS=$SLURM_CPUS_PER_TASK MEMORY2=1G
```

Expand All @@ -560,8 +578,6 @@
FENSAP-ICE is a fully integrated ice-accretion and aerodynamics
simulator.

Currently FENSAP-ICE is only available on Mahuika and in ANSYS 19.2.

The following FENSAP solvers are compatible with MPI

- FENSAP
Expand All @@ -574,7 +590,7 @@

#### With GUI

If you have set up X-11 forwarding, you may launch the FENSAP ice using

Check warning on line 593 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.x11_forwarding

'Write 'X11 forwarding', not 'X-11 forwarding'. Fix: 'X11 forwarding'.'
the command `fensapiceGUI` from within your FENSAP project directory.

1. Launch the run and select the desired
Expand All @@ -589,7 +605,6 @@
--account nesi99991
--mem-per-cpu memory
--time time
--licenses
```

Note: All these parameters will be applied to each individual
Expand All @@ -599,14 +614,14 @@

You may close your session and the job will continue to run on the
compute nodes. You will be able to view the running job at any time by
opening the GUI within the project folder.

Check warning on line 617 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.directory

'Write 'directory' ('project directory', 'home directory', 'subdirectory'), not 'project folder'. Fix: 'project directory'.'

!!! info
Submitting your job through the use of the GUI has disadvantages and
may not be suitable in all cases.
- Closing the session or losing connection will prevent the next
stage of the job starting (currently executing step will continue
to run). It is a good idea to launch the GUI inside a tmux/screen

Check warning on line 624 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Prose

typography.punctuation.spacing

'Inconsistent spacing after period (1 vs. 2 spaces).'
session then send the process to background to avoid this.
- Each individual step will be launched with the same parameters
given in the GUI.
Expand Down Expand Up @@ -634,7 +649,7 @@

## ANSYS-Electromagnetic

ANSYS-EM jobs can be submitted through a slurm script or by

Check warning on line 652 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.batch_script

'Write 'batch script', not 'slurm script'.'
[interactive session](../../Interactive_Computing/Slurm_Interactive_Sessions.md).

### RSM
Expand Down Expand Up @@ -727,6 +742,10 @@
export SLURM_EXCLUSIVE="" # don't share CPUs
echo "CPUs: Coupler:1 Struct:$MECHANICAL_CPUS Fluid:$FLUID_CPUS"

# Write the host file for Fluent before anything else holds the CPUs
HOSTFILE=${PWD}/hosts.${SLURM_JOB_ID}
srun hostname -s | sort > ${HOSTFILE}

echo "STARTING SYSTEM COUPLER"

cd Coupling
Expand Down Expand Up @@ -782,7 +801,7 @@
cd FluidFlow

# Run Fluent in the background, alongside the system coupler and ANSYS.
fluent 3ddp -g -t$FLUID_CPUS \
fluent 3ddp -g -t$FLUID_CPUS -cnf=${HOSTFILE} \
-scport=$port -schost=$node -scname="$fluentsolname" \
-i "fluidFlow.jou" > fluent.out || scancel $SLURM_JOBID &
cd ..
Expand All @@ -800,6 +819,11 @@
from the main memory to the GPU memory. We therefore suggest that you
only use them for larger jobs, unless benchmarking reveals otherwise.

### MPI

Intel MPI is the default and recommended MPI for ANSYS on Mahuika, so you do not need an `-mpi` flag.
The ANSYS module also sets the parallel start method for CFX.

### Interactive use

It is best to use journal files *etc* to automate ANSYS so that you can
Expand All @@ -811,7 +835,11 @@
instead of `sbatch`, for example:

``` bash
salloc -A nesi99991 -t 30 -n 16 -C avx --mem-per-cpu=512MB bash -c 'module load ANSYS; fluent -v3ddp -t$SLURM_NTASKS'
salloc --account nesi99991 --time 30 --ntasks 16 --mem-per-cpu 512MB \
bash -c 'module load ANSYS/2025R2
HOSTFILE=${PWD}/hosts.${SLURM_JOB_ID}
srun hostname -s | sort > ${HOSTFILE}
fluent 3ddp -t${SLURM_NTASKS} -cnf=${HOSTFILE}'
```

As with any job, you may have to wait a while before the resource is
Expand All @@ -821,4 +849,4 @@
### Simultaneous Multithreading

Utilising [simultaneous multithreading](../Parallel_Computing/Simultaneous_Multithreading.md) (i.e.: adding the `--hint=multithread` sbatch directive and doubling the number of tasks) will give a small speedup on most jobs with less than 8 cores, but also doubles the number of
`aa_r_hpc` license tokens required.

Check warning on line 852 in docs/Software/Available_Applications/ANSYS.md

View workflow job for this annotation

GitHub Actions / Check Prose

reannz.licence

'Write 'licence' for the noun ('license' is the verb), not 'license tokens'.'
Loading