Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 19 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,50 +9,58 @@ HPX-FFT depends on [HPX](https://hpx-docs.stellar-group.org/latest/html/index.ht
### Install dependencies

All dependencies can be installed using [Spack](https://github.com/spack/spack).
For HPX we recommend: `spack install hpx@1.10.0 networking={tcp/mpi/lci} max_cpu_count=256`
For HPX we recommend: `spack install hpx@1.11.0 networking={tcp/mpi/lci/all} max_cpu_count=256`
For FFTW we recommend `spack install fftw@3.3.10 +mpi +openmp`

Note that the LCI variant of HPX currently requires to add the [Octo-Tiger repo](https://github.com/G-071/octotiger-spack) to spack.
Alternatively, both librarires can be built from source.
The requirements can be also installed using the provided spack environments.
The spack-repo directory add support for installing HPX with the LCI or all parcelports at once.

## How to Compile HPX-FFT

We provide a simple compilation script for HPX-FFT. The script assumes an gcc compiler version 14.2.0. However, it can be easily tailored towards specific needs.

## How to Compile Examples
## How to Compile Examples:

HPX-FFT comes with several examples for HPX-FFT itself as well as for a FFTW reference. The examples can be compiled with the respective compilation scripts.

## How to Run Examples

### HPX-FFT

Shared examples:
```
1=Base size of the problem
2=FFTW Planning flag (estimate/measure)
3=Print info into runtime file (true/false)
4=Number of HPX threads
./hpxfft_shared_loop --nx=$1 --ny=$1 --plan=$2 --header=$3 --hpx:threads=$4
```

Distributed examples with slurm:
```
1=Base size of the problem
2=FFTW Planning flag (estimate/measure)
3=Print info into runtime file (true/false)
4=Collective operation (scatter/all_to_all)
5=Number of HPX threads
./hpxfft_distributed_loop --nx=$1 --ny=$1 --plan=$2 --header=$3 --run=$4 --hpx:threads=$5
6=Number of HPX localities
7=Partition

srun --mpi=pmix -p $6 -N 2 -n $6 -c $5 ./examples/hpxfft/build/hpxfft_distributed_loop --nx=$1 --ny=$1 --plan=$2 --header=$3 --run=$4 --hpx:ini=hpx.parcel.mpi.enable=1 --hpx:ini=hpx.parcel.tcp.enable=0 --hpx:ini=hpx.parcel.lci.enable=0
```

### FFTW

Shared examples:
```
// threads N_X N_Y plan header
./fftw_omp 1 8 14 estimate 0
```

Distributed examples with srun or mpirun:
```
// nodes ranks prog threads N_X N_Y plan header
srun -N 2 -n 4 fftw_mpi_omp 4 8 14 estimate 0
mpirun -n 4 fftw_mpi_omp 4 8 14 estimate 0
// nodes ranks prog threads N_X N_Y plan header
srun --mpi=pmix -N 2 -n 4 fftw_mpi_omp 4 8 14 estimate 0
mpirun -n 4 fftw_mpi_omp 4 8 14 estimate 0
```

## How to Run Benchmarks (wip)
Expand All @@ -73,17 +81,8 @@ Executables only:
`sbatch -p $PARTITION -N 1 -n 1 -c $THREADS run_fftw_shared.sh executable_name estimate/measure $THREAD_POW`

### Distributed
All:

- `./distributed_benchmark.sh estimate/measure partition_name`

Executables only:

- fftw_hpx_loop/fftw_hpx_tasg_agas:
`sbatch -p $PARTITION -N $NODES -n $NODES -c $THREADS run_hpx_dist.sh executable_name estimate/measure $NODES`

- fftw_mpi_threads/fftw_mpi_omp:
`sbatch -p $PARTITION -N $NODES -n $NODES -c $THREADS run_fftw_dist.sh executable_name estimate/measure $NODES $THREADS`
- `./distributed_benchmark.sh estimate/measure scatter/all_to_all tcp/mpi/lci`

## The Team

Expand All @@ -95,6 +94,7 @@ We specifically thank the follow contributors:

- [Alexander Strack](https://www.ipvs.uni-stuttgart.de/de/institut/team/Strack-00001/):
Maintainer and [initial framework](https://doi.org/10.1007/978-3-031-32316-4_5).
- Heiko Häfner: Testing and CI.

## How To Cite

Expand Down
52 changes: 26 additions & 26 deletions benchmark/distributed_benchmark.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# Benchmark script for shared memory
# $1: FFTW planning flag (estimate/measure)
# $2: Collective (scatter/all_to_all)
# $3: HPX Parcelport (tcp/mpi/lci)
################################################################################
# Config
################################################################################
Expand All @@ -21,16 +22,11 @@ FFTW_EXECUTABLES=(

# Get current hostname
HOSTNAME=$(hostname -s)
if [[ "$HOSTNAME" == "ipvsmisc" ]]; then
echo "tbd."
elif [[ "$HOSTNAME" == "rostam" ]]; then
echo "tbd."
elif [[ "$HOSTNAME" == "login1" ]]; then
PARTITION=short
NODES_POW=1
if [[ "$HOSTNAME" == "rostam1" ]]; then
PARTITION=buran
NODES_POW=4
THREADS=48
elif [[ "$HOSTNAME" == "simcl1" ]]; then
echo "tbd."
module load gcc/14.2.0
else
echo "Hostname is $HOSTNAME — no action taken."
exit 1
Expand All @@ -42,22 +38,30 @@ FFTW_PLAN=$1
# Check if FFTW_PLAN was provided
if [ -z "$FFTW_PLAN" ]; then
echo "Error: FFTW_PLAN parameter not set."
echo "Usage: distributed_benchmark.sh estimate/measure scatter/all_to_all"
echo "Usage: distributed_benchmark.sh estimate/measure scatter/all_to_all tcp/mpi/lci"
exit 1
fi
# Set Collective
COLLECTIVE=$2
# Check if COLLECITVE was provided
if [ -z "$COLLECTIVE" ]; then
echo "Error: COLLECTIVE parameter not set."
echo "Usage: distributed_benchmark.sh estimate/measure scatter/all_to_all"
echo "Usage: distributed_benchmark.sh estimate/measure scatter/all_to_all tcp/mpi/lci"
exit 1
fi
# Set HPX Parcelport
PARCELPORT=$3
# Check if COLLECITVE was provided
if [ -z "$PARCELPORT" ]; then
echo "Error: Parcelport parameter not set."
echo "Usage: distributed_benchmark.sh estimate/measure scatter/all_to_all tcp/mpi/lci"
exit 1
fi

################################################################################
# Run benchmarks
################################################################################
RESULT_DIR=$TOP_DIR/distributed_benchmark_on_$HOSTNAME
RESULT_DIR=$TOP_DIR/distributed_benchmark_on_$PARTITION
SCRIPT_DIR=$TOP_DIR/benchmark/sbatch_scripts

################################################################################
Expand All @@ -68,7 +72,7 @@ mkdir -p $RESULT_DIR/strong_scaling/plans
cd $RESULT_DIR/strong_scaling

# Config
LOOP=2
LOOP=10
BASE_SIZE=16384
START_NODES_POW=0

Expand All @@ -77,9 +81,8 @@ RUN_SCRIPT="$SCRIPT_DIR/run_hpxfft_strong_distributed.sh"
for EXE in "${HPXFFT_EXECUTABLES[@]}"; do
echo "Submitting job for executable: $EXE"

sbatch -p $PARTITION -N $NODES -n $NODES -c $THREADS \
$RUN_SCRIPT \
$EXE $FFTW_PLAN $START_NODES_POW $NODES_POW $BASE_SIZE $COLLECTIVE $THREADS $LOOP
$RUN_SCRIPT \
$EXE $FFTW_PLAN $START_NODES_POW $NODES_POW $BASE_SIZE $COLLECTIVE $THREADS $LOOP $PARTITION $PARCELPORT
done

# Loop over FFTW executables
Expand All @@ -88,9 +91,8 @@ RUN_SCRIPT="$SCRIPT_DIR/run_fftw_strong_distributed.sh"
for EXE in "${FFTW_EXECUTABLES[@]}"; do
echo "Submitting job for executable: $EXE"

sbatch -p $PARTITION -N $NODES -n $NODES -c $THREADS \
$RUN_SCRIPT \
$EXE $FFTW_PLAN $START_NODES_POW $NODES_POW $BASE_SIZE $THREADS $LOOP
$RUN_SCRIPT \
$EXE $FFTW_PLAN $START_NODES_POW $NODES_POW $BASE_SIZE $THREADS $LOOP $PARTITION
done

################################################################################
Expand All @@ -101,7 +103,7 @@ mkdir -p $RESULT_DIR/weak_scaling/plans
cd $RESULT_DIR/weak_scaling

# Config
LOOP=2
LOOP=10
BASE_SIZE=512
START_THREAD_POW=0

Expand All @@ -110,9 +112,8 @@ RUN_SCRIPT="$SCRIPT_DIR/run_hpxfft_weak_distributed.sh"
for EXE in "${HPXFFT_EXECUTABLES[@]}"; do
echo "Submitting job for executable: $EXE"

sbatch -p $PARTITION -N $NODES -n $NODES -c $THREADS \
$RUN_SCRIPT \
$EXE $FFTW_PLAN $START_NODES_POW $NODES_POW $BASE_SIZE $COLLECTIVE $THREADS $LOOP
$RUN_SCRIPT \
$EXE $FFTW_PLAN $START_NODES_POW $NODES_POW $BASE_SIZE $COLLECTIVE $THREADS $LOOP $PARTITION $PARCELPORT
done

# Loop over FFTW executables
Expand All @@ -121,7 +122,6 @@ RUN_SCRIPT="$SCRIPT_DIR/run_fftw_weak_distributed.sh"
for EXE in "${FFTW_EXECUTABLES[@]}"; do
echo "Submitting job for executable: $EXE"

sbatch -p $PARTITION -N $NODES -n $NODES -c $THREADS \
$RUN_SCRIPT \
$EXE $FFTW_PLAN $START_NODES_POW $NODES_POW $BASE_SIZE $THREADS $LOOP
$RUN_SCRIPT \
$EXE $FFTW_PLAN $START_NODES_POW $NODES_POW $BASE_SIZE $THREADS $LOOP $PARTITION
done
42 changes: 14 additions & 28 deletions benchmark/message_benchmark.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/bash
# Benchmark script for message size scaling between two nodes
# $1: Collective (scatter/all_to_all)
# $2: HPX Parcelport (tcp/mpi/lci)
################################################################################
# Config
################################################################################
Expand All @@ -12,23 +13,13 @@ HPXFFT_EXECUTABLES=(
"$EXAMPLES_DIR/hpxfft/build/hpxfft_distributed_loop"
"$EXAMPLES_DIR/hpxfft/build/hpxfft_distributed_agas"
)
# FFTW implementations
FFTW_EXECUTABLES=(
"$EXAMPLES_DIR/fftw/build/fftw_mpi_omp"
"$EXAMPLES_DIR/fftw/build/fftw_mpi_threads"
)

# Get current hostname
HOSTNAME=$(hostname -s)
if [[ "$HOSTNAME" == "ipvsmisc" ]]; then
echo "tbd."
elif [[ "$HOSTNAME" == "rostam" ]]; then
echo "tbd."
elif [[ "$HOSTNAME" == "login1" ]]; then
PARTITION=short
if [[ "$HOSTNAME" == "rostam1" ]]; then
PARTITION=buran
THREADS=48
elif [[ "$HOSTNAME" == "simcl1" ]]; then
echo "tbd."
module load gcc/14.2.0
else
echo "Hostname is $HOSTNAME — no action taken."
exit 1
Expand All @@ -42,11 +33,18 @@ if [ -z "$COLLECTIVE" ]; then
echo "Usage: message_benchmark.sh scatter/all_to_all"
exit 1
fi

# Set HPX Parcelport
PARCELPORT=$2
# Check if COLLECITVE was provided
if [ -z "$PARCELPORT" ]; then
echo "Error: Parcelport parameter not set."
echo "Usage: message_benchmark.sh scatter/all_to_all tcp/mpi/lci"
exit 1
fi
################################################################################
# Run benchmarks
################################################################################
RESULT_DIR=$TOP_DIR/message_benchmark_on_$HOSTNAME
RESULT_DIR=$TOP_DIR/message_benchmark_on_$PARTITION
SCRIPT_DIR=$TOP_DIR/benchmark/sbatch_scripts

################################################################################
Expand All @@ -65,18 +63,6 @@ RUN_SCRIPT="$SCRIPT_DIR/run_hpxfft_message_size.sh"
for EXE in "${HPXFFT_EXECUTABLES[@]}"; do
echo "Submitting job for executable: $EXE"

sbatch -p $PARTITION -N 2 -n 2 -c $THREADS \
$RUN_SCRIPT \
$EXE $BASE_SIZE $STOP_POW $COLLECTIVE $THREADS $LOOP
done

# Loop over FFTW executables
RUN_SCRIPT="$SCRIPT_DIR/run_fftw_message_size.sh"
# Loop over executables
for EXE in "${FFTW_EXECUTABLES[@]}"; do
echo "Submitting job for executable: $EXE"

sbatch -p $PARTITION -N 2 -n 2 -c $THREADS \
$RUN_SCRIPT \
$EXE $BASE_SIZE $STOP_POW $THREADS $LOOP
$EXE $BASE_SIZE $STOP_POW $COLLECTIVE $THREADS $LOOP $PARTITION $PARCELPORT
done
43 changes: 0 additions & 43 deletions benchmark/sbatch_scripts/run_fftw_message_size.sh

This file was deleted.

1 change: 0 additions & 1 deletion benchmark/sbatch_scripts/run_fftw_size_shared.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
#SBATCH --job-name=fftw_job # Job name
#SBATCH --output=fftw_job.log # Standard output and error log
#SBATCH --mail-type=NONE # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=alexander.strack@ipvs.uni-stuttgart.de # Where to send mail
#SBATCH --time=2:00:00 # Time limit hrs:min:sec
#SBATCH --exclusive # Exclusive ressource access

Expand Down
15 changes: 7 additions & 8 deletions benchmark/sbatch_scripts/run_fftw_strong_distributed.sh
Original file line number Diff line number Diff line change
@@ -1,10 +1,4 @@
#!/bin/bash
#SBATCH --job-name=fftw_job # Job name
#SBATCH --output=fftw_job.log # Standard output and error log
#SBATCH --mail-type=NONE # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=alexander.strack@ipvs.uni-stuttgart.de # Where to send mail
#SBATCH --time=2:00:00 # Time limit hrs:min:sec
#SBATCH --exclusive # Exclusive ressource access

# Benchmark script for distributed memory strong scaling
# $1: Executable name
Expand All @@ -14,6 +8,7 @@
# $5: Base size
# $6: Number of threads per node
# $7: Number of runs
# $8: Partition

# Log Info
pwd; hostname; date
Expand All @@ -23,21 +18,25 @@ POW_STOP=$4
BASE_SIZE=$5
THREADS=$6
LOOP=$7
PARTITION=$8
# Get run command
COMMAND="srun -N $((2**POW_START)) -n $((2**POW_START)) -c $THREADS"
COMMAND="srun --mpi=pmix -p $PARTITION --nodelist=$PARTITION[00-0$((2**POW_START - 1))] -N $((2**POW_START)) -n $((2**POW_START)) -c $THREADS"
EXECUTABLE=$1
ARGUMENTS="$BASE_SIZE $BASE_SIZE $2"
# Strong scaling loop from 2^pow_start to 2^pow_stop nodes
echo 'Submiting:' $COMMAND $EXECUTABLE $THREADS $ARGUMENTS
$COMMAND $EXECUTABLE $THREADS $ARGUMENTS 1
for (( j=1; j<$LOOP; j=j+1 ))
do
echo 'Submiting:' $COMMAND $EXECUTABLE $THREADS $ARGUMENTS
$COMMAND $EXECUTABLE $THREADS $ARGUMENTS 0
done
for (( i=2**($POW_START+1); i<=2**$POW_STOP; i=i*2 ))
do
COMMAND="srun -N $i -n $i -c $THREADS"
COMMAND="srun --mpi=pmix -p $PARTITION --nodelist=$PARTITION[00-0$(($i - 1))] -N $i -n $i -c $THREADS"
for (( j=0; j<$LOOP; j=j+1 ))
do
echo 'Submiting:' $COMMAND $EXECUTABLE $THREADS $ARGUMENTS
$COMMAND $EXECUTABLE $THREADS $ARGUMENTS 0
done
done
Expand Down
Loading
Loading