Skip to content

Fix benchmark result filenames #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bnbailey-psl wants to merge 2 commits into from
Closed

Fix benchmark result filenames #79

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fb9316b
Improve benchmarks script
bnbailey-psl Jul 13, 2025
cc0f2de
Include version in benchmark reports
bnbailey-psl Jul 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,22 @@ Helios is a C++ library for 3D physical simulation of plant and environmental sy

In order to build and compile the core library, you will need to install a C/C++ compiler (recommended are the GNU C compilers version 7.0+), and CMake. In order to run many of the model plug-ins, you will need to install NVIDIA CUDA 10.2+, and a GPU with compute capability 5.0+. The software has been tested on Linux, Mac, and Windows platforms. The YouTube channel linked above has a number of tutorials for getting started.

![Almond Reconstruction](doc/images/AlmondVarietyReconstruction.png)
![Almond Reconstruction](doc/images/AlmondVarietyReconstruction.png)

## Benchmarking

To compare runtime across different machines run the benchmark script on each
system. You can optionally provide a list of thread counts or log the output:

```bash
./utilities/run_benchmarks.sh "1,4,16" --log-file my_benchmark.log --verbose
```

This produces one results file per benchmark in `benchmarks/report` named
`<benchmark>_<hostname>_<version>_<gpu>.txt`. The first line of each
results file records the Helios version. After collecting reports from multiple
machines they can be plotted with:

```bash
python3 utilities/plot_benchmarks.py benchmarks/report/*.txt
```
191 changes: 124 additions & 67 deletions utilities/run_benchmarks.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,42 +1,108 @@
#!/bin/bash

# Run Helios benchmarks and collect runtime information.
#
# Usage:
# ./benchmark_script.sh [THREAD_COUNTS]
# $(basename "$0") [THREAD_COUNTS] [--verbose] [--log-file <file>] [--help]
#
# Arguments:
# THREAD_COUNTS - Optional. A comma or space-separated list of thread counts to use
# (e.g., "1,2,4,8,16" or "1 2 4 8 16").
# If not provided, the benchmark will run with 1 thread only.
#
# Examples:
# ./benchmark_script.sh "1,2,4,8,16" # Run with 1, 2, 4, 8, and 16 threads
# ./benchmark_script.sh "1 4 16" # Run with 1, 4, and 16 threads
# ./benchmark_script.sh # Run with 1 thread only (default)
# THREAD_COUNTS - Optional comma or space separated list of thread counts
# (e.g., "1,2,4" or "1 2 4"). If omitted the benchmarks run
# with a single thread.
# --verbose - Print build and run output to the console.
# --log-file - Write all output to the specified file. If used together
# with --verbose the output is also echoed to the console.
# --help - Show this help message and exit.
#
# Output:
# Results are saved to individual files in the benchmarks/report directory.
# The output includes benchmark name, GPU, build type, thread count,
# benchmark label, and runtime.
# Results are written to individual files in benchmarks/report named
# <benchmark>_<hostname>_<version>_<gpu>.txt.
#
# Environment:
# Requires NVIDIA GPU with drivers and CUDA toolkit installed.
# Sets OMP_NUM_THREADS environment variable to control OpenMP threading.
# Requires NVIDIA GPU with drivers and CUDA toolkit installed. The
# OMP_NUM_THREADS environment variable is used to set the thread count.
#

# Function to display usage information
usage() {
sed -n '3,18p' "$0"
exit 1
}

# Helper to run commands with optional logging
run_command() {
if [ -n "$LOG_FILE" ]; then
if [ "$VERBOSE" = "ON" ]; then
"$@" 2>&1 | tee -a "$LOG_FILE"
return ${PIPESTATUS[0]}
else
"$@" >> "$LOG_FILE" 2>&1
fi
elif [ "$VERBOSE" = "ON" ]; then
"$@"
else
"$@" >/dev/null 2>&1
fi
}

SAMPLES=("radiation_homogeneous_canopy")

BUILD_TYPES=("Debug" "Release")

# Default thread counts if not specified
THREAD_COUNTS=(1)

# Parse command-line arguments for thread counts
if [ $# -gt 0 ]; then
# Clear default and use provided values
# Parse arguments
VERBOSE="OFF"
LOG_FILE=""
THREAD_STRING=""

while [ $# -gt 0 ]; do
case $1 in
--verbose)
VERBOSE="ON"
;;
--log-file)
[ -z "$2" ] && usage
if [[ $2 = /* ]]; then
LOG_FILE="$2"
else
LOG_FILE="$(pwd)/$2"
fi
shift
;;
--help|-h)
usage
;;
*)
if [[ -z "$THREAD_STRING" ]]; then
THREAD_STRING="$1"
else
echo "Unknown option: $1" >&2
usage
fi
;;
esac
shift
done

if [ -n "$LOG_FILE" ]; then
> "$LOG_FILE"
fi

if [ -n "$THREAD_STRING" ]; then
THREAD_COUNTS=()
# Split the argument by commas and spaces
IFS=', ' read -r -a THREAD_COUNTS <<< "$1"
IFS=', ' read -r -a THREAD_COUNTS <<< "$THREAD_STRING"
fi

# Validate that all thread counts are positive integers
for t in "${THREAD_COUNTS[@]}"; do
if ! [[ $t =~ ^[1-9][0-9]*$ ]]; then
echo "Error: invalid thread count '$t'" >&2
exit 1
fi
done

# Get the directory script is in
SCRIPT_PATH="$(readlink -f "$0")"
SCRIPT_DIR="$( dirname "$SCRIPT_PATH" )"
Expand Down Expand Up @@ -67,116 +133,105 @@ if [ "${GPU_NAME}" == "" ];then
exit 1
fi

# Determine the current Helios version from git
HELIOS_VERSION=$(git describe --tags 2>/dev/null)
if [ -z "$HELIOS_VERSION" ]; then
HELIOS_VERSION="unknown"
fi
SANITIZED_VERSION=$(echo "$HELIOS_VERSION" | tr ' /:*?"<>|\\' '_')

echo "---------------------------------------------"
echo "Benchmarking on ${GPU_NAME}"
echo "Benchmarking Helios ${HELIOS_VERSION} on ${GPU_NAME}"
echo "Using thread counts: ${THREAD_COUNTS[*]}"
echo "---------------------------------------------"

ERROR_COUNT=0
for i in "${SAMPLES[@]}"; do

SAMPLE_DIR="${SCRIPT_DIR}/../benchmarks/${i}"

if [ ! -e "${SAMPLE_DIR}" ]; then
echo "Sample ${i} does not exist."
if [ ! -d "${SAMPLE_DIR}" ]; then
echo "Sample ${i} does not exist." >&2
exit 1
fi
if [ ! -e "${SAMPLE_DIR}/build" ]; then
mkdir -p "${SAMPLE_DIR}/build"
fi

cd "${SAMPLE_DIR}" || exit 1

# Create a new consolidated results file for this benchmark
RESULTS_FILE="${SAMPLE_DIR}/results/runtime.txt"
# Initialize the file (clear it if it already exists)
: > "${RESULTS_FILE}"

RUNTIME_FILE="../results/runtime.txt"

# Create sanitized GPU name for filename (replace spaces and special chars with underscores)
# Sanitize GPU and hostname for filenames
SANITIZED_GPU_NAME=$(echo "${GPU_NAME}" | tr ' /:*?"<>|\\' '_')
HOST_NAME=$(hostname)
SANITIZED_HOST_NAME=$(echo "${HOST_NAME}" | tr ' /:*?"<>|\\' '_')

# Define output file with benchmark name and GPU name
OUTPUT_FILE="${SCRIPT_DIR}/../benchmarks/report/${i}_${SANITIZED_GPU_NAME}.txt"
git describe --tags > "${OUTPUT_FILE}"
OUTPUT_FILE="${SCRIPT_DIR}/../benchmarks/report/${i}_${SANITIZED_HOST_NAME}_${SANITIZED_VERSION}_${SANITIZED_GPU_NAME}.txt"
echo "$HELIOS_VERSION" > "${OUTPUT_FILE}"

for build in "${BUILD_TYPES[@]}"; do
UNIQUE_ID="$(date +%s)_$$"
WORK_DIR="${SAMPLE_DIR}/run_${UNIQUE_ID}"
BUILD_DIR="${WORK_DIR}/build"
RESULT_DIR="${WORK_DIR}/results"

cd "${SAMPLE_DIR}"/build || exit 1

rm -rf "${SAMPLE_DIR}"/build/*glob*
mkdir -p "${BUILD_DIR}" "${RESULT_DIR}"
cd "${BUILD_DIR}" || exit 1

echo -ne "Building benchmark ${i} using build type ${build}...\n"

cmake .. -DCMAKE_BUILD_TYPE="${build}" -DENABLE_OPENMP=ON &>/dev/null

run_command cmake "${SAMPLE_DIR}" -DCMAKE_BUILD_TYPE="${build}" -DENABLE_OPENMP=ON
if (($? == 0)); then
echo -e "\r\x1B[32mBuilding benchmark ${i}...done.\x1B[39m"
else
echo -e "\r\x1B[31mBuilding benchmark ${i}...failed.\x1B[39m"
ERROR_COUNT=$((ERROR_COUNT + 1))
rm -rf "${SAMPLE_DIR}"/build/*glob*
rm -rf "${WORK_DIR}"
continue
fi

echo -ne "Compiling benchmark ${i}..."

cmake --build ./ --target "${i}" --config "${build}" &>/dev/null

run_command cmake --build ./ --target "${i}" --config "${build}"
if (($? == 0)); then
if [ -e "${i}" ]; then
if [ -e "${i}" ] || [ -e "${i}.exe" ]; then
echo -e "\r\x1B[32mCompiling benchmark ${i}...done.\x1B[39m"
else
echo -e "\r\x1B[31mCompiling benchmark ${i}...failed.\x1B[39m"
ERROR_COUNT=$((ERROR_COUNT + 1))
rm -rf "${SAMPLE_DIR}"/build/*glob*
rm -rf "${WORK_DIR}"
continue
fi
else
echo -e "\r\x1B[31mCompiling benchmark ${i}...failed.\x1B[39m"
ERROR_COUNT=$((ERROR_COUNT + 1))
rm -rf "${SAMPLE_DIR}"/build/*glob*
rm -rf "${WORK_DIR}"
continue
fi

# Loop over different thread counts
for threads in "${THREAD_COUNTS[@]}"; do
echo -ne "Running benchmark ${i} with ${threads} threads..."

# Set the OpenMP threads environment variable
export OMP_NUM_THREADS="${threads}"

if [[ "${OSTYPE}" == "msys"* ]]; then
"./${i}.exe" &>/dev/null
run_command "./${i}.exe"
exit_code=$?
else
"./${i}" &>/dev/null
run_command "./${i}"
exit_code=$?
fi

if [[ $exit_code -ne 0 ]]; then
echo -e "\r\x1B[31mRunning benchmark ${i} with ${threads} threads...failed with exit code ${exit_code}.\x1B[39m"
echo -e "\r\x1B[31mThe benchmark ${i} did not run successfully.\x1B[39m"
return 1
ERROR_COUNT=$((ERROR_COUNT + 1))
continue
fi

echo -e "\r\x1B[32mRunning benchmark ${i} with ${threads} threads...done.\x1B[39m"

if [ ! -e "../results/runtime.txt" ]; then
if [ ! -e "${RESULT_DIR}/runtime.txt" ]; then
echo -e "\r\x1B[31mResults file was not produced for ${threads} threads.\x1B[39m"
ERROR_COUNT=$((ERROR_COUNT + 1))
continue
fi

# Process runtime.txt and append to consolidated results file
while IFS=',' read -r label runtime; do
# Append with GPU name, build type, and thread count
echo "${i},${GPU_NAME},${build},${threads},${label},${runtime}" >> "${OUTPUT_FILE}"
done < "${RUNTIME_FILE}"
done < "${RESULT_DIR}/runtime.txt"
rm -f "${RESULT_DIR}/runtime.txt"
done

rm -rf "${SAMPLE_DIR}"/build/*glob*
rm -rf "${WORK_DIR}"

done #loop over build types

Expand All @@ -188,5 +243,7 @@ cat "${OUTPUT_FILE}"
echo -e "\n\x1B[32mResults have been saved to individual files in the benchmarks directory.\x1B[39m"
for i in "${SAMPLES[@]}"; do
SANITIZED_GPU_NAME=$(echo "${GPU_NAME}" | tr ' /:*?"<>|\\' '_')
echo " - ${SCRIPT_DIR}/../benchmarks/report/${i}_${SANITIZED_GPU_NAME}.txt"
done
HOST_NAME=$(hostname)
SANITIZED_HOST_NAME=$(echo "${HOST_NAME}" | tr ' /:*?"<>|\\' '_')
echo " - ${SCRIPT_DIR}/../benchmarks/report/${i}_${SANITIZED_HOST_NAME}_${SANITIZED_VERSION}_${SANITIZED_GPU_NAME}.txt"
done
Loading