Skip to content

Read write per timestep timing #141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

Read write per timestep timing #141

wants to merge 1 commit into from

Conversation

@TheAssembler1
Copy link

@TheAssembler1 TheAssembler1 commented Oct 24, 2025

Adds per step times in CSV for h5bench write/read. If async is enabled the wait times per step are also written to CSV. The following results are from Perlmutter single node 32 ranks.

Config/output when async is enabled:

{
    "mpi": {
        "command": "srun",
        "ranks": "32",
        "configuration": "-N 1 -n 32"
    },
    "vol": {
        "library": "/pscratch/sd/n/nlewi26/src/vol_connectors/lib:/pscratch/sd/n/nlewi26/src/vol-async/install/lib:/pscratch/sd/n/nlewi26/src/argobots-1.2/install/lib:/pscratch/sd/n/nlewi26/src/hdf5/install/lib:/pscratch/sd/n/nlewi26/src/vol-cache/install/lib:",
        "path": "/pscratch/sd/n/nlewi26/src/vol_connectors/lib",
        "connector": "async under_vol=0;under_info={}"
    },
    "file-system": {
        "lustre": {
            "stripe-size": "1M",
            "stripe-count": "1"
        }
    },
    "directory": "/pscratch/sd/n/nlewi26/h5bench_data",
    "benchmarks": [
        {
            "benchmark": "write",
            "file": "test.h5",
            "configuration": {
                "MEM_PATTERN": "CONTIG",
                "FILE_PATTERN": "CONTIG",
                "TIMESTEPS": "5",
                "DELAYED_CLOSE_TIMESTEPS": "1",
                "COLLECTIVE_DATA": "YES",
                "COLLECTIVE_METADATA": "YES",
                "EMULATED_COMPUTE_TIME_PER_TIMESTEP": "20 s", 
                "NUM_DIMS": "1",
                "DIM_1": "8388608",
                "DIM_2": "1",
                "DIM_3": "1",
                "CSV_FILE": "write_output.csv",
                "MODE": "ASYNC"
            }
        },
        {
            "benchmark": "read",
            "file": "test.h5",
            "configuration": {
                "MEM_PATTERN": "CONTIG",
                "FILE_PATTERN": "CONTIG",
                "TIMESTEPS": "5",
                "DELAYED_CLOSE_TIMESTEPS": "1",
                "COLLECTIVE_DATA": "YES",
                "COLLECTIVE_METADATA": "YES",
                "EMULATED_COMPUTE_TIME_PER_TIMESTEP": "20 s", 
                "NUM_DIMS": "1",
                "DIM_1": "8388608",
                "DIM_2": "1",
                "DIM_3": "1",
                "CSV_FILE": "read_output.csv",
                "MODE": "ASYNC"
            }
        }
    ]
}

Write benchmark:

metric, value, unit
operation, write, 
ranks, 32, 
collective data, YES, 
collective meta, YES, 
subfiling, NO, 
total compute time, 80.000, seconds
total size, 40.000, GB
raw time, 0.002, seconds
raw rate, 25.852, TB/s
metadata time, 27.027, seconds
observed rate, 1.464, GB/s
observed time, 107.315, seconds
timestep 0 data time, 0.001, seconds
timestep 1 data time, 0.000, seconds
timestep 2 data time, 0.000, seconds
timestep 3 data time, 0.000, seconds
timestep 4 data time, 0.000, seconds
timestep 0 metadata time, 0.006, seconds
timestep 1 metadata time, 14.011, seconds
timestep 2 metadata time, 0.002, seconds
timestep 3 metadata time, 0.002, seconds
timestep 4 metadata time, 0.002, seconds
timestep 0 data wait time, 0.000, seconds
timestep 1 data wait time, 0.000, seconds
timestep 2 data wait time, 0.000, seconds
timestep 3 data wait time, 0.000, seconds
timestep 4 data wait time, 0.000, seconds
timestep 0 metadata wait time, 0.000, seconds
timestep 1 metadata wait time, 0.000, seconds
timestep 2 metadata wait time, 0.000, seconds
timestep 3 metadata wait time, 0.000, seconds
timestep 4 metadata wait time, 13.004, seconds

Read benchmark:

metric, value, unit
operation, read, 
ranks, 32, 
collective data, YES, 
collective meta, YES, 
subfiling, NO, 
total compute time, 80.000, seconds
total size, 40.000, GB
raw time, 0.001, seconds
raw rate, 33.443, TB/s
metadata time, 14.044, seconds
observed rate, 2.838, GB/s
observed time, 94.094, seconds
timestep 0 data time, 0.000, seconds
timestep 1 data time, 0.000, seconds
timestep 2 data time, 0.000, seconds
timestep 3 data time, 0.000, seconds
timestep 4 data time, 0.000, seconds
timestep 0 metadata time, 0.001, seconds
timestep 1 metadata time, 7.036, seconds
timestep 2 metadata time, 0.002, seconds
timestep 3 metadata time, 0.002, seconds
timestep 4 metadata time, 0.002, seconds
timestep 0 data wait time, 0.000, seconds
timestep 1 data wait time, 0.000, seconds
timestep 2 data wait time, 0.000, seconds
timestep 3 data wait time, 0.000, seconds
timestep 4 data wait time, 0.000, seconds
timestep 0 metadata wait time, 0.000, seconds
timestep 1 metadata wait time, 0.000, seconds
timestep 2 metadata wait time, 0.000, seconds
timestep 3 metadata wait time, 0.000, seconds
timestep 4 metadata wait time, 7.003, seconds

output when sync is enabled.

{
    "mpi": {
        "command": "srun",
        "ranks": "32",
        "configuration": "-N 1 -n 32"
    },
    "vol": {
        "library": "/pscratch/sd/n/nlewi26/src/vol_connectors/lib:/pscratch/sd/n/nlewi26/src/vol-async/install/lib:/pscratch/sd/n/nlewi26/src/argobots-1.2/install/lib:/pscratch/sd/n/nlewi26/src/hdf5/install/lib:/pscratch/sd/n/nlewi26/src/vol-cache/install/lib:",
        "path": "/pscratch/sd/n/nlewi26/src/vol_connectors/lib",
        "connector": ""
    },
    "file-system": {
        "lustre": {
            "stripe-size": "1M",
            "stripe-count": "1"
        }
    },
    "directory": "/pscratch/sd/n/nlewi26/h5bench_data",
    "benchmarks": [
        {
            "benchmark": "write",
            "file": "test.h5",
            "configuration": {
                "MEM_PATTERN": "CONTIG",
                "FILE_PATTERN": "CONTIG",
                "TIMESTEPS": "5",
                "DELAYED_CLOSE_TIMESTEPS": "1",
                "COLLECTIVE_DATA": "YES",
                "COLLECTIVE_METADATA": "YES",
                "EMULATED_COMPUTE_TIME_PER_TIMESTEP": "20 s", 
                "NUM_DIMS": "1",
                "DIM_1": "8388608",
                "DIM_2": "1",
                "DIM_3": "1",
                "CSV_FILE": "write_output.csv",
                "MODE": "SYNC"
            }
        },
        {
            "benchmark": "read",
            "file": "test.h5",
            "configuration": {
                "MEM_PATTERN": "CONTIG",
                "FILE_PATTERN": "CONTIG",
                "TIMESTEPS": "5",
                "DELAYED_CLOSE_TIMESTEPS": "1",
                "COLLECTIVE_DATA": "YES",
                "COLLECTIVE_METADATA": "YES",
                "EMULATED_COMPUTE_TIME_PER_TIMESTEP": "20 s", 
                "NUM_DIMS": "1",
                "DIM_1": "8388608",
                "DIM_2": "1",
                "DIM_3": "1",
                "CSV_FILE": "read_output.csv",
                "MODE": "SYNC"
            }
        }
    ]
}

Write benchmark:

metric, value, unit
operation, write, 
ranks, 32, 
collective data, YES, 
collective meta, YES, 
subfiling, NO, 
total compute time, 80.000, seconds
total size, 40.000, GB
raw time, 60.167, seconds
raw rate, 680.775, MB/s
metadata time, 0.022, seconds
observed rate, 679.188, MB/s
observed time, 140.307, seconds
timestep 0 data time, 12.248, seconds
timestep 1 data time, 12.020, seconds
timestep 2 data time, 11.987, seconds
timestep 3 data time, 12.006, seconds
timestep 4 data time, 11.907, seconds
timestep 0 metadata time, 0.017, seconds
timestep 1 metadata time, 0.002, seconds
timestep 2 metadata time, 0.002, seconds
timestep 3 metadata time, 0.001, seconds
timestep 4 metadata time, 0.001, seconds

Read benchmark:

metric, value, unit
operation, read, 
ranks, 32, 
collective data, YES, 
collective meta, YES, 
subfiling, NO, 
total compute time, 80.000, seconds
total size, 40.000, GB
raw time, 27.535, seconds
raw rate, 1.452, GB/s
metadata time, 0.049, seconds
observed rate, 1.449, GB/s
observed time, 107.589, seconds
timestep 0 data time, 6.290, seconds
timestep 1 data time, 5.490, seconds
timestep 2 data time, 5.459, seconds
timestep 3 data time, 5.086, seconds
timestep 4 data time, 5.210, seconds
timestep 0 metadata time, 0.038, seconds
timestep 1 metadata time, 0.002, seconds
timestep 2 metadata time, 0.003, seconds
timestep 3 metadata time, 0.003, seconds
timestep 4 metadata time, 0.002, seconds

@TheAssembler1 TheAssembler1 marked this pull request as draft October 24, 2025 19:01
@TheAssembler1 TheAssembler1 force-pushed the read_write_per_timestep_timing branch from 4b96a79 to ba3edb1 Compare October 27, 2025 20:58
@TheAssembler1 TheAssembler1 changed the title Draft: Read write per timestep timing Read write per timestep timing Oct 27, 2025
@TheAssembler1 TheAssembler1 marked this pull request as ready for review October 27, 2025 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@TheAssembler1