Memory Leak in Dataset Creation

[stormmathisen]

The following code will leak about 120 MB. Changing GROUPS to 400000 leaks 1.2 GB. heaptrack suggests the problem lies in creating the plists for the dataset. I've tried calling the underlying hdf5 close functions on the file, group, dataset and plist without solving it, as well as trying to call the underlying hdf5 garbage collection function.

use hdf5;

const PER_GROUP: u64 = 10;
const GROUPS: u64 = 40000;


fn main() {
    let fake_data: [u16; 512] = [255; 512];
    //Create file
    let hdffile = hdf5::File::create("AFile.h5").unwrap();
    
        
    for i in 0..GROUPS
    {
        let groupname = format!("{}", i);
        //Create group
        let hdfgroup = hdffile.create_group(&groupname).unwrap();
        

        for i in 0..PER_GROUP {
            //Write PER_GROUP 8kb arrays as individual datasets
            let ds_name = format!("{}", i);

            hdfgroup
                .new_dataset_builder()
                .with_data(&fake_data)
                .create(&*ds_name)
                .unwrap();

            /*heaptrack will show that the call to create will cause the majority
            of memory leaks, when it creates the plist for access properties*/
        }
    }
    //Close the HDF file
    hdffile.close().unwrap();
    println!("Finished!");
    loop {
        //Hang about until quit.
        //Memory useage will remain high, would expect dropping the file to stop that
        std::hint::spin_loop();
    }
}

[mulimoen]

This is with the most recent hdf5? I was hoping we had fixed this with #139

[stormmathisen]

It was with the current release (1.12.2) built from the source available on the website. I haven't tried with 1.13.2 from the git repository

[smutch]

I just wanted to add that I think I am also experiencing this issue with

• cargo 1.65.0-nightly (9809f8ff3 2022-08-16)
• hdf5(-rust)=0.8.1
• hdf5=1.12.2

Happy to help debug if that would be at all useful.

[mulimoen]

I have managed to track the leak down to H5Pget_class:

#include <H5public.h>
#include <H5Ppublic.h>
#include <assert.h>
#include <stdio.h>


#define ITERATIONS 4000000

int main() {
    herr_t status;

    status = H5open();
    assert(!status);

    hid_t dataset_access = H5Pcreate(H5P_DATASET_ACCESS);
    assert(dataset_access > 0);

    for (int i=0; i<ITERATIONS; i++) {
        status = H5Pget_class(dataset_access);
        // printf("%d\n", status);
        assert(status > 0);
    }

    printf("Finished\n");

    while(true);

    status = H5Pclose(dataset_access);
    assert(!status);

    status = H5close();
    assert(!status);

    return 0;
}