some questions about the MultiModalIterableDataset

[howitry]

Great open-source work! I have some questions about MultiModalIterableDataset. According to the code http::www.com, MultiModalIterableDataset is a sharded version of HFDataset. HFDataset actually reads the entire list of files at once. Wouldn't this lead to insufficient memory when reading a very large training set?

[kcz358]

Hi, that would be the case if you use a json or jsonl dataset which indeed reads all the files into memory. If you use parquet and arrow format, I think the backend logic for the hf dataset uses an on-the-fly logic to read the data so would be okay to iterate over large dataset.

[howitry]

Hi, that would be the case if you use a json or jsonl dataset which indeed reads all the files into memory. If you use parquet and arrow format, I think the backend logic for the hf dataset uses an on-the-fly logic to read the data so would be okay to iterate over large dataset.

It seems that Parquet format is not read on-the-fly. Example code is as follows:

import psutil
import timeit
from datasets import load_dataset
import os

file_type='parquet'
dataset_path="debug.parquet" #7.5G
kwargs = {'split': 'train', 'streaming':False, 'num_proc': None, 'cache_dir': None}
mem_before = psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024)
dataset = load_dataset(file_type,data_files=dataset_path, **kwargs)
mem_after = psutil.Process(os.getpid()).memory_info().rss / (1024 * 1024)
print(f"RAM memory used: {(mem_after - mem_before)} MB")
print(dataset)

The results are as follows:

Generating train split: 5548571 examples [01:04, 85470.87 examples/s] 
Loading dataset shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 41/41 [00:00<00:00, 1067.29it/s]
RAM memory used: 1019.8984375 MB
Dataset({
    features: ['key1', 'key2', 'key3', 'key4', 'key5'],
    num_rows: 5548571
})

Loading a 7.5GB Parquet file consumed approximately 1GB of RAM. With stream=False, due to RAM limitations, it seems impossible to load very large files, such as 500GB+.
RAM consumption is acceptable if streaming=True. However, IterableDataset does not support automatic sharding like in the code https://github.com/EvolvingLMMs-Lab/lmms-engine/blob/main/src/lmms_engine/datasets/iterable/multimodal_iterable_dataset.py#L139.

RAM memory used: 25.5546875 MB
IterableDataset({
    features: ['key1', 'key2', 'key3', 'key4', 'key5'],
    num_shards: 1
})

datasets Version: 4.4.1