-
Notifications
You must be signed in to change notification settings - Fork 13
Open
Labels
enhancementNew feature or requestNew feature or request
Description
neural-flows uses a type of matfile that allows for partial i/o operations. With current SSD w/r speeds it shouldn't be a problem. This choice was made to keep memory usage to minimum. The disadvantage is that this feature may bring an HPC down if the storage directory is on a network drive rather than scratch space.
Describe the solution you'd like
For small datasets, use a struct() with equivalent hierarchy and organisation as the current file_object. Keep everything in memory and write at the end.
Describe alternatives you've considered
Memory mapped files, but they don't work very well with parallelised loops.
Pick any two: minimise bookkeping complexity, enable parallel computations, minimise memory usage.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request