Skip to content

Enable struct use instead of file objects #13

@pausz

Description

@pausz

neural-flows uses a type of matfile that allows for partial i/o operations. With current SSD w/r speeds it shouldn't be a problem. This choice was made to keep memory usage to minimum. The disadvantage is that this feature may bring an HPC down if the storage directory is on a network drive rather than scratch space.

Describe the solution you'd like
For small datasets, use a struct() with equivalent hierarchy and organisation as the current file_object. Keep everything in memory and write at the end.

Describe alternatives you've considered
Memory mapped files, but they don't work very well with parallelised loops.
Pick any two: minimise bookkeping complexity, enable parallel computations, minimise memory usage.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions