Skip to content

Commit 0944257

Browse files
committed
Optimizes Blosc2 MLArray data loading
Refactors the in-memory data loading process for Blosc2 MLArrays from disk. Previously, data was loaded via an intermediate cframe conversion. Now, an empty Blosc2 array is created in memory, and data chunks along with metadata are explicitly copied. This ensures a more direct and controlled preparation of the array in memory, better supporting optimizations for subsequent operations.
1 parent 6a797da commit 0944257

1 file changed

Lines changed: 13 additions & 2 deletions

File tree

mlarray/mlarray.py

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1540,8 +1540,19 @@ def _load(
15401540
self.support_metadata = True
15411541
dparams = MLArray._resolve_dparams(dparams)
15421542
ondisk = blosc2.open(str(filepath), dparams=dparams, mode="r")
1543-
cframe = ondisk.to_cframe()
1544-
self._store = blosc2.ndarray_from_cframe(cframe, copy=True)
1543+
in_memory = blosc2.empty(
1544+
ondisk.shape,
1545+
dtype=ondisk.dtype,
1546+
chunks=ondisk.chunks,
1547+
blocks=ondisk.blocks,
1548+
cparams=ondisk.schunk.cparams,
1549+
dparams=ondisk.schunk.dparams,
1550+
)
1551+
for i in range(ondisk.schunk.nchunks):
1552+
in_memory.schunk.update_chunk(i, ondisk.schunk.get_chunk(i))
1553+
for key, value in ondisk.schunk.vlmeta.getall().items():
1554+
in_memory.schunk.vlmeta[key] = value
1555+
self._store = in_memory
15451556
self._backend = "blosc2"
15461557
self.mode = None
15471558
self.mmap_mode = None

0 commit comments

Comments
 (0)