-
Notifications
You must be signed in to change notification settings - Fork 78
Fix: Load structs with large number of fields #230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
src/MAT_HDF5.jl
Outdated
| end | ||
| write_attribute(g, "MATLAB_fields", HDF5.VLen(k)) | ||
| total_chars = sum(length, k) | ||
| if total_chars < 4096 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This code should probably be added to more struct-like writing? MatlabClassObject, MatlabOpaque and MatlabStructArray perhaps?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, I forgot about MatlabStructArray, it should be added there. I don't think it needs to be added for the others but we can do it to be on the safer side
|
A couple of changes to support any type of struct data:
The test is failing because for some reason the fields are going wildly out of order between save-load. Not sure what's going on here or how to resolve it. The data is intact though |
|
I've made isapprox testing possible, and made sure it doesn't care about name ordering in MatlabStructArray isapprox. |
|
Seems to be a quirk of HDF5 files. Keys are being written based on hash order of the Dict, but HDF5 stores/retrieves them lexicographically I guess |
Fixes #78 bug where structs with large number of fields could not be written in v7.3 format.
_write_field_reference!()to write field name attributes as a dataset and return HDF5 referencestruct_large.matStruct field names are written as attributes in v7.3 format. However, HDF5 has a maximum allowable attribute size of 64KB. When the number of fields is large, MATLAB instead writes a reference to a dataset, and writes the field names under
#refs#.The threshold for this write mechanism is when the total character length of all field names of the struct is >= 4096