Issue with Keras / function mae

[jmcdonal]

Hi,
I'm trying to help some users use the isonet.py program. We are running on a Rocky Linux 9 platform (RHEL linux 9 clone). The user is trying to run a refine job and we're seeing errors like the ones below. In a naive google search looking at the issue it seems that some of the loss functions need to be serialized unless I should be using a different package version (below is the full list of packages in the conda environment).
Please advise,
Jeff

$ isonet.py refine ./subtomo.star --gpuID 0 --preprocessing_ncpus 16 --iterations 4 --noise_start_iter 10,15,20,25 --noise_level 0.05,0.1,0.15,0.2
07-10 15:44:31, INFO
######Isonet starts refining######

07-10 15:44:34, INFO Start Iteration1!
07-10 15:44:34, WARNING The results folder already exists
The old results folder will be renamed (to results~)
/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument alpha is deprecated. Use negative_slope instead.
warnings.warn(
07-10 15:44:36, WARNING You are saving your model as an HDF5 file via model.save() or keras.saving.save_model(model). This file format is considered legacy. We recommend using instead the native Keras format, e.g. model.save('my_model.keras') or keras.saving.save_model(model, 'my_model.keras').
07-10 15:44:39, INFO Noise Level:0.0
07-10 15:45:22, INFO Done preparing subtomograms!
07-10 15:45:22, INFO Start training!
07-10 15:45:23, ERROR Traceback (most recent call last):
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/IsoNet/bin/refine.py", line 128, in run
history = train_data(args) #train based on init model and save new one as model_iter{num_iter}.h5
^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/IsoNet/models/unet/train.py", line 93, in train_data
history = train3D_continue('{}/model_iter{:0>2d}.h5'.format(settings.result_dir,settings.iter_count),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/IsoNet/models/unet/train.py", line 38, in train3D_continue
model = load_model( model_file)
^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/saving_api.py", line 189, in load_model
return legacy_h5_format.load_model_from_hdf5(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/legacy/saving/legacy_h5_format.py", line 155, in load_model_from_hdf5
**saving_utils.compile_args_from_training_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/legacy/saving/saving_utils.py", line 143, in compile_args_from_training_config
loss = _deserialize_nested_config(losses.deserialize, loss_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/legacy/saving/saving_utils.py", line 202, in _deserialize_nested_config
return deserialize_fn(config)
^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/losses/init.py", line 149, in deserialize
return serialization_lib.deserialize_keras_object(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/serialization_lib.py", line 575, in deserialize_keras_object
return deserialize_keras_object(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/serialization_lib.py", line 678, in deserialize_keras_object
return _retrieve_class_or_fn(
^^^^^^^^^^^^^^^^^^^^^^
File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/serialization_lib.py", line 812, in _retrieve_class_or_fn
raise TypeError(
TypeError: Could not locate function 'mae'. Make sure custom classes are decorated with @keras.saving.register_keras_serializable(). Full object config: {'module': 'keras.metrics', 'class_name': 'function', 'config': 'mae', 'registered_name': 'mae'}

---

$ conda list

packages in environment at /panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
absl-py 2.1.0 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
bzip2 1.0.8 hd590300_5 conda-forge
ca-certificates 2024.6.2 hbcca054_0 conda-forge
certifi 2024.6.2 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
fire 0.6.0 pypi_0 pypi
flatbuffers 24.3.25 pypi_0 pypi
gast 0.6.0 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.64.1 pypi_0 pypi
h5py 3.11.0 pypi_0 pypi
idna 3.7 pypi_0 pypi
imageio 2.34.2 pypi_0 pypi
keras 3.4.1 pypi_0 pypi
lazy-loader 0.4 pypi_0 pypi
ld_impl_linux-64 2.40 hf3520f5_7 conda-forge
libclang 18.1.1 pypi_0 pypi
libexpat 2.6.2 h59595ed_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 14.1.0 h77fa898_0 conda-forge
libgomp 14.1.0 h77fa898_0 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libsqlite 3.46.0 hde9e2c9_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.3.1 h4ab18f5_1 conda-forge
markdown 3.6 pypi_0 pypi
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
ml-dtypes 0.3.2 pypi_0 pypi
mrcfile 1.5.0 pypi_0 pypi
namex 0.0.8 pypi_0 pypi
ncurses 6.5 h59595ed_0 conda-forge
networkx 3.3 pypi_0 pypi
numpy 1.26.4 pypi_0 pypi
openssl 3.3.1 h4ab18f5_1 conda-forge
opt-einsum 3.3.0 pypi_0 pypi
optree 0.11.0 pypi_0 pypi
packaging 24.1 pypi_0 pypi
pillow 10.4.0 pypi_0 pypi
pip 24.0 pyhd8ed1ab_0 conda-forge
protobuf 4.25.3 pypi_0 pypi
pygments 2.18.0 pypi_0 pypi
pyqt5 5.15.10 pypi_0 pypi
pyqt5-qt5 5.15.14 pypi_0 pypi
pyqt5-sip 12.13.0 pypi_0 pypi
python 3.12.4 h194c7f8_0_cpython conda-forge
readline 8.2 h8228510_1 conda-forge
requests 2.32.3 pypi_0 pypi
rich 13.7.1 pypi_0 pypi
scikit-image 0.24.0 pypi_0 pypi
scipy 1.14.0 pypi_0 pypi
setuptools 70.1.1 pyhd8ed1ab_0 conda-forge
six 1.16.0 pypi_0 pypi
tensorboard 2.16.2 pypi_0 pypi
tensorboard-data-server 0.7.2 pypi_0 pypi
tensorflow 2.16.1 pypi_0 pypi
termcolor 2.4.0 pypi_0 pypi
tifffile 2024.6.18 pypi_0 pypi
tk 8.6.13 noxft_h4845f30_101 conda-forge
tqdm 4.66.4 pypi_0 pypi
typing-extensions 4.12.2 pypi_0 pypi
tzdata 2024a h0c530f3_0 conda-forge
urllib3 2.2.2 pypi_0 pypi
werkzeug 3.0.3 pypi_0 pypi
wheel 0.43.0 pyhd8ed1ab_1 conda-forge
wrapt 1.16.0 pypi_0 pypi
xz 5.2.6 h166bdaf_0 conda-forge

[procyontao]

Hi,

Thanks for reporting. I have been using tensorflow2.5 to test everything. If you can roll back to a older version, that would be great.

We are working on reproducing your issue and update IsoNet so that it can matches latest versions of dependencies.

[mpm896]

Hi,

I ran into this issue as well, as I'm running tensorflow2.17. Im able to apparently overcome it by modifying models/unet/train.py:

at top of file:
from tensorflow.keras.losses import MeanAbsoluteError

Lines 34 & 38:
model = load_model( model_file) --> model = load_model(model_file, custom_objects={ "mae": MeanAbsoluteError() })