TF Object Detection API: Failed assertion during transfer learning: New volume:<val1> should match old volume: <val2>

[shreypareek1991]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/object_dection/model_lib_v2.py

2. Describe the bug

Trying to perform transfer learning using EfficiencentDetD0/D1/D2 models from this link.
I keep getting the following error:

I0515 15:47:52.399605 12364836864 api.py:459] feature_map_spatial_dims: [(64, 64), (32, 32), (16, 16), (8, 8), (4, 4)]
I0515 15:48:05.215337 12440334336 api.py:459] feature_map_spatial_dims: [(64, 64), (32, 32), (16, 16), (8, 8), (4, 4)]
/AppleInternal/Library/BuildRoots/9941690d-bcf7-11ed-a645-863efbbaf80d/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSNDArray/Kernels/MPSNDArrayIdentity.mm:795: failed assertion `New volume: 8192 should match old volume: 1048576 [reshapeWithCommandBuffer] MPSNDArrayIdentity.'

3. Steps to reproduce

Steps to reproduce the behavior.

python model_main_tf2.py
  --pipeline_config_path=<path to your config file>
  --model_dir=<path to a directory with your model>

4. Expected behavior

Code should proceed to start training.

5. Additional context

I0515 15:47:52.399605 12364836864 api.py:459] feature_map_spatial_dims: [(64, 64), (32, 32), (16, 16), (8, 8), (4, 4)]
I0515 15:48:05.215337 12440334336 api.py:459] feature_map_spatial_dims: [(64, 64), (32, 32), (16, 16), (8, 8), (4, 4)]
/AppleInternal/Library/BuildRoots/9941690d-bcf7-11ed-a645-863efbbaf80d/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSNDArray/Kernels/MPSNDArrayIdentity.mm:795: failed assertion `New volume: 8192 should match old volume: 1048576 [reshapeWithCommandBuffer] MPSNDArrayIdentity.'
Fatal Python error: Aborted

Thread 0x00000001f7439b40 (most recent call first):
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 52 in quick_execute
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/monomorphic_function.py", line 381 in call
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/monomorphic_function.py", line 1757 in _call_flat
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 143 in __call__
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 959 in _call
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 894 in __call__
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 150 in error_handler
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/distribute/mirrored_run.py", line 84 in call_for_each_replica
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/distribute/mirrored_strategy.py", line 696 in _call_for_each_replica
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py", line 2895 in call_for_each_replica
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py", line 1316 in run
  File "/Users/shreypareek/Desktop/Projects/dev/tf_object_detection/models/research/object_detection/model_lib_v2.py", line 176 in _ensure_model_is_built
  File "/Users/shreypareek/Desktop/Projects/dev/tf_object_detection/models/research/object_detection/model_lib_v2.py", line 401 in load_fine_tune_checkpoint
  File "/Users/shreypareek/Desktop/Projects/dev/tf_object_detection/models/research/object_detection/model_lib_v2.py", line 605 in train_loop
  File "src/train.py", line 115 in main
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/absl/app.py", line 254 in _run_main
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/absl/app.py", line 308 in run
  File "/Users/shreypareek/mambaforge/envs/tf_object_detection/lib/python3.8/site-packages/tensorflow/python/platform/app.py", line 36 in run
  File "src/train.py", line 124 in <module>

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

16 inch 2021 MacBook Pro M1
MacOS: Ventura 13.3.1 (22E261)
Python 3.8.16

• TensorFlow version (use command below):

tensorflow-deps           2.10.0                        0    apple
tensorflow-estimator      2.12.0                   pypi_0    pypi
tensorflow-io             0.32.0                   pypi_0    pypi
tensorflow-macos          2.12.0                   pypi_0    pypi
tensorflow-metal          0.8.0                    pypi_0    pypi

• CUDA/cuDNN version:
• GPU model and memory:
  • Apple M1 GPU. Metal 3

[wafiq1211]

I'm also encountering this issue. While running the model_builder_tf2_test.py works with GPU, model training only works when I uninstall tensorflow-metal and running using CPU only.

Did you manage to solve this issue?

[shreypareek1991]

Unfortunately no. I gave up on tf od and switched to keras-cv for the time being.
I will circle back to this in a few months.