ssd_mobilenet_v1_coco_2017_11_17 object detection running out of memory

[tinkerbeast]

Please go to Stack Overflow for help and support:

http://stackoverflow.com/questions/tagged/tensorflow

Also, please understand that many of the models included in this repository are experimental and research-style code. If you open a GitHub issue, here is our policy:

System information

• What is the top-level directory of the model you are using: ssd_mobilenet_v1_coco_2017_11_17
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Stock example (https://github.com/tensorflow/models/blob/master/research/object_detection/object_detection_tutorial.ipynb)
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): v2.0.0-9-g1cf0898dd4 2.0.0
• Bazel version (if compiling from source): NA
• CUDA/cuDNN version: 7.6.3
• GPU model and memory: GeForce RTX 2060, 5931 MiB
• Exact command to reproduce: Run https://github.com/tensorflow/models/blob/master/research/object_detection/object_detection_tutorial.ipynb

Describe the problem

I am running the standard example as mentioned above. However, it's running out of memory. I don't understand how a model which can be run on a mobile device can run out of memory with my GPU.

Note: I tried the issue with both config.gpu_options.allow_growth enabled, as well as disabled.

Source code / logs

2020-02-15 17:27:35.418822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5101 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:08:00.0, compute capability: 7.5)
2020-02-15 17:27:43.190194: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-15 17:27:44.590501: I tensorflow/stream_executor/cuda/cuda_driver.cc:830] failed to allocate 2.86G (3069050880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
[I 17:27:48.185 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
kernel 5d9a5941-82c1-4272-a225-6418c43eba41 restarted
[I 17:28:33.030 NotebookApp] Saving file at /workspace/misc_mobilenetssd_people.ipynb

[sglvladi]

@tinkerbeast even though your GPU has 6GB VRAM, it is possible that some other program may be using most of it, meaning that TensorFlow is unable to allocate the 2.86GB that it seems to be trying to do. When this happens, I would suggest opening the resource monitor to see how much of your VRAM is being used. Seeing as you are running on Ubuntu, you can try running nvidia-smi in a Terminal.