How can we optimize models to speed up session creation and cuda memory allocation for reducing initial load time and saving some time in overall evaluation computation ?

[AnuragYadav171]

[saikumarchalla]

@AnuragYadav171 Could you please elaborate your query so that we can suggest you in a better way. Thanks!

[AnuragYadav171]

@saikumarchalla , Thank you for your response !

Basically I'm trying to use Tensorflow Object Detection API as web based, where I will have an api which will trigger training and and another api for evaluation.

So for training it is running good for 3-4 hours as per the GPU capability!

But for Evaluation when we trigger that API then the trained model takes time for loading the model and creating sessions and cuda memory allocation then running actual inference on the image. This takes at least 30 seconds for the evaluation API to end.

The above step is again repeated for another evaluation request from the API.

When we have a video to run the inference on then it gives at least 16fps as per the GPU after taking initial time for loading the model and creating sessions and cuda memory allocation and then running actual inference on the video frames.

What I want to do is somehow save time for loading the model and creating sessions and cuda memory allocation for every API call on the image evaluation so that the overall time for an image evaluation API can be reduced to a good amount as running inference on image/frame takes just 1/16th second as per 16fps for this model.

[AnuragYadav171]

Hi,
I hope the description was giving proper insight to the opened issue.