How can SAHI do faster inference?

[rusvagzur]

Hi @fcakyon ,

I am currently using SAHI with YOLOv8. I am not completely sure that SAHI runs batch inference, since I have observed in the code that only num_batch=1 is supported and that inferences can take up to seconds per complete image depending on the number of slices.

I was wondering how could this package be improved in order to run batch inference in the GPU. Probably also doing the slices in the GPU directly when the original image is loaded.

Is batch inference dependant on YOLOv8? or is SAHI creating the bottleneck? Several seconds for the prediction of a single image doesn't justify higher small object detection for some real-world/real-time applications like security applications.

Best regards

[andressrodrl]

@rusvagzur

I´ve modified it to run inference with slice batches: https://github.com/andresinsitu/sahi_custom/tree/batch_inf
Added the argument batch to the predict and get_sliced_prediction function. It is working for me speeding it a bit.

At the moment the batch has to be a divisor of the number of slices, otherwise it leaves some slice without inference(but it works regardless)

[bshakya77]

Hello @dceluis,

Sure, I will take a look at it once.

Thank you.

Regards,
Bijay

[bshakya77]

Hello @dceluis,

I am getting this error when I try to run your code.

from sahi_batched.models import Yolov8DetectionModel
from sahi_batched import get_sliced_prediction_batched

I searched for that file in the original sahi repository. But, I could not find it. Could you please clarify the issue?

Thank you.

Regards,
Bijay

[dceluis]

Make sure SAHI is also installed. Current sahi versions may not be compatible, so it's possible you would have to lock to an older version of SAHI or tweak the batched library to be compatible with current versions.

[bshakya77]

@dceluis,

When I reviewed your code and ran the prediction using the default and your get_prediction_batched() method, it does reduce the inference time. But, when I printed the shift amount, it uses the default value [[0,0]]. I would like to know how it is performing the batch processing of the image.

I appreciate your feedback.

Regards,
Bijay

[dceluis]

@bshakya77

Hi Bijay,

If I recall correctly, I’m also optimizing by skipping some PIL image conversions (e.g., using PseudoPIL), so you might still see improvements even if the image is too small to slice (as indicated by the [[0,0]] shift values).