About speed of Rapid OCR when config cls_batch_num, rec_batch_num

[Iambestfeed]

I using RapidOCR openvino
When a config cls_ batch_num and rec_batch_num on rapidocr_openvinoconfig.yaml like that:

Cls:
    use_cuda: false

    model_path: models/ch_ppocr_mobile_v2.0_cls_infer.onnx

    cls_image_shape: [3, 48, 192]
    cls_batch_num: 12
    cls_thresh: 0.9
    label_list: ['0', '180']

Rec:
    use_cuda: false

    model_path: models/ch_PP-OCRv4_rec_infer.onnx

    rec_img_shape: [3, 48, 320]
    rec_batch_num: 12

Speed when infer like that:

Det time: 457.0133686065674 ms
Cls time: 26.304006576538086 ms
Rec time: 307.06143379211426 ms
Image MapleToursInvoice.png : 829.031005859375 ms
Det time: 113.49296569824219 ms
Cls time: 42.07015037536621 ms
Rec time: 470.7663059234619 ms
Image 412332467_1308191803226563_4387928075065407285_n.png : 636.244873046875 ms
Det time: 1420.362949371338 ms
Cls time: 45.60732841491699 ms
Rec time: 744.2312240600586 ms
Image Invoice-IDESS.jpg : 2298.78662109375 ms
Det time: 576.7419338226318 ms
Cls time: 50.64249038696289 ms
Rec time: 659.1346263885498 ms
Image 407243551_928739715470039_2440639408020678642_n.jpg : 1331.058349609375 ms
Det time: 1421.5641021728516 ms
Cls time: 40.178537368774414 ms
Rec time: 500.38623809814453 ms
Image A242-910-23.jpg : 2044.842529296875 ms

But when a config cls_ batch_num and rec_batch_num on rapidocr_openvinoconfig.yaml like that:

Cls:
    use_cuda: false

    model_path: models/ch_ppocr_mobile_v2.0_cls_infer.onnx

    cls_image_shape: [3, 48, 192]
    cls_batch_num: 6
    cls_thresh: 0.9
    label_list: ['0', '180']

Rec:
    use_cuda: false

    model_path: models/ch_PP-OCRv4_rec_infer.onnx

    rec_img_shape: [3, 48, 320]
    rec_batch_num: 6

Speed when infer like that:

Det time: 460.5863094329834 ms
Cls time: 29.515743255615234 ms
Rec time: 166.76664352416992 ms
Image MapleToursInvoice.png : 696.150146484375 ms
Det time: 112.9906177520752 ms
Cls time: 50.59456825256348 ms
Rec time: 317.4917697906494 ms
Image 412332467_1308191803226563_4387928075065407285_n.png : 490.988037109375 ms
Det time: 1406.1636924743652 ms
Cls time: 54.07381057739258 ms
Rec time: 508.3744525909424 ms
Image Invoice-IDESS.jpg : 2053.3955078125 ms
Det time: 536.2746715545654 ms
Cls time: 56.66613578796387 ms
Rec time: 479.9783229827881 ms
Image 407243551_928739715470039_2440639408020678642_n.jpg : 1114.74365234375 ms
Det time: 1409.6944332122803 ms
Cls time: 49.158573150634766 ms
Rec time: 379.0264129638672 ms
Image A242-910-23.jpg : 1921.259521484375 ms

So will cls_batch_num and rec_batch_num make the infer speed slower?

[SWHL]

When the batch is increased, machine resources may become insufficient, leading to a decrease in inference speed.

It is not recommended to change the number of batches.

[Iambestfeed]

But my server has 48 threads and A5000 GPU 🥲. GPU is only 30% used and I think I have enough resources for OCR..

[SWHL]

😅If you want to inference faster by GPU, I suggest to use rapidocr_paddle package.

[Iambestfeed]

Hmm, I've had a few bad experiences with paddles but I think I'll probably try again. But I still hope to deploy with openvino because I feel there is potential in implementing with this library.

[SWHL]

Openvino'GPU inference is only valid for its own GPU.
I don't have an Intel GPU device on hand, so I haven't experimented with GPU inference effects.

About paddlepaddle, you can install the package according to the docs
Then, pip install rapidocr_paddle, and use it like this:

import cv2

from rapidocr_paddle import RapidOCR

engine = RapidOCR()
image_path = "tests/test_files/ch_en_num.jpg"
result, elapse_list = engine(image_path)

print(result)
print(elapse_list)

See details for docs

[Iambestfeed]

I tested it on colab and everything is quite ok but I think there will be many problems when deploying on our local.
Thanks.

[SWHL]

If you have specific questions, welcome to discuss and exchange.