Dynamic Batching Export

[Ceachi]

Can we export, the trained model to ONNX or TensorRT to support dynamic batching?

[SkalskiP]

Hi @Ceachi 👋🏻 Thanks for your interest in RF-DETR! Have you tried exporting using the code from this PR? #79

[valavanisleonidas]

@SkalskiP I tried this PR model and it seems to be working. When i try to convert from the dynamic onnx to TRT model though I get

 Failed to parse ONNX model:
In node 211 with name: /backbone/backbone.0/projector/stages_sampling.0.0/stages_sampling.0.0.0/ConvTranspose and operator: ConvTranspose (parseNode): INVALID_NODE: Invalid Node - /backbone/backbone.0/projector/stages_sampling.0.0/stages_sampling.0.0.0/ConvTranspose
ITensor::getDimensions: Error Code 4: API Usage Error (/backbone/backbone.0/projector/stages_sampling.0.0/stages_sampling.0.0.0/ConvTranspose: IDeconvolutionLayer number of channels in `input` tensor must not be dynamic.)
ITensor::getDimensions: Error Code 4: API Usage Error (Output shape can not be computed for node /backbone/backbone.0/projector/stages_sampling.0.0/stages_sampling.0.0.0/ConvTranspose.)

I guess TensorRT does NOT support ConvTranspose unless both input shape and output channels are fully static. but not sure how to fix that. any ideas ?

[zhouzq-thu]

I converted the model to ONNX with dynamic batch size and image size, see https://github.com/zhouzq-thu/DETR-ONNX.

[Borda]

Consolidating TensorRT/ONNX dynamic batching discussions into #376. PR #853 is actively developing TensorRT engine inference support.

For immediate use: .export() for ONNX, then trtexec for TRT conversion. TensorRT export documentation was added in v1.5.0.

Closing — please follow #376 for updates. Feel free to reopen if your specific use case isn't covered!