YOLOv8 letterbox edge detections can disappear when bbox slightly crosses active padded image boundary

[Moizcode]

Hi Marcos,

First, thank you for maintaining this repository. We are using marcoslucianops/DeepStream-Yolo with a YOLOv8 model and found a reproducible edge case around letterboxed inputs.

Environment:

• DeepStream: 7.1
• CUDA: 12.6
• TensorRT: 10.x
• Model: YOLOv8 exported using the repo YOLOv8 export flow
• Network input: 640x640
• nvstreammux output: 1920x1080
• Config:
  • maintain-aspect-ratio=1
  • symmetric-padding=1
  • cluster-mode=2
  • pre-cluster-threshold=0.25
  • nms-iou-threshold=0.45

Problem:

The YOLO output tensor contains a valid detection, and PyTorch / ONNXRuntime / standalone TensorRT all detect it, but DeepStream object metadata did not include it.

For a 1920x1080 frame letterboxed into 640x640, the real image area is:

resized image: 640x360
top padding: 140
active image y range: 140..500

One missed YOLOv8 candidate looked like this in network coordinates:

class_id: 0
confidence: 0.954179
bbox: [308.406, 376.330, 463.912, 500.436]

Projected back to the mux frame, this is a valid person detection:

[925.219, 708.990, 1391.735, 1080.0]

The issue seems to happen because y2=500.436 is slightly outside the active letterboxed image bottom, which is 500, but still inside the full network height, which is 640.

In the current parser, boxes are clipped to the full network bounds:

x1 = clamp(x1, 0, netW);
y1 = clamp(y1, 0, netH);
x2 = clamp(x2, 0, netW);
y2 = clamp(y2, 0, netH);

For letterboxed input, this keeps y2=500.436 unchanged because it is valid inside 0..640. However, the actual image content ends at y=500. In our pipeline, this caused DeepStream to miss the object in NvDsObjectMeta.

What fixed it locally:

We modified the parser to clip boxes to the active letterboxed image area instead of only clipping to 0..networkInfo.width/height.

For 1920x1080 -> 640x640, we clamp to:

x: 0..640
y: 140..500

After clipping y2=500.436 to 500.0, DeepStream started returning the detection correctly in both raw and tracked metadata.

Before fix:

DeepStream raw YOLO detections missed the seated person.

After fix:

DeepStream raw YOLO detections include:

{
"class_id": 0,
"confidence": 0.9541794061660767,
"bbox_xyxy": [925.219482421875, 708.989990234375, 1391.7350463867188, 1080.0000305175781]
}

Question / suggestion:

Would it make sense for the parser or postprocess path to account for the active letterboxed image bounds when maintain-aspect-ratio=1 and symmetric-padding=1 are used?

The parser currently receives networkInfo.width and networkInfo.height, but for letterboxed inference the valid image region may be smaller than the full network canvas. Objects touching the image edge can produce coordinates slightly outside the active image region, such as y2=500.436 when the real image bottom is y=500.

The model output itself is correct. The issue appears to happen after inference, during DeepStream parsing/post-processing/object metadata creation.

I may be missing a preferred DeepStream-side way to handle this, so I wanted to report the reproducible behavior and the local fix.