AI inference receives BGR instead of RGB (red/blue channels swapped for SAM / YOLO-World / SAM3)

[wkentaro]

This was generated by AI during a kaizen pass.

Summary

Every AI feature (AI-Points, AI-Box, AI Text-to-Annotation) feeds a BGR image to the underlying vision model, not the RGB the models expect. The red and blue channels are silently swapped for the whole inference pipeline (SAM, EfficientSAM, YOLO-World, SAM3).

Root cause

img_qt_to_arr (labelme/_utils/image.py:52-60) returns the QImage's raw memory. For the 32-bit formats Qt uses when loading a PNG/JPEG (Format_RGB32 / Format_ARGB32), that raw layout is BGRA on little-endian, because Qt packs each pixel as 0xAARRGGBB and little-endian byte order puts blue first. The two AI call sites then take [:, :, :3], which yields B, G, R and hands it to the model as if it were RGB.

Affected call sites:

• labelme/_widgets/canvas.py:412-414 (_shapes_from_ai_points, drives AI-Points and AI-Box)
• labelme/_app.py:1354 (_submit_ai_prompt -> get_bboxes_from_texts, drives AI Text-to-Annotation)

The image flows straight into osam (OsamSession._get_or_compute_embedding -> model.encode_image) and the text-detection path with no compensating channel swap anywhere in labelme/_automation/, so the swap reaches the model uncorrected.

Not affected: labelme/_widgets/canvas.py:1808 (load_pixmap) also calls img_qt_to_arr, but only to hash the bytes, so channel order is irrelevant there.

Reproduction (confirmed empirically)

Real load path QImage.fromData -> QPixmap.fromImage -> toImage() -> img_qt_to_arr, on examples/tutorial/apc2016_obj3.jpg, at a strongly-colored pixel:

QImage format      = Format_RGB32
PIL ground-truth RGB = [227  35  46]
img_qt_to_arr raw    = [ 46  35 227 255]   # BGRA
arr[:, :, :3]->model = [ 46  35 227]        # BGR, not RGB
matches RGB?           False
matches reversed BGR?  True

Why this needs a human call (not auto-shipped)

The diagnosis is unambiguous, but the fix is a behavioral change to the core AI pipeline, so it wants maintainer sign-off:

1. Blast radius: it silently changes the output of every AI inference for all users. Channel order is cheaply testable, but whether the corrected color actually improves real-world segmentation/detection quality is not provable by a cheap unit test.
2. Contract coordination: in-flight PR test(_utils/image): cover img_qt_to_arr pixel layout and scanline padding #2328 deliberately documents the BGRA layout as img_qt_to_arr's intended contract (characterization test test_img_qt_to_arr_returns_bgra_pixels). So the fix should go at the two callers, not in img_qt_to_arr. That intersection is worth a conscious decision.
3. Prior history: closed issue segment_anything.py中对图像做归一化的部分是不是顺序反了？ #1284 (2023) already asked whether the channel order into segment_anything was reversed. There may be context on why it was closed.

Options / tradeoffs

• (A) Fix at the two callers (recommended): convert to true RGB before extracting the array, e.g. .convertToFormat(QtGui.QImage.Format.Format_RGB888), then drop the now-redundant [:, :, :3]. Surgical, leaves img_qt_to_arr's BGRA contract (and PR test(_utils/image): cover img_qt_to_arr pixel layout and scanline padding #2328) untouched.
• (B) Add an RGB-returning helper (e.g. img_qt_to_rgb_arr) used by the AI call sites, keeping the raw img_qt_to_arr for the hashing caller. Slightly more code, clearer intent at the call site.
• (C) Change img_qt_to_arr to always return RGB: collides with PR test(_utils/image): cover img_qt_to_arr pixel layout and scanline padding #2328 and any other raw-layout caller; not recommended.

Verification

Cheap and additive: stub propose_shapes / get_bboxes_from_texts (existing tests already stub the AI path in tests/unit/widgets/canvas_test.py), load a pixmap of a known solid color, and assert the captured image kwarg's pixel equals true RGB rather than the swapped value.