Screenshot recognition failed to capture the right coordinates

[leo6211]

Describe the bug
The image recongnition always return the wrong coordinates.

Configuration (please complete the following information):

• Using Claude code (I also find that claude desktop can not find the available devices))
• OS: Macos 15.5
• Device used: Android 16.0 arm64 API 36.0 in Android studio

To Reproduce
Steps to reproduce the behavior:

1. Open the device
2. Write a random prompt, my prompt is "go to the settings, and remove the animation in the accessibility setting"
3. The image recognition are always the first choice to capture the screen information, and always return the wrong coordinates. Then after two or three failed tries, it turns to the GUI hierarchy based method and find the correct coordinates.

Expected behavior
The image recognition can functions well.

I guess this may be related to the environment

My conversation in Claude code:

click 'storage' in the screen

⏺ I can see the "Storage" option at the bottom of the screen. Let me click
  on it for you.

⏺ mobile - mobile_click_on_screen_at_coordinates (MCP)(x: 353, y: 1550)
  ⎿  Clicked on screen at coordinates: 353, 1550

⏺ I've clicked on the "Storage" option at the bottom of the settings
  screen. This should take you to the storage settings where you can manage
   device storage, see storage usage, and configure storage-related
  options.

> use hierarchy method to capture the screen and click the 'storeage' 
  buttion

⏺ I'll use the hierarchy method to capture the screen elements and then
  click on the Storage button.

⏺ mobile - mobile_list_elements_on_screen (MCP)(noParams: {})
  ⎿ Found these elements on screen: [{"type":"android.widget.TextView","tex
    t":"Search Settings","label":"","identifier":"com.android.settings:id/s
    earch_bar_title","coordinates":{"x":189,"y":143,"width":389,"height":71
    … +44 lines (ctrl+r to expand)




⏺ Perfect! I can see the Storage element in the hierarchy. Now I'll click
  on the Storage button using its coordinates.

⏺ mobile - mobile_click_on_screen_at_coordinates (MCP)(x: 210, y: 1690)
  ⎿  Clicked on screen at coordinates: 210, 1690

⏺ I've successfully clicked on the "Storage" button using the hierarchy
  method. The Storage element shows "35% used - 5.20 GB free" and I clicked
   on its title coordinates at (210, 1690). This should take you to the
  storage settings page.

I tries two kinds of screen size (10802400 / 10801920), none of them works with the image-based way

[krismuhi]

it should use the accessibility tree by default. great report

[lezhi12]

in my opinion only screenshots can't get coordinator, but screenshots +screensize can do it. Looking forward to your ending plan.

[kihaki]

For me it helped when I told claude that the screenshot coordinates he receives might be misaligned and he should check the device screensize and figure it out - he mostly got it then. Device accessibility tree works much better though, still. Click coordinates from screenshots are not aligned properly somehow - my guess is that screenshots are somehow of a different size than the actual device.

[DavidEure]

Same problem here, I need to instruct Claude Code to use mobile_list_elements_on_screen before any click action

[Marcwab]

Confirming this issue, the root cause is a coordinate space mismatch between screenshot image pixels and actual device pixels.

I reproduced this on an Android 16 emulator (Medium Phone, 1080×2400) using Claude.ai with mobile-mcp.

When the LLM takes a screenshot via 'mobile_take_screenshot', the returned image is scaled down (in my case to roughly 480×1065 pixels). However, 'mobile_click_on_screen_at_coordinates' expects coordinates in the real device pixel space (1080×2400). The LLM doesn't know about this scaling factor, so it estimates tap coordinates from the visual screenshot which consistently land in the wrong area.

For example, a Google cookie consent button visually located at ~y=866 in the screenshot image actually corresponds to ~y=1950 in device coordinates. Every tap missed until I switched to 'mobile_list_elements_on_screen', which returns real device coordinates.

I recommend 'list_elements_on_screen' as the primary method for click targeting and use screenshots as a visual-only tool for understanding page state, not for deriving tap coordinates.

[gmegidish]

@Marcwab that's true. List elements on screen should be the way to go. It's also suggested in the mcp tool itself when you take a screenshot.

The LLM not knowing of the scaling factor is a bug. I will get to fix it this weekend and add a test. We should have added a scaling factor to the device info and/or to the screenshot response itself (if it's base64).

Thank you so much for adding some light to this issue in Android. This is not an issue with iOS, because in iOS we definitely put the scaling value in device info to equal to the retina factor (1, 2, or 3).

You rock!

[Marcwab]

Thank you @gmegidish for the quick response! Looking forward to the weekend fix. You all rock, big fan of this MCP!