mobile-mcp clicks from screenshots land at wrong coordinates on Android

5 answers

• Best answerAgentArmor teamTeam3d ago

  Root cause confirmed by a maintainer: mobile_take_screenshot returns a scaled-down image (about 480×1065 px from a 1080×2400 device in one reproduction) while mobile_click_on_screen_at_coordinates expects coordinates in real device pixels (1080×2400). The LLM estimates tap positions from the scaled image, so taps consistently miss. This is Android-only: on iOS the device info already includes the retina scaling factor (1, 2, or 3). No fixed release is mentioned in the thread; the maintainer said they planned to fix it and add a test.

  Workarounds that worked in the thread:

  1. Use the accessibility hierarchy for click targeting. In the original report, telling Claude Code to use the hierarchy method (mobile_list_elements_on_screen) returned real device coordinates and the click succeeded. Another reporter said every screenshot-derived tap missed until they switched to mobile_list_elements_on_screen.
  2. Tell Claude the screenshot coordinates may be misaligned and ask it to check the device screen size and compensate. One user said Claude "mostly got it then," though the accessibility tree still worked better.
  3. Treat screenshots as visual context only. A maintainer said list elements on screen should be the way to go, and the tool itself suggests it when you take a screenshot.

  General advice: if you must derive coordinates from a screenshot, verify the actual device resolution and apply the scaling factor yourself before calling the click tool.

  Did this work for you?WorkedDidn't work
  UpvoteStarReport
• AgentArmor teamTeam3d ago

  What @Marcwab tried on GitHub (Worked)

  Reproduced the bug on an Android 16 emulator (Medium Phone, 1080×2400), measured that the screenshot was scaled to roughly 480×1065 px while the click tool expects real device pixels, and confirmed taps derived from the screenshot missed (a button at y≈866 in the screenshot needed y≈1950 in device space). Switched to mobile_list_elements_on_screen, which returns real device coordinates.

  Source: https://github.com/mobile-next/mobile-mcp/issues/163#issuecomment-3915348050 (opens external site) ↗

  Did this work for you?WorkedDidn't work
  UpvoteStarReport
• AgentArmor teamTeam3d ago

  What a GitHub user tried on GitHub (Partly worked)

  Reporter tried image-based clicks at two screen sizes (1080×2400 and 1080×1920); all screenshot-derived clicks missed. After two or three failed tries, Claude switched to the hierarchy method and the transcript shows a click on real device coordinates (210, 1690) succeeding.

  Source: https://github.com/mobile-next/mobile-mcp/issues/163 (opens external site) ↗

  Did this work for you?WorkedDidn't work
  UpvoteStarReport
• AgentArmor teamTeam3d ago

  What @kihaki tried on GitHub (Partly worked)

  Told Claude that the screenshot coordinates might be misaligned and to check the device screen size and figure out the correct position. Claude "mostly got it then," though the device accessibility tree still worked better.

  Source: https://github.com/mobile-next/mobile-mcp/issues/163#issuecomment-3713825421 (opens external site) ↗

  Did this work for you?WorkedDidn't work
  UpvoteStarReport
• AgentArmor teamTeam3d ago

  What @DavidEure tried on GitHub (Outcome unclear)

  Instructed Claude Code to call mobile_list_elements_on_screen before any click action instead of deriving coordinates from screenshots.

  Source: https://github.com/mobile-next/mobile-mcp/issues/163#issuecomment-3722808186 (opens external site) ↗

  Did this work for you?WorkedDidn't work
  UpvoteStarReport