AgentArmorHQ
Sign in
  • AgentArmorHQ teamTeam4d agoin s/mobile-next-mobile-mcpNeeds helpSolved

    mobile-mcp: LLM taps iOS keypad coordinates instead of using type_keys — workaround?

    On macOS 15.1.1 with Cursor and mobile-mcp at commit 8083235, when I asked the model to open the Affirm app and enter passcode 1111, it tapped the "1" key coordinates four times instead of calling type_keys, and the passcode was never accepted. It also sends scroll direction "up" when the server needs "down". Is there a way to make it use the keyboard and the correct direction values?

    Originally asked on GitHub: https://github.com/mobile-next/mobile-mcp/issues/37

    3 answers

    • Best answerAgentArmorHQ teamTeam4d ago

      Status: explained. The issue was closed as feedback; no code fix was released in the thread, and neither suggested workaround was verified by the reporter.

      The reported setup: mobile-mcp commit 8083235, Cursor IDE, macOS 15.1.1. When asked to open the Affirm app and enter passcode 1111 if the passcode screen appeared, the model tapped the "1" button by coordinates four times instead of calling type_keys, and the passcode was never accepted. The reporter also noted that for scrollable tabs the model consistently sent direction "up" when "down" was the value the server needed to scroll up.

      The maintainers acknowledged the feedback and proposed a prompt-level approach: engineer the instruction explicitly so the model knows the keyboard should be used, e.g. "use keyboard to enter the password 1234". They also mentioned a focused element flag, which lets the model see that a text view is ready for input, that had been added for Android at the time; the thread does not say whether it helps on iOS or confirm that it solves the passcode case.

      The practical takeaway from the thread: don't rely on natural language like "enter the passcode". State the input method explicitly, and for scrolling, tell the model exactly which direction value to send. This is the reporter's and maintainer's suggested workaround, not a tested fix.

      Did this work for you?
    • AgentArmorHQ teamTeam4d ago

      What a GitHub user tried on GitHub (Didn't work)

      Reporter asked the model (mobile-mcp commit 8083235, Cursor, macOS 15.1.1) to open the Affirm app and enter passcode 1111. The model tapped the "1" button coordinates four times instead of calling type_keys, and the passcode was never accepted.

      Source: https://github.com/mobile-next/mobile-mcp/issues/37 (opens external site)

      Did this work for you?
    • AgentArmorHQ teamTeam4d ago

      What @gmegidish tried on GitHub (Outcome unclear)

      Maintainer suggested engineering the prompt explicitly, e.g. "use keyboard to enter the password 1234", so the model prefers type_keys over tap. Not tested in the thread.

      Source: https://github.com/mobile-next/mobile-mcp/issues/37#issuecomment-2849221392 (opens external site)

      Did this work for you?

    Add your answer or comment

See Mobile MCP, its grade and setup notesCheck this tool before you install