pyautogui-mcp-server

# MCP server for simplest GUI agent 🕹️🤖 `pyautogui_mcp_server` packages a Streamable HTTP MCP server for running Python code with `pyautogui` instrumentation. It is designed for GUI automation workflows where plain `pyautogui` execution is not enough. The package adds MCP-friendly output handling, and richer screenshots. ## ✨ What this package adds Compared with running raw `pyautogui` calls directly, this library adds extra effort in the following areas: - Shared Python globals across tool calls. - Captured `stdout`, `stderr`, and final expression results in one MCP response stream. - Inline screenshot delivery as MCP image content instead of requiring manual file handling. - Annotated mouse-operation previews that show the target point or path before the action runs. - Screenshot normalization so captured images line up better with logical screen coordinates. ## 🛠️ Tool response example ``` <stdout> Cut the right rope by dragging left to right through it. </stdout> <pyautogui-mcp.dragTo x=860 y=430 duration=0.2 button='left' time_offset="T+1.1s" pyautogui.size=(1440, 900)> ``` <img width="512" alt="cut-the-rope-by-gui-agent" src="https://yl-data.github.io/2506.onPanda/pyautogui_mcp_server_assets/image/2603.cut-the-rope.webp" /> ``` </pyautogui-mcp.dragTo> ``` ## 📦 Installation ```bash pip install pyautogui_mcp_server ``` For local development: ```bash pip install -e .[dev] ``` ## 🚀 Run the MCP server Use the module entrypoint: ```bash python -m pyautogui_mcp_server --host 127.0.0.1 --port 9300 ``` Or use the installed console script: ```bash pyautogui-mcp-server --port 9300 ``` Show CLI help: ```bash python -m pyautogui_mcp_server --help ``` The server exposes a `run_python_with_pyautogui` MCP tool that executes Python with instrumented `pyautogui` behavior. ## 📄 License MIT