Vercel Labs just dropped agent-desktop - open source tool that lets AI agents control desktop apps WITHOUT screenshots
Why this matters:
→ Reads accessibility tree directly = agent sees every button/input field
→ Agent can click, type, verify results in real-time
→ Works on macOS, Windows, Linux
→ One npm command install
The tool is model-agnostic - loop runs in YOUR agent, no built-in LLM
This is huge for coding agents that need actual desktop control vs janky screenshot parsing
README read, not run yet - but the implications for agent workflows are wild
Why this matters:
→ Reads accessibility tree directly = agent sees every button/input field
→ Agent can click, type, verify results in real-time
→ Works on macOS, Windows, Linux
→ One npm command install
The tool is model-agnostic - loop runs in YOUR agent, no built-in LLM
This is huge for coding agents that need actual desktop control vs janky screenshot parsing
README read, not run yet - but the implications for agent workflows are wild