Software Engineer, CUA Control
Simular · Palo Alto, CA · 1 mo ago
On-siteEngineeringFull-time
What You'll Do
- Work on the low-level computer control stack: mouse/keyboard injection, screen capture, coordinate mapping, input simulation
- Implement UI element detection using accessibility APIs (AXUIElement, UI Automation), DOM/a11y trees, and visual grounding
- Help build the abstraction layer that lets our agent operate across OS platforms and application types
- Tackle reliability problems: element targeting under UI changes, window occlusion, resolution scaling, cross-app focus management
- Contribute to feedback loops: how does the agent know its action worked? How does it recover when something unexpected happens?
- Work closely with the model and planning team on the interface between intent and execution
What You Might Have
- Built OS-level input automation (CGEvent, SendInput, xdotool, or similar)
- Understood accessibility frameworks - AXUIElement on macOS, UI Automation on Windows, AT-SPI on Linux
- Dealt with flaky element selectors, timing issues, resolution-dependent coordinates
- Thought carefully about reliability and edge cases
- Worked with tools like Playwright, Appium, PyAutoGUI, Hammerspoon, or similar
Bonus Experience
- With screen reader internals
- Remote desktop protocols (RDP/VNC)
- Game automation
- LLM agent tool-use systems
- Mobile device automation (iOS UIAutomation / XCTest, Android UIAutomator / Accessibility)