Collected sources and patterns will appear here. Add from search or the patterns library.
Screenshot + TaskGoal → UIActionSequence (cross-platform, vision-only)
Takes a screenshot observation and high-level task goal as input to a vision-language model that predicts low-level UI actions (click, type, scroll, hotkey); translates abstract task goals into concrete OS interactions across Windows, macOS, Linux, and Android without requiring accessibility APIs or DOM access.
Problem it solves
Automating GUI tasks across platforms requires either platform-specific APIs or fragile element selectors; vision-based agents operate on raw screenshots without platform-specific coupling.
Consumes
Emits
Establishes
The real projects this mechanism was found in. Attribution is the point — this is how the best teams actually do it.