This lesson's interactive features are locked, please to keep using them
Computer Use
Coding agents are happiest when they can use precise tools. Reading a file is easier with a read tool than by opening an editor and dragging a mouse over the text. Sometimes, though, the graphical user interface (GUI) is the thing that needs to be tested.
Computer use lets a model inspect a screen and operate graphical applications with actions like clicking, typing, scrolling, and switching windows. ChatGPT Computer Use and Claude's computer use build these controls directly into their desktop applications.
OpenCode doesn't currently include a general-purpose desktop controller. That said, it can still operate browsers through external tools, which is where most coding-related computer use happens anyway!
Feedback Loop
Computer use gives an agent another way to gather runtime evidence:
- Observe the current interface.
- Perform an action.
- Inspect what changed.
- Repeat until the behavior is correct.
That's the same feedback loop used with tests, logs, and compiler errors, but now the evidence is "closer to the user" because it's the same graphical interface they interact with.
I recommend using a direct API, command, or structured tool for feedback when one exists. Having a model click through a UI is slower and less reliable, but sometimes it's the only reasonable option! If you're making changes to the UI of a website, it's great that the model can check that the DOM looks good, but that's not much consolation if the rendered button is bleeding halfway outside its container.
Computer-use tools can read visible data, use active browser sessions, click destructive controls, and submit forms. It's like giving another person access to your mouse and keyboard, so be very careful about what permissions you allow – particularly to an all-encompassing multimodal agent like Claude Cowork or ChatGPT Desktop.