This lesson's interactive features are locked, please to keep using them

Vision Models

A vision model can accept image input along with text. Instead of reading a semantic description of a page, it can inspect the screenshot's pixels.

OpenCode supports image attachments and passes image results from tools to models that can accept them. Not every model does, so be sure to check a model's capabilities before you pay for a bunch of tokens or a subscription.

Pixels sometimes expose problems that structured browser data simply can't:

Awkward spacing
Clipped or overlapping content
Broken responsive layouts
Charts, canvases, and other graphical output
A page that technically works but looks like a Craigslist ad from 2004

I actually find it refreshing when a page looks like 20-year-old basic HTML, but that's beside the point...

Pixels Are Bad Selectors

Images are great for judging the rendered result, but they're a ~~bad~~ slow and clunky way for an agent to actually interact with the page.

A vision model can estimate where a button sits and click those coordinates. That usually works, but the coordinates are brittle, and frankly, computer use is pretty slow compared to interacting with structured browser data. Let your agent use browser data to find and operate normal controls, then have it use image input to judge the rendered result.

The accessibility tree handles the clicks, and the vision model answers, "Does this page look right?" To be fair, most agents' idea of "right" is probably different from yours, but it should at least be able to notice and fix obvious visual problems.

Assignment

Fix the playground's output pane at narrow screen widths. The text should stay readable without making the page wider than the browser window.

Tell OpenCode to install [email protected] in .opencode with Bun, and to download the Chromium browser with bunx --bun playwright install chromium from that directory.

On Linux, if you run into an error where browser system libraries are missing, use bunx --bun playwright install --with-deps chromium.
Have OpenCode start the playground on port 8080 if it isn't running. Open http://localhost:8080, run hallucinate(“the slop must flow”)✨ to print some text, and resize the browser to about 375 pixels wide. Capture a screenshot showing the overflowing output pane.
Drag and drop the screenshot file into OpenCode, or give the agent the path to the screenshot.
Tell OpenCode to fix the layout in playground/index.html, keeping output readable at both narrow and desktop widths.
Reload the page and compare the result at both widths.

Tell OpenCode to run and submit the CLI tests from the project root.