Architecture

Why we read the screen with OCR instead of trusting selectors

2026-08-29 · The QA Reef team · 7 min read

Every browser-automation tool, ours included, starts from the same default: find an element with a selector — an id, a CSS path, an accessibility role and name, a test-id attribute someone added on purpose — and act on it. That default is correct. Playwright's locator API is fast, precise, and gives you a real handle you can assert against. When we generate a Playwright test from a recorded flow, that's exactly what we emit: page.getByRole('button', { name: 'Sign in' }), not a coordinate.

The problem isn't the destination. It's getting there the first time, on a page that doesn't cooperate.

The case selectors don't cover

A meaningful slice of the software worth testing has no stable selector to reach for yet: an internal tool where nobody added data-testid because it was built for five people in a Slack channel, a third-party embed you don't control the markup of, a canvas-rendered map or chart where the "button" is a region of pixels and not a DOM node at all, or a UI mid-redesign where class names churn every deploy. In all of these, a selector-first recorder either can't record anything useful, or records something so brittle it breaks the next time a developer touches an unrelated component.

A person doesn't have this problem. A person looks at the screen, reads the label, and clicks it. That's the capability we built the Operator around: look, decide, act — read the screen with OCR, ask a cheap model which visible label matches the goal, move the mouse there on a human-like path, and click.

What "read the screen" actually means

Concretely, a single full-page OCR pass misses things. Dense UIs — a map with a legend, a data table, a settings page with a dozen small controls — lose text at the edges of a naive one-shot OCR call, and small labels near busy backgrounds get skipped entirely. We tile the screenshot into overlapping crops, OCR each tile, and merge overlapping detections by IoU before handing the result to the model. In our own internal testing against a DOM ground truth on a dense, map-heavy screen, that tiling step was the difference between two lines of recovered text and thirty-four — the OCR engine underneath didn't change, only whether it got a fair look at the whole screen. Measured word recall across a range of busy screens landed between 86% and 100%, with a 94% mean.

That's on macOS, using Apple's on-device Vision framework — fast, free, no network round trip. Off macOS, the same interface is served by a vision-model call instead (we run a 27B-class model reachable via Ollama internally); the tool is meant to degrade gracefully rather than require a specific OS.

Deciding is a separate, cheap step

OCR gives you words with pixel boxes. Something still has to decide, from a goal like "sign in with {email}/{password}," which of those words to click next. We use a small, cheap text model for that (DeepSeek- or Qwen-class, via OpenRouter) — not a large multimodal model reasoning over raw pixels on every step, because the OCR pass already did the seeing. The model's job is narrower: given a goal and a list of visible labels, pick the next action. That keeps a full agent-driven mapping session inexpensive, and — because every one of those decisions gets logged with the exact model id and its cost (decided_by) — auditable after the fact. If a run clicked the wrong thing, you can see which model made that call and what it cost, not just that "the AI did it."

What this deliberately isn't

It's worth being precise about what a vision-driven operator is not, because "AI watches the screen" gets oversold elsewhere. It's not a continuously running model narrating every frame — a screenshot is taken on demand, once per step, the same "on demand, not continuous" pattern documented for at least one well-known closed platform's own coordinate-based agent runner (see the compare page for sourcing and specifics). It's not a replacement for DOM automation once a flow is known to work — the whole point of turning a mapping session into a generated Playwright test is that the test uses real locators, not OCR, so a passing test runs fast and doesn't re-pay the OCR/LLM cost on every CI run. And it's not self-healing in the sense of silently patching a broken selector at test time; if a generated locator breaks, that's a maintenance signal, not something to paper over with another vision call mid-test.

Where we'd still reach for a selector first

If your app already has stable roles, labels, or test ids, the operator will happily use them — OCR reads visible text, and an accessible label is visible text. The point isn't "vision instead of the DOM," it's "vision as the option of last resort, when the DOM hasn't given you anything durable yet." Most flows in a mature app should generate selector-based tests directly from a driven mapping session, no agent decision-making involved at all. The Operator exists for the other case: the page nobody's instrumented, the map, the third-party widget, the redesign in progress — the places selector-first tools currently can't record anything at all.