we run screen-driven agents against web forms in production and the failure mode that took us longest to find wasn't navigation, it was commits that don't commit. a react controlled select can render the right value after a click while the framework's internal state never updated, so every pixel says done and the submitted payload says null. vision-only verification passes because the screen genuinely looks correct.
curious how you handle that class without DOM access. screenshot-after-action catches missing UI feedback, but when the UI itself is lying about form state the only reliable tells we found were downstream: the confirmation page, an outbound request, an email arriving. do your verification events ever consume anything besides pixels, or do you lean on the human approval gates for the risky commits?
Great observation on the "commits that don't commit" failure mode. We ran into the exact same class of bug building document extraction for regulated financial workflows — turns out vision-only verification has a blind spot that shows up whether you're parsing a screen or a PDF.
In tax preparation, documents have a lot of "the screen lies" equivalents: a PDF renderer can display perfectly aligned columns while the underlying text layer has misaligned fields. We learned this the hard way when an OCR pipeline extracted "123 Main St" beautifully from a W-2 image, but the XML metadata had a completely different address. The screen looked right, the data was wrong.
Our approach was a second verification layer structurally independent from extraction: deterministic rules that reconcile AI output against the source document's known structure. For a W-2, that means cross-walking every box number against the IRS's published schema and flagging deviations — same principle as your "downstream tells" but formalized into a rules engine.
The key insight: in regulated domains, the verification layer must consume something structurally different from what extraction consumed. If both use the same pixel-based representation, they share the same failure modes. Using DOM state, API call logs, or document metadata schemas as the verification source catches the cases where "pixel says done, payload says null."
Defining explicit "verification invariants" (similar to Coasty's approach) has been the most practical defense — things like "total across all line items must equal reported total" or "taxpayer name must match across all documents in the return." Simple arithmetic checks that don't need AI but catch the expensive failure modes that vision alone misses.
Curious how Coasty handles multi-source reconciliation — e.g., when the data on screen needs to be verified against a separate document (like a PDF guidance doc) rather than just against itself?
Yep, this is a real failure mode, and screenshot-after-action alone does not solve it. A field can look populated while the application never commits the underlying value.
We try to define verification around the actual outcome of the task rather than the immediately preceding UI state. Depending on the workflow, that can mean checking the confirmation page, reopening the submitted record, verifying a status change elsewhere in the application, matching a generated reference ID, or confirming that a downstream artifact was created.
We can also verify against separate inputs such as a PDF, CSV, or another screen for example, confirming that the values entered into a portal match the source document and then checking the resulting record after submission.
Today, our core agent is screen-driven, so we do not claim that pixels can prove every hidden state transition. For high-consequence commits, the workflow should use explicit downstream invariants and, where those are unavailable, a human approval or review step. Longer term, we think verification should be able to consume whatever independent evidence the environment exposes, including network or application-level signals, rather than treating vision as the only source of truth but it will be quite slower.
The invariant we use is the downstream outcome, not the visual state. For a controlled select, that can be the request payload, confirmation state, or persisted record. I’m building AnnotateQA around capturing a real browser failure and creating that failing Playwright reproduction before any code changes, with a candidate PR left for human review. If you have a recent case, I’d be glad to try one.
curious how you handle that class without DOM access. screenshot-after-action catches missing UI feedback, but when the UI itself is lying about form state the only reliable tells we found were downstream: the confirmation page, an outbound request, an email arriving. do your verification events ever consume anything besides pixels, or do you lean on the human approval gates for the risky commits?