Repository navigation
Let a Bot choose from a dropdown, and carry on after a browser step fails - #833
Merged
Merged
Conversation
The computer's /type filled every field with Playwright's fill, which a native <select> does not take: it waited until the action timed out, and the server reported the timeout as an element gone from the page. The Bot took a fresh snapshot and tried the same thing again, so a form with a dropdown (a CRM's Industry or Stage) could never be completed. enterText now picks the option whose label is the text, ignoring case and surrounding spaces, and failing that the one whose value is, which fires the change a person's choice does. Text that names no enabled option is refused with a 400 that names neither the text nor the page's options.
A native <select>'s options carry no refs, so the snapshot dropped them, and its value is the list of options rather than text, so the combobox came through with no value either. A Bot saw combobox "Industry" with nothing it could choose and no way to tell what it was set to. The element now carries its options (at most 50) and, as its value, the one marked selected. A custom listbox's options keep their own refs and stay elements of their own, listed once.
… tests Drives a real form through an AI snapshot and the aria-ref locator, as the computer does: a label and a value are chosen, an unknown or disabled option is refused without changing the field, a text box is still filled, and the snapshot lists the dropdown's options. The parser tests pin the captured snapshot of two selects and a custom listbox. The browser test joins the computer job's list, which is the only CI job that sets OPENBOT_COMPUTER_BROWSER.
A click, typed entry or key press goes through the approval gate, and when the gate answered with a failure the wrapper threw. CopilotKit starts no follow-up run after a frontend tool throws, so the Bot's turn ended on the spot: the step read "Didn't work" and the Bot never spoke again. The wrapper now answers the way callComputer already did for the steps that skip the gate, through the same failedComputerCall: a failed result with the reason, plus the refusal, stale-ref and person-has-control flags the model reads to choose its next step. A request that cannot reach the gate, or an answer with no result, is a failed result too.
Drives the handler the SDK keeps against a gate answering 409, 403, a request that cannot connect, and a 200 with no result. Each comes back as a failed result rather than a throw.
davidmckayv
requested review from
MikeRyanDev,
guidovizoso,
mxmzb and
tylerslaton
as code owners
October 23, 2026 23:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two bugs that stop a Bot partway through a form in its browser.
A Bot could not choose from a dropdown. The computer's
/typeused Playwright'sfillon every field, and a native<select>does not take it: the call waited until it timed out, and the server reported the timeout as an element gone from the page. The Bot snapshotted again and repeated itself, so a form with an Industry or Stage dropdown could never be saved. Typing into a dropdown now chooses the option by its label (ignoring case and surrounding spaces), then by its value. Text that names no enabled option gets a 400 saying so, without echoing the text or the page's options. Snapshots also listed a select as a barecomboboxwith no options and no value, because its options carry no refs; the element now carriesoptions(at most 50) and the selected option as itsvalue. Custom listboxes are unchanged: their options already have refs.A failed browser step ended the Bot's turn silently. Clicks, typed entries and key presses go through the approval gate, and when the gate answered with a failure, the wrapper in
approval-tools.tsxthrew. CopilotKit starts no follow-up run after a frontend tool throws, so the turn stopped with the step reading "Didn't work" and nothing more from the Bot. The wrapper now answers through the samefailedComputerCallthatcallComputeralready used for ungated steps: a failed result with the reason and the refusal, stale-ref or person-has-control flags. A gate that cannot be reached, or a 200 with no result, is a failed result too.Found running a Bot against a CRM form with no API: the first attempt stopped at the Industry dropdown with no reply; with both fixes, the Bot saved five accounts and recovered from two failed steps on the way.
Tests. Each new test fails on
mainand passes here.agent-computer/tests/enter-text.test.tsdrives a real form through an AI snapshot and thearia-reflocator, as the computer does. It is added to thecomputer (real browser)job's list, the only job that setsOPENBOT_COMPUTER_BROWSER.agent-computer/tests/aria-snapshot.test.tspins Playwright's captured snapshot of two selects and a custom listbox.app/tests/approval-tools.test.tsxdrives the handler the SDK keeps against a gate answering 409, 403, no connection, and a 200 with no result.No migration, and the server, computer and app can be upgraded in any order: an older server ignores the new snapshot field.