kubus/hatbot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Desktop: browser toolset next to the computer toolset #1

Merged
kubus merged 2 commits from t3code/browser-automation-use into main 2026-10-10 23:07:43 +00:00
Conversation 0 Commits 2 Files changed 11 +1703 -64
kubus commented 2026-10-10 23:07:28 +00:00
Owner
Copy link

Summary

  • Added browser_toolset_20260801 to desktop tasks alongside computer_toolset_20260801 for web page interactions
  • Browser toolset members include: navigate, read_page, find, form_input, left_click, get_page_text, screenshot, zoom, switch_tab, close_tab
  • Model reads pages as text via read_page and acts on elements by reference; takes screenshot only when needed for layout, images, or controls without refs
  • Added browserState() to report open tabs instead of always showing a screenshot at task start and after pauses
  • Browser element inspection checks policy before clicks/Enter, same as computer toolset
  • Halting: when a toolset call fails in a batch, remaining calls in that toolset don't execute
  • Removed Uptime Kuma confirmChecks waiting feature and kuma_waiting table; alerts start immediately
  • Updated PLAN.md (§5.5, §5.6, §6.1) with browser toolset flow and trust model

Testing

  • Browser navigation and page reading with filters (interactive, etc.)
  • Element location by reference and by viewport coordinate
  • Form input, tabs switching, and page scrolling
  • Browser screenshot capture to timeline
  • Policy check and approval for consequential browser clicks and Enter
  • Takeover and approval return to browser state instead of screenshot
  • Consequential click by element ref and coordinate
  • Failed toolset call halts remaining calls in that toolset this turn
  • Uptime Kuma alert without wait (confirmChecks removed)
  • Desktop task with browser and computer actions in one task
  • Not run: scheduling tests removed (confirm_checks references dropped)
## Summary - Added `browser_toolset_20260801` to desktop tasks alongside `computer_toolset_20260801` for web page interactions - Browser toolset members include: navigate, read_page, find, form_input, left_click, get_page_text, screenshot, zoom, switch_tab, close_tab - Model reads pages as text via `read_page` and acts on elements by reference; takes screenshot only when needed for layout, images, or controls without refs - Added `browserState()` to report open tabs instead of always showing a screenshot at task start and after pauses - Browser element inspection checks policy before clicks/Enter, same as computer toolset - Halting: when a toolset call fails in a batch, remaining calls in that toolset don't execute - Removed Uptime Kuma `confirmChecks` waiting feature and `kuma_waiting` table; alerts start immediately - Updated PLAN.md (§5.5, §5.6, §6.1) with browser toolset flow and trust model ## Testing - Browser navigation and page reading with filters (interactive, etc.) - Element location by reference and by viewport coordinate - Form input, tabs switching, and page scrolling - Browser screenshot capture to timeline - Policy check and approval for consequential browser clicks and Enter - Takeover and approval return to browser state instead of screenshot - Consequential click by element ref and coordinate - Failed toolset call halts remaining calls in that toolset this turn - Uptime Kuma alert without wait (`confirmChecks` removed) - Desktop task with browser and computer actions in one task - Not run: scheduling tests removed (confirm_checks references dropped)
The desktop-agent drives Chromium over CDP (/browser, /browser/state,
/browser/inspect): pages are read as text and acted on by element ref,
with screenshots only when needed. A task starts from the open tabs
instead of a screenshot. Browser clicks and Enter go through the same
consequential-step check. A task stops and asks when a consequential
step depends on a choice the task leaves open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kubus merged commit 162d6b043b into main 2026-10-10 23:07:43 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kubus/hatbot!1
No description provided.