Guaardvark ships AgentBrain — a three-tier neural router (Reflex <100ms, Instinct 1–3s, Deliberation full ReACT) — plus ReACT agents that reason, click, and type across real desktop applications. The DOM-metadata layer (Firefox CDP) extracts bounding boxes + labels so confident matches skip the vision model entirely, saving 1–3s per action and raising hit rates dramatically.
Reflex (<100ms, 0 LLM calls): instant pattern-matched commands. Instinct (1–3s, 1 call): most work with smart tool selection. Deliberation (5–30s): full multi-step ReACT when depth is warranted. Automatic escalation + Gemma4 direct path for screen tasks.
The Reason → Act → Observe loop lets agents decompose any task into steps, call the right tool at each step, and iterate until done — all without human intervention. Over 30 registered tools cover web, file, code, and desktop domains.
When DOM metadata is unavailable or ambiguous, the agent falls back to screenshot-based vision estimation. The servo controller queries the vision model for pixel coordinates, clicks, and observes the result before deciding next steps.
The DOMMetadataExtractor service connects to Firefox via CDP on port 9222, enumerates every interactive element with its bounding box, label, and type, and caches the snapshot for one agent step. DOM-matched clicks skip the vision model entirely.
Every reasoning step, tool call, screenshot, and DOM snapshot stays on your machine. No cloud inference, no API keys, no telemetry. The agent runs inside an isolated display managed by start_agent_display.sh so it cannot escape its workspace.
At the core of Guaardvark’s agent system is the ReACT (Reason + Act) loop. When you hand the agent a task — “search YouTube for the top three tutorials on prompt engineering and summarise each one” or “open the project config file, bump the version number, and commit” — it does not respond with a single answer and stop. Instead it enters a cycle: it reasons about what information it needs, acts by invoking a tool, observes the result, and reasons again. Each iteration builds on the last until the task is complete or the agent determines it cannot proceed without more input.
The v2.6 release extends that loop into desktop applications via a new vision layer. The agent now manages a real Firefox instance running in an isolated X display at DISPLAY=:99. Screenshots flow into the reasoning loop alongside text, giving the agent a genuine visual understanding of what is on screen. Before v2.6, the agent had to estimate click targets from pixels alone — a process that succeeded roughly 19% of the time on complex UIs. With DOM metadata injected alongside the screenshot, the agent can read the element list, find the exact button by its label and bounding box, and route the click directly — raising hit rates dramatically and cutting per-action latency by 1–3 seconds on DOM-matched targets.
All of this happens locally. The agent’s tool registry spans over 30 capabilities: reading and writing files, running shell commands, fetching web pages, searching your document index, querying databases, executing code, and now clicking and typing in any application visible on screen. Because every model call, tool invocation, and intermediate result stays on your hardware, you can direct the agent at sensitive documents, proprietary codebases, and internal services without any data leaving your network.
A neural router that decides how much work a message deserves before tools or heavy reasoning fire. Saves time on simple questions and unlocks depth when needed.
| Tier | Name | Latency | LLM Calls | When It Fires |
|---|---|---|---|---|
| 1 | Reflex | <100ms | 0 | Media commands, greetings, exact-match recipes (pre-computed table) |
| 2 | Instinct | 1–3s | 1 | Most requests — single LLM call with relevant tools in scope + semantic selection |
| 3 | Deliberation | 5–30s | 3–10 | Multi-step reasoning (full ReACT loop). Automatic escalation from lower tiers. |
Gemma4 Direct Path: When the active model is Gemma4 (native vision + tool use) and the agent screen is active, the brain skips ReACT overhead and sends a minimal prompt with screenshot + task. Returns JSON actions directly.
Configurable via AGENT_BRAIN_ENABLED. Graceful fallback to legacy path.
DOMMetadataExtractor service. The new backend/services/dom_metadata_extractor.py is a singleton that connects to Firefox via GET http://localhost:9222/json/list, then opens a WebSocket to the active tab and runs a Runtime.evaluate call injecting a JavaScript snippet that enumerates every interactive element — a, button, input, textarea, select, and [role="button"]. Each element returns its tag, text or aria-label, type, bounding box in screen-space coordinates (corrected for the browser chrome offset via window.screenY + outerHeight - innerHeight), center point, id, name, href, and focused state. Results are cached for one second, so a single agent step reuses the same snapshot. The service fails gracefully — returning an empty result on any error — so it never blocks the reasoning loop.
Prefer-DOM, fallback-vision flow. The servo controller in backend/services/servo_controller.py now tries a DOM lookup before calling the vision model. _lookup_element_coordinates(target) fuzzy-matches the target description against the cached DOM element list; when confidence is high the coordinates are returned immediately with zero network latency. Only when no DOM match is found does the controller fall through to the vision-estimation path, which calls the configured vision model and parses its coordinate response. The Gemma4 model path in backend/services/agent_brain.py also receives the formatted DOM element list directly in its system prompt (gated on a model check), so the model can reference element coordinates in its chain-of-thought before the servo controller is even called.
# Firefox starts with CDP enabled at port 9222
# Configured in scripts/start_agent_display.sh
firefox --remote-debugging-port=9222 --profile /data/agent/firefox_profile ...
# DOM snapshot (compact format injected into agent prompt)
Page: YouTube (https://youtube.com/results?search_query=guaardvark)
Interactive elements (screen pixel coordinates):
[1] button "Search" at (640,95) 80x36
[2] input[search] "Search" at (400,95) 450x36 (focused)
[3] a "Guaardvark intro" at (350,250) 600x45
# Servo log when DOM match found:
[SERVO] DOM shortcut: target="search box" -> element[2] at (625,95)Direct the agent to open a web application, fill in a form, copy values from one page into another, or click through a multi-step wizard — expressed in plain English. The DOM-aware click layer in v2.6 makes previously error-prone GUI tasks reliable enough for routine use. No Selenium, no AutoHotkey, no brittle xpath selectors to maintain.
Give the agent a research question. It searches the web, retrieves pages, extracts key claims, cross-references them against your indexed document library via RAG search, flags contradictions, and synthesises a structured report — all in a single session. Because every tool runs locally, no part of your private document corpus ever reaches an external API.
Ask the agent to audit a module for security issues, trace all callers of a deprecated function, or add type hints to every public method. The agent reads source files, builds a mental model of the call graph, proposes diffs, and can apply them directly — all within the same reasoning session that identified the problem.
Cloud-hosted agent platforms — including proprietary operator-API tools and hosted ReACT wrappers — route every tool call, every screenshot, and every reasoning step through external servers. That means your data, your prompts, and your task context leave your network on every iteration of the loop. They also charge per token, impose rate limits that throttle long-running tasks, and lock you into their approved tool sets. Guaardvark agents run the entire ReACT loop locally: the model is loaded on your GPU, tool calls execute on your CPU, and desktop screenshots never leave localhost. You get unlimited iterations, zero per-token cost after model download, full access to your local filesystem and applications, and the new DOM-metadata layer that no hosted platform can replicate — because hosted platforms have no access to your browser’s live DOM.
The headline addition is structured DOM metadata. The agent now extracts Firefox’s live DOM via Chrome DevTools Protocol (CDP) on port 9222, getting every interactive element’s bounding box, label, and type alongside the screenshot. When a DOM match is confident, the servo controller skips the vision model call entirely, saving 1–3 seconds per action. A new DOMMetadataExtractor service (backend/services/dom_metadata_extractor.py) handles the CDP connection, JavaScript injection, screen-space coordinate conversion, and one-second result caching. The feature is gated to the Gemma4 model profile; other models are untouched.
Any Ollama-compatible model loaded into Guaardvark can drive the text-only ReACT loop. Vision-grounded clicks require a model that accepts image inputs; currently the Gemma4 profile is the primary tested path. DOM-metadata injection is gated behind a Gemma4 model check in agent_brain.py so other models see standard prompts without the element list. Model switching is handled through Settings → Agent Model.
The servo controller in backend/services/servo_controller.py calls _lookup_element_coordinates(target) first. This does a fuzzy-match of the target description against the DOMMetadataExtractor cache. If the match confidence is high, coordinates are returned immediately with zero latency — the log shows [SERVO] DOM shortcut. If confidence is low or the cache is empty (for example, when clicking a desktop icon outside Firefox), the controller falls through to the vision estimation path, which calls the vision model and parses its coordinate response. Both paths converge on the same pyautogui click call.
The agent runs in an isolated X display managed by scripts/start_agent_display.sh (default DISPLAY=:99) with its own Firefox profile at data/agent/firefox_profile/. This separates agent browser activity from your main desktop session. File-system access is limited by the tool registry; the agent cannot write outside designated output directories unless you explicitly configure additional permissions. The agent’s actions are fully logged in logs/agent.log for post-session review.
The tool registry lists every capability the agent can invoke in a single ReACT step. Categories include: web fetch, web search, file read/write, shell execute, code analysis, image generation, RAG search query, memory read/write, database query, screenshot capture, DOM metadata extraction, and desktop click/type. Each tool is registered with a name, description, input schema, and output format. The agent discovers all registered tools at startup and can consult the registry to choose the best tool for any sub-task. Adding a custom tool is as simple as dropping a new tool specification into the tools directory.