Interactive vision trainers let the agent practice clicking, typing, and reading your apps. Lesson Pearls distil every thumbs-up moment into an editable, persistent memory the agent carries forward into every future session.
Self-contained HTML trainer pages (vision_trainer.html, comments_trainer.html) run inside the agent's virtual display at 1280×720. Each presents a sequence of tasks as visible on-screen text. The agent reads the instruction via vision, clicks or types the correct element, and advances only on success — building accurate hand-eye coordination through repetition rather than memorised shortcuts.
Begin Lesson and End Lesson buttons in the chat UI bracket a training block. Every thumbs-up inside the lesson attaches to that lesson's ID. When you click End Lesson, a single structured distillation pass runs over the full pearl sequence, producing a titled summary with ordered steps that the agent can read back in future sessions — not scattered per-interaction reflections, but a coherent lesson with clear boundaries.
A floating LessonPearlsFloater panel shows pearls accumulating in real time during a lesson. Each pearl is the verbatim task that earned a thumbs-up. The panel shows a live count and elapsed timer so you can see forward progress. When End Lesson fires, the floater transitions to "Finalising…" state while the distillation runs, then closes when the lesson summary is ready for review.
The lesson summary modal presents the distilled steps as an editable list — reorder with up/down arrows, edit individual step text, delete noise. Save patches the memory record via PATCH /api/memory/<id>. The agent reads lesson summaries with a relaxed 1,000-character budget (vs. 300 for ordinary memories) so full multi-step workflows aren't truncated before reaching the model.
Guaardvark's agent uses vision to perceive its screen and a servo controller to click and type. Out of the box it performs reasonably well on generic interfaces, but it improves dramatically when trained on the specific apps you use. The Vision Trainer system provides structured practice arenas: self-contained HTML pages that rotate the agent through a scripted sequence of tasks against mockup interfaces, scoring each attempt and wiring successful completions back to the positive-feedback pipeline.
vision_trainer.html is the primary trainer — a general-purpose click target practice page that runs at 1280×720 on the agent's virtual display (DISPLAY=:99). comments_trainer.html is a specialised mockup of a video platform with a comment section, designed to build confidence on the click-to-focus, type, click-to-submit flow that appears across YouTube, TikTok, and similar interfaces. The comments trainer presents six deliberately different video card shapes (long-form, shorts, live stream, tutorial, music, podcast) so the agent practices visually diverse targets without relying on fixed selectors. Every correct task completion fires a positive feedback event to /api/agent-control/feedback, which routes into the pearl distillation system.
Lesson Pearls address a specific problem with per-interaction feedback: individual thumbs-up events produce low-quality self-reflection without clear lesson boundaries, making it hard to tell when the agent has genuinely learned something coherent. The solution is explicit lesson framing. You start a lesson, work through a training sequence, give thumbs-up on successful steps, and end the lesson. The Lesson Pearls system collects every thumbs-up within that boundary, runs a single structured LLM distillation over the full sequence, and stores the result as an AgentMemory row with source="lesson_summary". Future sessions load this memory into the agent's context with full fidelity, giving the agent access to the complete ordered workflow it learned.
Pearl storage and context injection. Lesson pearls are stored as ToolFeedback rows in SQLite with a nullable lesson_id column added by migration 004_lesson_pearls.py. The lesson state machine lives in backend/api/lessons_api.py: POST /api/lessons/start mints a UUID lesson ID and stores it in the module-level ACTIVE_LESSONS dict; POST /api/lessons/<id>/end queries all pearls for that lesson, calls _distill_lesson_pearls to run the LLM distillation, creates an AgentMemory row, and emits lesson:ended over Socket.IO. The memory context formatter in memory_api.py gives lesson summaries a 1,000-character budget and flattens the JSON steps into plain text before injecting into the agent's prompt, solving the truncation problem that causes lessons to save but silently fail to influence the model.
DOM metadata for precise clicking. The training system is complemented by structured DOM metadata extraction: a DOMMetadataExtractor service connects to Firefox via the Chrome DevTools Protocol on port 9222, enumerates interactive elements (buttons, inputs, links, ARIA roles), and returns each element's bounding box in screen coordinates. This metadata is injected into the agent's prompt alongside the screenshot, allowing it to click precisely on named elements rather than guessing pixel positions from vision alone. For elements with a clear DOM match, the servo controller skips the vision model call entirely, reducing click latency by one to three seconds per action.
// Comments trainer positive-feedback hook (fires on each task completion)
fetch('/api/agent-control/feedback', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({
session_id: SESSION_ID,
lesson_id: activeLessonId, // attached to active lesson if open
task: currentTask.instruction,
positive: true,
source: 'comments_trainer'
})
});Your company's ERP or CRM has an idiosyncratic UI that no general-purpose agent has ever seen. Open the Vision Trainer, navigate to your tool, run a lesson bracketing the core workflow steps, thumbs-up each correct action. After End Lesson, the agent has a structured memory of the exact click sequence required. Next time you ask it to file a support ticket or pull a weekly report, it follows the learned steps rather than guessing from first principles.
Run the Comments Trainer to give the agent confidence on video-and-comment interfaces before enabling Social Outreach. The trainer's mockup presents six video card types and ten interaction tasks covering search, navigate, click, type, and submit. A session of 50–100 correct completions produces several lesson pearls that prime the agent to operate reliably on real social platforms without accumulating a misclick history.
After a session where the agent struggled with a particular UI pattern, open the Memory Management panel in Settings, find the relevant lesson summary, and edit the steps to correct the mislearned behaviour. The editable summary modal lets you reorder steps, fix mischaracterised actions, or delete noise before the lesson influences the next session. This closes the feedback loop: you can audit exactly what the agent thinks it learned and correct it without rerunning the full training sequence.
Prompt engineering can tell an agent what to do in general terms but cannot encode the specific pixel-level workflows of your particular applications. Guaardvark's Training feature builds app-specific muscle memory through repetition and structured distillation. The resulting lesson summaries are stored as first-class memories with full CRUD access — you can inspect, edit, export, and delete them at any time from the Settings panel. No fine-tuning is required; the learning happens entirely at the context level, which means it works with any Ollama-compatible model without modifying weights. Lessons created against one model are immediately readable by any other model you switch to.
Yes — the Lesson Pearls system and the Comments Trainer ship in v2.6, released today, alongside the existing vision-trainer infrastructure and positive-feedback pipeline. Install Guaardvark to use them.
No. Lesson summaries are stored as AgentMemory records and injected into the agent's context at inference time. No fine-tuning or weight modification occurs. This means training works immediately with any Ollama-compatible model, lessons are portable across model switches, and you can delete a lesson and its effects disappear in the next session.
Distillation is a single LLM call over the pearls collected during the lesson. For a typical lesson of 5–15 pearls, distillation completes in 3–8 seconds depending on the model. The result is available immediately via the summary modal.
Yes. The trainer page format is documented and straightforward — a self-contained HTML file at 1280×720 with a task sequence, click detection, and a fire-and-forget POST to /api/agent-control/feedback on completion. The existing vision_trainer.html and comments_trainer.html serve as reference implementations. Place new trainer files in data/agent/files/ and navigate to them on the agent's virtual display.
Lesson summaries are stored as AgentMemory rows in data/database/system_analysis.db with source="lesson_summary". Browse them in Settings → Memory Management. Each row shows the lesson title, step count, and creation date. Click the edit icon to open the summary modal for editing. Click delete to permanently remove the lesson from the agent's context.