Token-by-token streaming conversations with any Ollama-compatible model. Switch models mid-session, toggle RAG on or off, control the system prompt, and pick up any previous session — all from a first-class interface running entirely on your machine.
Responses arrive token by token through Socket.IO, rendered by the StreamingMessage React component. Streaming begins within milliseconds of the first token, so even long responses feel immediate. The stream can be interrupted at any point with a stop button that sends a cancellation signal to the backend without leaving the model in a broken state.
Switch models at any point in a conversation from the model selector. RAG mode, direct-LLM mode (--no-rag), and agent mode are all accessible without starting a new session. The unified_chat_api.py dispatcher routes each message to the appropriate backend path — enhanced_chat_api.py for RAG, simple_chat_api.py for direct, agent_chat_api.py for agent — based on the current mode setting.
Set a custom system prompt for any conversation, or select from the library of saved Rules (system prompts). The system prompt is visible and editable at the top of the chat panel. Changes take effect on the next message. Assigning a Rule from the project's rule set is a one-click operation — no copy-paste required.
Every conversation is stored as a Conversation + Message record in SQLite, scoped to the active project. Resume any past session from the sidebar. Export a conversation as Markdown with llx chat --export. Session history persists across restarts, browser clears, and machine reboots — it's in the database, not the browser.
Chat is the central interface of Guaardvark, and every other surface — the code editor's chat panel, the voice interface, the floating chat overlay available on every page — is built on the same chat infrastructure. The EnhancedChatInterface React component handles the full conversation UX: message rendering with Markdown and code block support, streaming token display, tool call visualisation via ToolCallCard, file attachment from the ChatInput, and voice input via the useAudioRecorder hook. The floating chat (FloatingChatFAB) makes a non-modal chat available without leaving whichever feature page you're working on.
The backend chat routing layer handles three distinct modes transparently. In RAG mode, the message flows through enhanced_chat_api.py, which calls agent_router.py for intent detection, then routes to unified_chat_engine.py for hybrid retrieval, then to Ollama for completion. In direct mode, the message bypasses RAG and goes straight to simple_chat_api.py for a lower-latency round trip with no document context. In agent mode, the message goes to agent_chat_api.py, which initialises the ReACT loop with the full tool registry. All three modes stream responses over the same Socket.IO connection.
The CLI replicates the full chat experience in the terminal. llx chat "prompt" streams a response with Rich formatting. llx chat --no-rag "prompt" bypasses retrieval. llx chat --resume continues the last conversation. llx opens an interactive REPL with command history and session persistence. For automation workflows, the --json flag outputs plain JSON for piping to other tools. The CLI and the browser UI share the same backend API, so sessions started in the browser can be continued in the terminal and vice versa.
Chat API layer. Four Flask blueprints handle chat routing: enhanced_chat_api.py (primary, with RAG and streaming), agent_chat_api.py (agent mode with ReACT loop), simple_chat_api.py (direct LLM, no RAG), and unified_chat_api.py (dispatcher that routes based on mode parameter). All four blueprints stream responses via Socket.IO using socketio_events.py. The llm_service.py abstraction wraps Ollama's API so model switching doesn't require changes in the chat layer — just pass a different model identifier. The context_manager.py handles conversation context window management, including truncation strategies for long sessions.
Message storage and retrieval. The Conversation and Message SQLAlchemy models store full conversation history in SQLite. Message records include role (user/assistant/system/tool), content, token count estimate, and a JSON blob for tool call data. The chat sidebar queries conversations sorted by updated_at descending, so the most recently active conversation appears first. Conversation import/export uses the standard Markdown format with user/assistant headers, compatible with most chat archiving tools.
# CLI: chat with streaming
llx chat "Explain the hybrid RAG pipeline in this codebase"
# Resume last session
llx chat --resume "What were the key points again?"
# Direct LLM, no document context
llx chat --no-rag "Write a haiku about Python"
# Export last session as Markdown
llx chat --export > session_20260505.md
# Interactive REPL with history
llxUpload a PDF technical specification or a set of meeting transcripts to the Documents surface, then ask questions in Chat with RAG mode active. The hybrid BM25 + vector retrieval pipeline finds the relevant passages, the LLM synthesises an answer with citations, and the response is streamed token by token. Your documents stay on your machine; no text is sent to a cloud service.
Start a conversation about a complex problem, work through it over several hours or multiple days by resuming the session, and export the final conversation as Markdown documentation of the reasoning process. The conversation history persists in the database through restarts and browser clears — you can resume from your terminal on a different day using llx chat --resume and the model has the full context of the prior exchanges.
Use llx chat --json "prompt" in shell scripts to integrate Guaardvark's LLM into automated pipelines. Pipe file content in, parse the JSON response with jq, and use the result in downstream processing. Because the CLI and browser share the same backend, you can set up a complex conversation context in the browser UI, then continue it in a scripted workflow via the terminal.
Cloud chat products send every message to a remote server. Your questions about proprietary documents, sensitive business context, and unreleased work are processed on infrastructure you don't control. Guaardvark Chat runs Ollama on your own hardware: every token is generated locally, no message leaves your network, and your conversation history is stored in a local SQLite file that you own and control. There are no usage limits, no subscription tiers, and no per-message costs. The RAG integration means your conversations are grounded in your own documents, not the model's training data — answers are specific to your context, not generic.
Any model served by Ollama is available. Pull a model with ollama pull model-name and it appears in the model selector dropdown. Common choices include Llama 3.2, Mistral, Gemma, Phi-4, and DeepSeek variants. Vision-capable models (Llava, Gemma4) enable image understanding in chat. The llx models list command shows all available models with their sizes and capabilities.
Context window length is determined by the model you select. Most current Ollama models support 4,096 to 128,000 tokens. For long sessions, Guaardvark's context manager applies a sliding window strategy: recent messages are always included, and older messages are summarised or truncated based on token budget. The strategy is configurable in Settings → Chat.
Yes. The ChatInput component includes a file attachment button. Attached images are processed by a vision-capable model if one is active. Attached text files (PDF, Markdown, code) are extracted and included in the message context. For persistent document indexing that's available across sessions, upload to the Documents surface instead.
The floating chat (FloatingChatFAB) is a lightweight overlay available on every page. It shares the same backend API and Socket.IO streaming as the main chat, but maintains its own conversation state via useFloatingChatStore. Conversations started in the floating chat can be promoted to the main chat surface for more detailed work. The floating chat is intentionally minimal — input, streaming response, and a conversation history list. It doesn't expose model switching or system prompt controls; those are in the main chat.
Use llx chat --export from the terminal to export the current session as a Markdown file. In the browser UI, a conversation export button in the session panel downloads the full conversation as a Markdown file. The database itself (data/database/system_analysis.db) contains all conversation records and can be queried directly with any SQLite client for bulk export or analysis.