Skip to content

Agent System

Agents are Vector Companion's core intelligence layer: they consume messages, stream responses from Ollama, invoke tools through a security pipeline, and deliver sentences to the TTS engine. The entire system is config-driven -- add, remove, or replace agents by editing config/config.py without touching any Python source.

Where to Look
Concern File
Agent class (streaming, tool calling) core/agent/agent_classes.py
Lazy creation from config core/agent/agent_factory.py
Token streaming and sentence extraction core/agent/agent_generation_service.py
Layered prompt construction core/agent/prompt_builder.py
Tool call detection and dispatch core/agent/tool_invocation_service.py
Vector memory context core/agent/memory_context_service.py

Modular Architecture

Agents are data, not code. Each agent is defined as a configuration entry in config/config.py, containing properties like name, personality, system prompts, and mode preferences. At startup, the lifecycle manager calls build_agents(config, tool_host=tool_host), which reads that config data and instantiates Agent objects lazily -- one per entry.

This design means:

  • No hardcoded agent names. The system never assumes a fixed set. New agents appear automatically when added to config.
  • Security is mandatory. Every agent receives a RealToolHost instance at construction time. build_agents() raises ValueError if the tool host is None, ensuring no agent can bypass the security pipeline.
  • Agents share message history. All agents reference the same messages list persisted in SQLiteStore, but only one agent responds per turn (see Agent Routing below).

The Agent Class

The Agent class handles three core responsibilities: streaming token generation from Ollama, executing recursive tool calls, and yielding sentences to the TTS pipeline.

Response Flow

flowchart LR
    A[Message Queued] --> B[Agent Token Stream]
    B --> C{Tool Call Detected?}
    C -->|Yes| D[RealToolHost Execute]
    D --> E[Result Integration]
    E --> B
    C -->|No| F[Yield Sentence]
    F --> G[StreamingCoordinator]
    G --> H[TTS Synthesis]
    H --> I[Audio Playback]

The response flow follows these steps:

  1. Queueing. The main loop queues user messages via queue_agent_responses() in main.py.
  2. Streaming. The agent streams tokens from Ollama. As each token arrives, the generation service checks for tool call patterns in the output.
  3. Tool execution. When a tool call is detected, it dispatches through RealToolHost.execute() -- the 8-step security pipeline described in Security Pipeline. Results are integrated back into the conversation context, and streaming continues (recursive multi-turn reasoning within a single message turn).
  4. Sentence extraction. Normal text sentences are yielded to StreamingCoordinator, a bounded-queue (maxsize=8) producer/consumer that bridges agent generation and TTS synthesis with backpressure handling.
  5. Audio playback. The consumer side of StreamingCoordinator pushes sentences to TTS synthesis via run_in_executor, then plays audio serialized by the audio_playback_lock.

Sentence Splitting

Sentences extracted from the LLM stream are subject to constraints defined in the dialogue quality harness (see Prompt Architecture):

  • Maximum 4 sentences per response
  • Each sentence under 100 characters
  • No markdown, emoji, or stage directions
  • 18 stock phrases are blacklisted (e.g., "I'd be happy to help", "As an AI")

These constraints are enforced through the prompt's few-shot examples and quality harness, not through code-level validation. The agent's generated output is trusted to comply.

Context Window

The context window controls how many recent messages are included in the agent's prompt via messages[-message_length:] in select_prompt_config() within main.py.

  • Default: 35 recent messages
  • Configurable via the Parameters panel "Context Msgs" slider (range: 3-100)
  • Auto-chat override: When auto_chat_mode is active, the slice length is overridden to 125 for longer conversational context
  • Config attribute: message_slice_len in config/config.py

The slider fires set_param commands in real-time, which route through _apply_control_update() to update the config attribute immediately — no restart required.

Prompt Architecture

Every agent response is built on a minimal prompt constructed by build_contextual_information() in core/agent/generation_service.py. Modern models don't need section headers — the prompt is plain text:

Personality (system_prompt1)     — Agent-specific persona
Dialogue quality harness         — Shared TTS/plain-text rules
Sentence count                   — Between N and 4 sentences
Tool use guidance                — Inline (no announcement, consecutive calls ok)
Mode status                      — Prepended when any modes active

Dialogue Quality Harness

The dialogue_quality_harness in config/config.py is shared across all agents. It enforces TTS awareness, plain text output, and audio transcript rules in a concise (~100 char) block. Agent personas (system_prompt1) define tone, knowledge domain, and communication style — they appear first in the prompt so the agent's identity is front-and-center.

Mode System

Each agent operates within a set of 10 toggleable modes managed by ModeRegistry in core/modes/mode_registry.py. The registry is the single source of truth — voice commands and UI toggles propagate changes to main.py globals and config module attributes via callback for backward compatibility:

ModeRegistry (source of truth) --set_mode_flag callback--> main.py globals, config module

The Ten Modes

Mode Purpose
analysis_mode Deeper reasoning with extended tool use
mute_mode Suppresses audio output; text-only responses
auto_chat_mode Enables unsolicited agent-initiated conversations
memory_mode Activates ChromaDB vector retrieval and CrossEncoder reranking
control_mode Allows control-oriented tool execution
cloud_mode Routes requests through cloud LLMs instead of local Ollama
training_mode Enables learning from user corrections
remote_mic_mode Accepts audio input from remote sources
silence_mode Suppresses all output (agent is "listening only")
legacy_mode Falls back to older response patterns for compatibility

Mode Registry (single source of truth)

ModeRegistry is the canonical store for all ten mode flags. When a flag changes, set_mode_flag() callback propagates the update to main.py globals and config module for backward compatibility:

  • Voice commands pass set_mode_flag as a callback to process_mode_commands().
  • UI toggles (Tauri frontend) call apply_ui_controls() to update the registry, which triggers callbacks.

Think Mode

Think mode controls reasoning depth. It is normalized to one of four levels:

Level Meaning
False / "" No extended thinking
"medium" Standard reasoning with brief reflection
"high" Deep reasoning with multi-step chain-of-thought
True Equivalent to "high" (legacy boolean form)

Think mode is synchronized across all agents via _apply_control_update("think_mode"), which directly sets agent.think on every agent instance. Without this direct update, apply_agent_think() would skip agents with truthy .think values.

[!WARNING] Legacy modes removed read_mode and text_mode existed in earlier versions and have been removed. Do not reference them in config or code.

Agent Routing

When a message enters the system, Vector Companion determines which agent responds using a simple but effective rule:

  1. Named mention. If the user's message addresses a specific agent by name, only that named agent processes the message.
  2. No name mentioned. Only the first active agent in the list responds. After it generates a response, the loop breaks -- subsequent active agents do not process the same message.

This guarantees exactly one response per turn regardless of how many agents are active. Tool call attributions route through caller=f"agent:{self.agent_name}", so even tool execution is traceable to the responding agent.

Why single response?

Multi-agent chorus (where every active agent responds to every message) creates a chaotic experience: overlapping audio, conflicting information, and poor TTS pipeline utilization. The single-response model keeps conversations coherent while still supporting many agents with different specializations.

Web Research Protocol

The web research workflow uses a two-tool pattern that prevents common pitfalls like hallucinated search results or incomplete page extraction:

flowchart LR
    A[User Question] --> B[web_search]
    B --> C{Enough info?}
    C -->|No| D[fetch_url]
    D --> E[Chrome TLS impersonation]
    E --> F[Synthesize findings]
    F --> A
    C -->|Yes| G[Direct answer]

web_search returns DuckDuckGo titles and URLs only -- no page content or snippets. This is fast and gives the agent an overview of available information. The title-only approach minimizes context window usage while the trailing instruction directs the agent to use fetch_url on at least 2 URLs before answering.

Step 2: URL Fetch

fetch_url extracts full page content via curl_cffi with Chrome TLS impersonation (fingerprinting as a real Chrome browser). The agent must call this on at least 2 URLs after web_search before responding. A near-duplicate query guard prevents excessive searches for the same topic.

Protocol Constraints

The Web Research Protocol block in the prompt enforces:

  • Never synthesize from titles alone; always fetch the full page for substantive claims
  • Detect and avoid near-duplicate queries (similar search terms)
  • Guard against consecutive searches without integration (search, then fetch, then synthesize -- not search, search, search)
  • Maintain multi-topic awareness across research cycles

Embedding and reranker models were retired from the web_search and read_file paths; they are now used exclusively in memory_mode. See Data Persistence for vector memory details.

Think Mode: Reasoning Depth

Think mode adjusts how deeply an agent reasons before responding. It operates at the prompt level -- higher think levels trigger extended chain-of-thought instructions in the system prompt.

Normalization

Think values go through normalize_agent_think(), which maps any input to one of four canonical forms:

normalize_agent_think(True)      -> "high"
normalize_agent_think("high")    -> "high"
normalize_agent_think("medium")  -> "medium"
normalize_agent_think(False)     -> False
normalize_agent_think("")        -> False

UI and Code Sync

When the Tauri UI toggles think mode, set_param sends a command through the API server. The main loop's _apply_control_update("think_mode") handler iterates all agents and sets their .think attribute directly. This ensures the generation service sees the correct level during prompt construction.

Best Practices for Developers

Adding a New Agent

  1. Add an entry to config/config.py with name, personality, and system prompts.
  2. Run python config/export_config.py to serialize public attributes to config/config.json.
  3. Restart the application -- build_agents() picks up new config automatically.

Troubleshooting Agent Behavior

  • No response: Check that at least one agent is active in config and not in silence_mode.
  • Multiple responses: Verify there isn't a code path bypassing the single-response break (should only happen with named mentions).
  • Tool calls failing: The RealToolHost security pipeline catches all exceptions and returns success=False. Check the audit log at Security Pipeline for denial reasons.
  • Mode not applying: Check ModeRegistry.get(mode_name) — it is the source of truth. If correct there but not in behavior, verify the agent's mode-aware code paths.

Memory Mode Agents

When memory_mode is active, agents use ChromaDB for vector storage and CrossEncoder for reranking. Embedding queries surface relevant past messages before generation. The reranker is lazy-loaded on first use (saves ~250MB GPU memory if memory_mode is never enabled). See Data Persistence.