Architecture Overview

Vector Companion is an event-loop-driven Python application coordinated by main.py. Three async flows run concurrently — recording voice input, generating agent responses, and serving the Tauri 2 React desktop frontend. They communicate through shared state primitives, a typed event bus, and a local HTTP/SSE API server.

Key Statistics

Metric	Value
Core subpackages	17
Service protocols	22 @runtime_checkable classes
Event types	120 across 23+ domains
Tools	11 (all gated by RealToolHost)
Toggleable modes	10 managed by ModeRegistry

Three Concurrent Flows

The main event loop in main.py runs three independent async flows:

1. Audio Pipeline (audio/)

flowchart LR
    A[Microphone] --> B[Audio Recording]
    B --> C[Parakeet-TDT STT]
    C --> D[Sentence Splitting]
    D --> E[Agent Message Queue]

The audio pipeline records user speech, transcribes it via Parakeet-TDT STT (HuggingFace Transformers >= 5.0 AutoModelForTDT), and splits the transcript into sentences for agent generation. Short utterances (3 words or fewer) are discarded as noise before reaching the agent queue.

2. Agent Generation (core/agent/)

flowchart LR
    A[ModelBackend] --> B[Token Streaming]
    B --> C[Tool Call Detection]
    C --> D{Tool Needed?}
    D -->|Yes| E[RealToolHost Pipeline]
    D -->|No| F[TTS Sentences]
    E --> G[Result Integration]
    G --> B
    F --> H[StreamingCoordinator]

The agent generation flow streams tokens via ModelBackend Protocol (Ollama by default, llama.cpp optional), detects tool calls in the output stream, and executes them through RealToolHost's 8-step security pipeline. Results are integrated back into the conversation context, enabling recursive multi-turn reasoning within a single message turn. Final sentences are yielded to the TTS pipeline through StreamingCoordinator.

3. UI Controls (tauri/frontend/)

flowchart LR
    A[Tauri React Frontend] --> B[HTTP POST /commands]
    B --> C[API Server]
    C --> D[EventBus Publish]
    D --> E[Main Loop Wake]
    F[SSE GET /events] --> G[Vite Dev Server]
    G --> H[React Components]

The Tauri frontend reads and writes sampling parameters, mode toggles, and agent selection via the local API server at 127.0.0.1:8765. Messages sent from the UI wake the main loop through SharedState event primitives. Agent responses stream back as Server-Sent Events consumed by React hooks (useEvents()) and written to Zustand stores.

Data Flow

The primary user interaction follows this pipeline:

flowchart LR
    A[User Speaks] --> B[Audio Recording]
    B --> C[STT Transcription]
    C --> D[Message Queued]
    D --> E[Agent Generation]
    E --> F{Tool Call?}
    F -->|Yes| G[RealToolHost Pipeline]
    G --> H[Result Integration]
    H --> E
    F -->|No| I[Sentence Yield]
    I --> J[StreamingCoordinator]
    J --> K[TTS Synthesis]
    K --> L[Audio Playback]
    M[Tauri Text Input] --> D

Text messages from the Tauri frontend bypass audio recording entirely and enter at the message-queue step. The text_only_flag in SharedState tells the main loop to skip audio transcription paths when UI text is received.

Agent Routing

When the main loop processes a queued message, it determines which agent responds:

• Named mention: Only the named agent processes the message.
• No name mentioned: Only the first active agent responds (subsequent active agents are skipped via break after response).
• All tool call attributions route through caller=f"agent:{self.agent_name}".

Subpackages

The core/ directory contains 17 subpackages, each owning a distinct concern. The table below lists every subpackage with its responsibility and key entry points.

Subpackage	Responsibility	Key Entry Points
core/agent/	Agent lifecycle, generation service, prompt building, tool invocation, memory context	build_agents(), Agent class, GenerationService
core/security/	Tool execution gate, manifest validation, autonomy policy, approval workflow, audit logging, process isolation, permission boundaries	RealToolHost, ToolManifest, ApprovalWorkflow, IsolationManager
core/infrastructure/	Event publishing/subscribing, event type registry (120 types), service protocols (22 classes), service registry	EventBus, EventType, ServiceRegistry
core/runtime/	Startup lifecycle (12-phase init), top-level app container, runtime configuration, asyncio primitives singleton	LifecycleManager, RuntimeApp, RuntimeConfig, SharedState
core/api/	Local HTTP/SSE server, typed command schemas (36 Pydantic models), chat log streaming	ApiServer, COMMAND_SCHEMAS, push_ui_message()
core/streaming/	Bounded-queue producer/consumer for TTS pipeline with backpressure and interrupt handling	StreamingCoordinator, InterruptReason, StreamingResult
core/model/	ModelBackend Protocol (multi-provider: Ollama, llama.cpp), model lifecycle management, ChatStreamChunk normalization	ModelBackend, OllamaModelBackend, LlamaCppModelBackend, ModelManager, create_model_backend()
core/speech/	STT backend interface, Parakeet-TDT adapter, speech backend registry (VAD is scaffold only)	STTBackendAdapter, BackendRegistry
core/data/	SQLite canonical store (WAL mode), vector embedding, CrossEncoder reranking, backup/search	SQLiteStore, ChromaEmbedder, CrossReranker
core/modes/	Mode flag registry (10 modes), settings persistence, single source of truth with callback propagation	ModeRegistry, process_mode_commands()
core/ux/	First-run onboarding, accessibility checks, voice state machine, secrets manager	OnboardingManager, VoiceStateMachine, SecretsManager
core/packaging/	Windows installer generation, cross-platform specs, debug bundle, encrypted export	PackagingConfig, default_packaging_config()
core/migration/	Legacy JSON-to-SQLite migration, migration manifest, status tracking	MigrationManager, build_plan(), execute()
core/worker/	Conversational orchestrator, background task runner, web research protocol, priority task queue	Orchestrator, BackgroundTaskRunner, TaskQueue
core/logging/	Structured logger with JSON formatting, sensitive data redaction	StructuredLogger, LogRedactor
core/util/	Metrics collection (bounded ring buffer), lazy loading with timeouts, resource management, benchmarking	MetricsCollector, LazyLoader, ResourceManager
core/adapter/	MCP protocol adapter, plugin marketplace scaffolds, diagnostics	MCPAdapter, PluginManager

Cross-Cutting Components

Several components span multiple subpackages rather than belonging to one.

SharedState Singleton (core/runtime/shared_state.py)

Lazy-initialized singleton holding all cross-module asyncio primitives:

Primitive	Purpose
tts_interrupt_event	Signal TTS synthesis to stop mid-stream
can_speak_event	Wake main loop for new messages
stop_recording_event	Halt audio recording (UI text input)
text_only_flag	Skip audio transcription path
audio_playback_lock	Serialize TTS playback
image_lock	Coordinate screenshot captures

Avoids event-loop affinity issues by never creating new asyncio primitives outside this singleton.

Event Bus (core/infrastructure/event_bus.py)

Async pub/sub system with typed subscription. 120 event types across 23+ domains (agent, audio, mode, security, ui, worker, etc.). Components subscribe by EventType; publishers do not need to know subscriber identities. Per-source sequence numbers enable replay debugging.

StreamingCoordinator (core/streaming/streaming_coordinator.py)

Bounded-queue (maxsize=8) producer/consumer bridging agent generation and TTS synthesis:

• Producer yields sentences from the LLM stream; consumer pushes to TTS
• 5-second backpressure timeout prevents memory accumulation during long responses
• Supports both list[str] and AsyncGenerator[str, None] inputs
• Respects audio_playback_lock and publishes 10 STREAM_* events for UI sync
• Replaces the prior inline unbounded queue pattern in queue_agent_responses()

ModeRegistry (core/modes/mode_registry.py)

Manages 10 toggleable modes. ModeRegistry is the single source of truth; voice commands and UI toggles propagate changes to globals/config via callback for backward compatibility:

ModeRegistry (source of truth) --set_mode_flag callback--> main.py globals, config module

Voice commands pass set_mode_flag as a callback to process_mode_commands() for sync. UI toggles use apply_ui_controls(). The 10 modes are: analysis_mode, mute_mode, auto_chat_mode, memory_mode, control_mode, cloud_mode, training_mode, remote_mic_mode, silence_mode, legacy_mode.

SQLite Canonical Store (core/data/sqlite_store.py)

Single WAL-mode SQLite database for all persistent state: conversations, messages, settings, memories, attachments, audit events, tool grants, and background tasks. Replaces scattered JSON files with a transactional store supporting CRUD operations per entity type.

Inter-Process Communication

The backend (Python) and frontend (Tauri React) communicate over the local API server:

Direction	Mechanism	Purpose
UI -> Backend	HTTP POST /commands	Send messages, toggle modes, set parameters, query state
Backend -> UI	SSE GET /events	Stream agent responses, tool calls, control updates
Both	.runtime/ files	Voice commands, state synchronization fallback

The API server authenticates via constant-time session secret comparison (X-Session-Secret header) and uses CORS middleware to allow the Vite dev server during development. This design keeps the frontend as a thin presentation layer -- all intelligence remains in the Python backend process.

Startup Sequence

LifecycleManager.run() executes 12 initialization phases in order:

flowchart LR
    A[setup_environment] --> B[setup_logging]
    B --> C[detect_device]
    C --> D[setup_torch]
    D --> E[initialize_shared_state]
    E --> F[load_audio_model]
    F --> G[load_tts_model]
    G --> H[load_reranker deferred]
    H --> I[build_agents]
    I --> J[load_history]
    J --> K[load_tools]
    K --> L[ready]

Agents are built after the ToolHost is created, ensuring all agents receive a non-None ToolHost (raises ValueError if not). The build_agents() factory then creates agents lazily from config data.

Key Invariants

These rules are enforced by design and appear in tests:

• No fixed agent set -- Agents are config-driven; the system never hardcodes a list of agent names.
• ToolHost mandatory -- Every tool call routes through RealToolHost's security pipeline. Agent construction raises ValueError without a ToolHost.
• SharedState owns all asyncio primitives -- No module creates new events or locks outside the singleton.
• SQLiteStore is canonical persistence -- All durable state goes through the WAL-mode database, never raw JSON files.
• Bounded TTS pipeline -- StreamingCoordinator uses a bounded queue (maxsize=8) with backpressure; unbounded queues are prohibited.
• Ten modes only -- The valid set is fixed: analysis, mute, auto_chat, memory, control, cloud, training, remote_mic, silence, legacy. No read_mode or text_mode.

Architecture at a Glance

graph TB
    subgraph "User"
        A[Voice Input]
        B[Tauri Desktop UI]
    end

    subgraph "Audio Pipeline"
        C[Recording]
        D[Parakeet-TDT STT]
        E[Sentence Splitting]
    end

    subgraph "Core - Agent Layer"
        F[Agent Generation]
        G[Tool Invocation]
        H[Memory Context]
    end

    subgraph "Core - Security Layer"
        I[RealToolHost Pipeline]
    end

    subgraph "Core - Runtime"
        J[SharedState]
        K[EventBus]
        L[ModeRegistry]
    end

    subgraph "Core - Streaming"
        M[StreamingCoordinator]
    end

    subgraph "Data Layer"
        N[SQLiteStore]
        O[ChromaDB Memory Mode]
    end

    subgraph "Frontend"
        P[Tauri React App]
        Q[SSE Consumer]
    end

    subgraph "Local API Server"
        R[FastAPI Local Server]
    end

    A --> C --> D --> E --> J
    B --> P --> Q
    J --> F
    F --> G --> I
    I --> H
    H --> F
    F --> M
    M --> N
    K --> J
    L --> J
    R <--> B
    R --> K