Skip to content

Multi-Provider LLM Support

Vector Companion supports multiple LLM providers via the ModelBackend Protocol. Ollama is the default provider. In-process llama.cpp (via llama-cpp-python) is available as an alternative with ~5x faster Time-To-First-Token by eliminating HTTP overhead.

Provider Selection

The active provider is controlled by config.llm_provider:

Value Backend Description
"ollama" OllamaModelBackend Ollama Python client, connects to 127.0.0.1:11434 (default)
"llama_cpp" LlamaCppPythonBackend In-process inference via llama-cpp-python, loads GGUF directly into memory

Switch providers at runtime via the Parameters panel in the Tauri desktop UI (Provider dropdown), or by editing config/config.py.

Provider persistence: When you switch providers via the UI, the selection persists to SQLite (app.provider key). On startup, the backend loads the persisted provider before creating the ModelBackend, so your choice survives restarts. Per-provider model selections are also saved (app.models.ollama / app.models.llama_cpp) -- switching providers automatically restores that provider's saved model roles.

Cloud mode guard: Switching to llama_cpp automatically disables cloud_mode because Ollama-specific cloud endpoints (e.g., qwen3-vl:235b-cloud) are not available via in-process llama.cpp. A CONFIG_UPDATED event fires so the UI reflects the change immediately.

ModelBackend Protocol

The ModelBackend Protocol (core/model/model_backend.py) defines 8 methods that any provider backend must implement:

Method Purpose
chat_stream() Async streaming via async generator, yield raw tokens
chat_stream_structured() Yield ChatStreamChunk(content, thinking, tool_calls, done, extra) objects
chat_stream_structured_sync() Sync version for run_in_executor compatibility (agent_classes.py pattern)
chat() Non-streaming dict response, normalized across providers
embed() Generate embeddings, return list[list[float]]
list_models() List available models on the provider
health_check() Verify provider is reachable and healthy
cancel() Signal in-flight cancellation (best-effort; llama.cpp does not support mid-stream interrupt)

All methods return normalized structures identical across providers. _normalize_chunk() converts ChatStreamChunk to Ollama-style dicts for consumer compatibility.

Factory Pattern

from core.model.model_backend import create_model_backend

# Reads config.llm_provider automatically
backend = create_model_backend(backend=config.llm_provider, default_model=config.language_model)

# Register custom backends
from core.model.model_backend import register_backend
register_backend("my_custom", MyCustomBackend)

llama.cpp In-Process Setup

Vector Companion uses llama-cpp-python (v0.3.34+) for in-process inference -- no external server, no HTTP. The GGUF model loads directly into Python memory at backend construction time.

Installation

Install llama-cpp-python with CUDA support (requires Visual Studio Build Tools + CUDA toolkit):

# Force CMake build with CUDA (avoids cached CPU-only wheel)
$env:CMAKE_ARGS = "-DGGML_CUDA=on"
$env:FORCE_CMAKE = "1"
uv pip install llama-cpp-python

Without CMAKE_ARGS, the package installs a pre-built CPU-only wheel that will not offload layers to your GPU.

Verify GPU detection:

from llama_cpp import Llama
# Check that gpu_offload returns True (requires model load)

Configuring in Vector Companion

# config/config.py
llm_provider = "llama_cpp"

# REQUIRED for llama_cpp: absolute path to .gguf model file
llama_cpp_model_path = r"D:\path\to\model.gguf"

# GPU layer offload (-1 = all layers, 0 = CPU only)
llama_cpp_n_gpu_layers = -1

# Directory to scan for available .gguf models (used by list_models / UI model picker)
llama_cpp_model_dir = r"D:\path\to\models"

Deprecated (kept for backward compatibility): llama_cpp_host, llama_cpp_port -- no longer used with in-process backend.

Performance

In-process llama.cpp eliminates HTTP serialization/deserialization and network round-trips. Benchmarks on RTX PRO 6000 Blackwell Max-Q (Qwen3.6-27B Q8_0, same-same prompt):

Metric llama.cpp (in-process) Ollama (HTTP) Improvement
Model Load ~9s ~15s 1.7x faster
TTFT (warm) ~140 ms ~700 ms 5x faster
Throughput Comparable Comparable --

The TTFT advantage comes from direct C++ generator yield to Python (no HTTP framing, no JSON serialization). Decode throughput is comparable between the two (same underlying CUDA kernel).

Limitations

  1. Single model at a time -- Only one GGUF is loaded into memory. Hot-switching to a different GGUF requires unloading the old model (~5-10s reload).
  2. Embeddings via Ollama -- When llm_provider = "llama_cpp", embeddings still go through Ollama (embedding model qwen3-embedding:0.6b). Keep Ollama running for memory mode + vector search.
  3. DFlash models cannot run standalone -- DFlash is a draft model architecture for speculative decoding. It requires ctx_other (encoder context from the target model) and cannot be used as a general-purpose chat model. Use the full Qwen3.6-27B-Q8_0 GGUF for standalone inference.
  4. VRAM shared with PyTorch -- The llama.cpp CUDA context shares VRAM with STT/TTS models (PyTorch). Choose model sizes that fit alongside your audio models (~15 GB overhead for STT + TTS).

Advanced Sampling Parameters

When using llama.cpp, the following sampling parameters are available:

Supported Parameters

Parameter Default Description
temperature 1.0 Sampling temperature
top_p 0.95 Nucleus sampling threshold
top_k 64 Keep top K tokens
min_p 0.0 Minimum token probability
typical_p 1.0 Locally typical sampling (1.0 = disabled)
presence_penalty 0.0 Presence penalty for repeat tokens
frequency_penalty 0.0 Frequency penalty for repeat tokens
repeat_penalty 1.0 Repeat penalty multiplier

These parameters are passed directly through _build_sampling_kwargs() to create_chat_completion().

Unsupported Parameters (logged + skipped)

llama-cpp-python's high-level API does not yet expose all llama.cpp sampling features:

Parameter Status
dry_multiplier / dry_base / dry_allowed_length Not exposed by high-level Python API -- debug warning logged, param skipped
xtc_threshold / xtc_probability XTC not supported by llama-cpp-python -- debug warning logged, param skipped
top_n_sigma Not exposed -- silently skipped
dynatemp_range Not exposed -- silently skipped

If you need DRY or XTC sampling, use Ollama as the provider (Ollama passes all params to llama.cpp via HTTP, where they are fully supported).

The Parameters panel in the Tauri UI conditionally displays DRY/XTC/advanced groups only when llama_cpp provider is selected. These widgets are visible for UI consistency but the params will be skipped at runtime with debug-level log warnings.

Vision Format Adapter

Different providers use different image formats in multimodal messages:

Provider Format
Ollama "images": [bytes] key in message dict
llama.cpp (in-process) Uses native tokenizer -- images not supported for vision

_normalize_vision_messages() in vision/image_processing.py handles the conversion automatically. When _vision_backend is None (Ollama), messages pass through unchanged.

Backend Injection

During startup (main.py), the created backend is injected into multiple subsystems:

  1. ModelManager -- create_model_manager(backend=_model_backend) -- ensures list_local_models() queries the configured provider, not always Ollama
  2. Agent factory -- build_agents(config, model_backend=backend, tool_host=tool_host)
  3. Embedding -- set_embedding_backend(backend) in core/data/embedding.py (falls back to Ollama if NotImplementedError)
  4. Vision -- set_vision_backend(backend) in vision/image_processing.py

All subsystems use the provider-appropriate format automatically after injection. Backend creation happens early (after EventBus setup, before ModelManager) so all downstream components receive the correct backend.

Runtime Hot-Switch

Provider switching works at runtime without restarting main.py. When you change the provider in the Tauri UI or via the set_provider API command, the handler:

  1. Creates a new backend -- For llama_cpp, loads the GGUF model into memory (~5-10s blocking operation). For ollama, creates an httpx client (instant).
  2. Cancels in-flight generation -- shared_state.generation_cancel_event.set() signals any running agent to stop streaming. The agent's token loop checks this event every iteration (agent_classes.py:989).
  3. Swaps all consumer references:
  4. _model_backend module global (main.py)
  5. agent.model_backend for each agent instance
  6. _embedding_backend via set_embedding_backend()
  7. _vision_backend via set_vision_backend()
  8. _model_manager.backend on the ModelManager instance
  9. Runs a non-blocking health check -- 3-second timeout via asyncio.wait_for. On success, logs "Hot-swap to {provider} (healthy: ...)". On failure (timeout, connection refused), logs a warning but proceeds -- the error will surface on the first agent call.
  10. Waits for generation to settle -- polls shared_state.agent_generating (max 10 seconds), then clears the cancel event.

The response message indicates the outcome: "switched to {provider}" on success, or "restart required" if the swap threw an exception (very rare -- e.g., factory returns unknown backend type).

Frontend: ModelsPanel Reactivity

The ModelsPanel in the Tauri desktop UI is provider-reactive:

  • Catalog refetch -- useEffect([provider]) dependency ensures the model catalog refetches when the provider changes
  • Cloud role filtering -- language_model_cloud and vision_model_cloud roles are hidden for llama_cpp provider (Ollama-specific cloud endpoints)
  • Loading state -- Displays "Loading models from {provider}..." while fetching, with AbortController cleanup on unmount/provider change
  • Model availability -- Models not in the catalog are marked (unavailable: name) with a disabled dropdown option
  • Purpose filtering -- Embedding models only appear for the embedding_model role; vision models for vision roles

Fallback Behavior

For backward compatibility during the transition period, all code paths fall back to direct Ollama calls when model_backend is None. This means:

  • Existing tests that don't inject a backend continue to work
  • config.llm_provider = "ollama" (default) produces zero behavioral change
  • The system degrades gracefully if the factory cannot create a backend