Multi-Provider LLM Support

Vector Companion supports two LLM providers via the ModelBackend Protocol: Ollama (the default) and a llama.cpp router-mode HTTP server (LlamaServerBackend, OpenAI-compatible).

Provider Selection

The active provider is controlled by config.llm_provider:

Value	Backend	Description
"ollama"	OllamaModelBackend	Ollama Python client, connects to 127.0.0.1:11434 (default)
"llama_server"	LlamaServerBackend	llama.cpp router-mode llama-server daemon (OpenAI-compatible HTTP), multi-model resident with dynamic load/unload. User-started, app never spawns/kills it

Switch providers at runtime via the Parameters panel in the Tauri desktop UI (Provider dropdown), or by editing config/config.py.

Provider persistence: When you switch providers via the UI, the selection persists to SQLite (app.provider key). On startup, the backend loads the persisted provider before creating the ModelBackend, so your choice survives restarts. Per-provider model selections are also saved (app.models.ollama / app.models.llama_server) -- switching providers automatically restores that provider's saved model roles. A legacy persisted llama_cpp provider is transparently migrated to llama_server at startup (its orphaned app.models.llama_cpp key is deleted).

Cloud mode guard: Switching to llama_server automatically disables cloud_mode because Ollama-specific cloud endpoints (e.g., qwen3-vl:235b-cloud) are not available via the router. A CONFIG_UPDATED event fires so the UI reflects the change immediately.

ModelBackend Protocol

The ModelBackend Protocol (core/model/model_backend.py) defines 8 methods that any provider backend must implement:

Method	Purpose
chat_stream()	Async streaming via async generator, yield raw tokens
chat_stream_structured()	Yield ChatStreamChunk(content, thinking, tool_calls, done, extra) objects
chat_stream_structured_sync()	Sync version for run_in_executor compatibility (agent_classes.py pattern)
chat()	Non-streaming dict response, normalized across providers
embed()	Generate embeddings, return list[list[float]]
list_models()	List available models on the provider
health_check()	Verify provider is reachable and healthy
cancel()	Signal in-flight cancellation (best-effort; llama.cpp does not support mid-stream interrupt)

All methods return normalized structures identical across providers. _normalize_chunk() converts ChatStreamChunk to Ollama-style dicts for consumer compatibility.

Factory Pattern

from core.model.model_backend import create_model_backend

# Reads config.llm_provider automatically
backend = create_model_backend(backend=config.llm_provider, default_model=config.language_model)

# Register custom backends
from core.model.model_backend import register_backend
register_backend("my_custom", MyCustomBackend)

llama.cpp Router-Mode Server (llama_server)

LlamaServerBackend (core/model/llama_server_backend.py) talks OpenAI-compatible JSON to a user-started llama-server --models-preset ... daemon (default http://127.0.0.1:8080). The app never spawns or kills the router — the same contract as Ollama. If the daemon is down, health_check() returns an actionable status ("start it via the router launch script") instead of crashing.

Setup

1. Start the router as a daemon (e.g. llama-server --models-preset D:\ai\llamacpp\router\preset.ini --host 127.0.0.1 --port 8080). Models are HF-cache-style ids (e.g. Qwen/...-GGUF:Q8_0) — no alias map, the app uses the exact ids the router reports from GET /models.
2. Configure:

# config/config.py
llm_provider = "llama_server"
llama_server_base_url = "http://127.0.0.1:8080"     # router host/port
llama_server_preset_path = r"D:\ai\llamacpp\router\preset.ini"  # for num_ctx hot-apply
llama_server_model_dir = r"D:\ai\llamacpp\models"  # informational; models come from the router

Sampling

/v1/chat/completions accepts all 20 sampling keys per request (llama.cpp extends the OpenAI body), so every UI slider — DRY, XTC, sigma, dynatemp, penalties, repeat_last_n, typical_p — reaches the server. num_predict is mapped to max_tokens.

num_ctx (load-time)

num_ctx is not a request parameter — it is fixed per model at load time in the router preset INI (ctx-size). The [*] section is the default that every model inherits — including cache models that are not listed in the INI — while a per-model section can override it (per-model windows are legal; verified against a live router). A model already running keeps its context until it is unloaded (manually, by LRU, or by a preset change) and reloaded; the new value then applies at load time — the same deferred semantics as Ollama.

1. backend.rewrite_preset_ctx(n) rewrites ctx-size in the preset's [*] section (default for all models)
2. GET /models?reload=1 tells the router to re-read the preset; any loaded instance whose source changed is unloaded
3. The next time the model loads, it comes up with the new context

set_num_ctx() wraps 1+2; applied=true means the new value is committed and takes effect on the model's next load — it does not disturb currently-running models.

Thinking and tools

• Thinking arrives as a native reasoning_content field (no regex). think maps to chat_template_kwargs.enable_thinking (+ reasoning_effort for level strings).
• Tool calls arrive in OpenAI shape. llama.cpp streams each tool call as fragments — the first delta carries id + name, later deltas carry only index + argument slices — so the backend merges deltas by index and yields the complete call once, on the final chunk (finish_reason set). That matches Ollama's one-complete-tool-call-per-chunk contract, which the agent's per-chunk tool execution depends on. The agent also parses raw JSON-text arguments into dicts (_normalize_tool_call), so both providers deliver the same flat shape. The backend normalises the app's Ollama-style tool schemas ({"function": {...}}) to the OpenAI shape ({"type": "function", "function": {...}}) automatically — the router 500s on a missing type ("Failed to parse tools: Missing tool type").
• embed() uses the router's real /v1/embeddings endpoint; Ollama serves embeddings via its native API.

Embeddings require a router-registered embedding model

The router only serves models registered in its preset/cache, so for memory_mode (vector memory) to work under llama_server the embedding model role must be a router model id, not the Ollama tag. Register an embedding GGUF in the preset with pooling flags, then set the embedding_model role (ModelsPanel) to that id:

[Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0]
pooling   = last
embedding = true

Per-provider model roles persist in SQLite (app.models.llama_server), and startup restores them before backend creation, so the router id is used for embedding calls even on a direct boot into llama_server.

Reranking requires a router-registered reranker model

The same rule applies to the memory_mode reranker. The router's POST /v1/rerank endpoint answers 501 "Start it with --reranking" unless the serving model was started with embedding=true + pooling=rank. Under llama_server, ServerReranker (core/data/server_reranker.py) scores via that endpoint using the reranker_model_llama_server role (default Voodisss/Qwen3-Reranker-0.6B-GGUF-llama_cpp:Q8_0); under ollama the in-process CrossEncoder is used instead (see Data Persistence).

The router needs a preset section carrying the keys the spawned child process requires:

[Voodisss/Qwen3-Reranker-0.6B-GGUF-llama_cpp:Q8_0]
reranking = true
hf-repo = Voodisss/Qwen3-Reranker-0.6B-GGUF-llama_cpp:Q8_0
batch-size = 4096
ubatch-size = 4096

• reranking = true — a single key that sets both embedding and pooling=rank; without it /v1/rerank answers 501.
• hf-repo = <id> — without it the child is spawned with --alias only, and a llama-server with neither --model nor --hf-repo decides it is the router and never loads, so the router reports loading forever.
• batch-size = 4096 + ubatch-size = 4096 — a rerank query/document pair is scored as a single sequence that must fit in one physical batch (limit = the smaller of the two); the [*] defaults (1024/512) 500 with "input is too large to process" on real conversation chunks.

This is mostly automatic: on lazy-load, ServerReranker.ensure_ready() appends the section to llama_server_preset_path if it is missing and patches missing keys into existing sections (explicit user values are respected; the write is idempotent), then runs GET /models?reload=1 so the router re-reads the INI. If the router still doesn't know the id (no HF cache entry), ensure_ready() queues a download via POST /models and defers loading to the next lazy-load attempt — the first RAG turn in the meantime falls back to top-3 embedding results. A model the router already knows from the cache but has not started is explicitly loaded via POST /models/load.

Scoring is native: the request is Jina-shaped (query + documents, top_n), the response relevance_score is the model's raw "yes" logit, and the instruction ("Given a query...") is baked into the GGUF's rerank chat template, so the per-call task string is nominal under this provider.

Router model management (load / LRU / unload)

The router runs each model as its own llama-server child process and manages the fleet:

• Autoload — a request for an unloaded model loads it on demand (--models-autoload, default on), so the app never pre-loads anything.
• Concurrent cap — --models-max (default 4, 0 = unlimited) limits loaded models. When a request arrives and every slot is full, it queues (FIFO) instead of evicting; the least-recently-used loaded model is unloaded to make room — with one rule: a model actively serving a request is never evicted mid-request.
• Explicit control — POST /models/load / POST /models/unload with {"model": "<id>"}, plus DELETE /models?model=<id> to remove a cache-registered model entirely. GET /models?reload=1 re-reads the preset INI (see above).
• Preset vs cache — source: "preset" means the id has a section in the INI (only place a per-model ctx window can live); source: "cache" means it was discovered from the HF model cache. Both inherit the [*] defaults — per-model ctx-size override is the only reason to add a section; the embedding model needs one (pooling flags, above).
• Draft models — a speculative draft model (e.g. DFlash) is not a standalone loadable model; it is attached to its target via spec-* args, so it never owns a slot.

The app uses only autoload + ?reload=1 today; explicit load/unload is available from the terminal for memory management.

Advanced Sampling Parameters

Both providers expose the full 20-key llama.cpp sampler set — Ollama forwards all params via its options payload, and LlamaServerBackend maps each key into the router's OpenAI-compatible request body. The Parameters panel in the Tauri UI shows the full slider set for both providers.

Vision Format Adapter

Different providers use different image formats in multimodal messages:

Provider	Format
Ollama	"images": [bytes] key in message dict
llama.cpp server (router)	OpenAI content blocks with image_url (adapter converts from Ollama form)

_normalize_vision_messages() in vision/image_processing.py handles the conversion automatically. When _vision_backend is None (Ollama), messages pass through unchanged.

LlamaServerBackend is additionally self-contained for the main agent chat path: the agent's auto-screenshot attaches Ollama-style images: [bytes] to the user message (agent_classes.py), and the backend converts those into OpenAI content blocks before serialising the request body. Without this, the raw bytes would break json.dumps() of the OpenAI body. (embed() is unaffected — text-only.)

Backend Injection

During startup (main.py), the created backend is injected into multiple subsystems:

1. ModelManager -- create_model_manager(backend=_model_backend) -- ensures list_local_models() queries the configured provider, not always Ollama
2. Agent factory -- build_agents(config, model_backend=backend, tool_host=tool_host)
3. Embedding -- set_embedding_backend(backend) in core/data/embedding.py (falls back to Ollama if NotImplementedError)
4. Vision -- set_vision_backend(backend) in vision/image_processing.py

All subsystems use the provider-appropriate format automatically after injection. Backend creation happens early (after EventBus setup, before ModelManager) so all downstream components receive the correct backend.

Runtime Hot-Switch

Provider switching works at runtime without restarting main.py. When you change the provider in the Tauri UI or via the set_provider API command, the handler:

1. Creates a new backend -- For ollama, creates the Ollama client (instant); for llama_server, creates an httpx.AsyncClient pointed at the router URL (instant).
2. Cancels in-flight generation -- shared_state.generation_cancel_event.set() signals any running agent to stop streaming. The agent's token loop checks this event every iteration (agent_classes.py:989).
3. Swaps all consumer references:
4. _model_backend module global (main.py)
5. agent.model_backend for each agent instance
6. _embedding_backend via set_embedding_backend()
7. _vision_backend via set_vision_backend()
8. _model_manager.backend on the ModelManager instance
9. Runs a non-blocking health check -- 3-second timeout via asyncio.wait_for. On success, logs "Hot-swap to {provider} (healthy: ...)". On failure (timeout, connection refused), logs a warning but proceeds -- the error will surface on the first agent call.
10. Waits for generation to settle -- polls shared_state.agent_generating (max 10 seconds), then clears the cancel event.

The response message indicates the outcome: "switched to {provider}" on success, or "restart required" if the swap threw an exception (very rare -- e.g., factory returns unknown backend type).

Frontend: ModelsPanel Reactivity

The ModelsPanel in the Tauri desktop UI is provider-reactive:

• Catalog refetch -- useEffect([provider]) dependency ensures the model catalog refetches when the provider changes
• Cloud role filtering -- language_model_cloud and vision_model_cloud roles are hidden for non-ollama providers (Ollama-specific cloud endpoints)
• Draft model is backend-only -- draft_model (speculative decoding for the llama_server provider) remains a valid backend role in get_models/set_model, but is not exposed as a ModelsPanel UI control
• Loading state -- Displays "Loading models from {provider}..." while fetching, with AbortController cleanup on unmount/provider change
• Model availability -- Models not in the catalog are marked (unavailable: name) with a disabled dropdown option
• Purpose filtering -- Embedding models only appear for the embedding_model role; vision models for vision roles

Fallback Behavior

For backward compatibility during the transition period, all code paths fall back to direct Ollama calls when model_backend is None. This means:

• Existing tests that don't inject a backend continue to work
• config.llm_provider = "ollama" (default) produces zero behavioral change
• The system degrades gracefully if the factory cannot create a backend