Skip to content

core.speech

Speech modules: STT backend implementations, async adapter, and speech backend abstract interfaces.

STT Backends

core.speech.stt_backends

Concrete STT backend implementations for Vector Companion.

Provides ParakeetTDTBackend (Transformers-based NVIDIA Parakeet-TDT-0.6B-v3), Qwen3ASRBackend (wrapper around existing qwen_asr.Qwen3ASRModel), and FallbackSTTBackend (ordered chain of backends).

All implementations conform to the STTBackend abstract interface defined in core/speech_backends.py.

FallbackSTTBackend

Bases: STTBackend

Chains multiple STT backends, trying each in order.

If the first backend fails (import error, model load failure, inference error), falls back to the next one. Raises RuntimeError only if ALL backends fail.

ParakeetTDTBackend

Bases: STTBackend

NVIDIA Parakeet-TDT-0.6B-v3 via HuggingFace Transformers.

Uses AutoModelForTDT and AutoProcessor for cross-platform compatibility (works on Windows where NeMo is not officially supported).

Model loading is lazy - happens on the first transcribe() call in a background executor to avoid blocking startup.

transcribe(audio_data: bytes, **kwargs: Any) -> STTResult async

Transcribe audio data using Parakeet-TDT model.

Parameters

audio_data : Raw WAV file bytes (16kHz mono expected). **kwargs : Additional arguments (ignored).

Returns

STTResult with transcribed text.

Qwen3ASRBackend

Bases: STTBackend

Wrapper around existing qwen_asr.Qwen3ASRModel.

Accepts an already-loaded model instance (typically created by lifecycle manager or main.py). This preserves all existing Qwen3-ASR behavior including language detection and confidence scoring.

transcribe(audio_data: bytes, **kwargs: Any) -> STTResult async

Transcribe audio using Qwen3-ASR model.

Writes bytes to temp file, calls the existing model.transcribe() API.

create_stt_backend(name: str, model_name: str = '', device: str | None = None) -> STTBackend | None

Create a named STT backend instance.

Parameters

name : Backend identifier ("parakeet_tdt_v3" or "qwen3_asr"). model_name : Model name/identifier (e.g. "nvidia/parakeet-tdt-0.6b-v3"). device : Device string ("cuda", "cuda:0", "cpu").

Returns

STTBackend instance or None if backend is unavailable.

STT Adapter

core.speech.stt_adapter

STTBackendAdapter — bridges async STTBackend to the legacy Qwen3ASRModel interface.

Existing code (audio_processing.py::transcribe_audio) calls: result = model.transcribe(audio=filepath, language=None) text = result[0].text

This adapter wraps any STTBackend instance and presents that same sync file-path interface, so call sites require ZERO changes.

Event loop handling: - Most call sites wrap transcription in run_in_executor(None, lambda: ...) which runs the adapter on a worker thread with no event loop. The adapter creates a temporary event loop in this case. - If somehow called on a thread with a running loop, the adapter uses asyncio.run_coroutine_threadsafe() to schedule the async transcribe.

STTBackendAdapter

Wraps STTBackend to present Qwen3ASRModel-compatible interface.

Usage

adapter = STTBackendAdapter(backend) result = adapter.transcribe(audio="/path/to/file.wav", language="en") text = result[0].text # "hello world"

backend: STTBackend property

Access the wrapped backend.

is_available() -> bool

Delegate to wrapped backend.

transcribe(audio: str, language: str | None = None) -> list[_SimpleResult]

Transcribe a WAV file, returning Qwen3ASRModel-compatible result.

Skips transcription if the WAV file is silent or too short, saving ~3s of GPU inference on empty recordings.

When VAD (Silero) has already confirmed speech during the recording cycle, the RMS silence gate is bypassed to prevent quiet voice from being filtered out after neural detection validated it as speech.

Parameters

audio : File path to WAV file. language : Optional language hint (passed through; some backends ignore).

Returns

list[_SimpleResult] with at least one element having a .text attribute.

Speech Backends

core.speech.speech_backends

Speech backend interfaces for Vector Companion.

Defines abstract interfaces for STT, VAD, and TTS backends. Provides a factory system for backend selection and a registry for custom implementations. The current working path (Qwen3-ASR + Chatterbox) is preserved as the default until replacements are validated on Windows.

Components

  • SpeechBackendType — enum of backend categories
  • STTBackend — speech-to-text interface
  • VADBackend — voice activity detection interface
  • TTSBackend — text-to-speech interface
  • BackendInfo — metadata about an available backend
  • BackendRegistry — registry for backend implementations
  • create_stt_backend() — factory for STT backends
  • create_vad_backend() — factory for VAD backends
  • create_tts_backend() — factory for TTS backends

BackendInfo dataclass

Metadata about an available backend implementation.

Parameters

name : Backend identifier (e.g. "faster_whisper"). backend_type : Category (STT, VAD, TTS). is_available : Whether the backend is available in the current environment. priority : Selection priority (DEFAULT, PREFERRED, OPTIONAL). description : Human-readable description. requires_install : Whether the backend requires additional installation. metadata : Arbitrary additional context.

BackendPriority

Bases: Enum

Priority ordering for backend selection.

BackendRegistry

Registry for speech backend implementations.

get(name: str) -> BackendInfo | None classmethod

Get backend info by name.

get_preferred(backend_type: SpeechBackendType) -> BackendInfo | None classmethod

Get the highest-priority available backend for a type.

list_backends(backend_type: SpeechBackendType | None = None, available_only: bool = False) -> list[BackendInfo] classmethod

List registered backends, optionally filtered by type.

register(info: BackendInfo, factory: Callable[..., Any] | None = None) -> None classmethod

Register a backend implementation.

STTBackend

Bases: ABC

Abstract speech-to-text backend interface.

get_info() -> BackendInfo abstractmethod

Get backend metadata.

is_available() -> bool abstractmethod

Check if the backend is available.

transcribe(audio_data: bytes, **kwargs: Any) -> STTResult abstractmethod async

Transcribe audio data to text.

STTResult dataclass

Result of speech-to-text transcription.

Parameters

text : Transcribed text. confidence : Confidence score (0.0-1.0). duration_ms : Audio duration in milliseconds. language : Detected language code. word_timestamps : Optional word-level timestamps. metadata : Arbitrary additional context.

SpeechBackendType

Bases: Enum

Categories of speech backends.

TTSBackend

Bases: ABC

Abstract text-to-speech backend interface.

get_info() -> BackendInfo abstractmethod

Get backend metadata.

is_available() -> bool abstractmethod

Check if the backend is available.

synthesize(text: str, **kwargs: Any) -> TTSResult abstractmethod async

Synthesize text to speech.

TTSResult dataclass

Result of text-to-speech synthesis.

Parameters

audio_data : Synthesized audio as bytes. format : Audio format (wav, mp3, etc.). sample_rate : Sample rate in Hz. duration_ms : Synthesized audio duration in milliseconds. metadata : Arbitrary additional context.

VADBackend

Bases: ABC

Abstract voice activity detection backend interface.

detect(audio_data: bytes, **kwargs: Any) -> VADResult abstractmethod async

Detect voice activity in audio data.

get_info() -> BackendInfo abstractmethod

Get backend metadata.

is_available() -> bool abstractmethod

Check if the backend is available.

VADResult dataclass

Result of voice activity detection.

Parameters

is_speech : Whether speech is detected. confidence : Confidence score (0.0-1.0). start_ms : Start of speech segment (if applicable). end_ms : End of speech segment (if applicable). metadata : Arbitrary additional context.

register_default_backends() -> None

Register all known backends with the registry.