Skip to content

Audio Processing

The audio pipeline handles three operations concurrently: recording user speech, transcribing audio to text, and synthesizing agent responses back into speech. All three run on separate asyncio roles to avoid blocking the main event loop.

Recording Pipeline

The microphone recording runs in a dedicated asyncio task. Audio is captured in chunks using sounddevice, buffered, and written to disk for transcription.

Parameters

Recording duration adapts based on active modes:

Mode Duration (seconds) Behavior
Default 10 Standard recording window
mute_mode 30 Extended recording for stacking multiple utterances
auto_chat_mode 5 Shorter windows for ambient conversation detection

Transcript Filtering

After transcription, single-word utterances (< 2 words) are discarded as noise. VAD segments require at least 2 words to be accepted. This threshold (lowered from 3→2) prevents discarding legitimate short utterances like "El turismo." or "Thank you!" while still filtering coughs, background sounds, and accidental microphone triggers.

Voice Activity Detection (VAD)

Silero VAD — Neural Speech Detection

By default, the recording loop uses Silero VAD (neural network-based voice activity detection) instead of simple RMS energy thresholding. Silero provides:

  • LSTM stateful context — remembers acoustic patterns across chunks
  • Fewer false positives from background noise
  • Better detection of quiet speech
  • Merge window logic — short pauses (< 300ms) within an utterance are not treated as segment boundaries

Segment Accumulation

The SpeechSegmentAccumulator state machine collects VAD-classified audio chunks into complete utterance segments:

Parameter Default Purpose
chunk_ms 32ms Size of each VAD input slice (512 samples at 16 kHz)
min_chunks 24 Minimum speech chunks before committing a segment (~768ms)
merge_window_ms 300ms Sub-threshold gaps within speech are merged, not split (~9 chunks)

When a silence boundary is detected after enough speech chunks, the accumulated segment is written to a temporary WAV file and transcribed immediately — rather than waiting for the entire recording window to expire.

RMS Fallback

If silero_vad is not installed or fails to load, the system falls back to the existing RMS energy thresholding (rms >= THRESHOLD). The fallback is automatic and transparent:

if vad_backend is not None and vad_backend.is_available():
    # Neural VAD detection
    result = vad_backend.detect(sub_chunk)
else:
    # Legacy RMS fallback
    rms = audio_rms(data, 2)
    is_speech = rms >= THRESHOLD

Chunk Subdivision

PyAudio reads CHUNK=8192 samples (~512ms at 16kHz), but Silero VAD expects 32ms chunks (512 samples). The subdivide_for_vad() helper splits each PyAudio read into ~16 VAD-sized slices, preserving all byte content so concatenation reconstructs the original data.

Streaming Segment Callback

When a VAD-detected segment completes transcription mid-recording, the on_segment_complete callback fires:

  1. Event publication — USER_SEGMENT_COMPLETE event via EventBus (real-time UI updates)
  2. Sentence splitting — transcript broken into TTS-ready chunks via split_buffer_into_sentences(segment_text, flush=True)
  3. Immediate TTS playback — each sentence (≥2 words) is synthesized and played using the active agent's voice

The callback supports both sync and async functions. Failures are caught and logged — a broken callback never interrupts the recording loop.

VAD Silence Gate Bypass

Both transcribe_audio() and STTBackendAdapter.transcribe() have RMS-based silence gates (wav_is_silent() / _is_silent()) that skip expensive GPU inference when audio is silent or too short. However, when Silero VAD (neural network) has already confirmed speech, a redundant RMS check can block valid but quiet voice from reaching the STT model.

The bypass system covers all transcription paths:

  1. VAD segments: transcribe_segment() passes skip_silence_gate=True to transcribe_audio(), completely skipping the wav_is_silent() check for segments already validated by VAD.
  2. Voice recording WAV (voice_recording*.wav): A _vad_speech_detected flag is set when VAD first detects speech in a recording cycle. aggregate_user_input() in main.py reads this flag and passes skip_silence_gate=True to transcribe_audio().
  3. Loopback transcription (audio_transcript_output*.wav): record_audio_output() uses skip_silence_gate=False — loopback has its own independent RMS silence gate, not bypassed by microphone VAD state.
  4. RMS fallback inline: When the RMS silence boundary triggers (long silence after speech), inline transcription in record_audio() passes skip_silence_gate=_vad_speech_detected.
  5. STT adapter gate: STTBackendAdapter.transcribe() checks the VAD flag via _vad_speech_was_detected() and bypasses its own _is_silent() gate.

Critical architecture note: main.py has a local copy of aggregate_user_input() that shadows the refactored version in conversation_controller.py. Both must be updated independently — missing one was the root cause of quiet voice being silently dropped after VAD detection.

VAD Segment Deduplication

When VAD successfully transcribes speech segments into user_text_list, the voice recording WAV contains the exact same audio. Without deduplication, aggregate_user_input() would join the VAD segments AND transcribe the WAV, producing concatenated duplicates (e.g., "hello world hello world").

The fix checks two conditions before transcribing WAV files:

  1. _vad_speech_detected is True — VAD was active and detected speech this recording cycle
  2. user_text_list has content — VAD segments were actually captured (≥ min_chunks threshold)

When both are true, WAV transcription is skipped. If either condition fails (VAD inactive, or speech too brief to form a segment), the WAV transcription still runs as a rescue path.

Speech-to-Text (STT)

Parakeet-TDT Backend

The default STT backend uses Parakeet-TDT via HuggingFace transformers >= 5.0 AutoModelForTDT. It provides high-quality transcription with timing diarization (speaker identification).

# In main.py lifecycle: load_audio_model()
stt_backend = ParakeetTDTBackend(model_name="parakeet-tdt")
adapter = STTBackendAdapter(stt_backend)

Thread Safety

ParakeetForTDT.generate() uses instance-level mutable state (_encoder_finished, _step_durations, _symbols_at_frame) that is not thread-safe. When multiple transcription paths fire concurrently (e.g., a VAD segment transcription and a final recording transcription on different executor threads), one thread's cleanup can delete an attribute another thread still needs, resulting in AttributeError: '_encoder_finished'.

To prevent this, ParakeetTDTBackend wraps _transcribe_file() in a threading.Lock (_inference_lock) that serializes all model inference. The lock scope covers the entire inference pipeline (audio loading → preprocessing → model.generate() → decoding), ensuring only one thread accesses the model's mutable state at a time.

Fallback Chain

If Parakeet-TDT is unavailable (e.g., transformers not installed), the system falls back to FallbackSTTBackend which returns empty transcripts silently. The agent continues operating in text-only mode.

Transformers v5 Required

Parakeet-TDT requires transformers >= 5.0 for AutoModelForTDT. Earlier versions (4.x) do not support this model type. Install with uv pip install "transformers>=5.0".

Transcription Flow

Microphone → PyAudio chunks (8192 samples) → VAD subdivision (512 samples)
    → SpeechSegmentAccumulator (collects speech segments)
    → On silence: temp WAV → STT adapter → Parakeet-TDT → delta transcript
    → on_segment_complete callback → EVENT + TTS preview

The STTBackendAdapter bridges the async STT interface to synchronous file-path operations. Critical: aggregate_user_input() must be called asynchronously, and transcribe_audio() runs via run_in_executor() to avoid blocking the event loop.

With VAD streaming, transcription happens incrementally on each silence boundary rather than waiting for the full recording window. This provides lower-latency feedback: the agent can begin responding before the user finishes speaking (if they pause between thoughts).

Text-to-Speech (TTS)

Chatterbox Faster Fork

The TTS engine is a custom fork of Chatterbox from the faster branch. Key differences from the default branch:

Feature Default Branch Faster Fork
t3_params Not supported Accepted — spread into t3.inference() for optimization
max_cache_len Fixed Configurable (default 1500)
emotion_adv dtype Incorrect Fixed with dtype=ve_embed.dtype
language_id Not present Added parameter

Initialization

from chatterbox.tts import ChatterboxTTS

tts = ChatterboxTTS.from_pretrained(device="cuda:0")
t3_to(tts, torch.bfloat16)  # Cast T3 weights to bfloat16 for performance

The t3_to() helper in main.py casts T3 model weights and condition vectors to bfloat16, significantly improving inference speed on modern GPUs.

Typical Generate Call

audio = tts.generate(
    text="Hello, how can I help you?",
    t3_params={"initial_forward_pass_backend": "cudagraphs-manual"},
    audio_prompt_path="voice_samples/default.wav",
    exaggeration=agent.exaggeration,
    cfg_weight=agent.cfg_weight,
)
Parameter Purpose
t3_params Optimization flags for T3 inference engine
audio_prompt_path Reference voice sample for cloning
exaggeration Emotional expression intensity (0.0–1.0)
cfg_weight Classifier-free guidance weight for quality

Sentence Splitting

split_buffer_into_sentences() in audio/text_processing.py breaks agent output into TTS-ready chunks:

  • Streaming mode: Requires whitespace after punctuation (fast, used during generation)
  • Flush mode: Treats end-of-string as a boundary (used for final cleanup)

The splitter handles abbreviations ("Dr.", "U.S.A."), decimal numbers ("3.14"), and edge cases gracefully.

Interrupt Handling

Voice Interruption

When the user speaks while TTS is playing, the new speech triggers a recording cycle which naturally interrupts pending audio playback:

  1. tts_interrupt_event in SharedState is checked during playback — if set, synthesis stops
  2. audio_playback_lock is released, unblocking the next turn
  3. The main loop begins recording for the new utterance

Active interrupt monitoring disabled

An energy-based interrupt monitor (monitor_user_interrupt in audio/audio_processing.py) exists but is not wired into production — the asyncio.create_task(...) call in main.py is commented out. Only the explicit stop_generation API command and natural recording-cycle overlap currently trigger interruptions.

Desktop Text Interruption

When a text message arrives from the Tauri frontend:

  1. stop_recording_event is set — breaks the recording loop immediately
  2. text_only_flag is set — tells the main loop to skip audio transcription
  3. The typed text is processed directly by the agent pipeline

This enables instant switching between voice and text input without waiting for the current recording window to expire.

Screenshot Captioning

Periodic screenshots are captured during recording sessions. Each screenshot is sent to Ollama for captioning, and the descriptions are streamed to screenshot_description.txt for the agent to read as visual context.

Telemetry

Metric Description
audio.stt_ms End-to-end transcription time (with model label)
audio.tts_ms Synthesis time per sentence

Both metrics are recorded in the in-memory MetricsCollector singleton with zero-overhead failure handling.

See Also