Text → Speech & Music
ACE-Step 3.5B (full songs with vocals, Suno-compatible), Chatterbox (expressive with emotion/style), Kokoro-82M (fast multilingual), Piper fallback. All local, streamed or exported, with ffmpeg mixing for beds + FX.
Feature · Generation
Audio Foundry
Full local audio production studio: ACE-Step 3.5B full-song music (vocals, Suno-style prompts + polish), Chatterbox expressive neural TTS (style/emotion), Kokoro-82M multilingual, Stable Audio Open SFX/ambience, Piper fallback + Whisper.cpp STT, ffmpeg mixing & streaming. Dual-venv isolation, Celery batch jobs, zero cloud.
Speech → Text
Whisper.cpp — compiled from source on first startup — delivers fast, accurate transcription without a network connection. Feed it meeting recordings, interviews, or voice memos and get clean timestamped transcripts ready for editing or RAG indexing.
Stitch & Layer
Chain narration clips, music beds, and sound effects into a finished audio file using the integrated ffmpeg pipeline. Trim silence, set crossfades, control per-track volume levels, and export a single polished output.
Batch & Automate
Queue multiple audio jobs as Celery background tasks. Generate an entire podcast series from a script, transcribe a folder of recordings overnight, or batch-produce narration for every article in your content library without tying up the UI.
A complete local audio production pipeline
Audio Foundry brings together the three components already woven into Guaardvark's core — Whisper.cpp for speech-to-text, Piper TTS for text-to-speech, and ffmpeg for audio processing — into a single cohesive surface. Rather than wiring these tools together yourself on the command line, the Audio Foundry UI and API handle the plumbing: format detection, sample-rate normalisation, silence trimming, and multi-track mixing are all handled behind the scenes.
The transcription side is powered by Whisper.cpp built from source during start.sh's first-run sequence. The binary lives at backend/tools/voice/whisper.cpp/build/bin/whisper-cli alongside the model weights in backend/tools/voice/whisper.cpp/models/. Because the binary runs natively on your CPU (or GPU with CUDA if available), even large model variants complete in a fraction of real time. Transcripts can be automatically ingested into LlamaIndex via the Documents surface, making every word searchable and available for RAG-grounded chat.
On the synthesis side, Piper TTS accepts plain text or SSML-annotated input and returns a WAV stream through the same WebSocket path used by live voice chat. For offline narration generation you call the same voice_api.py endpoint in non-streaming mode, which deposits the finished file directly into data/outputs/ and registers it in the Content Library. The ffmpeg layer then takes any collection of output files and applies your mix spec — specifying which tracks are foreground narration, which are ambient music beds, and where chapter titles should appear. The result is a single export-ready audio file, created entirely on your own hardware.
Under the hood
Voice pipeline stack. The speech processing chain is declared in backend/api/voice_api.py and uses two backing services: backend/tools/voice/ for the Whisper.cpp and Piper TTS binaries, and ffmpeg (required system dependency, validated at startup). Microphone audio arrives as WebSocket chunks, gets written to a temporary WAV file, is passed to the Whisper CLI binary, and the resulting transcript text is returned over the same socket. TTS works in reverse — text arrives, Piper outputs a WAV byte stream, which is forwarded through the socket to the browser audio context for playback, or saved to disk for offline use.
Background job execution. Long-running audio tasks — bulk transcription passes, multi-chapter narration generation, and multi-track mixing — run as Celery workers backed by Redis. Progress events are emitted over Socket.IO so the dashboard and job panel update in real time. Every generated audio file is catalogued in the data/outputs/ directory and linked to a Job record in the SQLite database, so you can retrieve, re-download, or delete outputs at any time from the Content Library. The unified_task_executor.py dispatcher handles queue prioritisation so audio jobs don't block image or video generation.
# Example: transcribe a file via the llx CLI
llx generate audio transcribe meeting.mp4 --output transcript.md
# Example: generate narration from text
llx generate audio narrate article.md --voice en_US-lessac-medium --output narration.wav
# Example: mix narration + music
llx generate audio mix narration.wav music.mp3 --narration-vol 1.0 --music-vol 0.2 --output final.mp3Who uses Audio Foundry and why
Podcast & explainer video production
Write a script in Guaardvark's Documents editor, send it to Piper TTS for narration, combine the audio with a music bed in ffmpeg, and hand the finished MP3 to the Video Editor for a full talking-head or slides-over-narration video — without touching a single cloud service. Each step is a Celery job, so you can queue an entire episode batch and come back to finished files.
Pairs with: Video Editor, Video Text Overlay
Meeting transcription & knowledge extraction
Drop a folder of MP4 or MP3 recordings into Guaardvark, run a batch transcription job, and have every word indexed into your LlamaIndex vector store. The resulting transcripts become first-class Documents, searchable via semantic query and available for RAG-grounded chat. Ask "what did we decide about the Q3 roadmap?" and get an answer with citations back to the exact recording timestamp.
Accessible content & silent-autoplay captions
Use Whisper.cpp to auto-transcribe any video clip, then pipe the transcript to Video Text Overlay to burn captions directly into the video file. The result plays clearly without sound — critical for social media autoplay and accessibility compliance. The same transcript can be saved as a subtitle file or ingested into the content library for reuse.
Pairs with: Video Text Overlay, Content Library
Guaardvark vs. cloud audio APIs
Cloud transcription and TTS services charge per minute of audio, impose rate limits, and send your voice data to third-party servers. Guaardvark Audio Foundry runs Whisper.cpp and Piper TTS on your own machine: the cost per hour of audio is zero beyond electricity, there are no rate limits, and your recordings never leave your hardware. For teams handling sensitive conversations — legal, medical, financial, or simply private — that constraint is non-negotiable. Batch processing is limited only by your CPU or GPU, not by API concurrency quotas. And because everything runs through the same Celery queue as image and video generation, you get unified job management, progress tracking, and output archiving across every media type.
Audio Foundry — common questions
Which Whisper model sizes are supported?
Guaardvark downloads and manages Whisper.cpp model weights in backend/tools/voice/whisper.cpp/models/. Tiny, base, small, medium, and large variants are all supported. Tiny runs fastest on CPU; large gives the highest accuracy. Model selection is configurable in Settings → Voice, and the startup script builds the Whisper.cpp binary from source automatically on first run so you don't need to handle the C++ toolchain manually.
What audio formats are accepted for transcription?
Whisper.cpp natively processes WAV. Guaardvark uses ffmpeg to pre-convert any format ffmpeg understands — MP3, MP4, M4A, FLAC, OGG, WEBM — before handing audio to Whisper. If ffmpeg can decode it, Audio Foundry can transcribe it. Conversion is handled transparently; you just upload the file.
Can I use custom Piper TTS voice models?
Yes. Piper TTS supports ONNX voice models. Place your model files in the Piper models directory and they appear in the voice selector dropdown. The Guaardvark Settings panel exposes which voice models are available, and the llx generate audio narrate command accepts a --voice flag for scripted workflows.
Does audio processing require a GPU?
No. Whisper.cpp and Piper TTS both run on CPU. GPU acceleration is available for Whisper.cpp if CUDA is present, which can reduce transcription time significantly for large files. ffmpeg audio mixing is always CPU-based. The system detects available hardware at startup via backend/services/hardware_service.py and routes accordingly.
How do I access generated audio files?
All generated audio outputs are saved to data/outputs/ and registered in the database as Job output records. You can browse them in the Content Library, download them directly from the Files page, or retrieve them via llx files list in the terminal. Each file is linked to the job that created it, so you can trace any output back to the parameters that produced it.