HuggingFace has released a repository called speech-to-speech that lets you build a complete voice assistant — the kind that listens, thinks, and talks back — entirely from open-source models [S2]. The project implements a four-stage pipeline: Voice Activity Detection, Speech-to-Text, a language model, and Text-to-Speech, each running in its own thread and connected by queues. Every single stage is swappable via command-line flags [S1][S2]. The repository already runs in production as the conversation backend for thousands of Reachy Mini robots [S2], and it was GitHub’s #1 trending repository of the day as of its latest README update [S2]. The install is a single line: pip install speech-to-speech [S2].
This is a development worth your attention if you build voice interfaces, work with local AI, or want to avoid lock-in to a single commercial voice API. Here is what the repository actually does, what changed and what did not, and whether it is worth your time.
What the Pipeline Actually Does
The speech-to-speech project is a cascade of four components, each running in its own thread and connected by queues [S2]:
- Voice Activity Detection (VAD): Silero VAD v5 detects speech boundaries and turn-taking. This is the component that decides when a user has started and stopped speaking, which is foundational for any conversational agent.
- Speech-to-Text (STT): The default is Parakeet TDT (nvidia/parakeet-tdt-0.6b-v3), a fast transcription model. The pipeline supports partial live transcripts so a downstream LLM can begin reasoning before the user finishes a turn.
- Language Model (LLM): The LLM generates the response, streaming text and tool calls. Critically, the LLM slot speaks OpenAI-compatible protocols, so you can point it at hosted OpenAI, HuggingFace Inference Providers, OpenRouter, a self-hosted vLLM server, or a local llama.cpp instance [S2].
- Text-to-Speech (TTS): The default is Qwen3-TTS (Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice), with optional backends including Kokoro-82M, Pocket TTS, ChatTTS, and MMS TTS installed via pip extras [S2].
Each stage has multiple interchangeable backends, selected via --stt, --llm_backend, and --tts flags [S2]. The code is designed for modification, with a focus on models available through Transformers and the Hugging Face Hub.
How It Runs and What Is Swappable
The project exposes itself through an OpenAI Realtime-compatible WebSocket API at ws://localhost:8765/v1/realtime [S2]. This is the key architectural decision: any client built against OpenAI’s Realtime voice protocol can connect to your self-hosted pipeline by switching an endpoint. The README explicitly frames this as “switching an OpenAI Realtime client endpoint from hosted OpenAI to a self-hosted speech-to-speech server” [S2].
The run modes matter because they map to different deployment scenarios [S2]:
- realtime (default): OpenAI Realtime protocol over WebSocket or WebRTC. This is for building apps or devices against a standard voice API.
- local: microphone and speakers. This is for direct, on-device use.
- raw-websocket and socket (TCP): alternative transports for custom integrations.
A fully local stack is genuinely possible. Instead of calling a commercial API, you can serve Gemma 4 with llama.cpp on your own hardware and point the pipeline at the local server [S2]. On macOS, the --local_mac_optimal_settings flag selects MPS, MLX LM, and MLX audio backends tuned for Apple Silicon [S2]. There is also an LLM proxy mode that exposes the configured LLM as an OpenAI-compatible endpoint for side tasks, which means the same voice pipeline can serve as a general-purpose local model server [S2].
The pipeline supports multi-language operation and includes an optional DeepFilterNet audio enhancement layer for the VAD stage, though the README notes a numpy version conflict between DeepFilterNet and Pocket TTS that requires manual handling [S2].
The Reachy Mini Production Signal
The README states plainly that “this pipeline runs in production as the conversation backend for thousands of Reachy Mini robots” [S2]. This is not a laboratory demo; it is a live fleet.
Reachy Mini is an open-source robot built by Pollen Robotics and HuggingFace, priced from $399, equipped with four microphones and a 5W speaker, and designed for human-robot interaction, creative coding, and AI experimentation [S3]. The robot is programmable in Python, comes as a kit, and uses HuggingFace’s open-source models for speech, vision, and personality [S3]. The robot blog was published on July 9, 2025, and describes the platform as covering hardware, software, and simulation environments, all released as open-source [S3].
The connection between the two projects is significant. A robot priced at $399 with four microphones and a speaker needs a conversation backend that is cheap to run, modifiable, and not locked to a commercial voice API with per-minute pricing. The speech-to-speech pipeline is that backend. If it can handle a fleet of thousands of conversation-driven robots, it is past the threshold of “interesting prototype” [S2][S3].
The repository’s community traction reinforces this. As of August 4, 2026, the GitHub repository has 10,943 stars, 1,349 forks, and 132 open issues [S2]. It was created on August 7, 2024, meaning it reached over 10,000 stars in roughly a year [S2]. The PyPI package is at version 0.2.11 and requires Python 3.10 or later [S2].
What to Watch and the Open Question
What did not change: voice agents are not new. Commercial offerings from OpenAI, Google, and others already provide end-to-end voice pipelines. What changed is that HuggingFace has produced an open, Apache-2.0-licensed reference architecture where every stage is replaceable, the API surface mimics OpenAI’s Realtime protocol, and it is proven in a production robot fleet [S2][S3].
The practical opportunity is real but conditional. If your use case needs a voice interface that runs on your own hardware, uses models you control, and avoids per-call pricing, this pipeline is a credible starting point. If your use case demands ultra-low latency or enterprise-grade reliability guarantees, you will need to benchmark and tune — the defaults are sensible but not necessarily optimal for every workload.
The honest risk is latency and model quality. The default STT, LLM, and TTS models are strong, but matching a commercial voice agent’s end-to-end responsiveness on commodity hardware may require tuning, a capable GPU, or substituting components. The CUDA notes in the README — which explain that the default Qwen3-TTS wheel targets CUDA 12.8 and that mismatched CUDA versions require a manual wheel install — are a reminder that “open and local” does not mean “zero configuration” [S2].
The Bottom Line: Is It Worth Your Time?
This project earns a 4/5 — Strong Look. It is an open-source, Apache-2.0-licensed, production-proven voice agent pipeline with a standard API surface, every component swappable, and a one-line install. The production deployment in thousands of Reachy Mini robots is evidence, not marketing. The reason it is not a 5/5 is that voice agent quality is highly dependent on your hardware, your latency budget, and your willingness to tune model defaults — the pipeline gives you the architecture, but the operational work falls to you. For the right developer, this is a strong starting point that removes vendor lock-in without removing the engineering effort.
Best use of your time today: Run pip install speech-to-speech, start the default server, and connect any OpenAI Realtime-compatible client to ws://localhost:8765/v1/realtime. The entire default pipeline runs locally in minutes. Safe to skip if you have no use case for a voice interface, no GPU or Apple Silicon machine, or an existing commercial voice stack that already meets your latency and quality requirements.
Sources
- [S1] Hugging Face put out a repo that lets you build a full voice assistant… — X / @techNmak (2026-08-03)
- [S2] Speech To Speech: Build voice agents with open-source models (huggingface/speech-to-speech README) — GitHub / huggingface (2026-08-04)
- [S3] Reachy Mini – The Open-Source Robot for Today's and Tomorrow's AI Builders — Hugging Face Blog (Thomas Wolf, Matthieu Lapeyre, Pollen Robotics) (2025-07-09)