> ## Documentation Index > Fetch the complete documentation index at: https://docs.inceptionlabs.ai/llms.txt > Use this file to discover all available pages before exploring further. # Realtime Voice Assistant: Mercury + ElevenLabs > Stream Mercury 2 in ElevenLabs for a low-latency voice assistant. This notebook demonstrates how to build a low-latency voice assistant using Mercury 2 for real-time intelligence combined with ElevenLabs' speech-to-text and text-to-speech models. We walk you through how to: 1. Convert text to speech using ElevenLabs TTS 2. Convert audio to text using ElevenLabs speech-to-text 3. Generate realtime responses with Mercury 2 4. Optimize latency using Mercury 2 streaming Directly via Google Colab or download the `.ipynb` to run it locally in Jupyter *** ## Dependencies Install packages and import the libraries required for the Mercury 2 SDK, ElevenLabs, and audio playback: ```python theme={null} %pip install \ "inceptionai" \ "elevenlabs>=2.15.0" \ "sounddevice>=0.5.1" \ "numpy>=1.26.0" \ "scipy>=1.16.2" \ "python-dotenv>=1.0.0" \ "pydub>=0.25.1" ``` ```python theme={null} import io import os import time import elevenlabs from dotenv import load_dotenv from inceptionai import Inception from IPython.display import Audio ``` ## API Keys Set up your API keys for both ElevenLabs and Inception Labs. 1. Create a `.env` in this directory 2. Add your API keys to the `.env`: * `INCEPTION_API_KEY` * `ELEVENLABS_API_KEY` The keys will be automatically loaded from the `.env` file. ```python theme={null} # Load environment variables from .env file load_dotenv() INCEPTION_API_KEY = os.getenv("INCEPTION_API_KEY") ELEVENLABS_API_KEY = os.getenv("ELEVENLABS_API_KEY") ``` ## Initialize Clients Create the ElevenLabs and Inception (Mercury 2) clients: ```python theme={null} assert INCEPTION_API_KEY is not None, ( "ERROR: INCEPTION_API_KEY not found. Please copy .env.example to .env and add your API keys." ) assert ELEVENLABS_API_KEY is not None, ( "ERROR: ELEVENLABS_API_KEY not found. Please copy .env.example to .env and add your API keys." ) mercury_client = Inception(api_key=INCEPTION_API_KEY) elevenlabs_client = elevenlabs.ElevenLabs( api_key=ELEVENLABS_API_KEY, base_url="https://api.elevenlabs.io" ) MERCURY_MODEL = "mercury-2" VOICE_ID = "21m00Tcm4TlvDq8ikWAM" # Rachel def mercury_completion(user_text, max_tokens=1000, temperature=0.75): return mercury_client.chat.completions.create( model=MERCURY_MODEL, messages=[{"role": "user", "content": user_text}], max_tokens=max_tokens, temperature=temperature, reasoning_effort="low", ) def mercury_stream(user_text, max_tokens=1000, temperature=0.75): stream = mercury_client.chat.completions.create( model=MERCURY_MODEL, messages=[{"role": "user", "content": user_text}], max_tokens=max_tokens, temperature=temperature, reasoning_effort="low", stream=True, ) for chunk in stream: if not chunk.choices: continue text = chunk.choices[0].delta.content if text: yield text ``` ## Create Input Audio We can simulate a user audio input using ElevenLabs text-to-speech model: ```python theme={null} audio = elevenlabs_client.text_to_speech.convert( voice_id=VOICE_ID, output_format="mp3_44100_128", model_id="eleven_v3", text="Hi, Mercury. ", ) audio_data = io.BytesIO() for chunk in audio: audio_data.write(chunk) Audio(audio_data.getvalue()) ``` ## Transcribe Speech into Text We then transcribe the audio input using ElevenLabs' speech-to-text model. We'll also measure the transcription latency: ```python theme={null} audio_data.seek(0) start_time = time.time() transcription = elevenlabs_client.speech_to_text.convert(file=audio_data, model_id="scribe_v1") end_time = time.time() transcription_time = end_time - start_time print(f"Transcribed text: {transcription.text}") print(f"Transcription time: {transcription_time:.2f} seconds") ``` ## Get a Response from Mercury 2 We send the transcribed text to Mercury 2 and measure the response time. We're using `mercury-2` for fast reasoning responses: ```python theme={null} start_time = time.time() completion = mercury_completion(transcription.text, max_tokens=1000, temperature=0.75) mercury_response = completion.choices[0].message.content end_time = time.time() non_streaming_response_time = end_time - start_time print(mercury_response) print(f"\nResponse time: {non_streaming_response_time:.2f} seconds") ``` ## Real-time with Streaming Improve response latency by using Mercury 2 streaming. This allows us to receive the first tokens more quickly, reducing user-perceived latency: ```python theme={null} start_time = time.time() first_token_time = None mercury_full_response = "" for text in mercury_stream(transcription.text, max_tokens=1000, temperature=0.75): mercury_full_response += text print(text, end="", flush=True) if first_token_time is None: first_token_time = time.time() streaming_time_to_first_token = first_token_time - start_time print( f"\n\nStreaming time to first token: {streaming_time_to_first_token:.2f} seconds - reducing response wait by {(non_streaming_response_time - streaming_time_to_first_token) * 100 / non_streaming_response_time:.2f}%" ) ``` We can also stream the text-to-speech response to reduce silence: ```python theme={null} start_time = time.time() first_audio_chunk_time = None audio_buffer = io.BytesIO() audio_generator = elevenlabs_client.text_to_speech.stream( voice_id=VOICE_ID, output_format="mp3_44100_128", text=mercury_full_response, model_id="eleven_turbo_v2_5", ) for chunk in audio_generator: if first_audio_chunk_time is None: first_audio_chunk_time = time.time() audio_buffer.write(chunk) streaming_tts_time_to_first_chunk = first_audio_chunk_time - start_time print(f"Streaming TTS time to first audio chunk: {streaming_tts_time_to_first_chunk:.2f} seconds") Audio(audio_buffer.getvalue()) ``` ## Streaming Mercury 2 Directly to TTS We've optimized Mercury 2 streaming and TTS separately. Now we can stream the model response and convert it to audio as soon as we have complete sentences, demarcated by `!`, `.`, `?`. ```python theme={null} import re sentence_pattern = re.compile(r"[.!?]+") sentence_buffer = "" audio_chunks = [] start_time = time.time() first_audio_time = None for text in mercury_stream(transcription.text, max_tokens=1000, temperature=0.75): print(text, end="", flush=True) sentence_buffer += text if sentence_pattern.search(sentence_buffer): sentences = sentence_pattern.split(sentence_buffer) # Process all complete sentences (all but the last element) for i in range(len(sentences) - 1): complete_sentence = sentences[i].strip() if complete_sentence: audio_gen = elevenlabs_client.text_to_speech.stream( voice_id=VOICE_ID, output_format="mp3_44100_128", text=complete_sentence, model_id="eleven_turbo_v2_5", ) sentence_audio = io.BytesIO() for chunk in audio_gen: if first_audio_time is None: first_audio_time = time.time() sentence_audio.write(chunk) audio_chunks.append(sentence_audio.getvalue()) sentence_buffer = sentences[-1] if sentence_buffer.strip(): audio_gen = elevenlabs_client.text_to_speech.stream( voice_id=VOICE_ID, output_format="mp3_44100_128", text=sentence_buffer.strip(), model_id="eleven_turbo_v2_5", ) sentence_audio = io.BytesIO() for chunk in audio_gen: if first_audio_time is None: first_audio_time = time.time() sentence_audio.write(chunk) audio_chunks.append(sentence_audio.getvalue()) sentence_streaming_time_to_first_audio = first_audio_time - start_time print(f"\n\nTime to first audio: {sentence_streaming_time_to_first_audio:.2f} seconds") combined_pcm = b"".join(audio_chunks) Audio(combined_pcm) ``` In this sentence-by-sentence approach, each chunk is synthesized independently, so the audio can sound less natural than a fully context-aware streaming setup. Check out how you can set up Mercury 2 with an ElevenLabs Agent for streaming.