Realtime Voice Assistant: Mercury + ElevenLabs

This notebook demonstrates how to build a low-latency voice assistant using Mercury 2 for real-time intelligence combined with ElevenLabs’ speech-to-text and text-to-speech models. We walk you through how to:

Convert text to speech using ElevenLabs TTS
Convert audio to text using ElevenLabs speech-to-text
Generate realtime responses with Mercury 2
Optimize latency using Mercury 2 streaming

Run the Voice Assistant

Directly via Google Colab or download the .ipynb to run it locally in Jupyter

Dependencies

Install packages and import the libraries required for the Mercury 2 SDK, ElevenLabs, and audio playback:

%pip install \
    "inceptionai" \
    "elevenlabs>=2.15.0" \
    "sounddevice>=0.5.1" \
    "numpy>=1.26.0" \
    "scipy>=1.16.2" \
    "python-dotenv>=1.0.0" \
    "pydub>=0.25.1"

import io
import os
import time

import elevenlabs
from dotenv import load_dotenv
from inceptionai import Inception
from IPython.display import Audio

API Keys

Set up your API keys for both ElevenLabs and Inception Labs.

Create a .env in this directory
Add your API keys to the .env:
- INCEPTION_API_KEY
- ELEVENLABS_API_KEY

The keys will be automatically loaded from the .env file.

# Load environment variables from .env file
load_dotenv()
INCEPTION_API_KEY = os.getenv("INCEPTION_API_KEY")
ELEVENLABS_API_KEY = os.getenv("ELEVENLABS_API_KEY")

Initialize Clients

Create the ElevenLabs and Inception (Mercury 2) clients:

assert INCEPTION_API_KEY is not None, (
    "ERROR: INCEPTION_API_KEY not found. Please copy .env.example to .env and add your API keys."
)
assert ELEVENLABS_API_KEY is not None, (
    "ERROR: ELEVENLABS_API_KEY not found. Please copy .env.example to .env and add your API keys."
)

mercury_client = Inception(api_key=INCEPTION_API_KEY)
elevenlabs_client = elevenlabs.ElevenLabs(
    api_key=ELEVENLABS_API_KEY, base_url="https://api.elevenlabs.io"
)

MERCURY_MODEL = "mercury-2"
VOICE_ID = "21m00Tcm4TlvDq8ikWAM"  # Rachel

def mercury_completion(user_text, max_tokens=1000, temperature=0.75):
    return mercury_client.chat.completions.create(
        model=MERCURY_MODEL,
        messages=[{"role": "user", "content": user_text}],
        max_tokens=max_tokens,
        temperature=temperature,
        reasoning_effort="low",
    )


def mercury_stream(user_text, max_tokens=1000, temperature=0.75):
    stream = mercury_client.chat.completions.create(
        model=MERCURY_MODEL,
        messages=[{"role": "user", "content": user_text}],
        max_tokens=max_tokens,
        temperature=temperature,
        reasoning_effort="low",
        stream=True,
    )

    for chunk in stream:
        if not chunk.choices:
            continue
        text = chunk.choices[0].delta.content
        if text:
            yield text

Create Input Audio

We can simulate a user audio input using ElevenLabs text-to-speech model:

audio = elevenlabs_client.text_to_speech.convert(
    voice_id=VOICE_ID,
    output_format="mp3_44100_128",
    model_id="eleven_v3",
    text="Hi, Mercury. ",
)

audio_data = io.BytesIO()
for chunk in audio:
    audio_data.write(chunk)

Audio(audio_data.getvalue())

Transcribe Speech into Text

We then transcribe the audio input using ElevenLabs’ speech-to-text model. We’ll also measure the transcription latency:

audio_data.seek(0)

start_time = time.time()

transcription = elevenlabs_client.speech_to_text.convert(file=audio_data, model_id="scribe_v1")

end_time = time.time()
transcription_time = end_time - start_time

print(f"Transcribed text: {transcription.text}")
print(f"Transcription time: {transcription_time:.2f} seconds")

Get a Response from Mercury 2

We send the transcribed text to Mercury 2 and measure the response time. We’re using mercury-2 for fast reasoning responses:

start_time = time.time()

completion = mercury_completion(transcription.text, max_tokens=1000, temperature=0.75)
mercury_response = completion.choices[0].message.content

end_time = time.time()
non_streaming_response_time = end_time - start_time

print(mercury_response)
print(f"\nResponse time: {non_streaming_response_time:.2f} seconds")

Real-time with Streaming

Improve response latency by using Mercury 2 streaming. This allows us to receive the first tokens more quickly, reducing user-perceived latency:

start_time = time.time()
first_token_time = None

mercury_full_response = ""

for text in mercury_stream(transcription.text, max_tokens=1000, temperature=0.75):
    mercury_full_response += text
    print(text, end="", flush=True)
    if first_token_time is None:
        first_token_time = time.time()

streaming_time_to_first_token = first_token_time - start_time
print(
    f"\n\nStreaming time to first token: {streaming_time_to_first_token:.2f} seconds - reducing response wait by {(non_streaming_response_time - streaming_time_to_first_token) * 100 / non_streaming_response_time:.2f}%"
)

We can also stream the text-to-speech response to reduce silence:

start_time = time.time()
first_audio_chunk_time = None

audio_buffer = io.BytesIO()

audio_generator = elevenlabs_client.text_to_speech.stream(
    voice_id=VOICE_ID,
    output_format="mp3_44100_128",
    text=mercury_full_response,
    model_id="eleven_turbo_v2_5",
)

for chunk in audio_generator:
    if first_audio_chunk_time is None:
        first_audio_chunk_time = time.time()
    audio_buffer.write(chunk)

streaming_tts_time_to_first_chunk = first_audio_chunk_time - start_time
print(f"Streaming TTS time to first audio chunk: {streaming_tts_time_to_first_chunk:.2f} seconds")

Audio(audio_buffer.getvalue())

Streaming Mercury 2 Directly to TTS

We’ve optimized Mercury 2 streaming and TTS separately. Now we can stream the model response and convert it to audio as soon as we have complete sentences, demarcated by !, ., ?.

import re

sentence_pattern = re.compile(r"[.!?]+")
sentence_buffer = ""
audio_chunks = []

start_time = time.time()
first_audio_time = None

for text in mercury_stream(transcription.text, max_tokens=1000, temperature=0.75):
    print(text, end="", flush=True)
    sentence_buffer += text

    if sentence_pattern.search(sentence_buffer):
        sentences = sentence_pattern.split(sentence_buffer)

        # Process all complete sentences (all but the last element)
        for i in range(len(sentences) - 1):
            complete_sentence = sentences[i].strip()
            if complete_sentence:
                audio_gen = elevenlabs_client.text_to_speech.stream(
                    voice_id=VOICE_ID,
                    output_format="mp3_44100_128",
                    text=complete_sentence,
                    model_id="eleven_turbo_v2_5",
                )

                sentence_audio = io.BytesIO()
                for chunk in audio_gen:
                    if first_audio_time is None:
                        first_audio_time = time.time()
                    sentence_audio.write(chunk)

                audio_chunks.append(sentence_audio.getvalue())

        sentence_buffer = sentences[-1]

if sentence_buffer.strip():
    audio_gen = elevenlabs_client.text_to_speech.stream(
        voice_id=VOICE_ID,
        output_format="mp3_44100_128",
        text=sentence_buffer.strip(),
        model_id="eleven_turbo_v2_5",
    )
    sentence_audio = io.BytesIO()
    for chunk in audio_gen:
        if first_audio_time is None:
            first_audio_time = time.time()
        sentence_audio.write(chunk)
    audio_chunks.append(sentence_audio.getvalue())

sentence_streaming_time_to_first_audio = first_audio_time - start_time
print(f"\n\nTime to first audio: {sentence_streaming_time_to_first_audio:.2f} seconds")

combined_pcm = b"".join(audio_chunks)
Audio(combined_pcm)

In this sentence-by-sentence approach, each chunk is synthesized independently, so the audio can sound less natural than a fully context-aware streaming setup. Check out how you can set up Mercury 2 with an ElevenLabs Agent for streaming.

Getting Started

API reference

Capabilities

Resources

Support

Realtime Voice Assistant: Mercury + ElevenLabs

Run the Voice Assistant

Dependencies

API Keys

Initialize Clients

Create Input Audio

Transcribe Speech into Text

Get a Response from Mercury 2

Real-time with Streaming

Streaming Mercury 2 Directly to TTS

Run the Voice Assistant

​Dependencies

​API Keys

​Initialize Clients

​Create Input Audio

​Transcribe Speech into Text

​Get a Response from Mercury 2

​Real-time with Streaming

​Streaming Mercury 2 Directly to TTS

Dependencies

API Keys

Initialize Clients

Create Input Audio

Transcribe Speech into Text

Get a Response from Mercury 2

Real-time with Streaming

Streaming Mercury 2 Directly to TTS