Getting Started With the Gemini Live API

If you've ever tried to build a voice assistant, you’ll know how clunky the traditional pipeline is. You record the user's voice, transcribe it with an STT model, send that text to an LLM, wait for a response, and then run it through a TTS engine. 

By the time your user hears an answer, several seconds of awkward silence have passed. It is less of a conversation and more of a slow request-and-response cycle. 

The Gemini Live API changes this flow by letting you stream audio and video continuously over a stateful WebSocket connection. Because the model processes and outputs audio natively, it can react immediately, adapt to natural vocal nuances, and even handle interruptions (barge-in) mid-sentence. 

In this tutorial, I will show you how to build a Real-Time AI Travel Concierge using the code-first Google Agent Development Kit (ADK). We'll equip our agent with Google Search for live destination weather lookups, connect it to a mock flight database, and spin up a local developer interface to test how it handles real-time dialogue and tool calling.

If you are new to the ecosystem, I recommend starting with the Building AI Agents with Google ADK course to establish your foundational knowledge.

What you will achieve: 

• Understand the inner workings, capabilities, and pricing of the Gemini Live API.
• Configure a Python environment with the Google Agent Development Kit. 
• Expose both built-in tools (Google Search) and custom business logic (flight status database lookups) to the voice assistant. 
• Launch a local web UI to talk to your travel agent live.

Prerequisites: 

• Basic familiarity with Python. 
• A Google Cloud Project (for Vertex AI access) or a Google AI Studio API key.

What is the Gemini Live API? 

At its core, the Gemini Live API is a developer tool from Google that allows you to build real-time voice and vision interactions with Gemini models. Instead of waiting for turn-based text responses, it enables continuous, natural conversation between a user and an AI agent. 

Technically, the Gemini Live API is designed for bidirectional, real-time streaming. Unlike standard HTTP REST APIs that use stateless request-response cycles, the Live API relies on a stateful, persistent WebSocket (WSS) connection. Through this open channel, the client streams raw audio, video (transmitted as image frames at ~1 FPS), and text, while the model streams back immediate, native audio and text.

Key capabilities of Gemini Live

• Barge-in (Interruptibility): In a natural human conversation, we don't wait for the other person to completely finish a paragraph before speaking if we need to clarify something. Gemini Live supports barge-in out of the box; users can interrupt the model's speech mid-sentence, and the server immediately stops streaming audio and adapts to the new input.
• Affective dialog and native audio: Instead of generating text and passing it to a Text-to-Speech (TTS) engine, models like gemini-live-2.5-flash-native-audio and gemini-3.1-flash-live-preview natively process and output audio. This preserves emotional nuance, laughter, tone adaptation, and speech rhythm, making interactions feel human-like. 
• Multimodal inputs and tool use: The Live API is not restricted to audio. It can "see" your environment through video frames and trigger external functions or tools (such as Google Search or custom APIs) in real-time, feeding the results back into the voice stream.

Gemini Live API Pricing

Because real-time streaming involves a continuous, full-duplex connection, the billing model differs from standard static text APIs. Google offers both a generous free tier for prototyping in Google AI Studio and a production-ready paid tier on Vertex AI. 

The model gemini-3.1-flash-live-preview is billed per million tokens or converted 

directly into a duration-based metric for audio and video media:

If you are prototyping, I highly recommend testing for free in Google AI Studio, which provides generous access subject to rate limits. 

When you are ready to scale, you can transition your project to Vertex AI for production Pay-as-you-go provisioning, which provides guaranteed throughput and enterprise-grade SLAs.

How To Access the Gemini Live API

As a developer, you have a few architectural choices when implementing the Live API. Because it relies on WebSockets, how you route the connection matters for latency and security.

Client-to-server vs server-to-server

• Client-to-server: Your frontend (e.g., a web or mobile app) connects directly to Google's WebSocket endpoint. This is the best approach for minimizing latency because the audio stream doesn't take extra hops. 
• However, you cannot expose your raw Google Cloud API keys in a frontend client; you must implement an authentication flow that generates temporary, ephemeral tokens for secure client access.
• Server-to-server: Your frontend streams audio to your own backend server, which then opens the WebSocket to the Gemini Live API. This is highly secure and allows you to inject backend logic (like user authentication or database lookups), but it adds a slight network delay to the audio stream.

SDKs and partner integrations

While you can write raw WebSocket code, the easiest way to start is by using Google’s GenAI SDK or the Agent Development Kit (ADK). 

For enterprise production, the ecosystem has rapidly evolved to support WebRTC integrations. Platforms like LiveKit, Pipecat by Daily, Twilio, and Voximplant offer robust infrastructure to handle network jitter, echo cancellation, and real-time audio routing directly into the Gemini Live API.

Building a Real-Time Voice Assistant With Gemini Live API

Let's put this into practice. We are going to build an AI Travel Concierge. 

This agent will use the Gemini Live API to converse with users over voice, but more importantly, it will have access to custom Python functions to look up real-time flight statuses mid-conversation.

We will use the Google Agent Development Kit (ADK) and the powerful gemini-3.1-flash-live-preview model, which is optimized for both standard text turns and real-time WebSocket streaming.

1. Setting up the environment

First, you need to set up a Python virtual environment and install the required ADK package. Run these commands in your terminal:

# Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate
# Install the Agent Development Kit
pip install google-adk

Next, configure your environment variables so the ADK can securely authenticate with your Google Cloud project:

export GOOGLE_CLOUD_PROJECT="your-project-id" 
export GOOGLE_CLOUD_LOCATION="us-central1"

If you are using a Gemini API Key from Google AI Studio instead, create a .env file in your root folder:

GEMINI_API_KEY=your_google_ai_studio_api_key

2. Initializing the Gemini Live agent 

Create a file named agent.py in your project directory. The ADK is designed to automatically detect and run the root_agent variable. In this code, we will define a custom Python function to check flight statuses, and then we will pass it directly into the agent's toolbelt. Initialize the agent directory: 

mkdir voice_assistant 
touch voice_assistant/__init__.py 
touch voice_assistant/agent.py

Expose the agent package inside voice_assistant/__init__.py:

from . import agent

3. Defining the agent and exposing custom tools 

Open voice_assistant/agent.py and write the travel agent configuration. In the code below, we define a standard Python function check_flight_status. 

In Google ADK, any Python function with proper type hints and a docstring is automatically converted into an agent tool. The LLM uses the docstring description and parameters to determine when and how to call the function. 

Model configuration 

Leveraging Gemini 3.1 Flash Live Preview To support both interaction modes in the ADK development server—standard text chat (which uses standard REST generateContent endpoints) and the real-time bidirectional WebSocket microphone interface—we explicitly set the model to gemini-3.1-flash-live-preview.

Unlike older streaming-only models, which return a 404 Not Found when triggered over standard HTTP REST requests, gemini-3.1-flash-live-preview natively supports both standard REST content generation and stateful WebSocket streaming.

from google.adk.agents.llm_agent import Agent
from google.adk.tools import google_search
def check_flight_status(flight_no: str) -> str:
   """Check the real-time flight status (e.g. delays, gate info, cancellations).
   Args:
       flight_no: The flight number to look up (e.g., AA120, DL450, UA990).
   """
   flight_no = flight_no.upper().strip()
   if "AA120" in flight_no:
       return "Flight AA120 to Paris is currently delayed by 45 minutes due to air traffic control. New departure time: 3:15 PM. Gate status: B22."
   elif "DL450" in flight_no:
       return "Flight DL450 to London is on time. Gate status: Boarding now at Gate C10."
   elif "UA990" in flight_no:
       return "Flight UA990 to Tokyo is cancelled due to adverse weather conditions at the destination. Passengers are advised to rebook at the counter."
   else:
       return f"Flight {flight_no} is currently scheduled on time. No delay reports found in the flight database."
# Define the root agent with search grounding and custom database flight checks
root_agent = Agent(
   model='gemini-3.1-flash-live-preview',
   name='root_agent',
   description='A real-time AI Travel Concierge with flight tracking and search grounding.',
   instruction=(
       'You are an expert AI Travel Concierge. You assist users with real-time flight status checkups, '
       'destination recommendations, and general travel planning. '
       'Always answer professionally and conversationally. '
       'Use the check_flight_status tool when a user asks about a specific flight status. '
       'Use Google Search to look up weather, transit conditions, or travel advisories at destinations.'
   ),
   tools=[google_search, check_flight_status],
)

4. Running and testing the interactive dev UI

The ADK includes a built-in development server and Web UI. This lets you test your agent locally without writing any frontend code. Start the web server from your workspace:

.venv/bin/adk web .

You should see output indicating the server has started:

1. Open http://127.0.0.1:8000 in your web browser.
2. Select your agent from the application list.
3. Click the Microphone button to start a live session.

Select root_agent from the application list to open the chat interface.

4. Speak into your microphone: Can you check the status of flight AA120? The agent will parse your request, recognize that it needs database information, trigger the custom check_flight_status Python function, and display the JSON tool call outcome before speaking the response back to you.

5. Testing Google Search Grounding:  Next, ask the concierge a destination weather question: What is the weather in London right now? The agent will activate the built-in google_search tool, crawl the web, and synthesize the live conditions into its audio output.

6. Testing the Barge-in Feature: Ask a broad question like: Tell me about the best historical sites to visit in Rome. As the agent begins listing spots, speak over it loudly: Actually, wait! Check flight UA990 instead!. The agent will immediately stop its speech stream, interrupt the Rome details, run the custom flight tracker tool, and announce that flight UA990 has been canceled. This showcases the low-latency, full-duplex nature of the Live API connection.

Best Practices for Gemini Live API Production

Building real-time voice applications introduces unique requirements around prompt architecture, network state management, audio ingestion, and resource scaling. Below are key best practices derived from official developer guidelines.

1. Designing clear system instructions

To get the best performance out of Gemini Live, structure your system instructions (SIs) in a strict hierarchy: Persona, Conversational Rules, and Guardrails.

• Separate personas: Keep each agent's role restricted to a single distinct system instruction rather than overloading one agent with multiple personas.
• Delineate conversation flows: Explicitly separate one-time elements (e.g., "ask the user for their name and ticket number once") from open-ended conversational loops (e.g., "answer questions about travel recommendations for as long as they want").
• Write tool invocations in sentences: Describe tool usage sequentially. For example: 'First, ask the user to provide their ticket number. Then invoke look_up_ticket with this number.'
• Enforce language consistency: Native audio models can drift in language if the context changes. If your agent is English-only, enforce it in your instruction: RESPOND IN English. YOU MUST RESPOND UNMISTAKABLY IN English.

2. Precise tool definitions and invocation conditions

When exposing tools to the Live API:

• Always include type annotations and detailed parameter descriptions.
• Document the Invocation Conditions explicitly inside the tool’s metadata description (e.g., "Invoke this tool only after the user has confirmed their departure date and destination"). This prevents the LLM from triggering functions prematurely.

3. Managing native audio context (Compression)

Native audio tokens accumulate rapidly, approximately 25 tokens per second of conversation. For long-running voice sessions, this can quickly fill the context window and increase latency.

• Enable ContextWindowCompressionConfig in your connection settings to drop older history once a threshold is reached:

from google.genai import types
live_config = types.LiveConnectConfig(
   context_window_compression=types.ContextWindowCompressionConfig(
       trigger_tokens=100_000,
       sliding_window=types.SlidingWindow(target_tokens=4_000),
   ),
)

Note: Context compression will cause old conversation history to be lost.

4. Transparent session resumption

Real-time mobile connections are prone to packet loss and temporary disconnections. Instead of starting a brand new conversation state from scratch, implement transparent session resumption:

• Configure your stream with SessionResumptionConfig(transparent=True).
• Listen for session_resumption_update events from the server and store the new_handle.
• Maintain a local buffer of outgoing messages. When the connection drops, catch the APIError or GoAway signal, reconnect using the saved handle, and replay only the unacknowledged client messages (using the server-reported last_consumed_client_message_index to prune the buffer).

5. Audio streaming optimization

• Client buffering: Do not buffer long audio frames on the client before sending. Stream small audio chunks (between 20 ms and 40 ms) continuously to minimize end-to-end latency.
• Microphone resampling: Ensure client-side audio capture resamples inputs (typically 44.1 kHz or 48 kHz) down to 16 kHz PCM before sending over the WebSocket.
• Explicit VAD signals: Let the model handle Voice Activity Detection (VAD). If your frontend needs to show immediate UI indicators (e.g., "AI is listening" vs "AI is thinking"), enable explicit_vad_signal=True in your configuration to receive VAD start/stop events directly in the server message flow.

6. Security (token management)

• Never hardcode master keys: Avoid exposing your AI Studio API key or Google Cloud Service Account credentials within client-side code (browsers or mobile applications).
• Deploy an ephemeral token exchange: Build an authentication endpoint on your secure backend. When a client wants to start a voice session, the backend generates a short-lived, restricted OAuth or API access token (e.g., via Vertex AI IAM) and passes it to the client. The client uses this temporary credential to open the direct WebSocket connection.

Conclusion: The Horizon of Conversational AI

The transition from turn-based, text-only LLMs to low-latency, multimodal streaming represents a paradigm shift in how we build human-computer interfaces. By processing audio and video natively through stateful WebSocket connections, the Gemini Live API allows us to build agentic experiences that feel less like static software programs and more like fluid human interactions.

Throughout this guide, we explored the inner architecture of Gemini Live, examined pricing models, and built a fully functional AI Travel Concierge utilizing the Google Agent Development Kit (ADK). We witnessed how ADK maps standard Python functions into active agent tools, giving our voice assistant access to both broad web-based search grounding and private internal database lookups.

What to Build Next? 

Now that you have the fundamentals of real-time voice agents down, consider extending your AI Travel Concierge: 

1. Expose Live Video Streams: Add support for camera image transmission to let the concierge "see" maps, boarding passes, or physical travel vouchers in real-time. 
2. Integrate Transactional Custom Tools: Implement a book_flight or reserve_hotel tool that mutates actual state, integrating authentication scopes for secure user confirmations. 
3. Production Middleware Routing: Move from local ADK testing to a WebRTC-based framework using LiveKit or Pipecat to stream production voice calls directly from web and mobile apps.

Ready to build more advanced conversational workflows? Check out DataCamp’s AI Agent Fundamentals skill track to deepen your knowledge of agentic architectures, tool use, and generative AI deployments.