How to Use GPT-4o Audio Preview With LangChain and ChatOpenAI

Advanced Use Cases of gpt-4o-audio-preview

In this section, I’ll cover how to bind tools to the gpt-4o-audio-preview model, handle more complex workflows, and even chain multiple steps together to create fully automated solutions. If you’ve ever wanted to build a voice assistant that not only transcribes audio but also fetches information from external sources, this section will be helpful.

Tool calling

Let’s start with tool calling. This is teaching your AI model how to use external tools or functions to improve its capabilities. Instead of just processing text or audio, the model can perform additional tasks like fetching data from APIs, doing calculations, or even looking up weather information.

OpenAI’s gpt-4o-audio-preview model can be combined with other tools and functions using LangChain’s bind_tools method. With this method, you can create a smooth workflow where the model decides when and how to use the tools you’ve provided.

I am going to walk you through a practical example of binding a tool to the model. In this case, we’re going to bind a tool that fetches the weather.

First, we define a Pydantic model that represents the tool we want to use. Then, we bind that tool to the gpt-4o-audio-preview model so it can be invoked when necessary. For this, you will need an API key from OpenWeatherMap.

import requests
from pydantic import BaseModel, Field

# Define a tool schema using Pydantic
class GetWeather(BaseModel):
   """Get the current weather in a given location."""
   location: str = Field(..., description="The city and state, e.g. Edinburgh, UK")
   def fetch_weather(self):
       # Using OpenWeatherMap API to fetch real-time weather
       API_KEY = "YOUR_API_KEY_HERE"  # Replace with your actual API key
       base_url = f"http://api.openweathermap.org/data/2.5/weather?q={self.location}&APPID={API_KEY}&units=metric"      
       response = requests.get(base_url)      
       if response.status_code == 200:
           data = response.json()
           weather_description = data['weather'][0]['description']
           temperature = data['main']['temp']
           return f"The weather in {self.location} is {weather_description} with a temperature of {temperature}°C."
       else:
           # Print the status code and response for debugging
           print(f"Error: {response.status_code}, {response.text}")
           return f"Could not fetch the weather for {self.location}."

# Example usage
weather_tool = GetWeather(location="Edinburgh, GB")  # Using city name and country code
ai_msg = weather_tool.fetch_weather()
print(ai_msg)

Let’s see what it’s going on:

• Defining the tool: We define a tool called GetWeather, which takes a location as input and returns the current weather for that location using the OpenWeatherMap API.
• Fetching weather in Edinburgh: This time, we are fetching the weather for Edinburgh, UK.
• Real-time weather data: The tool fetches real-time weather data using an external API and returns it.

In this particular example, I want to know the weather in Edinburgh. When I ran it, I got this output: The weather in Edinburgh, GB is broken clouds with a temperature of 13.48°C.

Chaining tasks: Building multi-step workflows

By chaining tasks, you can create multi-step workflows that combine multiple tools and model calls to handle complex requests. Imagine a scenario where you want your assistant to transcribe audio and then perform an action for the location mentioned in the audio. In this example, we’ll chain an audio transcription task with a weather lookup based on the transcribed location.

import base64
import requests
from pydantic import BaseModel, Field
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

# Define the tool schema for fetching weather
class GetWeather(BaseModel):
   """Get the current weather in a given location."""
   location: str = Field(..., description="The city and state, e.g. Edinburgh, UK")
   def fetch_weather(self):
       # Using OpenWeatherMap API to fetch real-time weather
       API_KEY = "YOUR_API_KEY_HERE"  # Replace with your actual API key
       base_url = f"http://api.openweathermap.org/data/2.5/weather?q={self.location}&appid={API_KEY}&units=metric"       
       response = requests.get(base_url)
       if response.status_code == 200:
           data = response.json()
           weather_description = data['weather'][0]['description']
           temperature = data['main']['temp']
           return f"The weather in {self.location} is {weather_description} with a temperature of {temperature}°C."
       else:
           return f"Could not fetch the weather for {self.location}."

# Instantiate the LLM model
llm = ChatOpenAI(
   model="gpt-4o-audio-preview"
)
# Function to handle audio transcription using the LLM
def audio_to_text(audio_b64: str) -> str:
   # Define the message to send for transcription
   messages = [
       (
           "human",
           [
               {"type": "text", "text": "Transcribe the following:"},
               {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
           ],
       )
   ]
   # Invoke the model and get the transcription
   output_message = llm.invoke(messages)
   # Return the transcription from the model's output
   return output_message.content

# Create a prompt template for transcription and weather lookup
prompt = ChatPromptTemplate.from_messages(
   [
       ("system", "You are an assistant that transcribes audio and fetches weather information."),
       ("human", "Transcribe the following and tell me the weather in the location mentioned in the audio."),
   ]
)

# Bind the tool to the model
llm_with_tools = llm.bind_tools([GetWeather])

# Chain the transcription and weather tool
chain = prompt | llm_with_tools

# Read and encode the audio file in base64
audio_file = "weather_input.wav" #Replace with your audio file
with open(audio_file, "rb") as audio_file:
   audio_b64 = base64.b64encode(audio_file.read()).decode('utf-8')

# Transcribe the audio to get the location
transcribed_location = audio_to_text(audio_b64)

# Print the transcription result for debugging
print(f"Transcribed location: {transcribed_location}")

# Check if transcription returned a valid result
if transcribed_location:
   # Fetch weather for the transcribed location
   weather_tool = GetWeather(location=transcribed_location)
   weather_result = weather_tool.fetch_weather()
   print(f"Weather result: {weather_result}")
else:
   print("No valid location was transcribed from the audio.")

Let’s break down the code:

• Audio transcription: The user provides an audio input (encoded in base64). This audio is sent to the model, which transcribes the content. For example, the transcription might say "Edinburgh”.
• Weather lookup: After transcribing the audio and determining the location, the model then calls the GetWeather tool to fetch the weather for that location.
• Fully automated workflow: This setup creates an automated multi-step workflow where the model first understands what was said in the audio (transcribing it) and then acts on that information by fetching relevant weather data.

For this example, I gave the model an audio file in which I said: “Edinburgh”. This is the output I got:Transcribed location: Edinburgh

Weather result: The weather in Edinburgh is broken clouds with a temperature of 13.47°C.

Fine-tuning gpt-4o-audio-preview

In some cases, you might want to fine-tune the model to handle specific tasks more effectively. For example, if you’re building an application that transcribes medical audio, you might want the model to have a deep understanding of medical terms and jargon. OpenAI allows you to fine-tune the model by training it on custom datasets.

Let’s see how to do this:

import base64
from langchain_openai import ChatOpenAI

# Step 1: Instantiate the fine-tuned audio-capable model
fine_tuned_model = ChatOpenAI(
    temperature=0,
    model="ft:gpt-4o-audio-preview:your-organization::model-id"
)

# Step 2: Capture and encode the input audio file (medical report)
audio_file = "medical_report.wav"
with open(audio_file, "rb") as audio_file:
    audio_b64 = base64.b64encode(audio_file.read()).decode('utf-8')

# Step 3: Create the message structure for transcription request
messages = [
    (
        "human",
        [
            {"type": "text", "text": "Please transcribe this medical report audio."},
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
        ],
    )
]

# Step 4: Invoke the fine-tuned model to transcribe the audio
result = fine_tuned_model.invoke(messages)

# Step 5: Extract and print the transcribed text from the response
transcription = result.content  # This contains the transcription
print(f"Transcription: {transcription}")

# Debugging: Print the full structure of the additional_kwargs for inspection
print(result.additional_kwargs)

This uses a fine-tuned model specifically trained to handle medical audio, but you can fine-tune models for other use cases as well, such as legal transcriptions, customer service recordings, or podcast editing. 

Practical Example: Building a Voice-Enabled Assistant

Finally, let’s look into a practical example where we build a voice-enabled assistant that listens to user queries through audio, generates a response, and replies back using audio. 

Workflow breakdown

This is the workflow we’re going to follow:

1. The user speaks into a microphone, and the audio is captured.
2. The model transcribes the audio.
3. The transcription is processed to generate an answer.
4. The model then generates an audio response that can be played back to the user.

Implementation

Let’s see how to code this:

import base64
from langchain_openai import ChatOpenAI

# Step 1: Instantiate the audio-capable model with configuration for generating audio
llm = ChatOpenAI(
   model="gpt-4o-audio-preview",
   temperature=0,
   model_kwargs={
       "modalities": ["text", "audio"],  # Enable both text and audio modalities
       "audio": {"voice": "alloy", "format": "wav"},  # Set the desired voice and output format
   }
)

# Step 2: Capture and encode the audio 
audio_file = "math_joke_audio.wav" #You can replace your audio file here
with open(audio_file, "rb") as audio_file:
   audio_b64 = base64.b64encode(audio_file.read()).decode('utf-8')

# Step 3: Create the message structure for transcription and audio response
messages = [
   (
       "human",
       [
           {"type": "text", "text": "Answer the question."},
           {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
       ],
   )
]

# Step 4: Invoke the model to transcribe the audio and generate a response
result = llm.invoke(messages)

# Step 5: Extract the audio response
audio_response = result.additional_kwargs.get('audio', {}).get('data')  # Safely check if audio exists

# Step 6: Save the audio response to a file if it exists
if audio_response:
   # Decode the base64 audio data and save it as a .wav file
   audio_bytes = base64.b64decode(audio_response)
   with open("response.wav", "wb") as f:
       f.write(audio_bytes)
   print("Audio response saved as 'response.wav'")
else:
   print("No audio response available")

How it works

Let’s break down how this works:

• Audio input: The user provides audio input, such as asking a question (e.g., "Tell me a math joke"). The audio is encoded into base64 and passed to the assistant.
• Transcription: The model transcribes the audio into text. This transcription is available if you want to log or display it.
• Response generation: Based on the transcription, the model generates a response using its internal knowledge (e.g., "The capital of France is Paris")
• Audio output: The response is converted into audio format. The audio is saved as a file (e.g., "response.wav"), which can be played back to the user using any audio player.

This assistant is made for talking with audio. It listens to the user's voice, understands the question, and then answers using its own voice. This setup is great for building simple voice assistants, and you can make it more advanced later if needed!

Conclusion

In this tutorial, we explored OpenAI's gpt-4o-audio-preview model, covering setup, audio input/output, advanced use cases, and even building a basic voice assistant. This powerful model provides a robust foundation for creating real-world audio-enabled applications.

If you want to improve your LangChain practical knowledge, I recommend these blogs:

• How to Build User Interfaces For AI Applications Using Streamlit And LangChain
• RAG With Llama 3.1 8B, Ollama, and Langchain: Tutorial
• Building LangChain Agents to Automate Tasks in Python