Skip to content

Building Multimodal AI Applications with LangChain & the OpenAI API

Goals

Videos can be full of useful information, but getting hold of that info can be slow, since you need to watch the whole thing or try skipping through it. It can be much faster to use a bot to ask questions about the contents of the transcript.

In this project, you'll download a tutorial video from YouTube, transcribe the audio, and create a simple Q&A bot to ask questions about the content.

  • Understanding the building blocks of working with Multimodal AI projects
  • Working with some of the fundamental concepts of LangChain
  • How to use the Whisper API to transcribe audio to text
  • How to combine both LangChain and Whisper API to create ask questions of any YouTube video

Before you begin

You'll need a developer account with OpenAI and a create API Key. The API secret key will be stored in your 'Environment Variables' on the side menu. See the getting-started.ipynb notebook for details on setting this up.

Task 0: Setup

The project requires several packages that need to be installed into Workspace.

  • langchain is a framework for developing generative AI applications.
  • yt_dlp lets you download YouTube videos.
  • tiktoken converts text into tokens.
  • docarray makes it easier to work with multi-model data (in this case mixing audio and text).

Instructions

Run the following code to install the packages.

# Install langchain
!pip install langchain==0.0.292
Hidden output 
# Install yt_dlp
!pip install yt_dlp==2023.7.6
Hidden output 
!pip install tiktoken==0.5.1
Hidden output 
!pip install docarray==0.38.0

Instructions

Task 1: Import The Required Libraries

‌
‌
‌