CrewAI: A Guide With Examples of Multi AI Agent Systems

AI agents are assistants that can perform tasks and interact with the world. Unlike traditional systems that follow fixed rules, they can learn and adapt to new situations. Think of them as intelligent robots that assist with a variety of tasks.

But what sets AI agents apart from other AI entities, like the popular language models we've all heard about?

In this article, I’ll address this question and introduce CrewAI, a free and open-source Python framework designed to simplify the development of multi-agent AI systems. We’ll explore the distinction between agents and language models, discuss why agent frameworks are important for building AI applications, and demonstrate how CrewAI enables agents to collaborate and achieve great results.

AI Agents vs. LLMs

Let us debunk a common misunderstanding regarding the difference between agents and large language models (LLMs). Both belong to the family of artificial intelligence, yet they possess distinct capabilities.

Language models, such as ChatGPT and Gemini, are highly adept at using language. They have undergone extensive training on vast amounts of text and code, which has endowed them with the ability to comprehend and produce language that closely resembles human communication.

LLMs are skilled wordsmiths of artificial intelligence, producing a wide range of content including translations, summaries, creative narratives, or even poetry. Their scope is typically limited to language-related tasks.

Agents, however, are primarily focused on taking action. They are capable of navigating, interacting with objects, and making decisions based on their perceptions. 

In short, language models are brains and agents are hands. Together, they form a powerful duo.

So, with agents playing such a vital role in AI applications, how do we manage their complexity when multiple agents need to work together? This is where agent frameworks come in.

The Need for an Agent Framework

The need for agent frameworks arises from the increasing complexity of AI applications, particularly those involving multiple agents working collaboratively to achieve a common goal. Let's see why agent frameworks are essential.   

Orchestration and coordination

As AI systems grow in scale, they often incorporate numerous agents with diverse capabilities. Managing these interactions and ensuring they work harmoniously becomes increasingly difficult.

Agent frameworks offer a structured environment that allows for the orchestration of agent activities, defining their roles and responsibilities, and improving communication.

In multi-agent systems, an important aspect is the efficient allocation of tasks to the most suitable agents and managing shared resources effectively. Agent frameworks provide dynamic mechanisms for task allocation, resource negotiation, and conflict resolution.

Modularity and reusability

Agent frameworks promote a modular approach to AI development, where agents are designed and implemented as independent components. This modularity allows for better code organization and enables the reuse of agent modules across various projects, thus simplifying the development and maintenance of complex systems. With this component-based structure, AI developers can focus on individual agents without disrupting the larger system.

In addition, agent frameworks often adopt a role-based architecture, allowing developers to assign specific roles to agents, which define their capabilities and permissions. This role-based design leads to a more organized and manageable system where agents can be added, removed, or modified with minimal impact on the overall architecture. This flexibility ensures that systems can evolve and adapt to new requirements without significant re-engineering.

Adaptability and learning

Real-world environments are often dynamic, unpredictable, and continuously evolving. Agent frameworks equip agents to perceive changes in their surroundings and adapt their behaviors accordingly. This adaptability ensures that agents can function effectively even in complex and constantly shifting scenarios, making them better at handling real-world challenges.

Moreover, agent frameworks frequently incorporate learning mechanisms that enable agents to improve their performance over time. By learning from feedback and experiences, agents can continuously optimize their decision-making processes and adapt to new challenges as they arise. This continuous improvement allows agents to become more effective and valuable, contributing to long-term system efficiency and success.

And speaking of agent frameworks, let's introduce one that's making waves in the AI community: CrewAI.

What Is CrewAI?

CrewAI is an open-source Python framework designed to support developing and managing multi-agent AI systems.

CrewAI improves these AI systems by assigning specific roles, enabling autonomous decision-making, and facilitating communication between agents. This approach allows them to tackle complex problems more effectively than individual agents working alone.

This framework consists of a range of tools, including web search engines and language models, enabling agents to engage with the outside world, collect information, and act to fulfil their objectives.

The design and scalability of CrewAI make it perfect for developing both basic and complex multi-agent applications, encouraging a collaborative method for tackling challenges and making decisions within AI systems.

Let's take a look at some of the key features that make CrewAI a powerful tool for building multi-agent systems.

• Agent orchestration: CrewAI ensures every agent knows its part in the performance. It provides the tools to define and coordinate agent behaviours, making sure everyone is playing in sync.
• Role-based architecture: Like assigning different instruments to musicians, CrewAI lets you assign specific roles to agents, defining their capabilities and permissions. This creates a modular and well-structured system, even when things get complex.
• Flexible communication: CrewAI supports various communication channels, allowing agents to exchange information seamlessly. Think of it as having a private chat, a group discussion, and a megaphone all rolled into one.
• Tool integration: CrewAI empowers agents to interact with the world through various tools. These tools enable agents to search the web, understand language, analyze data, and perform custom tasks.
• Scalability: CrewAI is designed to scale effortlessly, ensuring your multi-agent system remains responsive and efficient as it grows.

But what benefits does CrewAI bring to the table? Let's explore its advantages.

Benefits of Using CrewAI

Crew.ai enables multiple AI agents to collaborate, share knowledge, and coordinate their actions toward a common goal.

By automating task distribution and resource management, Crew.ai allows agents to concentrate on their specific roles with minimal overhead.

The framework also supports adaptability, allowing agents to adjust their behavior based on changing conditions or objectives.

Additionally, Crew.ai simplifies the development process with a user-friendly platform for creating and managing multi-agent systems.

Another key strength of CrewAI is its integration with a wide range of tools. This extends the capabilities of agents, allowing them to interact with the external world and gather information.

CrewAI supports tools like web search engines, language models, data analysis tools, and even custom-built functionalities. This enables agents to perform tasks beyond their core capabilities, such as retrieving information from the web or performing complex data analysis.

Hands-On: Building a Web Search Tool with CrewAI

Let's roll up our sleeves and build a workflow using CrewAI tools to scrape the content from the website and then perform RAG on it. 

We need to write the code that will make the tool work. Before we begin, let’s install the crewai-tools and crewai packages using pip:

pip install crewai-tools crewai

Once you have installed the packages, follow the steps below. In this example, we’ll use three different tools: ScrapeWebsiteTool to scrape the website, FileWriterTool to write the file, and TXTSearchTool to search the context for RAG. Let’s get started.

Step 1: Scraping a website

We first import the necessary libraries and initialize the ScrapeWebsiteTool. This tool is used to extract content from a website. In this case, it’s set to scrape content from Wikipedia's "Artificial Intelligence" page:

from crewai_tools import ScrapeWebsiteTool, FileWriterTool, TXTSearchTool
import requests

# Initialize the tool, potentially passing the session
tool = ScrapeWebsiteTool(website_url='https://en.wikipedia.org/wiki/Artificial_intelligence')  

# Extract the text
text = tool.run()
print(text)

Step2: Write the extracted text to a file

We now use the FileWriterTool to save the extracted text into a file named ai.txt. 

# Initialize the tool
file_writer_tool = FileWriterTool()

# Write content to a file in a specified directory
result = file_writer_tool._run(filename='ai.txt', content = text, directory = '', overwrite=True)
print(result)

Step 3: Set up the text search tool

We set up another tool to search the contents of the file ai.txt that we just saved. We also set the environment variable for the OpenAI API key.

import os
from crewai_tools import TXTSearchTool

os.environ['OPENAI_API_KEY'] = 'API-KEY'

# Initialize the tool with a specific text file, so the agent can search within the given text file's content
tool = TXTSearchTool(txt='ai.txt')

Step 4: Create an agent for the task and execute it

We create a data analyst agent with the role of an educator. This agent's task is to answer the question "What is Natural Language Processing?" based on the text we searched in the file.

from crewai import Agent, Task, Crew

context = tool.run('What is natural language processing?')

data_analyst = Agent(
    role='Educator',
    goal=f'Based on the context provided, answer the question - What is Natural Language Processing? Context - {context}',
    backstory='You are a data expert',
    verbose=True,
    allow_delegation=False,
    tools=[tool]
)

test_task = Task(
    description="Understand the topic and give the correct response",
    tools=[tool],
    agent=data_analyst,
    expected_output='Give a correct response'
)

crew = Crew(
    agents=[data_analyst],
    tasks=[test_task]
)

output = crew.kickoff()

Output:

Conclusion

CrewAI offers a practical platform for managing multi-agent systems. Whether it’s scraping content or delegating tasks, this framework supports collaboration between agents and helps them adapt and improve.

By integrating tools and providing an organized structure, CrewAI makes it easier for developers to manage agent interactions, distribute tasks, and ensure optimal performance.