Build an AI Competitor Intelligence Agent Team

For businesses looking to stay competitive, understanding the competition is crucial. But manually gathering and analyzing competitor data is time-consuming and often yields incomplete insights. What if we could automate this process using AI agents that work together to deliver comprehensive competitive intelligence?

In this tutorial, we'll build a multi-agent competitor analysis team that automatically discovers competitors, extracts structured data from their websites, and generates actionable insights. You'll create a team of specialized AI agents that work together to deliver detailed competitor analysis reports with market opportunities and strategic recommendations.

This system combines web crawling, data extraction, and AI analysis to transform raw competitor website data into structured insights. Using a team of coordinated AI agents, each specializing in different aspects of competitive analysis

Don’t forget to share this tutorial on your social channels and tag Unwind AI (X, LinkedIn, Threads, Facebook) to support us!

What We’re Building

This Streamlit application simulates a full-service market research team using multiple AI agents to analyze competitor websites and provide comprehensive competitive insights. Each agent represents a different competitive analysis specialist role, from data extraction and feature comparison to strategic analysis, working together to provide thorough market intelligence and recommendations.

Features

Our AI Agent Team

1. Research Agent: Leverages Perplexity AI's Sonar Pro or Exa AI's neural search to discover and validate competitor companies. Finds similar companies based on URLs or descriptions and ensures relevant matches.

2. Data Extraction Agent: Uses Firecrawl to perform deep website analysis, extracting structured information about pricing, features, tech stack, and customer feedback. References specific data points from competitor websites.

3. Market Analyst: Analyzes extracted data to identify market gaps, opportunities, and competitive advantages. Provides strategic insights and actionable recommendations.

4. Team Coordinator: Orchestrates the analysis process, combines insights from all agents, ensures comprehensive market coverage, and delivers structured comparative analysis. Acts as an Agent Team coordinator for all three agents.

Analysis Types

• Competitor Discovery - Done by Research Agent

  • URL-based competitor matching

  • Description-based company search

  • Competitor validation

• Feature Analysis - Done by Data Extraction Agent

  • Pricing model extraction

  • Key feature identification

  • Technology stack analysis

  • Marketing strategy assessment

• Market Analysis - Done by Market Analyst

  • Competitive positioning

  • Market opportunity identification

  • Strategic recommendations

• Comprehensive Reports - Done by Team Coordinator

  • Comparative analysis tables

  • Market gap analysis

  • Strategic planning suggestions

Tech Stack

Firecrawl serves as our web crawling powerhouse, taking URLs and transforming entire websites into clean, LLM-ready markdown format. It handles all the complex aspects of web scraping, including JavaScript rendering, anti-bot mechanisms, and content parsing, ensuring reliable data extraction from competitor websites.

Agno (former Phidata) provides the foundation for building and orchestrating our multi-agent system. It offers a clean framework for creating teams of specialized agents that can work together, share information, and leverage various tools while providing built-in monitoring and debugging capabilities.

Exa is the first meaning-based web search powered by embeddings. It uses advanced embeddings to find similar companies and relevant competitor URLs, going beyond simple keyword matching to identify truly relevant competitors.

Perplexity recently launched Sonar, a search API to integrate real-time, citation-backed search capabilities into your applications. The API comes in two tiers: Sonar and Sonar Pro, with the Pro version offering advanced querying, deeper context windows, and generating twice as many citations on average

For processing and analyzing the gathered data, we're using GPT-4o which provides the intelligence needed to generate meaningful insights and recommendations.

Prerequisites

Before we begin, make sure you have the following:

1. Python installed on your machine (version 3.10 or higher is recommended)

2. Your OpenAI, Firecrawl, and Exa or Perplexity Sonar API key

3. A code editor of your choice (we recommend VS Code or PyCharm for their excellent Python support)

4. Basic familiarity with Python programming

Step-by-Step Instructions

Setting Up the Environment

First, let's get our development environment ready:

1. Clone the GitHub repository:

git clone https://github.com/Shubhamsaboo/awesome-llm-apps.git

🌟 Don't forget to star the opensource repo to show your support.

2. Go to the ai_competitor_intelligence_agent_team folder:

cd ai_agent_tutorials/ai_competitor_intelligence_agent_team

3. Install the required dependencies:

pip install -r requirements.txt

4. Obtain your API keys:

   • Get your Firecrawl API key from Firecrawl's website

   • Get your Exa or Perplexity API key

   • Get your OpenAI API key from OpenAI's website

Creating the Streamlit App

Let’s create our app. Create a new file competitor_agent_team.py and add the following code:

1. Let's set up our imports:

import streamlit as st
from exa_py import Exa
from agno.agent import Agent
from agno.tools.firecrawl import FirecrawlTools
from agno.models.openai import OpenAIChat
from agno.tools.duckduckgo import DuckDuckGoTools
import pandas as pd
import requests
from firecrawl import FirecrawlApp
from pydantic import BaseModel, Field

2. Set up the Streamlit UI and configure search engine selection:

st.set_page_config(page_title="AI Competitor Intelligence Agent Team", layout="wide")

search_engine = st.sidebar.selectbox(
    "Select Search Endpoint",
    options=["Perplexity AI - Sonar Pro", "Exa AI"],
    help="Choose which AI service to use for finding competitor URLs"
)

3. Handle dynamic API key collection based on search engine:

st.sidebar.title("API Keys")
openai_api_key = st.sidebar.text_input("OpenAI API Key", type="password")
firecrawl_api_key = st.sidebar.text_input("Firecrawl API Key", type="password")

if search_engine == "Perplexity AI - Sonar Pro":
    perplexity_api_key = st.sidebar.text_input("Perplexity API Key", type="password")
else:  # Exa AI
    exa_api_key = st.sidebar.text_input("Exa API Key", type="password")

4. Define the CompetitorDataSchema using Pydantic:

class CompetitorDataSchema(BaseModel):
    company_name: str = Field(description="Name of the company")
    pricing: str = Field(description="Pricing details, tiers, and plans")
    key_features: List[str] = Field(description="Main features and capabilities")
    tech_stack: List[str] = Field(description="Technologies used")
    marketing_focus: str = Field(description="Marketing angles")
    customer_feedback: str = Field(description="Customer reviews")

5. Initialize the AI agents:

firecrawl_agent = Agent(
    model=OpenAIChat(id="gpt-4o", api_key=st.session_state.openai_api_key),
    tools=[firecrawl_tools, DuckDuckGoTools()],
    show_tool_calls=True,
    markdown=True
)

analysis_agent = Agent(...)
comparison_agent = Agent(...)

6. Implement competitor URL discovery with dual search engine support:

def get_competitor_urls(url: str = None, description: str = None) -> list[str]:
    if search_engine == "Perplexity AI - Sonar Pro":
        # Perplexity implementation
        perplexity_url = "https://api.perplexity.ai/chat/completions"
        # ... Perplexity API call logic
    else:  # Exa AI
        # Exa implementation
        result = exa.find_similar(...)

7. Implement competitor information extraction:

def extract_competitor_info(competitor_url: str) -> Optional[dict]:
    app = FirecrawlApp(api_key=st.session_state.firecrawl_api_key)
    url_pattern = f"{competitor_url}/*"
    
    extraction_prompt = """
    Extract detailed information about the company's offerings...
    """
    
    response = app.extract([url_pattern], {
        'prompt': extraction_prompt,
        'schema': CompetitorDataSchema.model_json_schema(),
    })

8. Generate comparison reports:

def generate_comparison_report(competitor_data: list) -> None:
    formatted_data = json.dumps(competitor_data, indent=2)
    system_prompt = f"""
    As an expert business analyst, analyze the following competitor data...
    """
    comparison_response = comparison_agent.run(system_prompt)

9. Create and display comparison tables:

# Create DataFrame from comparison data
df = pd.DataFrame(
    data_rows,
    columns=headers
)

# Display the table
st.subheader("Competitor Comparison")
st.table(df)

10. Generate analysis reports:

def generate_analysis_report(competitor_data: list):
    formatted_data = json.dumps(competitor_data, indent=2)
    report = analysis_agent.run(
        f"""Analyze the following competitor data and identify market opportunities...
        """
    )

11. Handle the main analysis workflow:

if st.button("Analyze Competitors"):
    if url or description:
        with st.spinner("Fetching competitor URLs..."):
            competitor_urls = get_competitor_urls(url=url, description=description)

12. Process competitor data:

competitor_data = []
for comp_url in competitor_urls:
    with st.spinner(f"Analyzing Competitor: {comp_url}..."):
        competitor_info = extract_competitor_info(comp_url)
        if competitor_info is not None:
            competitor_data.append(competitor_info)

13. Generate final reports:

if competitor_data:
    generate_comparison_report(competitor_data)
    analysis_report = generate_analysis_report(competitor_data)
    st.subheader("Competitor Analysis Report")
    st.markdown(analysis_report)

14. Error handling and validation:

try:
    # Processing logic
except Exception as e:
    st.error(f"Error: {str(e)}")
    # Error handling

Running the App

With our code in place, it's time to launch the app.

• In your terminal, navigate to the project folder, and run the following command

streamlit run competitor_agent_team.py

• Streamlit will provide a local URL (typically http://localhost:8501).

Working Application Demo

Conclusion

You've successfully built a powerful multi-agent competitor intelligence system that automates the entire process of competitor analysis. Your application can now discover relevant competitors, extract structured data from their websites, and generate actionable insights - all while coordinating multiple specialized AI agents.

This foundation can be enhanced further in several ways:

• Integrate additional data sources like social media profiles, news articles, and review sites to build a more comprehensive competitor profile.

• Create specialized analysis templates for different industries or business types, enabling more targeted competitive insights.

• Implement a monitoring system that tracks changes in competitor websites over time, helping identify strategic shifts and new opportunities.

• Add interactive charts and graphs to better visualize competitive landscapes and market positioning.

Keep experimenting and have fun watching these AI agents work together as a team to analyze the competition for you!

We share hands-on tutorials like this 2-3 times a week, to help you stay ahead in the world of AI. If you're serious about leveling up your AI skills and staying ahead of the curve, subscribe now and be the first to access our latest tutorials.

Don’t forget to share this tutorial on your social channels and tag Unwind AI (X, LinkedIn, Threads, Facebook) to support us!