Build a Deep Research Agent with OpenAI Agents SDK and Firecrawl

Need deep research capabilities but don't want to pay $200/month for ChatGPT Pro or $20/month for alternatives like Perplexity or Gemini Deep Research? Ok, you got to try it for free with Gemini. But did the 15-20 minutes of research and limited queries put you off? You're not alone. Today's research tools are powerful but expensive, leaving most of us searching for cheaper and faster alternatives.

In this tutorial, we'll show you how to create your own powerful Deep Research Agent that performs in minutes what might take human researchers hours or even days—all without the hefty subscription fees. Using OpenAI's Agents SDK and Firecrawl, you'll build a multi-agent system that searches the web, extracts content, and synthesizes comprehensive reports through a clean Streamlit interface.

OpenAI's Agents SDK is a lightweight framework for building AI applications with specialized agents that work together. It provides primitives like agents, handoffs, and guardrails that make it easy to coordinate tasks between multiple AI assistants.

Firecrawl’s new deep-research endpoint enables our agent to autonomously explore the web, gather relevant information, and synthesize findings into comprehensive insights.

Don’t forget to share this tutorial on your social channels and tag Unwind AI (X, LinkedIn, Threads, Facebook) to support us!

What We’re Building

This application implements a sophisticated multi-agent research system that utilizes OpenAI's Agents SDK and Firecrawl's web research capabilities to perform comprehensive research on any topic.

Features

1. Multi-Agent Architecture: The application uses two specialized agents -

   • Research Agent: Utilizes Firecrawl's deep research endpoint to gather comprehensive information from multiple web sources.

   • Elaboration Agent: Enhances the initial research by adding detailed explanations, examples, case studies, and practical implications.

2. Deep Web Research: Automatically searches the web, extracts content, and synthesizes findings

3. Enhanced Analysis: Second agent elaborates on initial research with additional context and insights

4. Interactive UI: Clean Streamlit interface for easy interaction

5. Downloadable Reports: Export research findings as markdown files

How The App Works

The Multi-Agent Deep Research system operates through a collaborative workflow:

1. Research Agent (Agent 1): The first agent uses Firecrawl to search multiple web sources, extract relevant content, and generate an initial research report. Think of this agent as your data gatherer and initial synthesizer.

2. Elaboration Agent (Agent 2): The second agent takes the baton, analyzing the initial report and transforming it into a more comprehensive document with additional context, examples, case studies, and implications. This agent acts as your content enhancer and expert editor.

3. Coordination Process: OpenAI's Agents SDK manages the handoff between these specialized agents, ensuring that each focuses on its specific strengths, similar to how a research team might have different specialists for data collection and analysis.

This multi-agent approach delivers better results than a single agent could provide—the first agent focuses entirely on gathering accurate information, while the second agent can dedicate its full attention to enhancing clarity, depth, and usefulness of the final report.

Prerequisites

Before we begin, make sure you have the following:

1. Python installed on your machine (version 3.10 or higher is recommended)

2. Your OpenAI and Firecrawl API key

3. A code editor of your choice (we recommend VS Code or PyCharm for their excellent Python support)

4. Basic familiarity with Python programming

Step-by-Step Instructions

Setting Up the Environment

First, let's get our development environment ready:

1. Clone the GitHub repository:

git clone https://github.com/Shubhamsaboo/awesome-llm-apps.git

🌟 Don't forget to star the opensource repo to show your support.

2. Go to the ai_deep_research_agent folder:

cd ai_agent_tutorials/ai_deep_research_agent

3. Install the required dependencies:

pip install -r requirements.txt

4. Grab your OpenAI and Firecrawl API keys.

Creating the Streamlit App

Let’s create our app. Create a new file deep_research_openai.py and add the following code:

1. Let's import our libraries:

import asyncio
import streamlit as st
from typing import Dict, Any, List
from agents import Agent, Runner, trace
from agents import set_default_openai_key
from firecrawl import FirecrawlApp
from agents.tool import function_tool

2. Set up the Streamlit app and configure the page:

# Set page configuration
st.set_page_config(
    page_title="Enhanced Research Assistant",
    page_icon="🔍",
    layout="wide"
)

# Initialize session state for API keys if not exists
if "openai_api_key" not in st.session_state:
    st.session_state.openai_api_key = ""
if "firecrawl_api_key" not in st.session_state:
    st.session_state.firecrawl_api_key = ""

3. Create a sidebar for API keys:

# Sidebar for API keys
with st.sidebar:
    st.title("API Configuration")
    openai_api_key = st.text_input(
        "OpenAI API Key",
        value=st.session_state.openai_api_key,
        type="password"
    )
    firecrawl_api_key = st.text_input(
        "Firecrawl API Key",
        value=st.session_state.firecrawl_api_key,
        type="password"
    )
    if openai_api_key:
        st.session_state.openai_api_key = openai_api_key
        set_default_openai_key(openai_api_key)
    if firecrawl_api_key:
        st.session_state.firecrawl_api_key = firecrawl_api_key

4. Create the main content area:

# Main content
st.title("🔍 Enhanced Deep Research Agent")
st.markdown("This multi-agent system performs comprehensive research using OpenAI Agents SDK and Firecrawl")

# Research topic input
research_topic = st.text_input("Enter your research topic:", placeholder="e.g., Latest developments in AI")

5. Define the deep research function tool:

@function_tool
async def deep_research(query: str, max_depth: int, time_limit: int, max_urls: int) -> Dict[str, Any]:
    """
    Perform comprehensive web research using Firecrawl's deep research endpoint.
    """
    try:
        # Initialize FirecrawlApp with the API key from session state
        firecrawl_app = FirecrawlApp(api_key=st.session_state.firecrawl_api_key)
        
        # Define research parameters
        params = {
            "maxDepth": max_depth,
            "timeLimit": time_limit,
            "maxUrls": max_urls
        }
        
        # Set up a callback for real-time updates
        def on_activity(activity):
            st.write(f"[{activity['type']}] {activity['message']}")
        
        # Run deep research
        with st.spinner("Performing deep research..."):
            results = firecrawl_app.deep_research(
                query=query,
                params=params,
                on_activity=on_activity
            )
        
        return {
            "success": True,
            "final_analysis": results['data']['finalAnalysis'],
            "sources_count": len(results['data']['sources']),
            "sources": results['data']['sources']
        }
    except Exception as e:
        st.error(f"Deep research error: {str(e)}")
        return {"error": str(e), "success": False}

6. Create the first agent - Research Agent:

# Create first agent in our multi-agent system - focused on gathering research
research_agent = Agent(
    name="research_agent",
    instructions="""You are a research assistant that can perform deep web research on any topic.
    When given a research topic or question:
    1. Use the deep_research tool to gather comprehensive information
    - Always use these parameters:
      * max_depth: 3 (for moderate depth)
      * time_limit: 180 (3 minutes)
      * max_urls: 10 (sufficient sources)
    2. The tool will search the web, analyze multiple sources, and provide a synthesis
    3. Review the research results and organize them into a well-structured report
    4. Include proper citations for all sources
    5. Highlight key findings and insights
    """,
    tools=[deep_research]
)

7. Create the second agent - Elaboration Agent:

# Create second agent in our multi-agent system - focused on enhancing content
elaboration_agent = Agent(
    name="elaboration_agent",
    instructions="""You are an expert content enhancer specializing in research elaboration.
    When given a research report:
    1. Analyze the structure and content of the report
    2. Enhance the report by:
    - Adding more detailed explanations of complex concepts
    - Including relevant examples, case studies, and real-world applications
    - Expanding on key points with additional context and nuance
    - Adding visual elements descriptions (charts, diagrams, infographics)
    - Incorporating latest trends and future predictions
    - Suggesting practical implications for different stakeholders
    3. Maintain academic rigor and factual accuracy
    4. Preserve the original structure while making it more comprehensive
    5. Ensure all additions are relevant and valuable to the topic
    """
)

8. Define the multi-agent research process function:

async def run_research_process(topic: str):
    """Run the complete multi-agent research process."""
    # Step 1: Initial Research with first agent
    with st.spinner("Agent 1: Conducting initial research..."):
        research_result = await Runner.run(research_agent, topic)
        initial_report = research_result.final_output
    
    # Display initial report in an expander
    with st.expander("View Initial Research Report (Agent 1 Output)"):
        st.markdown(initial_report)
    
    # Step 2: Enhance the report with second agent
    with st.spinner("Agent 2: Enhancing the report with additional information..."):
        elaboration_input = f"""
        RESEARCH TOPIC: {topic}
        INITIAL RESEARCH REPORT:
        {initial_report}
        Please enhance this research report with additional information, examples, case studies, 
        and deeper insights while maintaining its academic rigor and factual accuracy.
        """
        elaboration_result = await Runner.run(elaboration_agent, elaboration_input)
        enhanced_report = elaboration_result.final_output
    
    return enhanced_report

9. Implement the main research button and process:

# Main research process
if st.button("Start Multi-Agent Research", disabled=not (openai_api_key and firecrawl_api_key and research_topic)):
    if not openai_api_key or not firecrawl_api_key:
        st.warning("Please enter both API keys in the sidebar.")
    elif not research_topic:
        st.warning("Please enter a research topic.")
    else:
        try:
            # Create placeholder for the final report
            report_placeholder = st.empty()
            
            # Run the multi-agent research process
            enhanced_report = asyncio.run(run_research_process(research_topic))
            
            # Display the enhanced report
            report_placeholder.markdown("## Enhanced Research Report (Multi-Agent Output)")
            report_placeholder.markdown(enhanced_report)
            
            # Add download button
            st.download_button(
                "Download Report",
                enhanced_report,
                file_name=f"{research_topic.replace(' ', '_')}_report.md",
                mime="text/markdown"
            )
        except Exception as e:
            st.error(f"An error occurred: {str(e)}")

# Footer
st.markdown("---")
st.markdown("Powered by OpenAI Agents SDK and Firecrawl - Your own Deep Research solution without the subscription fees")

Running the App

With our code in place, it's time to launch the app.

• In your terminal, navigate to the project folder, and run the following command

streamlit run deep_research_openai.py

• Streamlit will provide a local URL (typically http://localhost:8501). Open this in your web browser, put in your API keys, enter a research topic, and click "Start Research" to begin the process.

Working Application Demo

Conclusion

You've successfully built a powerful multi-agent Deep Research system that delivers capabilities similar to premium services like Google, Perplexity, or OpenAI’s Deep Research but running on your own infrastructure with your own API keys.

Here are some ways you could enhance this project:

1. Adding a Third Agent for Fact-Checking: Create a verification agent that double-checks claims against additional sources.

2. Implementing Topic-Specific Agents: Create specialized versions of each agent for domains like science, finance, or technology with specialized knowledge and research methods.

3. Adding User Feedback Loop: Allow users to request specific revisions to the report, which gets routed to the appropriate agent.

4. Creating a Research History: Save previous research topics and results for quick reference and continuous improvement.

Keep experimenting with different agent configurations and features to build more sophisticated AI applications.

We share hands-on tutorials like this 2-3 times a week, to help you stay ahead in the world of AI. If you're serious about leveling up your AI skills and staying ahead of the curve, subscribe now and be the first to access our latest tutorials.

Don’t forget to share this tutorial on your social channels and tag Unwind AI (X, LinkedIn, Threads, Facebook) to support us!