🔬 News Research Agent

Build an AI research agent with ScrapeGraph and Tavily

Learn how to build a research agent that combines ScrapeGraph’s extraction capabilities with Tavily’s search API to gather and analyze news articles on specific topics.

Try it yourself in our News Research Agent Notebook

The Goal

We’ll extract the following information from news articles:

Field	Description
Title	Article headline
Link	URL to the article
Description	Article summary

Code Example

from pydantic import BaseModel, Field
from typing import List

# Schema for a single news item
class NewsItemSchema(BaseModel):
    title: str = Field(description="Title of the news article")
    link: str = Field(description="URL to the news article")
    description: str = Field(description="Summary/description of the news article")

# Schema that contains a list of news items
class ListNewsSchema(BaseModel):
    news: List[NewsItemSchema] = Field(description="List of news articles with their details")

# Agent implementation...
# 1. Uses Tavily to search for relevant articles
# 2. Extracts structured data using ScrapeGraph
# 3. Returns formatted news list

Example Output

{
  "news": [
    {
      "title": "Matternet adds ANRA's UTM tech to expand drone delivery",
      "link": "https://www.therobotreport.com/matternet-adds-anras-utm-tech-to-expand-drone-delivery/",
      "description": "This latest partnership follows Matternet's recent launch of a drone delivery operation in Silicon Valley."
    },
    {
      "title": "Helm.ai upgrades generative AI model to enrich autonomous driving data",
      "link": "https://www.therobotreport.com/helm-ai-upgrades-generative-ai-model-to-enrich-autonomous-driving-data/",
      "description": "Helm.ai said the new model enables automakers to generate diverse, realistic video data tailored to specific requirements."
    },
    {
      "title": "New research analyzes safety of Waymo robotaxis",
      "link": "https://www.therobotreport.com/new-research-analyzes-safety-of-waymo-robotaxis/",
      "description": "Waymo shared research with Swiss Re, one of the world's largest insurance providers, analyzing liability claims related to collisions from 25.3 million fully autonomous miles driven."
    },
    {
      "title": "From AI to humanoids: top robotics trends of 2024",
      "link": "https://www.therobotreport.com/from-ai-to-humanoids-top-robotics-trends-of-2024/",
      "description": "The Robot Report Podcast reflects on the successes and challenges that defined the robotics industry in 2024."
    },
    {
      "title": "Symbotic acquires OhmniLabs, maker of disinfection & telepresence robots",
      "link": "https://www.therobotreport.com/symbotic-buys-healthcare-robot-maker-ohmnilabs/",
      "description": "With the acquisition of OhmniLabs, Symbotic said it will be better positioned to expand its capabilities for supply chain customers."
    }
  ]
}

SmartScraper

Learn more about our AI-powered extraction service

Python SDK

Explore our Python SDK documentation

---

Have a suggestion for a new example? Contact us with your use case or contribute directly on GitHub.