SpiderTools is an open source web Scraper & Crawler that returns LLM-ready data. To start using Spider, you need an API key from the Spider dashboard.

Prerequisites

The following example requires the spider-client library.

pip install -U spider-client

Example

The following agent will run a search query to get the latest news in USA and scrape the first search result. The agent will return the scraped data in markdown format.

cookbook/tools/spider_tools.py
from phi.agent import Agent
from phi.tools.spider import SpiderTools

agent = Agent(tools=[SpiderTools()])
agent.print_response('Can you scrape the first search result from a search on "news in USA"?', markdown=True)

Toolkit Params

ParameterTypeDefaultDescription
max_resultsint-The maximum number of search results to return
urlstr-The url to be scraped or crawled

Toolkit Functions

FunctionDescription
searchSearches the web for the given query.
scrapeScrapes the given url.
crawlCrawls the given url.

Information

SpiderTools is an open source web Scraper & Crawler that returns LLM-ready data. To start using Spider, you need an API key from the Spider dashboard.

Prerequisites

The following example requires the spider-client library.

pip install -U spider-client

Example

The following agent will run a search query to get the latest news in USA and scrape the first search result. The agent will return the scraped data in markdown format.

cookbook/tools/spider_tools.py
from phi.agent import Agent
from phi.tools.spider import SpiderTools

agent = Agent(tools=[SpiderTools()])
agent.print_response('Can you scrape the first search result from a search on "news in USA"?', markdown=True)

Toolkit Params

ParameterTypeDefaultDescription
max_resultsint-The maximum number of search results to return
urlstr-The url to be scraped or crawled

Toolkit Functions

FunctionDescription
searchSearches the web for the given query.
scrapeScrapes the given url.
crawlCrawls the given url.

Information