SpiderTools is an open source web Scraper & Crawler that returns LLM-ready data. To start using Spider, you need an API key from the Spider dashboard.

​
Prerequisites

The following example requires the spider-client library. 
pip install -U spider-client

​
Example

The following agent will run a search query to get the latest news in USA and scrape the first search result. The agent will return the scraped data in markdown format.
cookbook/tools/spider_tools.py
from phi.agent import Agent
from phi.tools.spider import SpiderTools

agent = Agent(tools=[SpiderTools()])
agent.print_response('Can you scrape the first search result from a search on "news in USA"?', markdown=True)

​
Toolkit Params

ParameterTypeDefaultDescription
max_resultsint-The maximum number of search results to return
urlstr-The url to be scraped or crawled

​
Toolkit Functions

FunctionDescription
searchSearches the web for the given query.
scrapeScrapes the given url.
crawlCrawls the given url.

​
Information