Tools
Crawl4AI
Tools
Crawl4AI
Crawl4aiTools enable an Agent to perform web crawling and scraping tasks using the Crawl4ai library.
Prerequisites
The following example requires the crawl4ai library.
pip install -U crawl4ai
Example
The following agent will scrape the content from the https://github.com/phidatahq/phidata webpage:
cookbook/tools/crawl4ai_tools.py
from phi.agent import Agent
from phi.tools.crawl4ai_tools import Crawl4aiTools
agent = Agent(tools=[Crawl4aiTools(max_length=None)], show_tool_calls=True)
agent.print_response("Tell me about https://github.com/phidatahq/phidata."
Toolkit Params
| Parameter | Type | Default | Description |
|---|---|---|---|
max_length | int | 1000 | Specifies the maximum length of the text from the webpage to be returned. |
Toolkit Functions
| Function | Description |
|---|---|
web_crawler | Crawls a website using crawl4ai’s WebCrawler. Parameters include ‘url’ for the URL to crawl and an optional ‘max_length’ to limit the length of extracted content. The default value for ‘max_length’ is 1000. |