WebNavigator

A web navigator agentic object with tools to fetch, render, and screenshot web pages.

Description

WebNavigator gives the agent the ability to retrieve and analyze web content at multiple tiers:

• Fast HTTP fetch — retrieve and parse static HTML pages
• Headless Chrome rendering — execute JavaScript for SPAs and dynamic content
• Screenshot capture — save a visual snapshot of a rendered page

Tools

fetch_raw(url: str, timeout: int = 15) -> dict

Fetch the raw HTTP response including headers and body.

Returns a dict with keys: url, status_code, headers, content. Useful for debugging HTTP responses, testing server behavior, or inspecting the raw output before applying any parsing.

Args:

• url — The URL to fetch (must start with http:// or https://).
• timeout — Maximum seconds to wait for the HTTP response.

Returns: Dict with status_code, headers, and content.

fetch_text(url: str, timeout: int = 15) -> str

Fetch text content from a web page using HTTP and HTML parsing.

Uses httpx to retrieve the page HTML, then BeautifulSoup with lxml to extract clean text, stripping scripts, styles, and navigation elements. Also returns discovered links.

This is the fastest option and works well for most static HTML sites. It does NOT execute JavaScript, so single-page apps (React, Vue, etc.) may return empty or minimal content.

Args:

• url — The URL to fetch.
• timeout — Maximum seconds to wait for the HTTP response.

Returns: A string with the extracted text content, followed by a section listing all discovered links.

render_page(url: str, timeout: int = 30) -> str

Render a web page with a headless Chrome browser and extract text.

Spawns google-chrome (or chromium-browser) in headless mode with --dump-dom to execute JavaScript and capture the fully rendered DOM. BeautifulSoup then extracts clean text and links.

This is slower than fetch_text but works for JavaScript-rendered pages (SPAs, dynamically loaded content).

Args:

• url — The URL to render.
• timeout — Maximum seconds for Chrome to load the page.

Returns: A string with the extracted text content, followed by a section listing all discovered links.

take_screenshot(url: str, output_file: str | None = None, timeout: int = 30) -> str

Capture a screenshot of a web page using headless Chrome.

Spawns Chrome in headless mode with --headless=new --screenshot and saves the PNG to a file. Returns the path to the saved screenshot, which can then be read by another agentic object with image capabilities.

Args:

• url — The URL to screenshot.
• output_file — Optional output path. If not provided, a file is created in a temporary directory.
• timeout — Maximum seconds for Chrome to load the page.

Returns: A string with the path to the saved screenshot, or an error message.

Usage

To use the WebNavigator, derive from it as a parent class of your agentic object:

class MyAgent(WebNavigator, AgenticObject):
    """You are an agent with web navigation capabilities."""

The agent will have access to the fetch_raw, fetch_text, render_page, and take_screenshot tools through its system prompt.