> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/nearai/ironclaw/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Fetch

> Fetch URLs and extract content as clean Markdown

## Overview

The web fetch tool retrieves URLs and extracts content as clean Markdown. Purpose-built for reading web pages, articles, and documentation with automatic HTML-to-Markdown conversion.

<Note>
  For API calls with custom methods, headers, or authentication, use the `http` tool instead.
</Note>

## web\_fetch

Fetch a URL and extract its content as clean Markdown. Always uses GET method and automatically converts HTML to readable Markdown.

**Input Parameters**

<ParamField path="url" type="string" required>
  HTTPS URL to fetch. Must be a public URL (no localhost or private IPs).
</ParamField>

**Output**

<ResponseField name="url" type="string">
  Original URL requested
</ResponseField>

<ResponseField name="final_url" type="string">
  Final URL after following redirects (max 3 hops)
</ResponseField>

<ResponseField name="status" type="integer">
  HTTP status code
</ResponseField>

<ResponseField name="title" type="string">
  Page title extracted from `<title>` tag (HTML only)
</ResponseField>

<ResponseField name="content" type="string">
  Clean Markdown content (HTML converted via Readability)
</ResponseField>

<ResponseField name="word_count" type="integer">
  Number of words in the content
</ResponseField>

**Example**

```json theme={null}
{
  "url": "https://example.com/article"
}
```

**Response**

```json theme={null}
{
  "url": "https://example.com/article",
  "final_url": "https://example.com/article",
  "status": 200,
  "title": "How to Build APIs",
  "content": "# How to Build APIs\n\nAPIs are the backbone of modern web applications...",
  "word_count": 1250
}
```

## Features

### Automatic HTML to Markdown

When fetching HTML pages:

* Extracts main content using Readability algorithm
* Removes navigation, ads, footers, and clutter
* Converts to clean Markdown format
* Preserves headings, links, lists, and code blocks
* Falls back to raw HTML if conversion fails

### Redirect Following

* Follows up to 3 redirects automatically
* Each redirect hop is SSRF-validated
* Returns final URL after all redirects
* Relative redirects resolved correctly

### Security

All the same security features as the `http` tool:

* **HTTPS-only**: HTTP requests rejected
* **SSRF protection**: Localhost and private IPs blocked
* **DNS rebinding defense**: All resolved IPs validated
* **Cloud metadata blocked**: 169.254.169.254 rejected
* **Leak detection**: URLs scanned for secrets
* **Size limits**: 5MB maximum response size

## Approval

<Info>
  Web fetch is **always auto-approved**. The SSRF and leak protections are unconditional, and reading public web pages doesn't require confirmation.
</Info>

Approval requirement: `Never`

## Rate Limiting

* 30 calls per minute
* 500 calls per hour

## Error Conditions

<ResponseField name="InvalidParameters">
  * Missing or invalid URL
</ResponseField>

<ResponseField name="NotAuthorized">
  * HTTP URL (not HTTPS)
  * Localhost or private IP address
  * Hostname resolves to disallowed IP
  * Outbound leak detected (secrets in URL)
</ResponseField>

<ResponseField name="ExternalService">
  * Network error
  * Failed to read response body
</ResponseField>

<ResponseField name="Timeout">
  * Request exceeded 30 second timeout
</ResponseField>

<ResponseField name="ExecutionFailed">
  * Too many redirects (max 3)
  * Redirect missing Location header
  * Response Content-Length exceeds 5MB
  * Response body exceeds 5MB during streaming
</ResponseField>

## Examples

### Documentation Page

```json theme={null}
{
  "url": "https://docs.rs/tokio/latest/tokio/"
}
```

**Response:**

```json theme={null}
{
  "url": "https://docs.rs/tokio/latest/tokio/",
  "final_url": "https://docs.rs/tokio/latest/tokio/",
  "status": 200,
  "title": "tokio - Rust",
  "content": "# tokio\n\nA runtime for writing reliable asynchronous applications...",
  "word_count": 850
}
```

### Blog Article

```json theme={null}
{
  "url": "https://blog.rust-lang.org/2024/01/15/Rust-1.75.0.html"
}
```

### GitHub README

```json theme={null}
{
  "url": "https://github.com/rust-lang/rust/blob/master/README.md"
}
```

## Comparison with HTTP Tool

| Feature | web\_fetch | http |
| - | - | - |
| Methods | GET only | GET, POST, PUT, DELETE, PATCH |
| Headers | None (auto User-Agent) | Custom headers supported |
| Body | None | Request body supported |
| HTML Conversion | Always attempted | Optional (if enabled) |
| Output | Markdown-focused | Raw response |
| Approval | Never required | Required unless auto-approved |
| Use Case | Reading web content | API calls, authentication |

## Use Cases

### Reading Documentation

Fetch API docs, library documentation, or technical articles:

```json theme={null}
{
  "url": "https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API"
}
```

### Research

Gather information from multiple sources:

```json theme={null}
{
  "url": "https://en.wikipedia.org/wiki/REST"
}
```

### Content Extraction

Extract clean content from blog posts or news articles:

```json theme={null}
{
  "url": "https://example.com/blog/how-to-rust"
}
```

## Technical Details

### User Agent

Requests use a Chrome-like User-Agent to avoid bot detection:

```
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
```

### Accept Header

Prefers Markdown but accepts HTML:

```
text/markdown, text/html;q=0.9, */*;q=0.8
```

### Title Extraction

Extracts `<title>` tag from HTML using ASCII-safe case-folding to handle Unicode correctly.

### Content Type Detection

HTML conversion triggered when `Content-Type` header contains `text/html`.