Tools

HTML Fetch

html_fetch MCP

Get the HTML of a given URL.

Fetch the HTML of a URL, from the Botify SiteCrawler archive by default, or live through a real-time fetch or the rendering farm (which executes the page JavaScript) when `origin` asks for it. The HTML is cleaned before being returned: `remove_tags` drops the listed tags, `only_body` keeps the body only and `only_text` strips the markup down to text. Set `save_to_tmp_storage` to upload the HTML to temporary project storage and return its URL instead of the HTML inline. This option writes a new temporary object. A rendering-farm fetch leaves from the same network the project's own crawl renders from, resolved from the project settings against the execution environments the farm offers. It falls back to the farm's default environment when those settings or that list are out of reach, or when no environment serves the project's network. `rendering_farm_network` renders from a named network instead, and fails rather than falling back.

Playground

Organization Project

POST

item

url required

The URL to fetch HTML for. Temporary uploaded URLs on https://app.botify.com/:organization/:project/o/storage/tmp are allowed.

origin

The origin of the URL

country_code

The country code to use for the real-time fetch. 2 letters lowercase

fallback_real_time_fetch

Fallback to real_time_fetch_url if the URL is not found in SiteCrawler

only_text

Only return the text of the page

only_body

Only return the body of the page

remove_tags

Remove all tags from the HTML

config

save_to_tmp_storage

Save the fetched HTML to Botify temporary storage and return the temporary URL instead of the HTML. Use this to save output tokens. The returned URL can be passed to `tmp_storage_read`, `tmp_storage_grep`, `tmp_storage_jq`, and other HTML-processing tools.

Credits

Fixed cost per URL requested. A page already in your SiteCrawler crawl is read from it and adds nothing. A live page fetch is billed only when you ask for one, or when the page is missing from the crawl and you allowed the fallback. Pages that refuse an ordinary fetch are retried on harder routes, and only the route that got the page is billed: a protected page costs a protected-page fetch alone, and a hard-blocked one a hard-blocked fetch alone. Botify remembers for a day which route worked for a domain, so repeat fetches of the same site go straight to it.

These usage SKUs can be charged on a call.

SKU	Credits	Description
Tool call	1 per request	Charged once per successful item, on top of any usage below.
Live page fetch	5 per 10 page	Fetches a page from the live web as an ordinary visitor would.
Live page fetch, hard-blocked page	75 per 10 page	A page that refused every gentler attempt. Charged instead of them: the attempts it refused cost nothing.
Live page fetch, protected page	25 per 10 page	A page that refused an ordinary fetch and had to be retried. Charged instead of the ordinary fetch, not on top of it.

Schemas

item is what you send, config how the tool behaves, and the response is what you get back.

Item

object

Properties

url string required

The URL to fetch HTML for. Temporary uploaded URLs on https://app.botify.com/:organization/:project/o/storage/tmp are allowed.

origin OriginEnum

The origin of the URL

Type: OriginEnum

Default: "site_crawler"

country_code string

The country code to use for the real-time fetch. 2 letters lowercase

Default: "us"

fallback_real_time_fetch boolean

Fallback to real_time_fetch_url if the URL is not found in SiteCrawler

Default: false

only_text boolean

Only return the text of the page

Default: false

only_body boolean

Only return the body of the page

Default: false

remove_tags array

Remove all tags from the HTML

Default: ["script", "noscript", "style", "header", "footer", "nav", "aside", "form", "svg", "iframe"]

Example

{
  "url": "string",
  "origin": "site_crawler",
  "country_code": "us",
  "fallback_real_time_fetch": false,
  "only_text": false,
  "only_body": false,
  "remove_tags": [
    "script",
    "noscript",
    "style",
    "header",
    "footer",
    "nav",
    "aside",
    "form",
    "svg",
    "iframe"
  ]
}

Type Definitions

1 type

These are the available sub-object types that can be used in the properties above.

OriginEnum string

Values: site_crawler real_time_fetch_url rendering_farm temporary_upload

Configuration

object

Properties

save_to_tmp_storage boolean

Default: false

Example

{
  "save_to_tmp_storage": false
}

Response

object

Properties

origin OriginEnum

The origin of the HTML (site_crawler, real_time_fetch_url, rendering_farm, or temporary_upload)

Type: OriginEnum

Default: null

crawl string | null

Crawl Slug from SiteCrawler

Default: null

html string | null

HTML content of the page. Omitted when `save_to_tmp_storage` is True; see `html_tmp_storage_url` instead.

Default: null

html_tmp_storage_url string | null

Temporary storage URL of the HTML page. Returned instead of `html` when `save_to_tmp_storage` is True. Can be passed to `tmp_storage_read`, `tmp_storage_grep`, `tmp_storage_jq`, and other HTML-processing tools.

Default: null

status_code integer | null

HTTP status code of the response

Default: null

headers object | null

HTTP headers of the response

Default: null

iframe_allowed boolean | null

Whether iframes are allowed in the response

Default: null

reason string | null

Reason for failure, if any

Default: null

Example

{
  "origin": null,
  "crawl": null,
  "html": null,
  "html_tmp_storage_url": null,
  "status_code": null,
  "headers": null,
  "iframe_allowed": null,
  "reason": null
}

Type Definitions

1 type

These are the available sub-object types that can be used in the properties above.

OriginEnum string

Values: site_crawler real_time_fetch_url rendering_farm temporary_upload

Endpoints

POST http://agents.botify.com/{organization}/{project}/html_fetch/process

POST http://agents.botify.com/{organization}/{project}/html_fetch/batch_process

POST http://agents.botify.com/{organization}/{project}/html_fetch/async_process

POST http://agents.botify.com/{organization}/{project}/html_fetch/async_batch_process

See running a long job in the background for the polling flow.

cURL

curl -X POST "http://agents.botify.com/{organization}/{project}/html_fetch/process" \
  -H "Authorization: Bearer $BOTIFY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "item": {
    "url": "<url>"
  }
}'

Over MCP

This tool is exposed as the MCP tool html_fetch. An MCP client passes organization, project and item as tool arguments. Set up MCP →

Raw metadata

GET /agents/html_fetch — this page as JSON.
/agents/html_fetch/skill.md — Markdown for coding agents.