Agentic Platform
Documentation API reference Tools Skills

Tools

HTML Fetch

html_fetch MCP

Get the HTML of a given URL.

Fetch the HTML of a URL, from the Botify SiteCrawler archive by default, or live through a real-time fetch or the rendering farm (which executes the page JavaScript) when `origin` asks for it. The HTML is cleaned before being returned: `remove_tags` drops the listed tags, `only_body` keeps the body only and `only_text` strips the markup down to text.

Playground

Your organization, project and token stay in this browser's local storage and are sent straight to the API — this page never sees them.

POST
item
url required The URL to fetch HTML for. Temporary uploaded URLs on https://app.botify.com/:organization/:project/o/storage/tmp are allowed.
origin The origin of the URL
country_code The country code to use for the real-time fetch. 2 letters lowercase
fallback_real_time_fetch Fallback to real_time_fetch_url if the URL is not found in SiteCrawler
only_text Only return the text of the page
only_body Only return the body of the page
remove_tags Remove all tags from the HTML
save_to_tmp_storage Save the fetched HTML to Botify temporary storage and return the temporary URL instead of the HTML. Use this to save output tokens. The returned URL can be passed to other HTML-processing tools.

Billing

Fixed cost per URL requested. A page already in your SiteCrawler crawl is read from it and adds nothing. A live page fetch is billed only when you ask for one, or when the page is missing from the crawl and you allowed the fallback. Pages that refuse an ordinary fetch are retried on harder routes and every attempt is billed, so a protected page costs a live page fetch plus a protected-page fetch, and a hard-blocked one the whole sequence. Botify remembers for a day which route worked for a domain, so repeat fetches of the same site usually stay on the first.

These usage SKUs can be charged on a call.

SKU Credits Description
Tool call 1 per request Charged once per successful item, on top of any usage below.
Live page fetch 495 per 1,000 page Fetches a page from the live web as an ordinary visitor would.
Live page fetch, hard-blocked page 7,425 per 1,000 page A page that refused every gentler attempt. Charged on top of them, so a hard-blocked page costs the whole sequence.
Live page fetch, protected page 2,475 per 1,000 page A page that refused an ordinary fetch and had to be retried. Charged on top of the ordinary attempt, not instead of it.

Schemas

item is what you send, config how the tool behaves, and the response is what you get back.

Item

object
Properties
url string required

The URL to fetch HTML for. Temporary uploaded URLs on https://app.botify.com/:organization/:project/o/storage/tmp are allowed.

origin OriginEnum

The origin of the URL

Type: OriginEnum

Default: "site_crawler"

country_code string

The country code to use for the real-time fetch. 2 letters lowercase

Default: "us"

fallback_real_time_fetch boolean

Fallback to real_time_fetch_url if the URL is not found in SiteCrawler

Default: false

only_text boolean

Only return the text of the page

Default: false

only_body boolean

Only return the body of the page

Default: false

remove_tags array

Remove all tags from the HTML

Default: ["script", "noscript", "style", "header", "footer", "nav", "aside", "form", "svg", "iframe"]

save_to_tmp_storage boolean

Save the fetched HTML to Botify temporary storage and return the temporary URL instead of the HTML. Use this to save output tokens. The returned URL can be passed to other HTML-processing tools.

Default: false

Example
{
  "url": "string",
  "origin": "site_crawler",
  "country_code": "us",
  "fallback_real_time_fetch": false,
  "only_text": false,
  "only_body": false,
  "remove_tags": [
    "script",
    "noscript",
    "style",
    "header",
    "footer",
    "nav",
    "aside",
    "form",
    "svg",
    "iframe"
  ],
  "save_to_tmp_storage": false
}

Type Definitions

1 type

These are the available sub-object types that can be used in the properties above.

OriginEnum string
#

Values: site_crawler real_time_fetch_url rendering_farm temporary_upload

Response

object
Properties
origin OriginEnum

The origin of the HTML (site_crawler, real_time_fetch_url, rendering_farm, or temporary_upload)

Type: OriginEnum

Default: null

crawl string | null

Crawl Slug from SiteCrawler

Default: null

html string | null

HTML content of the page. Omitted when `save_to_tmp_storage` is True; see `html_tmp_storage_url` instead.

Default: null

html_tmp_storage_url string | null

Temporary storage URL of the HTML page. Returned instead of `html` when `save_to_tmp_storage` is True. Can be passed to other HTML-processing tools.

Default: null

status_code integer | null

HTTP status code of the response

Default: null

headers object | null

HTTP headers of the response

Default: null

iframe_allowed boolean | null

Whether iframes are allowed in the response

Default: null

reason string | null

Reason for failure, if any

Default: null

Example
{
  "origin": null,
  "crawl": null,
  "html": null,
  "html_tmp_storage_url": null,
  "status_code": null,
  "headers": null,
  "iframe_allowed": null,
  "reason": null
}

Type Definitions

1 type

These are the available sub-object types that can be used in the properties above.

OriginEnum string
#

Values: site_crawler real_time_fetch_url rendering_farm temporary_upload

Endpoints

POST http://agents.botify.com/{organization}/{project}/html_fetch/process
POST http://agents.botify.com/{organization}/{project}/html_fetch/batch_process
POST http://agents.botify.com/{organization}/{project}/html_fetch/async_process
POST http://agents.botify.com/{organization}/{project}/html_fetch/async_batch_process

See running a long job in the background for the polling flow.

cURL

curl -X POST "http://agents.botify.com/{organization}/{project}/html_fetch/process" \
  -H "Authorization: Bearer $BOTIFY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "item": {
    "url": "<url>"
  }
}'

Over MCP

This tool is exposed as the MCP tool html_fetch. An MCP client passes organization, project and item as tool arguments. Set up MCP →

Raw metadata