Agentic Platform
Documentation API reference Tools Skills

Tools

Site Crawler URL Detail

site_crawler_url_detail MCP Url data

Fetch site crawler crawl data for a given URL.

Get data for a URL from the latest successful site crawler analysis, including: * Core metadata: title, meta description, H1s, H2s * Technical details: HTTP status code, response time, content type * Link relationships: canonical URLs, hreflang tags * Link graph analysis: internal links, external links, incoming links * Page content: structured data, main content analysis * JavaScript rendering: rendered vs HTML content, JS dependencies * Crawl data: crawl date, crawl depth, crawl path

Playground

POST
item
url required

The URL to look up

config
save_to_tmp_storage

Save the bulky URL details to Botify temporary storage and return the temporary URL instead of the payload. Use this to save output tokens.

Credits

These usage SKUs can be charged on a call.

SKU Credits Description
Tool call 1 per request Charged once per successful item.

Schemas

item is what you send, config how the tool behaves, and the response is what you get back.

Item

object
Properties
url string required

The URL to look up

Example
{
  "url": "string"
}

Configuration

object
Properties
save_to_tmp_storage boolean

Save the bulky URL details to Botify temporary storage and return the temporary URL instead of the payload. Use this to save output tokens.

Default: false

Example
{
  "save_to_tmp_storage": false
}

Response

object
Properties
url_data object | null

Raw URL details from the latest successful crawl. Omitted when `save_to_tmp_storage` is True; see `url_data_tmp_storage_url` instead.

Default: null

url_data_tmp_storage_url string | null

Temporary storage URL of the raw URL details. Returned instead of `url_data` when `save_to_tmp_storage` is True.

Default: null

url_details_href string | null

A link to the detailed URL analysis in Botify Analytics

Default: null

error string | null

Set when crawl data could not be returned. `no_crawl_data`: the project has no successful crawl. `url_not_crawled`: the URL is missing from the latest successful crawl.

Default: null

Example
{
  "url_data": null,
  "url_data_tmp_storage_url": null,
  "url_details_href": null,
  "error": null
}

Endpoints

POST http://agents.botify.com/{organization}/{project}/site_crawler_url_detail/process
POST http://agents.botify.com/{organization}/{project}/site_crawler_url_detail/batch_process
POST http://agents.botify.com/{organization}/{project}/site_crawler_url_detail/async_process
POST http://agents.botify.com/{organization}/{project}/site_crawler_url_detail/async_batch_process

See running a long job in the background for the polling flow.

cURL

curl -X POST "http://agents.botify.com/{organization}/{project}/site_crawler_url_detail/process" \
  -H "Authorization: Bearer $BOTIFY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "item": {
    "url": "<url>"
  }
}'

Over MCP

This tool is exposed as the MCP tool site_crawler_url_detail. An MCP client passes organization, project and item as tool arguments. Set up MCP →

Raw metadata