Agentic Platform
Documentation API reference Tools Skills

Tools

URL Details from Botify SiteCrawler

site_crawler_url_details Url data

Find crawl data for a given URL.

Get comprehensive URL analysis from Botify's latest crawl, including: * Core metadata: title, meta description, H1s, H2s * Technical details: HTTP status code, response time, content type * Link relationships: canonical URLs, hreflang tags * Link graph analysis: internal links, external links, incoming links * Page content: structured data, main content analysis * JavaScript rendering: rendered vs HTML content, JS dependencies * Crawl data: crawl date, crawl depth, crawl path

Playground

POST
item
url required

The URL to analyze

question required

What information would you like to know about this URL

Credits

Fixed cost per URL requested. Crawl data comes from your own project and adds nothing; the summary is written by a model, so what you pay beyond the call scales with how much crawl data the question needs and how long the summary runs. Neither is capped.

These usage SKUs can be charged on a call.

SKU Credits Description
Tool call 1 per request Charged once per successful item, on top of any usage below.
GPT-4o (standard), input 2,500 per million tokens Tokens the model reads from the prompt you send.
GPT-4o (standard), output 10,000 per million tokens Tokens the model writes in its answer.
GPT-4o (standard), read from cache 1,250 per million tokens Tokens the model reads from a cached prompt. Cheaper than a fresh read.

Schemas

item is what you send, config how the tool behaves, and the response is what you get back.

Item

object
Properties
url string required

The URL to analyze

question string required

What information would you like to know about this URL

Example
{
  "url": "string",
  "question": "string"
}

Response

object
Properties
summary string required

A summary of the URL details

url_details_href string | null

A link to the detailed URL analysis in Botify Analytics

Default: null

Example
{
  "summary": "string",
  "url_details_href": null
}

Endpoints

POST http://agents.botify.com/{organization}/{project}/site_crawler_url_details/process
POST http://agents.botify.com/{organization}/{project}/site_crawler_url_details/batch_process
POST http://agents.botify.com/{organization}/{project}/site_crawler_url_details/async_process
POST http://agents.botify.com/{organization}/{project}/site_crawler_url_details/async_batch_process

See running a long job in the background for the polling flow.

cURL

curl -X POST "http://agents.botify.com/{organization}/{project}/site_crawler_url_details/process" \
  -H "Authorization: Bearer $BOTIFY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "item": {
    "url": "<url>",
    "question": "<question>"
  }
}'

Raw metadata