Agentic Platform
Documentation API reference Tools Skills

Tools

Sitemaps Detection

sitemaps_detection MCP Url data

Find the sitemaps declared in the robots.txt of a site's start URLs

Read the robots.txt of each origin of the given start URLs and return the sitemap URLs it declares (Sitemap: lines). Only absolute http(s) URLs are kept. Invalid start URLs and unreachable or failing robots.txt files are skipped (they yield no sitemap); the call errors only when no start URL is usable. The robots.txt is fetched like robots_txt_check does: without the project's user agent or authentication headers.

Playground

POST
item
start_urls required

Start URLs of the crawl. The robots.txt of each distinct origin (scheme, host, port) is read once. An explicit default port (:443 for https, :80 for http) is the same origin.

Credits

Fixed cost per call. A robots.txt we can fetch ourselves adds nothing. When an origin blocks the direct fetch, a proxied fetch may be billed for that origin.

These usage SKUs can be charged on a call, including SKUs from tools this one may call.

SKU Credits Description Used by
Tool call 1 per request Charged once per successful item, on top of any usage below.
  • This tool
Live page fetch 5 per 10 page Fetches a page from the live web as an ordinary visitor would.
Live page fetch, hard-blocked page 75 per 10 page A page that refused every gentler attempt. Charged instead of them: the attempts it refused cost nothing.
Live page fetch, protected page 25 per 10 page A page that refused an ordinary fetch and had to be retried. Charged instead of the ordinary fetch, not on top of it.

Schemas

item is what you send, config how the tool behaves, and the response is what you get back.

Item

object
Properties
start_urls array required

Start URLs of the crawl. The robots.txt of each distinct origin (scheme, host, port) is read once. An explicit default port (:443 for https, :80 for http) is the same origin.

min items 1, max items 20

Example
{
  "start_urls": [
    "string"
  ]
}

Response

object
Properties
sitemaps array required

Sitemap URLs declared in the robots.txt files, in order, without duplicates

robots_txt array required

One entry per origin, in the order of the start URLs

Items: RobotsTxtSource

Example
{
  "sitemaps": [
    "string"
  ],
  "robots_txt": [
    {
      "url": "string",
      "found": true,
      "status_code": 0
    }
  ]
}

Type Definitions

1 type

These are the available sub-object types that can be used in the properties above.

RobotsTxtSource object
#

The robots.txt read for one origin.

url string required

The robots.txt URL that was requested

found boolean required

Whether a robots.txt was returned (2xx)

status_code integer | null required

HTTP status, or null on transport failure

Endpoints

POST http://agents.botify.com/{organization}/{project}/sitemaps_detection/process
POST http://agents.botify.com/{organization}/{project}/sitemaps_detection/batch_process
POST http://agents.botify.com/{organization}/{project}/sitemaps_detection/async_process
POST http://agents.botify.com/{organization}/{project}/sitemaps_detection/async_batch_process

See running a long job in the background for the polling flow.

cURL

curl -X POST "http://agents.botify.com/{organization}/{project}/sitemaps_detection/process" \
  -H "Authorization: Bearer $BOTIFY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "item": {
    "start_urls": []
  }
}'

Over MCP

This tool is exposed as the MCP tool sitemaps_detection. An MCP client passes organization, project and item as tool arguments. Set up MCP →

Raw metadata