Agentic Platform
Documentation API reference Tools Skills

Tools

Robots.txt Check

robots_txt_check MCP Url data

Fetch a host's robots.txt and optionally test URLs against a user-agent

Fetch robots.txt for a domain and optionally check whether URLs are allowed or disallowed for a given user-agent. Matching uses Google's robots.txt parser. A 2xx body (including an empty file) is passed to the matcher. A 4xx response other than 429 is treated as no robots.txt, so every URL is allowed. A 429, a 5xx response, or a transport failure is treated as a temporary full disallow, so every URL is disallowed. The matcher does not fold a more specific bot such as Googlebot-Image into Googlebot. Pass the user-agent token you want to test.

Playground

POST
item
domain required

Hostname (www.example.com) or absolute URL. Reduced to an origin (scheme + host + port). https is assumed when the scheme is omitted.

urls

URLs to test against the fetched robots.txt. Absolute http(s) URLs, or paths (/private) resolved against the origin. Each URL must use the same scheme, host, and port. At most 20.

user_agent

User-agent token passed to the matcher. Required when urls is non-empty. Not defaulted; a specific bot such as Googlebot-Image is not folded into Googlebot.

Credits

Fixed cost per call. A robots.txt we can fetch ourselves adds nothing. When the origin blocks the fetch (401, 403, 407, or 429) or returns a server error, a live page fetch is billed. A 5xx may be retried on harder routes, and only the route that got the file is billed. A private, loopback, or unresolvable host is not retried.

These usage SKUs can be charged on a call.

SKU Credits Description
Tool call 1 per request Charged once per successful item, on top of any usage below.
Live page fetch 5 per 10 page Fetches a page from the live web as an ordinary visitor would.
Live page fetch, hard-blocked page 75 per 10 page A page that refused every gentler attempt. Charged instead of them: the attempts it refused cost nothing.
Live page fetch, protected page 25 per 10 page A page that refused an ordinary fetch and had to be retried. Charged instead of the ordinary fetch, not on top of it.

Schemas

item is what you send, config how the tool behaves, and the response is what you get back.

Item

object
Properties
domain string required

Hostname (www.example.com) or absolute URL. Reduced to an origin (scheme + host + port). https is assumed when the scheme is omitted.

urls array

URLs to test against the fetched robots.txt. Absolute http(s) URLs, or paths (/private) resolved against the origin. Each URL must use the same scheme, host, and port. At most 20.

Default: []

max items 20

user_agent string | null

User-agent token passed to the matcher. Required when urls is non-empty. Not defaulted; a specific bot such as Googlebot-Image is not folded into Googlebot.

Default: null

Example
{
  "domain": "string",
  "urls": [],
  "user_agent": null
}

Response

object
Properties
found boolean required

True when the fetch returned HTTP 2xx, including an empty file

content string | null required

robots.txt body on 2xx, truncated to 500 KiB. Null when found is false. Empty file is found=true and content=''.

status_code integer | null required

HTTP status of the robots.txt response, or null on transport failure

url string required

The robots.txt URL that was requested

checks array

One entry per input URL, in order. Empty when urls was omitted. allowed is true when the URL passes robots.txt.

Items: RobotsTxtUrlCheck

Default: []

Example
{
  "found": true,
  "content": "string",
  "status_code": 0,
  "url": "string",
  "checks": []
}

Type Definitions

1 type

These are the available sub-object types that can be used in the properties above.

RobotsTxtUrlCheck object
#

Whether one URL is allowed or disallowed by robots.txt.

url string required

The absolute URL that was matched

allowed boolean required

True when the URL passes robots.txt, false when it is disallowed

Endpoints

POST http://agents.botify.com/{organization}/{project}/robots_txt_check/process
POST http://agents.botify.com/{organization}/{project}/robots_txt_check/batch_process
POST http://agents.botify.com/{organization}/{project}/robots_txt_check/async_process
POST http://agents.botify.com/{organization}/{project}/robots_txt_check/async_batch_process

See running a long job in the background for the polling flow.

cURL

curl -X POST "http://agents.botify.com/{organization}/{project}/robots_txt_check/process" \
  -H "Authorization: Bearer $BOTIFY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "item": {
    "domain": "<domain>"
  }
}'

Over MCP

This tool is exposed as the MCP tool robots_txt_check. An MCP client passes organization, project and item as tool arguments. Set up MCP →

Raw metadata