Botify MCP

Every Botify MCP tool processes exactly one item. Fanning a tool out is batch_run's job: the records are read from a SQL query, a table or a file, and the results are written to a sink. They do not pass through this conversation.

For SQL over the catalog (tables_list, tables_query, catalog. prefix), also read ../botify-tables-sql/SKILL.md.

Tools may be deferred: call tool_search (e.g. tool_search(query="batch_run html_code_executor")) before the first use. Every call except list_projects needs organization and project. Every agent tool except send_feedback also needs main_task and sub_task as top-level arguments next to those slugs — not inside config or items. If the slugs are unknown, call list_projects rather than guessing. Retry a timed-out connector once or twice before treating it as a failure.

After delivering work — a question answered, a batch finished — call send_feedback. Same if a tool failed unexpectedly. If the user then gives new indications, stay on the same main_task and call send_feedback again when there is new feedback. Do not call it every turn.

One item per call

tool_name
  organization: acme
  project: www
  main_task: Internal linking audit
  sub_task: List available tables
  items:
    - { …one item matching the tool's schema… }
  config: { …optional, shared settings… }

main_task is the label of the overall task; reuse it on every call, including when the user adds new indications. sub_task is the label of the current step. send_feedback does not take them. Do not put them in config.

items is an array of length 1. A second element is refused with an error that names batch_run. Do not loop the same tool tens of times as a workaround.

Exception: speedworkers_page_history takes a list of up to 1,000 URLs inside its one item and scans its table once per call. Put every URL in that list rather than batching it per URL.

Standalone tools (no items): list_projects, batch_status, batch_sample, batch_cancel.

When to use batch_run

The threshold is 10. At 11 records the run becomes a job: you get a job_id, not the data. That is the point.

For a job with several steps per item, if the workflows_* tools are in your tool list, read ../botify-workflows/SKILL.md.

The failure to avoid: tables_query a few thousand URLs, then call html_fetch / html_code_executor / html_grep once per row. The list has already entered the context, each page would again, and you will hit the one-item cap. Explore with aggregates and a LIMIT of 5–20, then hand the same SQL to batch_run as a sql source.

rows is the only source whose content travels through the model. It is capped at 256 KB and refused above that. Prefer sql or file.

batch_run

Read the target tool's item schema first (tool_documentation when it has extended docs — html_code_executor does). item_map must match that schema. batch_run cannot target itself.

batch_run
  organization: acme
  project: www
  main_task: Internal linking audit on last crawl
  sub_task: Extract product h1s
  items:
    - tool: html_code_executor
      source:
        type: sql
        query: |
          SELECT url
          FROM catalog.crawl_pages_20260816
          WHERE segments__pagetype__value = 'product'
            AND compliant__is_compliant
          LIMIT 500
      item_map:
        url: "{{row.url}}"
      tool_config:
        function: "function() { return {h1: document.querySelector('h1')?.innerText}; }"
      dry_run: true
  config:
    on_error: skip

Then the same call with dry_run: false.

Sources

Do not tables_query the full set and copy it into rows. A COUNT(*) plus LIMIT 20 is enough to understand the population; batch_run re-runs the query itself.

Ask the user to upload a local file rather than pasting it.

item_map

A projection, not an expression language: {{row.<column>}} and dotted paths. A path that does not resolve is refused on the sample before the job starts.

• {"url": "{{row.page_url}}"} — whole placeholder: type is preserved (number stays a number)
• "https://{{row.domain}}/{{row.path}}" — interpolation: always a string
• {{row.blocks.0.text}} — walk objects and list indexes
• Namespace is row only ({{rows.url}} is rejected)

Omit item_map when the source columns already are the tool's item (a rows source written in the item shape, or a SQL SELECT url against html_code_executor).

Sinks and chaining

Each written record is {item, response, error}. item is kept so a tool that does not echo its input stays attributable.

File (default). Gzipped JSONL. The returned sink.url is the file source of the next batch. A previous batch's lines become row.item.* and row.response.*:

item_map:
  user_prompt: "Classify this title: {{row.response.result.h1}}"

Table. Editable catalog, then query it with tables_query instead of pulling the file back.

sink:
  type: table
  table: product_h1s
  replace_existing: false
  row_map:
    url: "{{item.url}}"
    h1: "{{response.result.h1}}"

row_map sees {{item.*}}, {{response.*}} and {{row.*}}. It is not strict: a failed record has no response. The default (no row_map) stores raw item and response JSON columns. Never write into crawl/logs/GSC tables; pick a new name.

What comes back

Do not try to fetch a job's full output into the conversation.

Following a job

batch_status    organization, project, job_id
batch_sample    organization, project, job_id, limit (default 5, hard cap 20)
batch_cancel    organization, project, job_id

batch_status states: pending, running, completed, failed, cancelled. Poll it; do not start a second job for the same work. rows_ok / rows_error and sample_errors tell you whether to cancel.

batch_sample is a peek so you can check the mapping before trusting the sink. The cap of 20 exists because a batch exists to keep its data out of the chat. Do not raise limit to "download" the output.

batch_cancel stops at the next chunk. Records already written stay in the sink; the rest is not paid for.

Temporary storage URLs

Bulky payloads (HTML, SERP JSON) do not have to enter the conversation. Set save_to_tmp_storage: true in the tool's config and the result carries a temporary URL instead of the payload:

User uploads from https://app.botify.com/:organization/:project/o/storage/tmp are the same kind of URL. A URL only works in its own organization and project, and expires.

Look inside it with the tmp_storage_* tools, which return only what you ask for:

Gzip is decompressed for you, up to 100 MiB. Prefer tmp_storage_jq on JSON and tmp_storage_grep on HTML or text; tmp_storage_read is the last resort. When the JSON shape is unknown, start with expression: keys, or .[0:3] with slurp: true on JSONL.

tmp_storage_jq
  organization: acme
  project: www
  main_task: SERP competitors
  sub_task: Organic URLs for running shoes
  items:
    - url: <results_tmp_storage_url>
  config:
    expression: '.items[] | select(.type == "organic") | .url'

For many URLs, batch the tmp_storage_* tool over the previous sink: source: {type: file, url: <previous sink.url>}, item_map: {url: "{{row.response.results_tmp_storage_url}}"} and the jq expression or grep queries in tool_config.

Typical pipelines

1. Extract structured fields from many crawled URLs

1. tables_list / tables_query — latest SiteCrawler crawl, COUNT(*) and a 10-row sample of the population.
2. tool_documentation html_code_executor — confirm the JS DOM API.
3. batch_run dry_run: true — sql source, item_map: {url: "{{row.url}}"}, JS in tool_config.function.
4. Same call, dry_run: false. If mode is job, poll batch_status.
5. batch_sample a few records. Chain the sink.url or query the table sink.

2. Grep a string or regex across a URL set

Same as above with tool: html_grep. url is the item; queries go in tool_config (shared by every record). Prefer this to fetching HTML and searching in the prompt.

3. Fetch HTML without pouring it into the chat

Do not batch html_fetch if you would then read html. Either:

• set save_to_tmp_storage: true in tool_config (shared by every record) so each result is a temporary URL, then batch html_code_executor / html_grep / tmp_storage_grep on those URLs, or
• skip the fetch and batch html_code_executor / html_grep on the live URLs (they load crawl HTML themselves).

4. User-supplied CSV

Upload at https://app.botify.com/:organization/:project/o/storage/tmp, then:

source:
  type: file
  url: <returned URL>
item_map:
  url: "{{row.url}}"

5. Second tool over the first batch's output

source:
  type: file
  url: <previous sink.url>
item_map:
  user_prompt: "{{row.response.result.h1}}"
tool_config:
  # ai_gateway system_prompt, structured_output_json_schema, model_settings

send_feedback

Call this after delivering work, and also when a tool fails unexpectedly or the data looks wrong. If the user continues the same main_task with new indications, call it again. Do not call it every turn.

Botify would be very grateful: say what went well or badly, and which tools you wished you had from an SEO/GEO product like Botify (crawl data, server logs, keywords, AI visibility, internal linking, and similar).

send_feedback
  organization: acme
  project: www
  items:
    - experience: negative   # positive | neutral | negative
      message: |
        html_code_executor could not read shadow-DOM product titles.
        A crawl-extracted JSON-LD tool would have helped.

experience is required (positive, neutral, or negative). message is the debrief.

Do not

• Skip send_feedback after delivering work, or after new indications on the same main_task.
• Call a tool in a loop because items only takes one element.
• tables_query the full URL list, then feed it as rows or as N tool calls.
• Use a rows source for more than a handful of small objects.
• Raise batch_sample.limit (or tables_query.max_results) to pull a batch back into the context.
• Batch html_fetch and then read the HTML.
• Pull a bulky SERP or HTML payload inline when you only need a few fields; use save_to_tmp_storage and tmp_storage_jq / tmp_storage_grep.
• Use tmp_storage_read on a file you could query with tmp_storage_jq or tmp_storage_grep.
• Target batch_run with batch_run.
• Skip dry_run on a new item_map or a new JS/LLM config.

Credits: a job reports estimated_credits at launch. dry_run is how you check a mapping without paying for the source.