--- name: botify-mcp description: > Call Botify MCP tools correctly, and fan a tool out with batch_run so lists of URLs or rows never enter the conversation. Use whenever the user is on the Botify MCP, asks to run a tool on many URLs, pages or rows, mentions batch_run, batch_status, batch_sample, batch_cancel, or wants to keep I/O out of the LLM. Also use as soon as there are more than about ten items to process, or when a tool returns a temporary storage URL (*_tmp_storage_url) to read with tmp_storage_jq, tmp_storage_grep or tmp_storage_read. After delivering work, call send_feedback; call it again if the user continues the same main_task. --- # Botify MCP Every Botify MCP tool processes **exactly one item**. Fanning a tool out is `batch_run`'s job: the records are read from a SQL query, a table or a file, and the results are written to a sink. They do not pass through this conversation. For SQL over the catalog (`tables_list`, `tables_query`, `catalog.` prefix), also read `../botify-tables-sql/SKILL.md`. Tools may be deferred: call `tool_search` (e.g. `tool_search(query="batch_run html_code_executor")`) before the first use. Every call except `list_projects` needs `organization` and `project`. Every agent tool except `send_feedback` also needs `main_task` and `sub_task` as top-level arguments next to those slugs — not inside `config` or `items`. If the slugs are unknown, call `list_projects` rather than guessing. Retry a timed-out connector once or twice before treating it as a failure. After delivering work — a question answered, a batch finished — call `send_feedback`. Same if a tool failed unexpectedly. If the user then gives new indications, stay on the same `main_task` and call `send_feedback` again when there is new feedback. Do not call it every turn. ## One item per call ``` tool_name organization: acme project: www main_task: Internal linking audit sub_task: List available tables items: - { …one item matching the tool's schema… } config: { …optional, shared settings… } ``` `main_task` is the label of the overall task; reuse it on every call, including when the user adds new indications. `sub_task` is the label of the current step. `send_feedback` does not take them. Do not put them in `config`. `items` is an array of length 1. A second element is refused with an error that names `batch_run`. Do not loop the same tool tens of times as a workaround. Exception: `speedworkers_page_history` takes a list of up to 1,000 URLs inside its one item and scans its table once per call. Put every URL in that list rather than batching it per URL. Standalone tools (no `items`): `list_projects`, `batch_status`, `batch_sample`, `batch_cancel`. ## When to use batch_run | Situation | What to do | |---|---| | One URL / one question | Call the tool directly | | 2–10 small results you will read now | `batch_run` is fine; the run stays **inline** and the results come back | | **More than ~10 records**, or any list you do not want in context | `batch_run` with a `sql`, `table` or `file` source | | Large HTML, a crawl URL set, an uploaded CSV | Same: `batch_run`. Never paste the rows into the prompt | The threshold is 10. At 11 records the run becomes a **job**: you get a `job_id`, not the data. That is the point. For a job with several steps per item, if the `workflows_*` tools are in your tool list, read `../botify-workflows/SKILL.md`. **The failure to avoid:** `tables_query` a few thousand URLs, then call `html_fetch` / `html_code_executor` / `html_grep` once per row. The list has already entered the context, each page would again, and you will hit the one-item cap. Explore with aggregates and a `LIMIT` of 5–20, then hand the **same SQL** to `batch_run` as a `sql` source. `rows` is the only source whose content travels through the model. It is capped at 256 KB and refused above that. Prefer `sql` or `file`. ## batch_run Read the target tool's item schema first (`tool_documentation` when it has extended docs — `html_code_executor` does). `item_map` must match that schema. `batch_run` cannot target itself. ``` batch_run organization: acme project: www main_task: Internal linking audit on last crawl sub_task: Extract product h1s items: - tool: html_code_executor source: type: sql query: | SELECT url FROM catalog.crawl_pages_20260816 WHERE segments__pagetype__value = 'product' AND compliant__is_compliant LIMIT 500 item_map: url: "{{row.url}}" tool_config: function: "function() { return {h1: document.querySelector('h1')?.innerText}; }" dry_run: true config: on_error: skip ``` Then the same call with `dry_run: false`. | Field | Role | |---|---| | `tool` | Target tool id (`html_code_executor`, `html_grep`, `html_fetch`, `ai_gateway`, …) | | `source` | Where records come from (`sql`, `table`, `file`, `rows`) | | `item_map` | Projection onto the tool's item. Omit when each record already has that shape | | `tool_config` | The target's **config**, shared by every record (JS function, grep queries, LLM schema) | | `sink` | Where results go. Default: a gzipped JSONL file whose URL is a valid `file` source | | `dry_run` | First 5 records only, results in full, nothing launched, no sink | | `config.on_error` | `skip` (default): record the failure and continue. `fail_job`: stop on the first error | ### Sources | `type` | Use when | Notes | |---|---|---| | `sql` | The list already lives in the catalog | BigQuery Standard SQL, `catalog.` prefix. Optional `limit` | | `table` | A whole catalog table | Bare name or `catalog.name`. Optional `limit` | | `file` | A CSV/TSV the user uploaded, or the previous batch's JSONL | URL from `https://app.botify.com/:organization/:project/o/storage/tmp`, a previous `sink.url`, or a public HTTP(S) URL. CSV, TSV, JSONL; gzip allowed. `format` defaults to `auto` | | `rows` | A handful of objects you already have | Last resort. This **is** I/O through the LLM | Do not `tables_query` the full set and copy it into `rows`. A `COUNT(*)` plus `LIMIT 20` is enough to understand the population; `batch_run` re-runs the query itself. Ask the user to upload a local file rather than pasting it. ### item_map A projection, not an expression language: `{{row.}}` and dotted paths. A path that does not resolve is refused on the sample **before** the job starts. - `{"url": "{{row.page_url}}"}` — whole placeholder: type is preserved (number stays a number) - `"https://{{row.domain}}/{{row.path}}"` — interpolation: always a string - `{{row.blocks.0.text}}` — walk objects and list indexes - Namespace is `row` only (`{{rows.url}}` is rejected) Omit `item_map` when the source columns already are the tool's item (a `rows` source written in the item shape, or a SQL `SELECT url` against `html_code_executor`). ### Sinks and chaining Each written record is `{item, response, error}`. `item` is kept so a tool that does not echo its input stays attributable. **File (default).** Gzipped JSONL. The returned `sink.url` is the `file` source of the next batch. A previous batch's lines become `row.item.*` and `row.response.*`: ``` item_map: user_prompt: "Classify this title: {{row.response.result.h1}}" ``` **Table.** Editable catalog, then query it with `tables_query` instead of pulling the file back. ``` sink: type: table table: product_h1s replace_existing: false row_map: url: "{{item.url}}" h1: "{{response.result.h1}}" ``` `row_map` sees `{{item.*}}`, `{{response.*}}` and `{{row.*}}`. It is not strict: a failed record has no `response`. The default (no `row_map`) stores raw `item` and `response` JSON columns. Never write into crawl/logs/GSC tables; pick a new name. ### What comes back | `mode` | When | In the tool result | |---|---|---| | `dry_run` | `dry_run: true` | 5 `{item, response, error}` objects. No job, no sink | | `inline` | ≤ 10 records | All results. A sink is still written if you asked for one | | `job` | > 10 records | `job_id`, `rows_total`, `estimated_credits`, `sink`. **`results` is null** | Do not try to fetch a job's full output into the conversation. ## Following a job ``` batch_status organization, project, job_id batch_sample organization, project, job_id, limit (default 5, hard cap 20) batch_cancel organization, project, job_id ``` `batch_status` states: `pending`, `running`, `completed`, `failed`, `cancelled`. Poll it; do not start a second job for the same work. `rows_ok` / `rows_error` and `sample_errors` tell you whether to cancel. `batch_sample` is a peek so you can check the mapping before trusting the sink. The cap of 20 exists because a batch exists to keep its data out of the chat. Do not raise `limit` to "download" the output. `batch_cancel` stops at the next chunk. Records already written stay in the sink; the rest is not paid for. ## Temporary storage URLs Bulky payloads (HTML, SERP JSON) do not have to enter the conversation. Set `save_to_tmp_storage: true` in the tool's `config` and the result carries a temporary URL instead of the payload: | Tool | Returns instead | |---|---| | `html_fetch`, `speedworkers_page_preview`, `speedworkers_page_snapshot` | `html_tmp_storage_url` (no `html`) | | `google_serp`, `google_ai_overview`, `google_ai_mode`, `google_people_also_ask`, `google_trends` | `results_tmp_storage_url` (no `results`) | User uploads from `https://app.botify.com/:organization/:project/o/storage/tmp` are the same kind of URL. A URL only works in its own organization and project, and expires. Look inside it with the `tmp_storage_*` tools, which return only what you ask for: | Tool | Use for | Limits | |---|---|---| | `tmp_storage_jq` | JSON / JSONL: pick fields, filter, count | `max_results` ≤ 1000 (default 100) | | `tmp_storage_grep` | Any text (HTML, CSV, JSON): a string or RE2 regex, with line number and context | ≤ 20 `queries`, ≤ 1000 matches per query, 30 s scan | | `tmp_storage_read` | The whole file, when it is small and you need all of it | Refused above 1 MiB decompressed | Gzip is decompressed for you, up to 100 MiB. Prefer `tmp_storage_jq` on JSON and `tmp_storage_grep` on HTML or text; `tmp_storage_read` is the last resort. When the JSON shape is unknown, start with `expression: keys`, or `.[0:3]` with `slurp: true` on JSONL. ``` tmp_storage_jq organization: acme project: www main_task: SERP competitors sub_task: Organic URLs for running shoes items: - url: config: expression: '.items[] | select(.type == "organic") | .url' ``` For many URLs, batch the `tmp_storage_*` tool over the previous sink: `source: {type: file, url: }`, `item_map: {url: "{{row.response.results_tmp_storage_url}}"}` and the jq expression or grep `queries` in `tool_config`. ## Typical pipelines **1. Extract structured fields from many crawled URLs** 1. `tables_list` / `tables_query` — latest SiteCrawler crawl, `COUNT(*)` and a 10-row sample of the population. 2. `tool_documentation` `html_code_executor` — confirm the JS DOM API. 3. `batch_run` `dry_run: true` — `sql` source, `item_map: {url: "{{row.url}}"}`, JS in `tool_config.function`. 4. Same call, `dry_run: false`. If `mode` is `job`, poll `batch_status`. 5. `batch_sample` a few records. Chain the `sink.url` or query the table sink. **2. Grep a string or regex across a URL set** Same as above with `tool: html_grep`. `url` is the item; `queries` go in `tool_config` (shared by every record). Prefer this to fetching HTML and searching in the prompt. **3. Fetch HTML without pouring it into the chat** Do not batch `html_fetch` if you would then read `html`. Either: - set `save_to_tmp_storage: true` in `tool_config` (shared by every record) so each result is a temporary URL, then batch `html_code_executor` / `html_grep` / `tmp_storage_grep` on those URLs, or - skip the fetch and batch `html_code_executor` / `html_grep` on the live URLs (they load crawl HTML themselves). **4. User-supplied CSV** Upload at `https://app.botify.com/:organization/:project/o/storage/tmp`, then: ``` source: type: file url: item_map: url: "{{row.url}}" ``` **5. Second tool over the first batch's output** ``` source: type: file url: item_map: user_prompt: "{{row.response.result.h1}}" tool_config: # ai_gateway system_prompt, structured_output_json_schema, model_settings ``` ## send_feedback Call this after delivering work, and also when a tool fails unexpectedly or the data looks wrong. If the user continues the same `main_task` with new indications, call it again. Do not call it every turn. Botify would be very grateful: say what went well or badly, and which tools you wished you had from an SEO/GEO product like Botify (crawl data, server logs, keywords, AI visibility, internal linking, and similar). ``` send_feedback organization: acme project: www items: - experience: negative # positive | neutral | negative message: | html_code_executor could not read shadow-DOM product titles. A crawl-extracted JSON-LD tool would have helped. ``` `experience` is required (`positive`, `neutral`, or `negative`). `message` is the debrief. ## Do not - Skip `send_feedback` after delivering work, or after new indications on the same `main_task`. - Call a tool in a loop because `items` only takes one element. - `tables_query` the full URL list, then feed it as `rows` or as N tool calls. - Use a `rows` source for more than a handful of small objects. - Raise `batch_sample.limit` (or `tables_query.max_results`) to pull a batch back into the context. - Batch `html_fetch` and then read the HTML. - Pull a bulky SERP or HTML payload inline when you only need a few fields; use `save_to_tmp_storage` and `tmp_storage_jq` / `tmp_storage_grep`. - Use `tmp_storage_read` on a file you could query with `tmp_storage_jq` or `tmp_storage_grep`. - Target `batch_run` with `batch_run`. - Skip `dry_run` on a new `item_map` or a new JS/LLM config. Credits: a job reports `estimated_credits` at launch. `dry_run` is how you check a mapping without paying for the source.