Data

Botify Graph

Every Botify product writes its data to the same place: crawls, server logs, Search Console, analytics, AI visibility, SpeedWorkers, PageWorkers, product feeds. The Botify Graph exposes all of it for a project as plain SQL tables, under one catalog. namespace, so a single query can join a crawl to its logs and its search traffic.

How it works

The Graph is read through the tables_* tools, which are regular tools: call them over MCP or over the API, with an organization and a project like any other. Queries are written in BigQuery Standard SQL and every table is addressed as catalog.<table>.

Table names are stable aliases you can read and type: the crawl of 13 July 2026 is catalog.crawl_pages_20260713, Google bot hits are catalog.logs_google_crawls. The tools map them to the physical storage and check that you have access to the project.

tables_list    →  which tables does this project have?
tables_schema  →  which columns does this table have?
tables_query   →  run the SQL

Product families

Tables come in families, one per product. Which ones exist depends on the products the project has enabled: tables_list is always the source of truth.

SiteCrawler

One set of tables per crawl, named after the crawl date.

The two most recent crawls are also listed as crawl_pages_latest and crawl_pages_previous (same for every crawl table), which is the easiest way to compare one crawl to the next.

LogAnalyzer

Server logs, split per search engine and per month. The engine is one of google, bing, yandex, naver, baidu, apple, openai and other_ai_bots (Anthropic, Perplexity, Meta, Mistral…).

Search Console (RealKeywords)

Analytics and Core Web Vitals

AI Visibility

SpeedWorkers and PageWorkers

AgenticCatalog

The push table mixes every source and destination of the project: always filter on source_id and destination.

Other

The tools

Run a query

On my-org/my-project, how many compliant URLs of the latest crawl
were never crawled by Googlebot last month? Break it down by segment.

curl -X POST "http://agents.botify.com/{organization}/{project}/tables_query/process" \
  -H "Authorization: Bearer $BOTIFY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "item": {
    "query": "SELECT http_code, COUNT(*) AS urls FROM catalog.crawl_pages_latest GROUP BY http_code ORDER BY urls DESC",
    "max_results": 50
  }
}'

Rules of the road

Prefix every table with catalog.

FROM catalog.crawl_pages_20260713, not FROM crawl_pages_20260713. A table that does not exist fails the query rather than returning no rows: check its name with tables_list. INFORMATION_SCHEMA is not available; use tables_schema instead.

Bound the tables that grow forever

Logs, Search Console, analytics, served pages, sitemaps, segmented log metrics, the AgenticCatalog push table and Merchant Center performance keep growing day after day. tables_list gives each of them a time_filter column, with the first and last days with data when known, and tables_query rejects a query that reads one without bounding that column in the WHERE clause of the SELECT reading it.

SELECT keyword, SUM(clicks) AS clicks
FROM catalog.search_console
WHERE date >= '2026-09-01'
GROUP BY keyword
ORDER BY clicks DESC

Log tables are stored month by month: their filter on ts selects the months read, so it must bound ts from below with constant values (DATE(ts) BETWEEN '2026-09-01' AND '2026-09-30').

Join on url_hash

Crawl, logs, logs_urls and PageWorkers tables share the same 64-bit url_hash, which is the cheapest way to join them. The logs crawls and visits tables have a virtual url column you can select, group by or count; to filter, sort or join on URLs, join logs_urls on url_hash instead. SpeedWorkers is the exception: its urlHash is a different hash, so join on its urlFnv64Hash column instead. Search Console tables join on url.

-- Compliant pages of the latest crawl that Googlebot never visited in September
SELECT c.url, c.depth
FROM catalog.crawl_pages_latest AS c
LEFT JOIN (
  SELECT DISTINCT url_hash
  FROM catalog.logs_google_crawls
  WHERE DATE(ts) BETWEEN '2026-09-01' AND '2026-09-30'
) AS g USING (url_hash)
WHERE c.compliant__is_compliant AND g.url_hash IS NULL
ORDER BY c.depth

Read enums from the schema

Many integer columns (bot_id, delivery reasons, devices…) are enums. tables_schema lists their values, and most come with a readable *_name column next to them (bot_name, botName…): filter on the id, display the name. Columns flagged as bitmasks are tested with col & flag != 0. Bot ids of the logs and of SpeedWorkers are two different catalogs: compare bot names, never ids, across the two.

Mind the cost

Queries are billed in credits from the data they process: select the columns you need, filter on dates, and set max_results to what you will actually read. When Botify has set up a GCP project for your organization, tables_query can run there instead, and Google bills the scan directly.

Create your own tables

Derived datasets — a list of orphan pages, LLM-enriched product data, a keyword mapping — can be saved next to Botify's tables and queried the same way, as catalog.<your_table>. Botify's own tables are read-only.

• From a query: run CREATE TABLE catalog.my_table AS SELECT … through tables_query, then INSERT INTO catalog.my_table SELECT … to append.
• From rows you already have: create the table with tables_create (columns, and an optional expiry in hours), then add rows with tables_insert.
• From a CSV file: call tables_load with a new table name. It creates the table and loads the file in one call, and refuses an existing table unless asked to replace it.

Go further

• botify-tables-sql — the full set of conventions, for your coding agent or MCP client.
• Query cookbook — crawl budget, AI bots, Search Console trends, orphan pages.
• The tables tools — schemas and an in-browser playground.