SpeedWorkers

SpeedWorkers is Botify's bot-facing delivery service: it keeps a pre-rendered cache of a site's pages and serves them to search and AI bots in a few hundred milliseconds. It is part of the Botify Activation Suite, available as an option on Pro and Enterprise plans alongside PageWorkers.

Two principles explain most of its behavior:

1. It answers bots, never users. Interception happens at the CDN based on the user-agent, so human traffic is untouched. This is why SpeedWorkers has no effect on Core Web Vitals, and why it is not cloaking — same URL, same content, dynamically rendered, which both Google and Bing recommend.
2. Value comes from removing fetch and some render cost from the bot's budget. Bots spend crawl budget on retrieving pages, and some of them also spend it on rendering. Serving a fully rendered page from cache means more pages crawled in the same window and deeper indexation of JavaScript-dependent content. Origin-load reduction is conditional: it only holds if SpeedWorkers does not refresh more often than the bots it replaces.

Use this file for the concept map and vocabulary. The reference files hold the detail people usually need precisely.

Core vocabulary

Most confusion comes from mixing these up, so keep them distinct.

• Page inventory — the set of URLs SpeedWorkers may fetch, render, cache and deliver. Being in inventory is not the same as being cached.
• Cache / refresh — fetching and rendering a page and storing it. "Refresh" is any subsequent update.
• Cache behavior (also called section in the data model) — the rules deciding how a page is fetched and rendered, and how long it stays fresh: refresh rate, stale window, priority, rendering configuration. A behavior matches a URL pattern.
• Scope — the pages a rule or behavior applies to.
• Cache miss — SpeedWorkers could not serve a cached page, so the request falls back to the origin server.
• Page states — Not Indexed (in inventory, never cached) → Indexed (cached and current) → Outdated / Stale (past the refresh period, still served) → Expired (no longer served). The optimal state is everything Indexed.
• Inventory optimization rule — a rule that short-circuits fetch and render for a URL pattern: block (do not serve), static 200 HTML, static 4xx (404 or 410), 3xx redirect, or a response computed by JavaScript. Keeps cache capacity focused on pages that matter.
• Advanced refresh rule — a rule that overrides refresh parameters at the behavior/section level: prevent indexation, or change refresh frequency based on conditions such as status code change, page property values, or status code stability.
• Quality control — a CSS-selector check run when a page is cached, with an action of fail and deliver (warn only) or fail and do not deliver (fall back to the origin). Quality controls are what let a site guarantee what bots receive.
• Page property — a value extracted from the page HTML (for example product availability) that can drive advanced cache rules.

How the cache gets populated

Inventory is filled mainly by bot discovery (bots requesting pages) and by the configured inventory sources: manual lists, remote files, bot-request rules, and links discovery. SpeedWorkers then fetches, renders and caches those URLs according to cache behaviors. Being in inventory is the prerequisite for being cached; cache behaviors and inventory optimization rules then decide how and whether a URL is actually served. See references/settings.md#inventory.

The five KPIs

Anchor any health discussion on these; they are the SpeedWorkers Overview page.

• Delivery Efficiency — share of in-scope bot hits served from cache rather than falling back to the origin. Blocked patterns don't count against it.
• Refresh Efficiency — share of delivered pages that were up to date rather than stale or expired.
• Fetch Speed — average pages per second refreshed from the origin, to be read against the Max Refresh Speed setting.
• Render Quality — share of pages passing the defined quality controls.
• Refresh Capacity — whether the whole inventory can be refreshed given inventory size, refresh rates, max refresh speed, and advanced refresh rules. Under 90% is safe, 90–100% is a warning, above 100% means some pages will never be refreshed. Remediation levers live in references/metrics-and-diagnostics.md.

Reading a project's state: the tools

When the question is about a specific project, look before you answer. Six read-only Botify MCP tools expose what the interface shows:

• speedworkers_config — the settings of the production version (default), the draft, or version N: cache behaviors, inventory optimization rules, JS snippets, rendering configs, inventory sources, Live Update feeds, quality controls.
• speedworkers_versions — the version list with push date, author and comment.
• speedworkers_inventory — daily counts of indexed, stale, expired and not indexed pages, by section, priority or refresh frequency, plus the rendering load against max speed (the tool's Refresh Capacity).
• speedworkers_page_preview — what a bot receives for a URL now, with the delivery reason, cache behavior, quality control results and speed gain.
• speedworkers_page_history — past indexations of up to 1,000 URLs in a single call; never fan it out per URL.
• speedworkers_page_snapshot — the HTML and rendering details stored during one past indexation, from an event_id of the history.

Inputs, outputs, limits and diagnostic recipes: references/tools.md.

How to reason about a SpeedWorkers question

• "Bots get an empty or incomplete page" → check in this order: is the user-agent actually routed to SpeedWorkers, is the URL in inventory, is it cached, did a quality control fail. speedworkers_page_preview answers the last three for one URL, and speedworkers_page_history shows when it started.
• "Some pages are never served" → first confirm the page is in inventory, matches a cache behavior, and is not caught by an inventory optimization rule. Then look at refresh capacity and inventory size (speedworkers_inventory), not at rendering.
• "Something changed on date X" → speedworkers_versions for a push around that date, then speedworkers_config on both versions.
• "Would this site benefit?" → good candidates: slow HTML fetch, slow or failing rendering, a large HTML-versus-JS crawl gap, a large long-tail inventory, or heavy/costly bot load on the origin. Poor candidates: fully indexed by Google, small, properly indexable, and fast. If the site relies heavily on JS, SpeedWorkers may only be useful for bots that don't render (OpenAI, Perplexity, Anthropic, and similar). Full fit assessment: references/measurement-and-rollout.md.
• "Can we roll out on only part of the site?" → generally not advised: it makes the impact of SpeedWorkers hard to evaluate, and may affect how Google crawls the site. Only one site has done a partial rollout to date (excluding a Google Merchant Center variant of their product pages). If it must be done, only when the URL structure allows a clean pattern; otherwise through deterministic hashing at the edge or server-side interception. See the measurement reference.
• Cost questions → refresh credits are calculated from page render time, and inventory optimization rules are the main way to avoid spending capacity on pages bots never request.

Where the numbers come from

Everything in the interface is built on two event tables that are also queryable directly in SQL: catalog.speedworkers_indexed_pages (one row per fetch/render/cache attempt — the write side) and catalog.speedworkers_served_pages (one row per bot request — the read side). Inventory Monitoring and Render Quality come from the first, Delivery Center and the speed figures from the second.

Reach for the tables when the interface and the tools can't answer the question: attributing failures to a specific cache behavior or inventory source, measuring how stale the delivered pages actually are, isolating one bot's behavior, quantifying render cost and third-party weight, or joining SpeedWorkers data with crawl and log data. See references/data-tables.md for fields, enums and ready-made queries.

Working conventions

• Settings are versioned (Draft / In Production / Archived). Never describe a settings change as live until it has been pushed to production; speedworkers_config reads production unless asked for draft or a version number.
• Serving Markdown to AI bots is a cache behavior option, not a separate product; traditional search bots keep receiving rendered HTML.
• When a site's CDN is not standalone — for example bundled with a commerce platform such as Shopify or SFCC — prefer SpeedWorkers CDN (the DNS integration). Raise it early with the Botify team rather than assuming a standard CDN integration is possible. See references/architecture.md.
• Cite support.botify.com articles when explaining behavior; they are public and precise. Links are listed at the end of references/measurement-and-rollout.md.
• If a question depends on what a specific account has contracted (which bots are allowed, max refresh speed, whether Live Update or Bing Content Submission are enabled), say so and point to the SEO Success Manager instead of guessing.