# Measuring impact, rollout and adjacent features

## The three value pillars

1. **Get strategic pages crawled, indexed and regularly refreshed.** Faster delivery means bots crawl more pages in the same window, and controlling what is visible to bots reduces site bloat.
2. **Index the full content of every page.** Bots receive the complete rendered page, including content that normally requires user interaction or comes from third parties such as review platforms — particularly valuable on client-side rendering frameworks. Quality controls then let the site guarantee what bots see.
3. **Protect the infrastructure.** Bot traffic becomes predictable and controllable, and cached pages keep being served during origin outages. Origin-load reduction is conditional: it only holds if SpeedWorkers does not refresh more often than the bots it replaces. When that holds, infrastructure cost drops because bots stop hitting the origin — along with any downstream services the origin calls to render a page.

A useful supporting point from Google's [Myths and facts about crawling](https://developers.google.com/crawling/docs/myths-about-crawling): serving more pages in a limited time lets Google crawl more of them, and time spent rendering counts as much as time spent requesting.

## Assessing fit

Good candidates:

- Slow HTML fetch, slow rendering, or rendering that outright fails.
- **HTML crawl versus JavaScript crawl** — compare word counts, links and content between the two: a large gap means bots are missing content today.
- **Large long-tail inventories**, because ordinary CDNs cannot keep long-tail pages in cache while SpeedWorkers can.
- **Heavy or expensive bot load on the origin**, especially when rendering a page triggers paid downstream APIs such as search or recommendation services. Serving cached HTML avoids those calls, and inventory optimization rules can stop unwanted URL patterns before they reach any downstream system.
- **Rendering-related visibility gaps in AI engines** — for example a brand that is frequently mentioned but rarely cited by domain in one engine — can point to content that bots cannot see.

Poor candidates: the site is fully indexed by Google, the page count is small, pages are properly indexable, and fetch/render is already fast.

If the site relies heavily on JavaScript, SpeedWorkers may only be useful for bots that don't render — OpenAI, Perplexity, Anthropic, and similar.

**CDN prerequisite**: confirm early that the CDN can be configured independently. For embedded CDNs (Shopify, SFCC), prefer SpeedWorkers CDN as described in `architecture.md`.

## Measuring impact

**Interception granularity determines what can be measured.** If URLs carry no indicator of the section they belong to — common on product detail pages — a specific category cannot be isolated cleanly. The alternatives are deterministic hashing at the edge, for example treating a fixed percentage of URLs with a hash rule such as `hash(url) % 100 < 20`, which requires edge-compute capability on the CDN, or handling interception server-side.

**The cleanest control group is a separate property** — a different domain or brand — because it removes any interaction between treated and untreated pages.

**When test and control live on the same site, expect and use the spillover.** Freeing crawl budget on treated pages benefits untreated ones too, so a clean isolated measurement is not achievable. Report three tiers instead:

- **Treated segment** — the direct effect of the optimization.
- **Control segment** — the indirect effect of newly freed crawl budget.
- **Site-wide** — a pre versus post launch comparison.

Read the results accordingly: if the control also improves, the treated-versus-control delta is the *lower bound* of the true effect, and the control's lift documents the systemic benefit. Difference-in-differences is a suitable statistical approach when segments cannot be separated.

**Evidence crawls.** Run a large HTML-only crawl and a JavaScript crawl before launch, then an equivalent crawl after, to show site structure without JavaScript, with JavaScript but no SpeedWorkers, and with SpeedWorkers. Plan crawl rate increases in advance, and keep very large scopes to HTML-only, since large JavaScript crawls are slower and more fragile.

**Metrics to track alongside**: crawl volume and crawl rate from log data, indexation and coverage, impressions and clicks by segment, AI bot hits and citations, and origin bot traffic or CDN cache hit ratio for the infrastructure argument.

## Rollout sequence

A typical enterprise rollout runs in this order:

1. **Implementation** — CDN configuration, inventory and cache behavior setup, quality controls, with interception initially restricted to the Botify test user-agent.
2. **Pre-production validation** — the site's own team crawls the SpeedWorkers cache and tests behavior against agreed criteria. This should pass before any change to production bot routing.
3. **CDN launch** — widen the interception rules to the contracted bots, typically search engines first, then AI bots.
4. **Monitoring and tuning** — watch delivery efficiency, refresh capacity and render quality in the first weeks, and adjust inventory sources, cache behaviors and quality controls from what the data shows.

Partial rollout (intercepting only some sections or variants) is generally not advised: it makes the impact of SpeedWorkers hard to evaluate, and may affect how Google crawls the site. Only one site has done it to date, excluding a Google Merchant Center variant of their product pages. Prefer a full rollout unless there is a hard product reason not to.

## Adjacent features

### PageWorkers with SpeedWorkers

The two are complementary: PageWorkers modifies on-page content and links for bots and users, while SpeedWorkers guarantees that strategic pages are fully rendered and accessible to bots.

- On a cache hit, SpeedWorkers runs PageWorkers on the page and serves the optimized render — always with the latest optimizations, regardless of how old the cache entry is.
- On a cache miss, the bot is served the unrendered page from the origin with the PageWorkers snippet installed: bots that execute JavaScript run it, bots that don't (including most AI bots) don't.
- The practical consequence: SpeedWorkers makes PageWorkers optimizations visible to non-JavaScript bots, and visible earlier and more often to Google.
- Updating an optimization does not require a cache refresh. Propagation generally takes between 30 seconds and five minutes.

### Live Update

Live Update keeps price and availability accurate on cached pages, for large sites that cannot raise their refresh rate enough to track frequent changes. Botify ingests the product feed, and on each bot request SpeedWorkers fetches the pre-rendered page, updates price and availability on the fly from the latest feed, and delivers it.

Requirements: a product feed at an agreed frequency, and the selectors to update. Google Shopping / Merchant Center feeds are supported by default, as is any CSV containing product URL, price, currency, availability, and optionally sales price. The feed can be self-hosted (S3, GCP or HTTP, with credentials shared with the SEO Success Manager) or hosted by Botify.

The robust implementation is standard attributes on product pages: `data-lu-target="price"`, `data-lu-target="availability"`, `data-lu-target="ld-json"` on the product structured data, and optionally `data-lu-target="sales-price"`. CSS selectors work as a fallback but break silently when the front end changes, in which case SpeedWorkers simply serves the cached values — so front-end changes should be flagged to Botify Support. Live Update is also what makes Google AdsBot support viable.

### Bing Content Submission

Pushes the fully rendered HTML of new and updated cached pages straight to Bing's index instead of waiting for Bingbot to crawl. Setup is a single step: grant `fastindex@botify.com` Read/Write access to the Bing Webmaster Tools account. The benefit is more indexed pages on Bing-powered surfaces, and therefore more active pages and organic visits.

When the goal is getting pages into Bing's index rather than serving crawls, [SmartIndex Push to Bing](https://support.botify.com/en/articles/10518692-configuring-push-to-bing) can do this more efficiently, with decision rules over which pages to push. If SpeedWorkers Bing Content Submission is already enabled, keep using it and do not also configure Push to Bing in SmartIndex.

## Public documentation

On `support.botify.com`:

- SpeedWorkers Overview — `/en/articles/9108707-speedworkers-overview`
- SpeedWorkers Terms — `/en/articles/9108710-speedworkers-terms`
- Getting Started — `/en/articles/9108709-getting-started-with-speedworkers`
- Implementation — `/en/articles/9133976-speedworkers-implementation`
- Page lifecycle — `/en/articles/9108733-understanding-the-page-life-cycle-in-speedworkers`
- "Not cached" reasons — `/en/articles/9108734-understanding-not-cached-reasons-in-speedworkers`
- Managing page inventory — `/en/articles/9108714-managing-speedworkers-page-inventory`
- Optimizing page inventory — `/en/articles/9108715-optimizing-speedworkers-page-inventory`
- Cache behaviors — `/en/articles/9108720-managing-speedworkers-cache-behaviors`
- Quality controls — `/en/articles/9108722-managing-speedworkers-quality-controls`
- CSS selectors — `/en/articles/9108723-using-css-selectors-in-speedworkers`
- Refresh capacity — `/en/articles/9108721-managing-refresh-capacity-in-speedworkers`
- Enabling and testing — `/en/articles/9108724-enabling-and-testing-speedworkers`
- Serving Markdown to AI bots — `/en/articles/15862980-serving-markdown-to-ai-bots`
- Live Update — `/en/articles/9197017-speedworkers-live-update-configuration`
- Bing Content Submission — `/en/articles/9108739-bing-content-submission-overview`
- SmartIndex Push to Bing — `/en/articles/10518692-configuring-push-to-bing`
- PageWorkers with SpeedWorkers — `/en/articles/15464354-how-pageworkers-works-with-speedworkers`
- Supported bots — `/en/articles/9108633-supported-bots`
- FAQs — `/en/articles/9108736-speedworkers-faqs`