# Architecture, CDN integration and bots ## Request flow 1. A bot requests a page. The CDN in front of the site — the customer's CDN for the recommended and advanced methods, Botify's CDN for SpeedWorkers CDN — inspects the user-agent (the main criterion) and applies secondary safeguards such as the remote IP. Requests that fail those checks are dismissed; see Bots below. 2. If the user-agent matches an allowed bot and the request is for an HTML page (not an asset), the CDN switches the origin to the SpeedWorkers endpoint, passing the original URL and an authentication token. 3. SpeedWorkers checks its cache: - **Hit** → it returns the fully rendered page, with PageWorkers optimizations applied at serve time if PageWorkers is installed. - **Miss** → depending on the implementation method below, either Botify's Cloudflare Worker or the customer's CDN retrieves the unrendered page from the origin. 4. Assets (JS, CSS, images, fonts, video) are always served by the site's own infrastructure. Never send bots to SpeedWorkers with a 3xx redirect — it exposes the token and has a strong negative SEO impact. Switch the origin instead. ## Implementation methods Three first-class methods: - **Recommended** — eligible bot traffic is routed to Botify's CDN (Cloudflare with a Worker), which calls SpeedWorkers. On a miss, the Cloudflare Worker retrieves the page from the origin and serves it. Simple configuration, compatible with the widest range of CDNs. Trade-offs: marginal extra latency on misses; origin fetches go back through the customer's CDN, generating additional requests and bandwidth cost; those fetches arrive from Botify's CDN IP ranges, which are third-party and therefore not guaranteed. - **Advanced** — the customer's CDN handles the fallback itself when a page is not in inventory. No latency penalty on misses, but the CDN configuration is more complex. - **SpeedWorkers CDN (DNS integration)** — integration at the DNS level, for customers with no CDN or no access to it (Shopify, SFCC and other embedded CDNs). Everything is handled by Botify's CDN. Downside: the *whole* traffic goes through Botify's CDN, which adds cost — acceptable when assets are served from a different domain than the pages. Upside: Botify gets complete logs and can ingest them into Log Analyzer. Rule of thumb: for embedded CDNs, prefer SpeedWorkers CDN. CDNs with documented integration guides: Akamai, Azure Front Door, Cloudflare, CloudFront, Fasterize, Fastly, Imperva, NGINX. The advanced method is documented for Akamai, Cloudflare and Fastly. Any CDN meeting the requirements below should be compatible — confirm with the Botify team. ## Endpoints, headers and authentication Endpoints depend on the method: - Recommended: `{adn-id}.sw.adn.cloud`. - Advanced: `{cluster}.speedworkers.com`. - SpeedWorkers CDN: Botify's CDN is the public origin; there is no separate customer-CDN origin switch. The ADN staging endpoint is rarely used. The normal setup is the **production** ADN endpoint for everything, including the client's own staging environment, which also points at ADN production. Authentication is per method, not chronological: - Recommended: an ADN token sent as the `x-sw-adn-token` header or as a query parameter. - Advanced: website ID + token, as `website_id` and `token` query parameters or the `X-Sw-Options-Auth` header. - SpeedWorkers CDN (DNS): no authentication. Original URL: the `x-sw-url` header, or a `url` query parameter. Required when subdomains are handled by SpeedWorkers. Other request headers used in integrations: `x-sw-client-ip`, `x-sw-if-modified-since`, and an `x-sw-agent` identifying the interceptor. Response signals are not equivalent: - `x-sw-status: success` means SpeedWorkers answered. - `x-sw-adn: success` (recommended integration) only means the recommended integration handled the request — the response may still come from the fallback. Anything else means the CDN should fall back to the origin (advanced method) or that the integration did not complete as expected. Integrate in a tool that respects DNS TTLs, since the endpoint IPs can change. Treat tokens and website IDs as secrets: rotate them if they are ever sent to a third-party debugging service. ## CDN requirements checklist - Filter to allowed bot user-agents only, and to HTML pages only. A typical asset exclusion pattern: `\.(js|map|css|jpg|jpeg|png|ico|gif|tiff|svg|woff|woff2|ttf|eot|mp4|otf|txt|xml)$` - Disable content caching on SpeedWorkers responses. - Disable retries when the SpeedWorkers origin returns an error code, and set a 15-second timeout before falling back. - Preserve headers from both the requester and the SpeedWorkers origin. - Do not route the Botify fallback user-agent — `Mozilla/5.0 (compatible; botify; http://botify.com) fallback` — to SpeedWorkers, and do not block the production endpoint's IPs from any environment, including the client's staging, since that environment also points at ADN production. - Requester IPs are appended to `X-Forwarded-For`. Review any rule based on that header, especially Geo-IP resolution, since Botify's IPs are North America based. - Once SpeedWorkers is live, origin logs will only show the fallback user-agent. Log ingestion for Botify must come from the CDN, and should include the SpeedWorkers origin and cluster log sources. ## Integration tests Tests run against the production endpoint. Force a specific SpeedWorkers outcome with the `X-Sw-Options` header, alongside `X-Sw-Options-Auth` carrying the website ID, using the test user-agent `botify-bot-sw-test`: - `always-success` → forces a cache hit even if the page is not cached. Expect a `Success` body and `x-sw-status: success`. Confirms SpeedWorkers is called and its reply returned. - `always-notfound` → forces a cache miss. Expect the site's own page and no `x-sw-*` headers (in the recommended setup, `x-sw-adn: success`). Confirms the fallback works. - `always-timeout` → delays the response past the configured timeout. Confirms the timeout fallback works. - Helpers: `passed-through`, `request-time`, and `echo-`, echoed back as `X-Sw-Echo`. Example: `curl -v -A "botify-bot-sw-test" -H "X-Sw-Options-Auth: WEBSITEID" -H "X-Sw-Options: always-success" https://www.example.com/` During initial integration, filter only the Botify test user-agent (`botify-bot-sw-*`) at the CDN, then widen to production bots once the tests pass. If a call fails, two techniques narrow it down: point the SpeedWorkers host at a request-inspection service to verify the outbound call itself (rotating the token first), and temporarily remove the failover rule so the CDN surfaces the real edge error. ## Bots - Interception is user-agent based and case-insensitive, with secondary checks such as the remote IP. Which bots are allowed is defined by the subscription. - Search bots commonly intercepted: `googlebot/`, `google-inspectiontool/`, `googleother`, `google-agent`, `bingbot/`, `yandexbot/`, `yandexmobilebot/`, `baiduspider/`, `applebot/`, `yeti/`, plus `botify-bot-sw-*`. - AI bots: `GPTBot/`, `OAI-SearchBot/`, `OAI-AdsBot/`, `ChatGPT-User/`, `ClaudeBot/`, `Claude-Web`, `Claude-User/`, `Claude-SearchBot/`, `anthropic-ai`, `PerplexityBot/`, `Amazonbot/`, `Bytespider`, `CCBot/`, `FacebookBot/`, `meta-externalagent/`, `Perplexity-User/`, `YouBot/`. Serving AI bots through SpeedWorkers requires Bing in the subscription. - Some bots do not publish their IP ranges — Bytespider and Brave — so their hits appear in reports but cannot be authenticated. Those are the known cases; any other bot that does not publish IP ranges has the same limitation. - Google StoreBot is not supported, since it would require exercising live checkout JavaScript on a pre-rendered page. Google AdsBot is supported when Live Update is properly integrated. - Some AI products fetch pages through server-side tools whose user-agent and IPs differ from the assistant's documented crawler and are not published. Those requests can bypass interception and land on an unrendered page. To identify one, request a unique cache-busting URL that has never been fetched, then find that query string in the origin logs: the matching user-agent and source IP are what needs adding to the routing rules. - If traffic is routed to SpeedWorkers for an unsupported bot: with the advanced configuration SpeedWorkers answers anyway; with the standard configuration the request is not served by SpeedWorkers and falls back to the origin.