# Metrics, monitoring and diagnostics

## Page lifecycle

A URL must enter the inventory before it can be cached and delivered. It stays as long as at least one source keeps it, according to that source's retention setting (up to 90 days after it was last seen).

States: **Not Indexed** (in inventory, never cached) → **Indexed** (cached and current) → **Outdated / Stale** (past the refresh period, still served) → **Expired** (was cached, but both the refresh and stale windows have elapsed; no longer served, and retried at the start of the next refresh cycle).

Common causes of expiry: the refresh batch was full, or the origin returned an error during the refresh. The page can also just be gone from the origin.

Discovery example, with a Bot request source set to two requests over five days: a URL requested three times by Googlebot on January 1–2 enters inventory on January 2, and leaves it four days after the last bot request if the refresh rate never brought it back.

## Overview KPIs

The Overview page shows four cards, plus Refresh Capacity (the fifth KPI; see the next section).

- **Delivery Efficiency** — the share of in-scope bot hits served from cache rather than redirected to the origin. Only in-scope pages count; blocked patterns are not treated as hits. A decline usually points to an inventory problem or a bottleneck in the rendering process.
- **Refresh Efficiency** — the share of pages delivered while up to date rather than stale or expired. Serving a stale page beats serving nothing, but this ratio should stay high.
- **Fetch Speed** — the average number of pages refreshed per second from the origin. Overlay the Max Refresh Speed setting: consistently close to the maximum means some inventory will go unrefreshed; well below means there is spare capacity.
- **Render Quality** — the share of pages passing quality controls at render time.

## Refresh Capacity

The percentage of the inventory that can be refreshed daily given inventory size, max refresh speed, cache behavior refresh rates, and advanced refresh rules. Shown on the Overview page and at the top of the Cache Behaviors settings, always as the *current* rate regardless of the reporting period.

- **Below 90% — safe**: the whole inventory should be refreshed if nothing changes.
- **90–100% — warning**: fine today, at risk if the inventory grows.
- **Above 100% — capacity exceeded**: some pages will not be refreshed. The interface shows how many, and the daily maximum.

Levers: reduce the inventory by removing pages bots never request, slow the refresh rate so pages stay cached longer, or request a higher max refresh speed from the SEO Success Manager.

## Inventory Monitoring

**Indexed URLs tab** — Total Inventory Overview (daily counts of indexed, outdated, expired and not indexed pages), Inventory Dimensions Over Time (by cache behavior, priority, inventory source or rendering configuration), and Current State of the Inventory (with selectable primary and secondary dimensions). Metrics: No. of Pages, Indexed, Outdated, Expired, Not Indexed.

The same daily counts, plus the rendering load against max speed, are available through `speedworkers_inventory` (see `tools.md`).

**Cache Refreshes tab** — Overview (Total Pages Cached in Period, unique desktop/mobile/responsive pages, Total Cache Fails), Total Fetched Pages by Date, Total Fetched Pages by Behavior, Reasons Why Pages Weren't Cached, and All HTTP Status Codes From Origin Server by Date. Every visualization drills through to the URL Explorer.

## Delivery Center

Cards: Summary (Total Requested Pages, Total Delivered Pages, Delivery Efficiency), All Requests From Allowed User Agents, All Successful Requests By Search Engine, By Device and By Section (following the custom cache behaviors), and All Types of Failure.

## Render Quality

Cards: Summary (Total Quality Controls, Failed & Delivered, Failed & Not Delivered, Render Quality), Quality Control Fails, Fails Over Time, and All Quality Controls (Total Executions, Total Fails, Total Fails Without Delivery, Fail Ratio).

## Preview

Available from the Preview URL box at the top of any SpeedWorkers page, or the preview icon in the URL Explorer. It shows the page as bots see it, plus the speed gain versus no SpeedWorkers (as a ratio and in milliseconds), the HTTP status code, the last refresh and next scheduled refresh, the cache behavior applied (including which bots receive Markdown), the quality controls in scope, the inventory source, the mobile and desktop versions, and the full HTML.

`speedworkers_page_preview` returns the same information for a URL, and `speedworkers_page_history` / `speedworkers_page_snapshot` go back in time to past indexations. See `tools.md`. Both tools report these reasons in snake_case (`not_cached`, `quality_controls_failed`…).

## "Not cached" reasons

Reached by drilling through the Not Cached metric or from the URL Explorer.

| Reason | Meaning |
|---|---|
| bad token | The token is invalid or lacks the right permissions |
| cached | Shown in the URL Explorer for pages that *were* cached |
| configuration | Current settings say the page should not be delivered |
| custom response | Live JavaScript returned a custom error code |
| no data | SpeedWorkers never attempted to add the page to inventory |
| internal error | A SpeedWorkers error, timeout, or overloaded server |
| origin server error | The origin errored while caching, fetching or rendering |
| request origin timeout | The origin timed out |
| quality controls failed | A defined quality control rule failed |
| invalid request | The incoming request contains bad parameters |
| delivery stopped | Delivery was stopped by the current configuration |
| ip filtered | The IP is not a search engine, is unresolved, or is explicitly blocked |
| js code error | An error occurred while executing the page's JavaScript |
| js redirect not enabled | The page is a JavaScript redirect and no configuration returns them as real redirects |
| js redirect same url | The JavaScript redirect target equals the requested URL |
| not html | No `text/html` content-type (a 302 has none but is still deliverable) |
| last refresh 4xx / 5xx | The last refresh returned a 4xx (other than 404) or a 5xx |
| out of scope | The domain is not allowed |
| throttled | Too many bot requests in a period, so a 429 was returned |
| unsupported method | Only GET and HEAD are accepted |
| websocket | WebSockets are not accepted |

## Recurring integration symptoms

- **A 5xx from the SpeedWorkers origin on some paths or bots while others work** — check the token and whether staging and production configurations differ, and whether the failing requests arrive through a CDN origin failover rather than direct routing. Failover paths sometimes rewrite the URI before it reaches SpeedWorkers.
- **A 200 returned for pages that don't exist**, typical on single-page applications — decide the intended contract rather than letting a fallback serve a soft 200. A 410 through an inventory optimization rule is usually the right answer for SEO.
- **An AI tool receives an empty shell** — the requester's user-agent is probably not in the routing rules. See the note on server-side fetchers in `architecture.md`.
- **Traffic reporting looks wrong after go-live** — origin logs now only see the Botify fallback user-agent; log ingestion has to move to the CDN.

## Frequently asked questions

- **Why 304 responses?** If a bot sends an `if-modified-since` header and the page has not been refreshed since that date, SpeedWorkers returns a 304 so the bot does not download identical HTML again. This speeds up content discovery and protects crawl budget.
- **Does it help Core Web Vitals?** No. Core Web Vitals measure user experience; SpeedWorkers measures bot experience.
- **Is it cloaking?** No. It dynamically renders the same pages to make content more accessible to bots, which both Google and Bing recommend.
- **How are refresh credits calculated?** From page render time.
- **How is speed measured?** As total fetch time plus render time when rendering is needed — bot-facing speed, since that is what affects crawl ratio and indexation quality.
- **How long are pages kept in inventory?** It depends on the cache behavior and source retention settings; see the page lifecycle above.