# Settings

Navigation: `Activation > SpeedWorkers >` Overview | Inventory Monitoring | Render Quality | Delivery Center | URL Explorer | Settings. The Settings tabs are **Inventory**, **Cache Behaviors**, **Quality Controls** and **Control Center**.

## Versioning

Every settings change lives in a version: **Draft** (one at a time, no effect until pushed), **In Production** (one at a time, read-only — editing it creates a new Draft and overwrites any existing one), and **Archived** (previous deployments, which can be reverted to or used as the basis for a new Draft). Push through *Review & Push to Production*, which shows a diff of settings added, deleted and changed, and lets you attach a note. Each section records who changed what and when.

`speedworkers_versions` lists the versions with their push date, author and note; `speedworkers_config` returns the full settings of production (default), the draft, or any version number. See `tools.md`.

## Inventory

Defines which URLs SpeedWorkers may cache and deliver. Two blocks: the sources ("List all URLs to cache") and the inventory optimization rules.

Each source shows its type, a summary of its rule, fetch and cache toggles, the number of URLs it contributes, and a preview link. Fetch and cache can be deactivated per source — for Bots Discovery, only fetch can be deactivated. Set one or more inventory domains; drop the `www` to include subdomains. Deleting a source removes its URLs from inventory unless they also come from another source.

### Source types

- **Manual list** — URLs typed in, or uploaded as a .txt/.csv file.
- **Fetch from a distant file** — a .csv or .xml file on S3, GCP or HTTP, one URL per line. Download strategies: file at a URL, last modified file matching a prefix, or all new files matching a prefix from a given date. Choose the download frequency and the action applied to the URLs found: *append to existing list* (which also resets the "last seen" date of known URLs), *overwrite*, *force refresh*, *purge from cache*, or *purge and refresh*. This action cannot be changed once saved. Retention after a URL is last seen: up to 90 days. A good first setup is to seed from recently crawled URLs in the logs, then switch to Bot request.
- **Bot request** — adds URLs bots actually requested, whether or not they were already in inventory. Configure N requests from a given bot over a period; a minimum of two is recommended. When configurations overlap, the one with the fewest requests over the longest window takes precedence, making the others redundant. Selectable per cache behavior.
- **Links discovery** — extracts links from pages already in inventory, skipping `rel="nofollow"`. Use with caution: it can add a very large number of URLs at once, including URLs blocked by robots.txt. Retention up to 90 days.

A URL found by several sources stays in inventory until it disappears from all of them. Reporting by inventory source counts such a URL once per source, by design.

### Inventory optimization rules

Scoped URL rules that avoid fetch and render entirely, preserving cache capacity:

- **Block URLs** — for non-HTML resources and patterns already blocked by robots.txt, such as checkout, facets and auto-generated URLs.
- **Static HTML** — serve a fixed HTTP 200 body on URLs that don't exist on the origin. Useful for domain ownership verification, or for returning one page across many URLs.
- **Static 4xx** — serve 404 (the resource is not here) or 410 (it existed and is gone, which search engines de-index faster), with custom HTML.
- **3xx redirect** — serve 301 or 302, either to a single URL or through a **custom JavaScript redirection** that derives the destination from the requested URL. The JavaScript can be tested against sample URLs in the interface.
- **JS-computed response** — serve a response computed by JavaScript rather than a static body or redirect.

Conditions combine with And/Or. **Order matters: the first matching rule wins** and later ones are skipped, so reorder deliberately. Deletion cannot be undone. The blocked-assets rule set up during onboarding should be left in place.

## Cache Behaviors

Controls how pages are fetched and rendered and how long they stay valid. One **default behavior** plus ordered **custom behaviors** that override it on a defined scope (again, first match wins; custom behaviors inherit the default values until edited). A banner at the top shows the current refresh capacity.

### Default and custom rules

- **Refresh rate** — minimum 4 hours, maximum 60 days.
- **Stale window** — how long to keep serving a page past its refresh period when a refresh fails. Roughly 25% of the refresh rate is a good default (weekly refresh → 2 days stale). It prevents pages dropping straight to Expired.
- **Refresh priority** — which pages are refreshed first.
- **Mobile and desktop** — either one responsive version (recommended for responsive sites) or separate desktop and mobile versions, each with its own rendering configuration: default JavaScript, HTML, or no rendering.
- **Ignored parameters** — parameters to strip so SpeedWorkers caches clean URLs, with a chosen behavior for the stripped URLs. Optionally also strip them from links, canonicals and hreflang attributes on all in-scope pages.
- **Content-Type** — leave at `text/html` unless there is a specific reason.
- **JavaScript redirection** — a JS-driven URL change without a page reload. Choose not to cache the page, or to cache it as a 301 or 302.

### Advanced rules

Avoid wasted refreshes by branching on state: current or previous HTTP status code, the count of consecutive identical status codes, how long the same status code has persisted (in minutes), current or previous page property, or the inventory source. Each rule sets its own refresh rate, stale window and priority, or a *do not update cache* (prevent indexation) flag. These rules feed Refresh Capacity alongside inventory size, refresh rates and max refresh speed.

Examples: a page back to 200 after a 404 could be refreshed daily; a URL returning 404 for more than a week could drop to every 15 days.

### Markdown for AI bots

Per behavior — default or custom — choose which AI bots receive Markdown instead of HTML. The rationale: because speed matters in real-time retrieval, AI bots may deprioritize hard-to-parse HTML, and a simplified content-first view maximizes the chance they access, understand and summarize the content accurately. Notes:

- Traditional search bots continue receiving fully rendered HTML from the cache.
- If a bot explicitly requests Markdown, SpeedWorkers returns Markdown only. A global "Markdown-only response" option honors any bot that asks for it, regardless of the selection.
- Same URL, same content, so this is not cloaking.
- Commonly enabled for editorial and content pages rather than product listing and detail pages.
- Any cached page can be previewed as HTML, HTML+Markdown, or Markdown only, even when the option is disabled.

## Quality Controls

Quality controls ensure pages served to bots contain the content blocks needed for indexation. A control is **scope** → **goal** → **action**.

- **Goal by element** — a CSS selector must exist, or occur at or above a threshold of occurrences.
- **Goal by text** — a CSS selector whose first match, or all matches, contain given text.
- Test the selector against real URLs in the interface: Success, Fail, or Not Cached (the URL is outside the control's scope).
- **Action** — *deliver the page* (report the failure only) or *do not deliver the page* (report it and forward the bot request to the origin server).
- Controls can be saved without activating, then toggled on and off, and deleted.
- CSS selector guidance: prefer an `#id`, then a `.class`, then `[attribute='value']`. Selectors copied from browser dev tools tend to be brittle and break with every release. Botify Assist can also generate the right selector from a plain-language description of the element.
- Timing subtlety worth stating explicitly: controls run **when a page is cached**, but the Render Quality charts count **deliveries**. A page cached with a failing control therefore adds a failure every time it is served — intentional, since what matters is what bots actually received.

When adjusting a control, record the previous metrics, label the new settings version, then wait for in-scope pages to be re-rendered *and* delivered before comparing. There is always a lag.

## Control Center

- **Delivery toggle** — disabling stops delivery to bots; fetch, render and cache keep running. To stop fetching, deactivate it per inventory source instead.
- **Service status tests** — run automatically every 10 minutes, or on demand, using a Botify bot against a specific URL. Rule types: **Cache Hit**, **Cache Miss**, **Force Timeout** (with a wait in minutes), and **Out of Scope**, which cannot be combined with any other type. Recommended: at least one of each, plus several Cache Hit tests across templates such as product, listing and support pages. A failing test raises a warning banner across SpeedWorkers, with an explanation on hover.