# Botify Agent: Speedworkers Inventory

**ID:** `speedworkers_inventory`

Get the daily SpeedWorkers inventory of the project (pages in index, stale, expired, not indexed) and whether the rendering speed keeps up with it


    Get the SpeedWorkers inventory of a project: how many pages are in each state, day
    by day, and whether the rendering capacity is enough to keep them fresh.

    The inventory is a daily snapshot of the pages SpeedWorkers knows about, counted by
    state:
    - in index: fresh pages, delivered
    - stale: past their refresh time but within their max stale time, still delivered
      while waiting to be re-rendered
    - expired: past their max stale time, no longer delivered
    - not indexed: in the inputs but never successfully rendered

    There is one row per day, broken down by the requested dimensions (by section by
    default):
    - section_name: SpeedWorkers section
    - config_id: SpeedWorkers config id
    - priority: refresh priority (emergency, high, medium or low)
    - refresh_frequency: refresh frequency of the pages, in minutes
    An empty list of dimensions returns one total per day. Rows can include up to 5
    sample URLs per state.

    The crawl stats compare today's inventory to the max rendering speed of the
    production version: the required speed is what refreshing every page at its
    section refresh frequency needs, and a load above 1 means pages will go stale or
    expire. They are null when the inventory has never been computed, which is the
    case until the cache refresh is enabled on the production version.

    Use the speedworkers_config tool to see the sections and their refresh settings.
    

## Authentication

All endpoints require a **Bearer token** in the `Authorization` header.

```
Authorization: Bearer <your-token>
```

## Calling this agent

This agent supports the following processing modes:

| Mode | Type | Description |
|------|------|-------------|
| **Process** | Synchronous | Single-item processing. Best for real-time requests with immediate response. |
| **Batch Process** | Synchronous | Process multiple items in a single request for efficiency. |
| **Async Process** | Asynchronous | Single-item processing for long-running tasks that exceed timeout limits. |
| **Async Batch Process** | Asynchronous | Large-scale batch jobs with background processing. |

# Synchronous Processing

Synchronous calls block until the result is ready. Use these for quick operations where you need immediate results.

## Single Item Processing

Process a single input and receive the result immediately.

```
POST https://agents.botify.com/{org}/{project}/speedworkers_inventory/process
```

### Request body

```json
{
  "item": {}
}
```

### Response

Returns a single processed item (HTTP 200).

### cURL example

```bash
curl -X POST "https://agents.botify.com/{org}/{project}/speedworkers_inventory/process" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "item": {}
}'
```

## Batch Processing

Process multiple inputs in a single synchronous request. More efficient than making individual calls. Items are processed concurrently on the server.

```
POST https://agents.botify.com/{org}/{project}/speedworkers_inventory/batch_process
```

### Request body

```json
{
  "items": [
    {}
  ]
}
```

### Response

Returns an array of results. Each element is either a successful processed item (`"status": "success"`) or an error (`"status": "error"`).

### cURL example

```bash
curl -X POST "https://agents.botify.com/{org}/{project}/speedworkers_inventory/batch_process" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {}
  ]
}'
```

### When to use synchronous calls

- Processing completes within timeout limits (typically 30 seconds)
- You need immediate results for user-facing features
- Small to medium batch sizes (up to ~100 items)

# Asynchronous Processing

Asynchronous calls return immediately with a batch ID. You then poll for results using the `async_batches/` endpoints.

### Async processing flow

1. **Submit job** -- `POST` to `async_process` or `async_batch_process`
2. **Receive batch ID** -- response contains `batch_id`
3. **Poll status** -- `HEAD` request to check readiness (lightweight, no body)
4. **Get results** -- `GET` request to retrieve processed results

## Async Single Item

Submit a single item for background processing. Use the `/single` endpoint to retrieve the result.

```
POST https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_process
```

### Request body

```json
{
  "item": {}
}
```

### cURL example

```bash
# Step 1: Submit the job
curl -X POST "https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_process" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "item": {}
}'

# Response: { "batch_id": "abc123" }

# Step 2: Check if batch is ready (HEAD request -- lightweight check)
curl -I -X HEAD "https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_batches/abc123" \
  -H "Authorization: Bearer $TOKEN"
# Returns 200 if ready, 204 if still processing

# Step 3: Get the result (for single-item async)
curl -X GET "https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_batches/abc123/single" \
  -H "Authorization: Bearer $TOKEN"
# Returns 200 with result, 202 if still processing, 404 if batch not found
```

## Async Batch Processing

Submit multiple items for background processing. The request body uses **JSONL** (newline-delimited JSON) format.

```
POST https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_batch_process
```

### Request body (JSONL, `Content-Type: text/plain`)

The first line contains `config` and `batch_config`. Each subsequent line is an item with a unique `id`.

```
{"config": {}, "batch_config": {}}
{"id": "item_1", "item": {}}
```

### Response

Returns a JSON object with `batch_id`, `status`, and a `Location` header.

### cURL example

```bash
# Step 1: Submit the batch job
curl -X POST "https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_batch_process" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: text/plain" \
  -d '{"config": {}, "batch_config": {}}\n{"id": "item_1", "item": {}}'

# Response: { "batch_id": "xyz789" }

# Step 2: Check if batch is ready (HEAD request)
curl -I -X HEAD "https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_batches/xyz789" \
  -H "Authorization: Bearer $TOKEN"
# Returns 200 if ready, 204 if still processing

# Step 3: Get all results
curl -X GET "https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_batches/xyz789" \
  -H "Authorization: Bearer $TOKEN"
```

## Checking Batch Status

Use a lightweight `HEAD` request to check if your batch is ready without transferring data.

```
HEAD https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_batches/{batch_id}
```

```bash
curl -I -X HEAD "https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_batches/{batch_id}" \
  -H "Authorization: Bearer $TOKEN"

# Response headers indicate status:
# HTTP/1.1 200 OK        -> Batch is complete, results ready
# HTTP/1.1 204 No Content -> Still processing
# HTTP/1.1 404 Not Found -> Batch does not exist
```

## Retrieving Results

Once the batch is ready, fetch results using the appropriate endpoint.

```bash
# For async_process (single item) -- use /single endpoint
curl -X GET "https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_batches/{batch_id}/single" \
  -H "Authorization: Bearer $TOKEN"

# For async_batch_process (multiple items) -- use base endpoint
curl -X GET "https://agents.botify.com/{org}/{project}/speedworkers_inventory/async_batches/{batch_id}" \
  -H "Authorization: Bearer $TOKEN"
# Returns streaming JSONL where each line is a processed item or error
```

### Async response codes

| Code | Endpoint | Meaning | Action |
|------|----------|---------|--------|
| 200 | `HEAD` / `GET` | Batch complete, results ready | Read results from response body |
| 204 | `HEAD` | Still processing | Continue polling |
| 202 | `GET /single` | Still processing | Continue polling |
| 400 | `GET /single` | All items failed (client error) | Check error in response body |
| 404 | `HEAD` / `GET` | Batch does not exist | Verify `batch_id` is correct |

> **HEAD vs GET:** Use `HEAD` requests for lightweight status checks (no response body). Use `GET` only when ready to retrieve results to minimize bandwidth.

### When to use asynchronous calls

- Processing takes longer than 30 seconds
- Large batch sizes (100+ items)
- Background processing where immediate results aren't required
- Integration with job queues or workflow systems

## Python Example

A complete Python example showing both synchronous and asynchronous patterns.

```python
import requests
import time

API_TOKEN = "your_api_token"
BASE_URL = "https://agents.botify.com/{organization}/{project}"
AGENT = "speedworkers_inventory"

headers = {
    "Authorization": f"Bearer {API_TOKEN}",
    "Content-Type": "application/json"
}

single_payload = {
  "item": {}
}


def process_sync(payload: dict) -> dict:
    """Synchronous single-item processing."""
    response = requests.post(
        f"{BASE_URL}/{AGENT}/process",
        headers=headers,
        json=payload
    )
    response.raise_for_status()
    return response.json()


def process_async(
    payload: dict,
    poll_interval: int = 5,
    max_retries: int = 60
) -> dict:
    """Asynchronous single-item processing with polling."""
    # Submit job
    response = requests.post(
        f"{BASE_URL}/{AGENT}/async_process",
        headers=headers,
        json=payload
    )
    batch_id = response.json()["batch_id"]

    # Poll for results using HEAD (lightweight check)
    for _ in range(max_retries):
        status = requests.head(
            f"{BASE_URL}/{AGENT}/async_batches/{batch_id}",
            headers=headers
        )

        if status.status_code == 200:
            result = requests.get(
                f"{BASE_URL}/{AGENT}/async_batches/{batch_id}/single",
                headers=headers
            )
            return result.json()
        elif status.status_code == 204:
            time.sleep(poll_interval)
        else:
            raise Exception(f"Unexpected status: {status.status_code}")

    raise Exception("Max retries exceeded")


def process_async_batch(
    items: list,
    config: dict | None = None,
    poll_interval: int = 5
) -> dict:
    """Asynchronous batch processing with polling."""
    import json as _json

    # Build JSONL payload: first line is config, subsequent lines are items
    lines = [_json.dumps({"config": config or {}, "batch_config": {}})]
    for i, item in enumerate(items):
        lines.append(_json.dumps({"id": f"item_{i}", "item": item}))
    body = "\n".join(lines)

    # Submit batch (JSONL, text/plain)
    response = requests.post(
        f"{BASE_URL}/{AGENT}/async_batch_process",
        headers={**headers, "Content-Type": "text/plain"},
        data=body
    )
    batch_id = response.json()["batch_id"]

    # Poll until ready
    while True:
        status = requests.head(
            f"{BASE_URL}/{AGENT}/async_batches/{batch_id}",
            headers=headers
        )

        if status.status_code == 200:
            results = requests.get(
                f"{BASE_URL}/{AGENT}/async_batches/{batch_id}",
                headers=headers
            )
            return results.json()

        time.sleep(poll_interval)


# Example usage
# result = process_sync(single_payload)
# result = process_async(single_payload)
# results = process_async_batch([{"input": "item1"}, {"input": "item2"}])
```

## Billing

Fixed cost per item requested. This tool reads data Botify already holds for your project and calls no outside service, so nothing is added to the per-call price.

These usage SKUs can be charged on a call.

| SKU | Credits | Description |
| --- | ------- | ----------- |
| Tool call | 1 per request | Charged once per successful item. |


## Schemas

### Item

```json
{
  "type": "object",
  "properties": {
    "date_start": {
      "anyOf": [
        {
          "format": "date",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Inclusive start of the inventory period (ISO date, e.g. 2026-09-01). Defaults to 29 days before date_end.",
      "title": "Date Start"
    },
    "date_end": {
      "anyOf": [
        {
          "format": "date",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Inclusive end of the inventory period (ISO date, e.g. 2026-09-30). Defaults to today (UTC).",
      "title": "Date End"
    },
    "dimensions": {
      "default": [
        "section_name"
      ],
      "description": "Dimensions the daily inventory is broken down by, on top of the date (always included). Allowed values: 'section_name' (SpeedWorkers section), 'config_id' (SpeedWorkers config id), 'priority' (refresh priority: emergency, high, medium or low), 'refresh_frequency' (refresh frequency of the pages, in minutes). Each dimension added multiplies the number of rows. An empty list returns one total per day. Defaults to ['section_name'].",
      "items": {
        "$ref": "#/$defs/SpeedworkersInventoryDimension"
      },
      "title": "Dimensions",
      "type": "array"
    },
    "include_samples": {
      "default": false,
      "description": "Whether to return up to 5 sample URLs per page state (in index, stale, expired, not indexed) for each row",
      "title": "Include Samples",
      "type": "boolean"
    }
  },
  "required": [],
  "$defs": {
    "SpeedworkersInventoryDimension": {
      "enum": [
        "section_name",
        "config_id",
        "priority",
        "refresh_frequency"
      ],
      "type": "string"
    }
  }
}
```

### Config

This agent uses the default config (no custom fields).

### Response

```json
{
  "$defs": {
    "CrawlStats": {
      "properties": {
        "max_speed": {
          "anyOf": [
            {
              "type": "number"
            },
            {
              "type": "null"
            }
          ],
          "description": "Max rendering speed of the production version, in pages/s",
          "title": "Max Speed"
        },
        "required_speed": {
          "anyOf": [
            {
              "type": "number"
            },
            {
              "type": "null"
            }
          ],
          "description": "Rendering speed needed to refresh the whole inventory at the sections refresh frequencies, in pages/s",
          "title": "Required Speed"
        },
        "max_pages_refresh": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "description": "Pages that can be rendered per day at max speed",
          "title": "Max Pages Refresh"
        },
        "required_pages_refresh": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "description": "Pages that must be rendered per day to stay fresh",
          "title": "Required Pages Refresh"
        },
        "pages_in_inventory": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "description": "Pages in today's inventory",
          "title": "Pages In Inventory"
        },
        "load": {
          "anyOf": [
            {
              "type": "number"
            },
            {
              "type": "null"
            }
          ],
          "description": "required_speed / max_speed: above 1, the website cannot refresh its inventory in time and pages go stale or expire",
          "title": "Load"
        },
        "info": {
          "$ref": "#/$defs/CrawlStatsInfo",
          "description": "success, or why the speeds could not be computed (no max speed, no pages in today's inventory, a section without refresh frequency...)"
        },
        "sections": {
          "description": "Load of each delivered section",
          "items": {
            "$ref": "#/$defs/CrawlStatsSection"
          },
          "title": "Sections",
          "type": "array"
        }
      },
      "required": [
        "max_speed",
        "required_speed",
        "max_pages_refresh",
        "required_pages_refresh",
        "pages_in_inventory",
        "load",
        "info",
        "sections"
      ],
      "title": "CrawlStats",
      "type": "object"
    },
    "CrawlStatsInfo": {
      "enum": [
        "success",
        "no_config_js",
        "no_website_max_speed",
        "no_pages_in_inventory",
        "invalid_refresh_frequency"
      ],
      "type": "string"
    },
    "CrawlStatsSection": {
      "properties": {
        "name": {
          "description": "SpeedWorkers section",
          "title": "Name",
          "type": "string"
        },
        "load": {
          "anyOf": [
            {
              "type": "number"
            },
            {
              "type": "null"
            }
          ],
          "description": "Share of the website max speed this section requires",
          "title": "Load"
        }
      },
      "required": [
        "name",
        "load"
      ],
      "title": "CrawlStatsSection",
      "type": "object"
    },
    "InventorySample": {
      "properties": {
        "url": {
          "description": "Sample URL",
          "title": "Url",
          "type": "string"
        },
        "device": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "description": "Device the URL is cached for",
          "title": "Device"
        }
      },
      "required": [
        "url",
        "device"
      ],
      "title": "InventorySample",
      "type": "object"
    },
    "InventoryStatsRow": {
      "properties": {
        "date": {
          "description": "Day of the inventory snapshot",
          "format": "date",
          "title": "Date",
          "type": "string"
        },
        "section_name": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "SpeedWorkers section (only with the section_name dimension)",
          "title": "Section Name"
        },
        "config_id": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "SpeedWorkers config id (only with the config_id dimension)",
          "title": "Config Id"
        },
        "priority": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Refresh priority: emergency, high, medium or low (only with the priority dimension)",
          "title": "Priority"
        },
        "refresh_frequency": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Refresh frequency of the pages, in minutes (only with the refresh_frequency dimension)",
          "title": "Refresh Frequency"
        },
        "count": {
          "description": "Number of pages in the inventory",
          "title": "Count",
          "type": "integer"
        },
        "count_in_index": {
          "description": "Pages that are fresh and delivered by SpeedWorkers",
          "title": "Count In Index",
          "type": "integer"
        },
        "count_in_stale": {
          "description": "Pages past their refresh time but within their max stale time: still delivered, waiting to be re-rendered",
          "title": "Count In Stale",
          "type": "integer"
        },
        "count_expired": {
          "description": "Pages past their max stale time: no longer delivered",
          "title": "Count Expired",
          "type": "integer"
        },
        "count_not_indexed": {
          "description": "Pages in the inputs that were never successfully rendered (render pending or failed)",
          "title": "Count Not Indexed",
          "type": "integer"
        },
        "samples_in_index": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/InventorySample"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Sample in index URLs (only with include_samples)",
          "title": "Samples In Index"
        },
        "samples_in_stale": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/InventorySample"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Sample stale URLs (only with include_samples)",
          "title": "Samples In Stale"
        },
        "samples_expired": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/InventorySample"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Sample expired URLs (only with include_samples)",
          "title": "Samples Expired"
        },
        "samples_not_indexed": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/InventorySample"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Sample not indexed URLs (only with include_samples)",
          "title": "Samples Not Indexed"
        }
      },
      "required": [
        "date",
        "count",
        "count_in_index",
        "count_in_stale",
        "count_expired",
        "count_not_indexed"
      ],
      "title": "InventoryStatsRow",
      "type": "object"
    }
  },
  "properties": {
    "date_start": {
      "description": "Inclusive start of the inventory period",
      "format": "date",
      "title": "Date Start",
      "type": "string"
    },
    "date_end": {
      "description": "Inclusive end of the inventory period",
      "format": "date",
      "title": "Date End",
      "type": "string"
    },
    "inventory_stats": {
      "description": "Daily inventory snapshots, one row per date and combination of dimension values, ordered by date. A day without a snapshot has no row. When truncated, a day only partly within the limit is left out, unless it is the only one.",
      "items": {
        "$ref": "#/$defs/InventoryStatsRow"
      },
      "title": "Inventory Stats",
      "type": "array"
    },
    "truncated": {
      "description": "Whether the oldest days were left out to stay within 2000 rows: shorten the period or use fewer dimensions",
      "title": "Truncated",
      "type": "boolean"
    },
    "crawl_stats": {
      "anyOf": [
        {
          "$ref": "#/$defs/CrawlStats"
        },
        {
          "type": "null"
        }
      ],
      "description": "Rendering capacity against today's inventory. Null when the inventory has never been computed for the website, i.e. the cache refresh is not enabled on the production version."
    }
  },
  "required": [
    "date_start",
    "date_end",
    "inventory_stats",
    "truncated",
    "crawl_stats"
  ],
  "title": "SpeedworkersInventoryProcessedItem",
  "type": "object"
}
```

## Best Practices

**Use exponential backoff for polling.** Start with short intervals (1-2 seconds) and increase the delay between polls to reduce API load.

**Set reasonable timeouts.** For synchronous calls, configure your HTTP client with appropriate timeout values (30-60 seconds).

**Handle rate limits gracefully.** Implement retry logic with backoff when you receive 429 (Too Many Requests) responses.

**Batch when possible.** Use batch endpoints to reduce the number of API calls and improve throughput.