Skip to main content
Docs API Ref
POST /ai/{crawl,scrape,search,browser,links,unblocker}

AI API

Every AI endpoint takes a plain English prompt alongside the parameters you already use on the standard API. Spider plans the work, cleans the page, and returns data shaped by that prompt. Six routes cover crawling, scraping, search, browser automation, link discovery, and unblocking, all on one request and response contract.

Base URL
https://api.spider.cloud
Method
POST
Auth
Bearer API key
Access
AI Studio subscription
On this page

Routes

Six routes, all POST only. Any other method returns 400. Each one also answers on a /v1 alias, so /v1/ai/crawl is equivalent to /ai/crawl.

How AI routes differ

The request and response contract matches the standard API. Six things behave differently once a prompt is involved.

  • A prompt, not a selector

    Describe the data you want in plain language. There is no CSS path to maintain when the markup changes.

  • Everything the standard route takes

    Each AI route accepts the full parameter surface of its non-AI counterpart, plus the AI fields.

  • Structured output

    Results arrive on metadata.extracted_data, shaped by extraction_schema when you supply one.

  • Always a real browser

    AI routes cache, but they never skip the browser, so the model reads live page content.

  • Separate billing

    AI routes require an active AI Studio subscription, and each request spends credits against it.

  • Model cost passed through

    ai_cost is the raw vendor token spend and is excluded from domain markup.

Authentication

Send your API key as a bearer token against https://api.spider.cloud. The same key works across every Spider endpoint.

AI routes also require an active AI Studio subscription , and every request spends credits against it. A key that works on /crawl can still be refused here. See Errors for what that looks like.

Authorization header required

Bearer YOUR_API_KEY. Requests without it fail with an empty 400.

Content-Type header required

application/json.

Common parameters

These apply across the AI routes. Each route additionally accepts everything its standard counterpart takes, so the Parameters reference covers proxies, geo-location, caching, headers and the rest.

prompt string required

Natural language description of what to do and what to pull back. An empty prompt is rejected with a 400.

extraction_schema object optional

Shape for the structured result. Takes `name` (required), `description` (optional) and `schema`, which may be a JSON object or a JSON encoded string.

cleaning_intent "extraction" | "action" | "general" optional, crawl / scrape / links / unblocker only

How aggressively the HTML is reduced before the model reads it. Use "extraction" for data pulls, "action" to keep interactive elements, "general" for a balanced clean. Not accepted on /ai/search or /ai/browser.

metadata boolean default false

Return page metadata such as title and description. Note that `metadata.extracted_data` is attached to the response either way, so structured output never depends on this flag.

return_format string optional

Content shape for each page, for example "markdown", "raw", "text" or "commonmark".

limit number optional

Maximum pages to process. The value is lifted internally into a crawl budget of `{"*": limit}`.

An extraction_schema pins the output shape. Pass schema as an object, or as a JSON encoded string if that is easier to carry through your client.

{
  "extraction_schema": {
    "name": "pricing_tier",
    "description": "One pricing tier from the plans table.",
    "schema": {
      "type": "object",
      "properties": {
        "tier": { "type": "string" },
        "monthly_cost": { "type": "number" }
      },
      "required": ["tier", "monthly_cost"]
    }
  }
}

Response

Every AI route returns a JSON array of page objects, even when you asked for a single URL. Two fields catch people out, so they are called out below.

url string always

The URL this result came from.

status number always

HTTP status returned by the target page, not by the Spider API.

error string | null always

Per page error. Null on success. A page can fail while the request as a whole returns 200.

duration_elasped_ms number always

Time spent on this page. Note the spelling: "elasped" is part of the public contract, so reading duration_elapsed_ms returns undefined.

costs object | null always

Cost breakdown in USD. It is null, not an object of zeros, whenever the request bills nothing, so check before reading total_cost.

metadata.extracted_data object always

The structured result. It is attached whether or not you passed metadata: true.

content string conditional

Page content in the shape named by return_format. Omitted when no content was requested.

links string[] conditional

Discovered links. Always present on /ai/links, which forces link return on.

Endpoints

Six references, each one listing only what is specific to that route. Everything in Common parameters applies to all of them.

AI Crawl

POST /ai/crawl

Crawl a site from one starting URL and let the prompt decide what matters on each page. Returns an array of pages with structured data attached.

route parameter
depth number optional

How many link levels below the starting URL to follow.

post /ai/crawl
import os

import requests

payload = {
  "url": "https://spider.cloud",
  "prompt": "Collect every pricing tier with its monthly cost and included credits.",
  "cleaning_intent": "extraction",
  "depth": 2,
  "limit": 25,
  "return_format": "markdown",
  "metadata": True
}

response = requests.post(
    "https://api.spider.cloud/ai/crawl",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())
application/json
[
  {
    "url": "https://spider.cloud/pricing",
    "error": null,
    "status": 200,
    "content": "# Pricing\n\nStarter, Pro and Enterprise plans.",
    "metadata": {
      "title": "Pricing",
      "description": "Spider Cloud plans and included credits.",
      "extracted_data": {
        "tiers": [
          { "tier": "Starter", "monthly_cost": 0, "included_credits": 200 },
          { "tier": "Pro", "monthly_cost": 99, "included_credits": 50000 }
        ]
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 4821
  }
]

AI Scrape

POST /ai/scrape

Fetch a single URL and extract exactly what the prompt asks for. Pair it with an extraction_schema when you need a stable shape.

post /ai/scrape
import os

import requests

payload = {
  "url": "https://spider.cloud/pricing",
  "prompt": "Pull the Pro plan name, its monthly cost, and whether annual billing is selected.",
  "cleaning_intent": "extraction",
  "extraction_schema": {
    "name": "pricing_tier",
    "description": "One pricing tier from the plans table.",
    "schema": "{\"type\":\"object\",\"additionalProperties\":false,\"properties\":{\"tier\":{\"type\":\"string\"},\"monthly_cost\":{\"type\":\"number\"},\"is_annual\":{\"type\":\"boolean\",\"default\":true,\"description\":\"Annual billing selected\"}},\"required\":[\"tier\",\"monthly_cost\"]}"
  },
  "return_format": "markdown",
  "metadata": True
}

response = requests.post(
    "https://api.spider.cloud/ai/scrape",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())
application/json
[
  {
    "url": "https://spider.cloud/pricing",
    "error": null,
    "status": 200,
    "content": "# Pricing\n\nStarter, Pro and Enterprise plans.",
    "metadata": {
      "title": "Pricing",
      "extracted_data": {
        "tier": "Pro",
        "monthly_cost": 99,
        "is_annual": false
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 2140
  }
]

AI Search

POST /ai/search

Run a web search driven by the prompt and return ranked results. Pass `search` yourself to skip query generation, or leave it off and the prompt becomes the query.

route parameters
search string optional, search only

The literal search query. When omitted, a query is generated from the prompt.

fetch_page_content boolean default false, search only

Fetch and return the content of each result page instead of the result listing alone.

num number optional, search only

Number of search results to return.

post /ai/search
import os

import requests

payload = {
  "prompt": "Find recent benchmarks comparing Rust web crawlers on throughput.",
  "search": "rust web crawler throughput benchmark",
  "num": 10,
  "fetch_page_content": True,
  "return_format": "markdown",
  "limit": 10
}

response = requests.post(
    "https://api.spider.cloud/ai/search",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())
application/json
[
  {
    "url": "https://example.com/rust-crawler-benchmarks",
    "error": null,
    "status": 200,
    "content": "# Rust crawler benchmarks\n\nThroughput measured across 12 runs.",
    "metadata": {
      "title": "Rust crawler benchmarks",
      "extracted_data": {
        "results": [
          { "title": "Rust crawler benchmarks", "rank": 1 },
          { "title": "Comparing async crawlers", "rank": 2 }
        ]
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 3675
  }
]

AI Browser

POST /ai/browser

Drive a real browser session with a natural language prompt, so clicks, toggles and form steps happen before extraction. Best for pages that only reveal data after interaction.

route parameter
wait_for object optional

A WaitFor struct, not a number. Fields are `selector` and `dom` (each `{ timeout, selector }`), `idle_network`, `idle_network0` and `almost_idle_network0` (each `{ timeout }`), `delay` (`{ timeout }`) and the boolean `page_navigations`. Every timeout is a duration object of `{ secs, nanos }`.

post /ai/browser
import os

import requests

payload = {
  "url": "https://spider.cloud/pricing",
  "prompt": "Switch the billing toggle to annual, then read back each plan name with its price.",
  "wait_for": {
    "selector": {
      "timeout": {
        "secs": 10,
        "nanos": 0
      },
      "selector": "#pricing"
    },
    "idle_network": {
      "timeout": {
        "secs": 15,
        "nanos": 0
      }
    },
    "page_navigations": True
  },
  "return_format": "markdown",
  "metadata": True
}

response = requests.post(
    "https://api.spider.cloud/ai/browser",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())
application/json
[
  {
    "url": "https://spider.cloud/pricing",
    "error": null,
    "status": 200,
    "content": "# Pricing\n\nAnnual billing selected.",
    "metadata": {
      "title": "Pricing",
      "extracted_data": {
        "billing_period": "annual",
        "plans": [
          { "plan": "Pro", "price": 79 }
        ]
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 9310
  }
]

AI Links

POST /ai/links

Collect the links on a page that match the prompt. Link return is forced on for this route, so every page object carries a `links` array.

route parameter
return_page_links boolean forced on

Always enabled for this route. You do not need to send it, and sending false does not disable it.

post /ai/links
import os

import requests

payload = {
  "url": "https://spider.cloud",
  "prompt": "Find every documentation link under the API reference.",
  "cleaning_intent": "general",
  "limit": 50,
  "return_format": "raw"
}

response = requests.post(
    "https://api.spider.cloud/ai/links",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())
application/json
[
  {
    "url": "https://spider.cloud",
    "error": null,
    "status": 200,
    "links": [
      "https://spider.cloud/docs/api",
      "https://spider.cloud/docs/quickstart",
      "https://spider.cloud/docs/ai"
    ],
    "metadata": {
      "title": "Spider Cloud",
      "extracted_data": {
        "docs_links": [
          { "label": "API reference", "url": "https://spider.cloud/docs/api" },
          { "label": "Quickstart", "url": "https://spider.cloud/docs/quickstart" }
        ]
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 1988
  }
]

AI Unblocker

POST /ai/unblocker

Fetch a page that is behind bot protection and hand the cleaned result to the model. Use it when a normal scrape returns a challenge page.

post /ai/unblocker
import os

import requests

payload = {
  "url": "https://example.com/protected-article",
  "prompt": "Return the article headline and body text without navigation or ads.",
  "cleaning_intent": "general",
  "return_format": "markdown",
  "metadata": True
}

response = requests.post(
    "https://api.spider.cloud/ai/unblocker",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())
application/json
[
  {
    "url": "https://example.com/protected-article",
    "error": null,
    "status": 200,
    "content": "# The article title\n\nBody text without navigation or ads.",
    "metadata": {
      "title": "The article title",
      "extracted_data": {
        "headline": "The article title",
        "word_count": 1284
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 6402
  }
]

Errors

Three failures account for nearly everything you will see. The status code is the whole signal on the first one, since it arrives with no body.

400
Validation failed

Returned with an empty body. There is no message to parse. It covers malformed JSON, a missing or empty prompt, an unrecognised field, a missing Authorization header, and an unresolvable API key. If a request fails with nothing in the body, check those first.

402
AI Studio subscription required

The account has no active AI Studio subscription, or the subscription has no credits left to spend. Plans are on the AI Studio pricing page .

{
  "error": "AI Studio subscription required",
  "code": "ai_subscription_required",
  "url": "https://spider.cloud/ai/pricing"
}
429
Rate limit exceeded

Carries a Retry-After header in seconds, and retry_after_ms in the body. Back off for that long rather than retrying immediately.

{
  "error": "Rate limit exceeded",
  "retry_after_ms": 1000
}

Rate limits

AI routes are limited by requests per second on the account's AI Studio tier, separately from the monthly credit allowance.

TierRequests per secondCredits per month
Starter1600
Lite53,000
Standard1012,500
Scale2560,000

Looking for the non-AI routes? Start at the API reference . For plans and included credits, see AI Studio pricing .