No, credits never expire. There is no monthly subscription — buy once and use them whenever you need.

What if I need more credits?

Buy more anytime. Volume purchases of $500+ unlock bonus credits up to 30% extra — no negotiations needed.

What are the rate limits?

Every account can make up to 50,000 core API requests per second.

The Web Crawler for AI Agents and LLMs

Turn any URL into clean, structured data with one API call. No proxies to manage, no parsing to debug, no infrastructure to run.

Get started free Read the docs

2,500 free credits on signup. No card required.

spider.cloud

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

json_data = {
  "url": "https://spider.cloud",
  "return_format": "markdown"
}

response = requests.post('https://api.spider.cloud/scrape',
  headers=headers, json=json_data)

print(response.json())

const response = await fetch('https://api.spider.cloud/scrape', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: "https://spider.cloud",
    return_format: "markdown"
  })
});

console.log(await response.json());

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

json_data = { "limit": 5, "url": "https://spider.cloud" }

response = requests.post('https://api.spider.cloud/crawl',
  headers=headers, json=json_data)

print(response.json())

const response = await fetch('https://api.spider.cloud/crawl', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    limit: 5,
    url: "https://spider.cloud"
  })
});

console.log(await response.json());

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

json_data = { "limit": 5, "url": "https://spider.cloud" }

response = requests.post('https://api.spider.cloud/screenshot',
  headers=headers, json=json_data)

# returns base64 encoded image
print(response.json())

const response = await fetch('https://api.spider.cloud/screenshot', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    limit: 5,
    url: "https://spider.cloud"
  })
});

// returns base64 encoded image
console.log(await response.json());

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

json_data = { "limit": 5, "url": "https://spider.cloud" }

response = requests.post('https://api.spider.cloud/links',
  headers=headers, json=json_data)

print(response.json())

const response = await fetch('https://api.spider.cloud/links', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    limit: 5,
    url: "https://spider.cloud"
  })
});

console.log(await response.json());

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

json_data = { "search": "AI web crawler" }

response = requests.post('https://api.spider.cloud/search',
  headers=headers, json=json_data)

print(response.json())

const response = await fetch('https://api.spider.cloud/search', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    search: "AI web crawler"
  })
});

console.log(await response.json());

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

json_data = {
  "url": "https://spider.cloud",
  "return_format": "markdown"
}

response = requests.post('https://api.spider.cloud/scrape',
  headers=headers, json=json_data)

print(response.json())

const response = await fetch('https://api.spider.cloud/scrape', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    url: "https://spider.cloud",
    return_format: "markdown"
  })
});

console.log(await response.json());

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

json_data = { "limit": 5, "url": "https://spider.cloud" }

response = requests.post('https://api.spider.cloud/crawl',
  headers=headers, json=json_data)

print(response.json())

const response = await fetch('https://api.spider.cloud/crawl', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    limit: 5,
    url: "https://spider.cloud"
  })
});

console.log(await response.json());

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

json_data = { "limit": 5, "url": "https://spider.cloud" }

response = requests.post('https://api.spider.cloud/screenshot',
  headers=headers, json=json_data)

# returns base64 encoded image
print(response.json())

const response = await fetch('https://api.spider.cloud/screenshot', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    limit: 5,
    url: "https://spider.cloud"
  })
});

// returns base64 encoded image
console.log(await response.json());

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

json_data = { "limit": 5, "url": "https://spider.cloud" }

response = requests.post('https://api.spider.cloud/links',
  headers=headers, json=json_data)

print(response.json())

const response = await fetch('https://api.spider.cloud/links', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    limit: 5,
    url: "https://spider.cloud"
  })
});

console.log(await response.json());

import requests, os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

json_data = { "search": "AI web crawler" }

response = requests.post('https://api.spider.cloud/search',
  headers=headers, json=json_data)

print(response.json())

const response = await fetch('https://api.spider.cloud/search', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    search: "AI web crawler"
  })
});

console.log(await response.json());

Built into the leading AI frameworks

Get any data, from any site

Other crawlers break on the first bot check. Spider doesn't. Antidetect browsers, proxy rotation, and vision AI that actually works.

UNMATCHED SPEED

Crawl 100 pages in under 2 seconds. Results stream back the instant they land so your pipeline never stalls.

$ spider crawl example.com --limit 100

▶ Throughput: 0 pages/sec

> complete in 5.00s

/about

/pricing

/docs/api

/blog/1

/careers

/contact

/faq

/terms

PAY PER USE

Zero commitments. Pay per page, starting under a tenth of a cent. Price drops automatically as you scale.

Cost per crawl LIVE

$ 0 . 0 0 1 1 4 4 2 1

RELIABILITY

99.9% success rate, even on protected sites. Rotating proxies and anti-bot bypass on every request.

99.9%

success rate

30-day uptime

AI EXTRACTION

Describe what you need in plain English. Vision models read the rendered page and return structured JSON.

prompt "Get every listing with price and rating"

Source

<div class="listing">
  <h3>MacBook Air M4</h3>
  <span>$1,099</span>
  <span>4.8 ★</span>
</div>

Structured

[
  { "title": "MacBook Air M4",
    "price": "$1,099",
    "rating": 4.8 }
]

INTEGRATIONS

SDKs for Python, Node, Rust, and Go. Native plugins for LangChain, LlamaIndex, CrewAI, and more. Up and running in minutes.

+12

WEB SEARCH API

One API call. Real search results, fully scraped, with citations. Your AI agent goes from question to grounded answer in under 3 seconds.

OpenAI o3 vs Claude 4 benchmarks 5 results

arxiv.org 412ms

Benchmarking Large Language Models: O3 and Claude 4

We evaluate performance across MMLU, HumanEval, and GPQA. Claude 4 scores 92.1% on reasoning tasks while o3 leads in...

artificialanalysis.ai 580ms

LLM Benchmark Comparison: Speed, Quality, and Price

Side-by-side comparison of output quality, latency, and cost per token across frontier models including o3, Claude 4...

simonwillison.net 310ms

Hands-on with o3 and Claude 4: what actually changed

After testing both models extensively on real-world tasks, the practical differences come down to tool use and long...

AI BROWSER

The highest-rated stealth browser on the market. Act, extract, and observe on any site with automatic CAPTCHA solving.

protected-site.com/dashboard stealth

navigate protected-site.com/login 120ms

captcha hCaptcha solved automatically 1.2s

act Click "Export CSV" button 85ms

extract 42 rows of structured data 310ms

observe Dashboard loaded, 3 tables found 95ms

done 6 steps, 0 errors 2.8s

SPEED TEST

tailwindcss.com · 06/2024

Three modes, one API. Smart mode figures out which pages need a browser and which don't, so you don't have to.

HTTP

2s 405 pages

SMART

5s 190 browser, 405 HTTP

BROWSER

8s 595 pages

HTTP → SMART → BROWSER

Teams trust Spider to collect the web

Powering data pipelines for AI companies, agencies, and developers worldwide.

GitHub discussions Discord chat

@iammerrick

Rust based crawler Spider is next level for crawling & scraping sites. So fast. Their cloud offering is also so easy to use. Good stuff. https://github.com/spider-rs/spider

@WilliamEspegren

Web crawler built in rust, currently the nr1 performance in the world with crazy resource management Aaaaaaand they have a cloud offer, that's wayyyy cheaper than any competitor Name a reason for me to use anything else? github.com/spider-rs/spid…

@gasa

@spider_rs is the best crawling tool i have used. I had a complicated project where i needed to paste url and get the website whole website data. Spider does it in an instant

@Ashpreet Bedi

@spider_rs is THE best crawler out there, give it a try

@Troyusrex

I found a new tool, Spider-rs, which scrapes significantly faster and handles more scenarios than the basic scraper I built did. Our use of Spider-rs and AWS infrastructure reduced the scraping time from four months to under a week.

@Dify.AI

🕷️ Spider @spider_rust can be used as a built-in tool in #Dify Workflow or as an LLM-callable tool in Agent. It allows fast and affordable web scraping and crawling when your AI applications need real-time web data for context.

Built for production

Rust core. Open source. Battle-tested at billions of pages.

100K+ pages per second

Written in Rust with async concurrency from the ground up. Crawl entire sites in seconds, not hours. Results stream back the instant they land.

Open source core

The crawler that powers this API is open source with 2K+ GitHub stars. No vendor lock-in. Audit the code, self-host, or use our cloud.

AI-native output

Returns clean markdown, structured JSON, or screenshots. Vision models extract data from any page. Your LLM gets exactly what it needs, nothing it doesn't.

Gets past bot detection

Antidetect browsers, residential proxies, and fingerprint rotation built in. Other crawlers break on the first challenge. Spider doesn't.

Frequently Asked Questions

Everything you need to know about Spider.

What is Spider?

Spider is a fast web scraping and crawling API designed for AI agents, RAG pipelines, and LLMs. It supports structured data extraction and multiple output formats including markdown, HTML, JSON, and plain text.

How can I try Spider?

What formats can Spider convert web data into?

Spider outputs HTML, raw, text, and various markdown formats. It supports JSON, JSONL, CSV, and XML for API responses.

Can you crawl all pages?

Yes, Spider accurately crawls all necessary content without needing a sitemap ethically. We rate-limit individual URLs per minute to balance the load on a web server.

Does it respect robots.txt?

Yes, compliance with robots.txt is default, but you can disable this if necessary.

What if a crawl fails?

Failed requests cost nothing. You only pay for successful responses that return data.

What if I get blocked?

Spider includes an unblocker with stealth mode, rotating proxies, and automatic retries. For heavily protected sites, the browser cloud provides full browser sessions with anti-detection built in.

How does billing work?

Each request is billed for bandwidth ($1/GB) plus compute ($0.001/min). Most pages cost a fraction of a cent. You can estimate your costs with the pricing calculator above.