Website Crawler
Fast HTTP pull of public HTML with no browser session. Ideal when the first response already contains the content you need.
Credit cost: 1 credit per call
ApiKeyAuthx-api-key<token>Your API key. Include it in the x-api-key header for every request.
query*stringPublic URL to retrieve with the Crawler tier (plain HTTP, no JavaScript). Include http:// or https://.
Example:
https://books.toscrape.com/Pick a tier:
- Static or SSR HTML: Website Crawler (this endpoint).
- Needs JavaScript: Website Rendering.
- Bot wall on an approved domain: Website WebUnlocker.
Guide: fetch tier comparison.
allow_redirects?booleanWhether BrowserSolver should chase HTTP redirects for this fetch.
true: follow the redirect chain (default)false: stop on the first status and return that response
Tip: set false when you need the intermediate 301/302 hop itself.
truereturn_page_source?booleanChoose between processed markup and the exact bytes from the origin.
false: return BrowserSolver's processed HTML (default)true: return the untouched source from the web server
Tip: enable true for debugging parsers against the raw document.
falsefields?stringComma-separated list of response fields to return, with dot notation for nested fields (e.g. name,location.city).
is_async?booleanRun the request as a background job: the API answers 202 with a job_id to poll on GET /v1/jobs/{id}.
falsesoft_fail?booleanReturn HTTP 200 even when the request fails; the error is in the error field of the body.
falseSuccessful response
response?stringapplication/jsoncurl -X GET 'https://api.browsersolver.com/v1/website/crawler?query=string'"<!DOCTYPE html><html lang=\"en\"><head><meta charset=\"utf-8\"><title>Welcome to Python.org</title><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><link rel=\"stylesheet\" href=\"/static/stylesheets/style.css\"></head><body><div id=\"touchnav-wrapper\"><header class=\"main-header\"><h1>Python</h1><nav><a href=\"/downloads/\">Downloads</a><a href=\"/doc/\">Documentation</a></nav></header><main><section class=\"launch-shell\"><h2>Get Started</h2><p>Python is a programming language that lets you work quickly.</p></section></main></div></body></html>"