WebCrawler API

WebCrawler API sits in the managed-scraping layer, and its differentiation is output quality plus the 2026 agent features rather than raw crawl throughput. The core loop is simple: POST a URL to /v2/scrape with output_formats: ["markdown", "links"] and get back both the cleaned article body and a structured hyperlink array. Cleaning is not a post-processing afterthought — menus, cookie banners, footers and ads are removed server-side, and the vendor's own FAQ claims the markdown is clean enough to pass straight into an LLM prompt or vector store. That's the claim to test on your own pages, but the pipeline design is right. The strongest additions of 2026 are Wagent and change detection. Wagent (June 19, 2026) is an agent that performs web searches, reads pages, decides which links are relevant, follows them and extracts data into a JSON Schema you supply — you give it a prompt, seed URLs, an output_schema, a model choice (the changelog lists openai/gpt-5.4-mini, anthropic/claude-sonnet-4.6 and google/gemini-3.1-flash-lite-preview) and a required max_spend_usd cap. Billing is per LLM token plus per page scraped, so the spending cap isn't optional garnish; it's the control that keeps a sprawling agent run from surprising you. Change detection feeds take the opposite approach to freshness: instead of a polling loop, you subscribe to a site and receive only changed pages with diffs, which is the correct shape for keeping a documentation knowledge base current without burning tokens. Operationally, the boring parts are handled: residential proxies, automatic retries, rate-limit handling, real headless browsers, JavaScript rendering, CAPTCHA solving and anti-bot bypass, with fastest-path selection per request. The published numbers are 98% extraction success, 2.5s average crawling time and 99.98% uptime, with cached pages around 0.9s versus 4.7s uncached — pass max_age=0 when you must bypass cache. Job cost is visible per job in the dashboard and filterable by status, and GET /organization/balance lets you check spendable balance programmatically. Where it strains. The /markdown LLM-cleaning endpoint adds one to two seconds of latency per page and silently falls back to raw uncleaned markdown if the LLM step fails — you need to detect that fallback if clean output is a hard requirement. The synchronous /v2/scrape endpoint times out after three minutes (raised from two in May 2026), so the heaviest pages still belong on the async job path or webhooks. Parallel request ceilings scale with plan — 5 on pay-as-you-go, 10 on the $29/mo Starter, 20 on the $99/mo Professional, 50 on the $499/mo Business — and that ceiling, not the per-page rate, is usually what limits a large backfill. This is also an indie-built product with support handled directly by the founder, which is fine for a developer tool and less fine if you need a named support contact or a formal security review. And the credit model means you're prepaying for a service whose upstream target