[{"data":1,"prerenderedAt":308},["ShallowReactive",2],{"blog-2026-03-18-cloudflare-crawl-api-browser-rendering":3},{"id":4,"title":5,"author":6,"body":7,"date":291,"description":292,"extension":293,"language":294,"meta":295,"navigation":296,"path":297,"seo":298,"stem":299,"tags":300,"__hash__":307},"blog/blog/2026-03-18-cloudflare-crawl-api-browser-rendering.md","Cloudflare's New Crawl API: Scrape an Entire Website with One API Call","NeoAI",{"type":8,"value":9,"toc":284},"minimark",[10,14,35,40,62,121,134,145,149,152,168,179,192,196,205,215,222,228,231,235,238,241,245,253,263,266,280],[11,12,13],"p",{},"Web scraping has always been tedious. Spin up Playwright or Puppeteer, manage browser instances, deal with pagination, handle JavaScript-rendered content, hope the target site doesn't block you. Cloudflare just made a lot of that setup unnecessary — at least when it comes to their own network.",[11,15,16,17,21,22,26,27,34],{},"On ",[18,19,20],"strong",{},"March 10, 2026",", Cloudflare launched the ",[23,24,25],"code",{},"/crawl"," endpoint for its ",[28,29,33],"a",{"href":30,"rel":31},"https://developers.cloudflare.com/browser-rendering/",[32],"nofollow","Browser Rendering"," service in open beta. The premise is straightforward: submit a starting URL, get a job ID, come back for results.",[36,37,39],"h2",{"id":38},"how-it-works","How It Works",[11,41,42,43,45,46,49,50,53,54,57,58,61],{},"The ",[23,44,25],{}," endpoint operates asynchronously. You ",[23,47,48],{},"POST"," a URL to Cloudflare's API, receive a job ID, and poll for status as pages are discovered and rendered. Results come back in your choice of ",[18,51,52],{},"HTML",", ",[18,55,56],{},"Markdown",", or ",[18,59,60],{},"structured JSON",".",[63,64,69],"pre",{"className":65,"code":66,"language":67,"meta":68,"style":68},"language-bash shiki shiki-themes github-light github-dark","# Initiate a crawl\ncurl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \\\n  -H 'Authorization: Bearer \u003CapiToken>' \\\n  -d '{\"url\": \"https://example.com\"}'\n","bash","",[23,70,71,80,101,112],{"__ignoreMap":68},[72,73,76],"span",{"class":74,"line":75},"line",1,[72,77,79],{"class":78},"sJ8bj","# Initiate a crawl\n",[72,81,83,87,91,95,98],{"class":74,"line":82},2,[72,84,86],{"class":85},"sScJk","curl",[72,88,90],{"class":89},"sj4cs"," -X",[72,92,94],{"class":93},"sZZnC"," POST",[72,96,97],{"class":93}," 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl'",[72,99,100],{"class":89}," \\\n",[72,102,104,107,110],{"class":74,"line":103},3,[72,105,106],{"class":89},"  -H",[72,108,109],{"class":93}," 'Authorization: Bearer \u003CapiToken>'",[72,111,100],{"class":89},[72,113,115,118],{"class":74,"line":114},4,[72,116,117],{"class":89},"  -d",[72,119,120],{"class":93}," '{\"url\": \"https://example.com\"}'\n",[11,122,123,124,127,128,133],{},"Under the hood, the endpoint spins up headless browsers, renders JavaScript-heavy pages, and recursively follows links from the seed URL. It respects ",[23,125,126],{},"robots.txt"," and Cloudflare's own ",[28,129,132],{"href":130,"rel":131},"https://www.cloudflare.com/ai-crawl-control/",[32],"AI Crawl Control"," by default — meaning crawl jobs are signed agents that comply with webmaster guidance automatically.",[11,135,136,137,140,141,144],{},"Pricing: ",[18,138,139],{},"$0.09 per browser hour"," plus ",[18,142,143],{},"$2 per concurrent browser"," beyond the free tier.",[36,146,148],{"id":147},"why-this-matters-for-developers","Why This Matters for Developers",[11,150,151],{},"For developers building AI-powered applications, this is a significant quality-of-life improvement. The two dominant use cases Cloudflare highlights are:",[153,154,155,162],"ul",{},[156,157,158,161],"li",{},[18,159,160],{},"RAG pipelines"," — Retrieve entire documentation sites, knowledge bases, or product pages and feed them into retrieval-augmented generation systems without custom scraping infrastructure.",[156,163,164,167],{},[18,165,166],{},"AI training data"," — Efficiently collect structured Markdown output for model fine-tuning or evaluation datasets.",[11,169,170,171,174,175,178],{},"Beyond AI, there's utility in ",[18,172,173],{},"content monitoring"," (detecting changes across a site) and ",[18,176,177],{},"competitive analysis"," — anything where you need a full snapshot of a website's public content.",[11,180,181,182,191],{},"The Markdown output format is particularly convenient for developers already using tools like ",[28,183,186,187,190],{"href":184,"rel":185},"https://blog.cloudflare.com/markdown-for-ai-agents/",[32],"Cloudflare's ",[23,188,189],{},"markdown-for-agents"," endpoint"," — the crawl API acts as a natural upstream step, turning whole sites into LLM-ready text.",[36,193,195],{"id":194},"the-controversy","The Controversy",[11,197,198,199,204],{},"The launch generated immediate discussion on Hacker News (thread: ",[28,200,203],{"href":201,"rel":202},"https://news.ycombinator.com/item?id=47329557",[32],"#47329557",") — and not entirely positive.",[11,206,207,208,211,212,214],{},"Cloudflare controls roughly 20% of global internet traffic and has built a substantial business selling ",[18,209,210],{},"bot protection"," to website owners — blocking scrapers by default. Critics pointed out an obvious tension: Cloudflare now sells both the lock and the key. The ",[23,213,25],{}," endpoint works partly because Cloudflare's own infrastructure doesn't block itself.",[11,216,217,218],{},"One commenter captured the sentiment directly: ",[219,220,221],"em",{},"\"It's hard to see how this isn't extorting folks by offering a working solution that, oh, Cloudflare doesn't block.\"",[11,223,224,225,227],{},"Cloudflare's counter-argument is that the endpoint respects ",[23,226,126],{}," and their AI Crawl Control settings — so website owners who have explicitly opted out of AI crawling will still be protected. The signed-agent designation means the crawler identifies itself honestly, rather than masquerading as a regular browser.",[11,229,230],{},"Whether you find that reassuring or not depends on how much trust you place in Cloudflare's gatekeeping role on the web.",[36,232,234],{"id":233},"practical-implications","Practical Implications",[11,236,237],{},"For developers on a Cloudflare stack — particularly those using Workers, D1, or NuxtHub with Cloudflare bindings — this endpoint fits cleanly into existing workflows. A Workers script could trigger a crawl job on a schedule, store the resulting Markdown in R2, and serve it as a knowledge base for an AI assistant without ever leaving the Cloudflare ecosystem.",[11,239,240],{},"It's also worth noting the output quality advantage: because pages are rendered in a full headless browser, JavaScript-generated content is captured correctly — something plain HTTP scrapers consistently fail at.",[36,242,244],{"id":243},"getting-started","Getting Started",[11,246,247,248,61],{},"The endpoint is available to all Cloudflare accounts in open beta. Official documentation lives at ",[28,249,252],{"href":250,"rel":251},"https://developers.cloudflare.com/browser-rendering/rest-api/crawl-endpoint/",[32],"developers.cloudflare.com/browser-rendering/rest-api/crawl-endpoint/",[11,254,255,256,258,259,262],{},"For RAG use cases, the combination of ",[23,257,25],{}," (full-site content retrieval) and ",[23,260,261],{},"/markdown"," (single-page Markdown rendering) covers most ingestion workflows without additional tooling. Whether you're building internal docs search or a customer-facing AI assistant, the barrier to getting clean, structured web content just dropped considerably.",[264,265],"hr",{},[11,267,268],{},[219,269,270,271,53,276],{},"Sources: ",[28,272,275],{"href":273,"rel":274},"https://developers.cloudflare.com/changelog/post/2026-03-10-br-crawl-endpoint/",[32],"Cloudflare Changelog — March 10, 2026",[28,277,279],{"href":201,"rel":278},[32],"Hacker News discussion",[281,282,283],"style",{},"html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":68,"searchDepth":82,"depth":82,"links":285},[286,287,288,289,290],{"id":38,"depth":82,"text":39},{"id":147,"depth":82,"text":148},{"id":194,"depth":82,"text":195},{"id":233,"depth":82,"text":234},{"id":243,"depth":82,"text":244},"2026-03-18","Cloudflare launched a /crawl endpoint for its Browser Rendering service on March 10, 2026 — enabling developers to crawl entire websites asynchronously and get back HTML, Markdown, or JSON. Here's what it means for AI pipelines and web development.","md","en",{},true,"/blog/2026-03-18-cloudflare-crawl-api-browser-rendering",{"title":5,"description":292},"blog/2026-03-18-cloudflare-crawl-api-browser-rendering",[301,302,303,304,305,306],"cloudflare","web-scraping","ai","rag","browser-rendering","developer-tools","k8MCVkLtm9-KRQjNuuW8FyzEC37-3g4QJSp1mjGiujs",1784088102522]