How to Perform Large-Scale Website Crawls (1M+ URLs)

Crawling a hundred pages is trivial. Crawling a million is an engineering problem. At that scale the things that break are memory, patience, and the assumption that your crawl will finish in one uninterrupted run. This guide covers how to plan, size, run, and export a crawl of a million URLs or more without losing your work halfway through.

Why Large Crawls Fail

Almost every failed million-URL crawl we have seen falls into one of four categories.

Memory exhaustion. Every crawler holds extracted data per URL in memory: status code, title, meta description, headings, word count, response time, the links found on the page. Individually this is tiny. Multiplied by a million it becomes the dominant cost of the crawl. If you have not measured your per-URL footprint, you have no idea whether your machine can finish, and you find out around URL 600,000 when the process gets killed.

Crashes that lose everything. An in-memory-only crawler treats a crash as total data loss. Twelve hours of crawling, one laptop that goes to sleep, and you start again from URL one. The most demoralizing failure mode at scale, and entirely avoidable.

Export out-of-memory errors. Cruel, because it happens after the crawl succeeds. You finish 1.2 million URLs, click export to CSV, and the process dies building the whole file in memory before writing it. The data was there. You just could not get it out.

Getting banned. A million requests is real traffic. Run it too aggressively and you trip rate limiting, a WAF rule, or a Cloudflare challenge, and your crawl quietly turns into a million 429s. On a client's production site you may also degrade it for actual users. Speed is not the only goal.

Planning the Crawl

The most effective optimization for a large crawl is not crawling URLs you did not need. Ten minutes of planning routinely removes 40% of a crawl's volume.

Define scope before you start

Start by asking what question the crawl answers. "Audit the whole site" is not a question. "Do all 400,000 product pages have unique titles and valid canonicals" is, and it implies a much narrower crawl. LibreCrawl gives you three levers worth setting deliberately:

• Include patterns - Restrict the crawl to URLs matching specific patterns. If you only care about product and category templates, say so up front instead of crawling every author archive.
• Exclude patterns - Cut the parts of the site that generate infinite URL space: faceted navigation, calendar widgets, sort parameters, session IDs, internal search, print views. On e-commerce sites, excluding filter parameters often halves the crawl on its own.
• Extension filters - Skip file types you do not need. If the audit is about HTML pages, do not spend requests and memory on PDFs, images, CSS, JS, and fonts.

Decide whether you actually need JavaScript rendering

This is the biggest single decision affecting crawl duration. Fetching raw HTML is one HTTP request. Rendering means launching a browser context, executing JavaScript, and waiting for the network to settle, which in practice runs 10 to 50 times slower per URL depending on how heavy the front end is. Applied to a million URLs, that is the gap between an overnight crawl and a fortnight.

So test rather than defaulting. Crawl a few hundred representative URLs both ways. If titles, meta descriptions, headings, body content, and internal links all appear in the raw HTML, you do not need rendering and should not pay for it. If the raw HTML is an empty div and a bundle, you do.

A middle path works well on large sites: run the full million-URL crawl unrendered to map structure, status codes, and metadata, then run a much smaller rendered crawl over a sample of each template. Our guide to crawling JavaScript websites covers how to run that comparison and what to look for in the diff.

Seed from sitemaps

Link discovery is breadth-first, so deep pages surface late. Sitemaps give you the URL inventory directly instead of making you traverse to it. LibreCrawl discovers them automatically, including those declared in robots.txt and nested sitemap index files, so you usually get this without configuring anything.

Two caveats. Sitemaps are frequently incomplete or stale, so sitemap URLs alone are not a full crawl. And the comparison itself is valuable: sitemap URLs the crawl never reached through links are effectively orphaned, while crawled URLs missing from the sitemap point at a broken generation process.

Memory Management

This is where large crawls are won or lost, and it is the part most people guess at. You do not have to guess.

Measure, then extrapolate

LibreCrawl's live memory panel reports three numbers while the crawl runs: the current size of the crawl data, the average KB per URL, and a projected size for one million URLs. That third number is the useful one, because it turns a question about the future into arithmetic you can do in the first ten minutes. The workflow we use on every new large site:

1. Configure scope and start the crawl with your intended settings.
2. Let it reach roughly 10,000 URLs, enough to average out template differences.
3. Read the KB per URL figure from the memory panel.
4. Multiply by your expected total. The panel already projects 1M for you, so scale that up or down.
5. Compare against the RAM you actually have available, not the RAM installed in the machine.

Typical pages come out at a handful of KB per URL of extracted data, which is comfortable: a million URLs lands in the low gigabytes, and most modern workstations handle that without complaint. But "typical" is doing real work in that sentence. Sites with enormous navigation menus, thousands of internal links per page, or aggressive extraction settings run several times higher, and that is exactly the case you want to catch at URL 10,000 rather than URL 800,000.

When the projection is too big

• Tighten scope - Reduce URL count rather than data per URL. Exclude patterns and extension filters are free performance.
• Split the crawl - One crawl per site section or template, then combine exports. Three 400,000 URL crawls are far easier to manage than one 1.2 million URL crawl, and each is independently recoverable.
• Cap with max URLs - Stop at a known size instead of at whatever point the machine gives out.
• Get more memory - Sometimes the honest answer. RAM is cheaper than a week of restarted crawls.

Sizing hardware from that math

Once you know your KB per URL, sizing stops being folklore. Take the projected size and leave real headroom: the OS, browser processes if you are rendering, the UI, and export buffers all want memory too. We plan for crawl data occupying no more than half of available RAM. In practice:

• Up to ~100,000 URLs - Any modern laptop with 8GB, rendering included if you are patient.
• Up to ~500,000 URLs - 16GB is a reasonable target for HTML-only crawling.
• 1 million URLs and beyond - 32GB gives room to work, keep the UI responsive, and export without drama. Budget more if you are rendering at this scale, because browser processes are the larger consumer.
• CPU and network - HTML-only crawls are usually network bound, so bandwidth and latency to the target matter more than core count. Rendered crawls flip that: browsers are CPU hungry and cores become the limit.
• Disk - Persistence writes continuously, so use an SSD. Spinning disks turn batch saves into a bottleneck.

Treat these as starting points. Your own measurement beats any general recommendation, including ours.

Persistence: The Safety Net

Million-URL crawls used to be nerve-wracking because they were all-or-nothing. LibreCrawl writes crawl data to the database in batches, every 50 URLs or every 30 seconds, whichever comes first, and checkpoints the pending queue alongside it. The practical consequence is that a crash costs you seconds of work, not hours. If the process dies at URL 840,000, the database already holds essentially all of it plus the queue state needed to continue, and you resume from the dashboard rather than starting over.

Batching matters here: writing every single URL individually would make the database the bottleneck on a fast crawl, so batching keeps persistence close to free while still keeping worst-case loss tiny. We wrote up the design, including how the same storage layer powers content duplication detection across the whole crawl, in the post on database persistence and duplication detection.

Two habits make the most of this. Plan long crawls as resumable rather than continuous, so stopping deliberately at the end of a working day is routine instead of a loss. And keep historical crawls: on a large site the month-over-month diff is usually more actionable than the raw snapshot.

Politeness and Speed

At a million requests you are no longer a rounding error in someone's traffic logs. These settings are partly about finishing on time and partly about not causing harm.

Concurrency

LibreCrawl defaults to 5 concurrent requests and lets you configure it. Five is deliberately conservative and almost never hurts a site. Raising it is the most direct way to speed up a large crawl, but the ceiling is set by the target server, not your machine. Raise it gradually and watch response times: if the average climbs as you add concurrency, you are the cause and should back off. Ten concurrent requests against a healthy server is fine. The same ten against shared hosting with a slow database is an accidental load test.

Delay and rate limiting

A configured delay between requests is the blunt but reliable instrument, and the arithmetic is worth doing before you commit. At 10 requests per second you cover a million URLs in a bit under 28 hours of continuous crawling. At 3 per second the same crawl takes over four days. Pick the rate that gets you the data on time and that the site can absorb, then let persistence handle the interruptions a multi-day crawl will hit.

Respect robots.txt, and watch for 429s

Leave robots.txt compliance on. Beyond the ethics, disallowed paths are usually disallowed for a reason, and they are often the same infinite URL spaces you wanted to exclude anyway. If a crawl delay is specified, honor it.

Keep the status code breakdown visible during the crawl. A rising count of 429s means you are being rate limited, and every one is a URL you did not actually collect data for. Reduce concurrency, add delay, and re-crawl the affected block at a sustainable rate. A sudden wall of 403s usually means a WAF has decided you are a bot, which often needs a user agent and a rate adjustment together.

Configuration Specifics

• Max URLs - Configurable up to 5 million. Set it slightly above your expected total rather than leaving it wide open, so a URL generation bug on the target site cannot run your crawl into the ground.
• Max depth - A safety valve against infinite spaces. Most sites keep their meaningful content within a handful of clicks of the homepage, and depth 15 discoveries are frequently parameter noise. Check the depth distribution on your sample crawl before setting this.
• Timeouts - Long enough for the site's slowest legitimate responses, short enough that a few hanging URLs do not stall the crawl. Rendered crawls need considerably more, since the timeout has to cover script execution.
• Retries - Worth having, because over a million requests you will hit transient failures. Worth keeping modest, because retrying a genuinely broken endpoint multiplies wasted requests.
• Extraction settings - Everything you extract costs memory per URL. At a million URLs, turning off extractions you will not analyze is a real saving, not a micro-optimization.

Exports at Scale

Building an entire CSV in memory before writing the first byte doubles peak memory at exactly the moment your machine is already full of crawl data. Use the streaming export endpoint instead: it writes rows out as it reads them, so the whole dataset never sits in memory at once and the download starts almost immediately. On a million-row export this is the difference between a file and a crash. The API docs cover the endpoint if you want to drive exports from a script.

• Export filtered subsets where you can. If the deliverable is "all pages with duplicate titles", export that, not all 1.2 million rows.
• Mind what opens the file. Spreadsheets cap out around a million rows and get slow long before that. A million-row CSV is a database import, not something to scroll.
• Export in sections. One file per site section keeps each file workable and makes it obvious where a finding came from.

Analysis at Scale

A million rows of crawl data is not something you read, it is something you query. LibreCrawl's tables use virtual scrolling, so only visible rows are rendered and the interface stays responsive with very large result sets. That is the fast path for exploratory work: filter to 404s, sort by response time, isolate a URL pattern, see results immediately.

For grouping, joining against other data sources, or repeatable reporting, export and analyze elsewhere: pandas, DuckDB, SQLite, BigQuery, whatever your team already uses. Crawl data joined against analytics sessions and log file hits is where the genuinely valuable large-site findings come from, and no crawler UI will do that join for you.

One note on visualization: the site structure graph caps at 500 nodes, deliberately. A force-directed graph of a million nodes is not a diagram, it is a grey rectangle, and it takes a long time to render before telling you nothing. Use it on a representative subset, a single section or the top levels of the hierarchy. For whole-site questions, distributions and aggregate numbers are the honest tool.

Worked Example: Auditing a 2 Million URL E-commerce Site

A large retailer, roughly 2 million crawlable URLs: products, categories, and a very large volume of faceted filter combinations. The brief is a technical SEO audit with two weeks to deliver.

1. Scope to what matters, first

The 2 million figure is mostly facets. Product and category templates hold the SEO value and the audit findings, so the first crawl targets those with include patterns, plus exclude patterns for filter and sort parameters, internal search, and account pages. Extension filters drop PDFs and media. In-scope volume after filtering: around 700,000 URLs. Facets are deferred rather than ignored. They are their own question (are these pages indexable, canonicalized, and blocked appropriately) answered later with a separate shallow crawl.

2. Sample crawl and extrapolate

Start the crawl and stop looking at the URL counter. At 10,000 URLs, read the KB per URL figure and the 1M projection, multiply out to 700,000, and compare against available RAM. On a 32GB workstation with a typical per-URL footprint this comfortably fits, and the decision takes two minutes. Check at the same mark whether raw HTML contains product titles, prices, and canonicals. It usually does for products and often does not for facet-driven category listings, which argues for keeping the main crawl unrendered and following up with a rendered sample of category templates. If the projection had come back too large, the fix here is cheap: tighten excludes, or split into one crawl per top-level category. Discovering the same problem at 500,000 URLs is not.

3. Set the pace, run overnight

Start conservative on concurrency and raise it only while response times stay flat. Confirm robots.txt compliance is on and keep the status breakdown visible so 429s show up immediately. Then let it run overnight: at a sustainable rate, 700,000 URLs is a single long session rather than a multi-day project.

4. Resume after the interruption

Something will interrupt it: a dropped VPN, a system update, a laptop that slept, the target's own maintenance window. In our experience this happens on most crawls above half a million URLs. With batch saves every 50 URLs or 30 seconds and a checkpointed queue, recovery is: open the dashboard, resume, lose seconds. This is the step that turns large-scale crawling from a high-stakes operation into routine work.

5. Export and analyze

Use the streaming export rather than materializing 700,000 rows in memory. Export the full dataset once for the record, then filtered subsets per finding: duplicate titles, missing canonicals, thin product descriptions, redirect chains, products in the sitemap but never linked. The full export goes into DuckDB and gets joined against server logs and Search Console data. That join is usually where the headline finding lives, because it is the only way to say which of the site's 400,000 product pages Google actually crawls. Then run the deferred facet crawl, shallow and capped.

Two weeks is comfortable for this. The parts that cost time are the planning and the analysis. The crawl itself, once scoped and sized correctly, mostly runs while you sleep.

Conclusion

Large-scale crawling is less about raw crawler speed than about three unglamorous disciplines: crawling only what you need, measuring your memory footprint early enough to act on it, and making interruption survivable. Get those right and a million URLs is a scheduling exercise rather than a gamble.

The tooling side is largely solved. Live memory projection tells you at 10,000 URLs whether the crawl will fit. Batch persistence and queue checkpoints mean a crash costs seconds. Streaming exports mean the data comes out as easily as it went in. What remains is judgment: what is in scope, whether you need rendering, and what rate gets you the data without harming the site you are auditing.

Key takeaways:

• Scope aggressively with include patterns, exclude patterns, and extension filters before you start
• Crawl 10,000 URLs, read the KB per URL figure, and extrapolate before committing to the full run
• Only enable JavaScript rendering if you have verified you need it, since it runs 10 to 50 times slower
• Rely on batch persistence and queue checkpoints so interruptions cost seconds, then resume from the dashboard
• Keep concurrency and delay at rates the target server can absorb, and back off on 429s
• Use the streaming export endpoint for large datasets, and analyze outside the UI when you need joins