Database Persistence and Content Duplication Detection in LibreCrawl

For a long time, LibreCrawl had one embarrassing weakness: your crawl data lived in memory, and only in memory. A crashed server, a closed terminal, or an accidental Ctrl+C meant hours of crawling gone. That era is over. LibreCrawl now persists every crawl to a SQLite database automatically, recovers from crashes mid-crawl, and keeps your full crawl history one click away. And while we were rebuilding the storage layer, we added something SEO tools usually reserve for paid tiers: content duplication detection across your entire site.

Why We Built Persistence

If you have ever crawled a large site, you know the specific anxiety of watching a URL counter climb past 40,000 and realizing that everything depends on nothing going wrong. A server restart, an out-of-memory kill, a laptop going to sleep at the wrong moment, and the crawl is simply gone. You start over from URL zero.

We heard this from users constantly, and we felt it ourselves. Our own large-scale crawling guide spends a lot of words on keeping crawls alive precisely because losing one used to be so painful. Paid crawlers like Sitebulb and DeepCrawl solved this years ago with project databases and cloud storage, and credit where it's due, their persistence stories have always been solid. But you shouldn't need a subscription to keep your own crawl data safe on your own machine.

So we picked the most boring, reliable option available: SQLite. No database server to install, no configuration, no connection strings. A single file on disk that survives anything short of the disk itself failing. It is the same choice that browsers, phones, and half the software you use daily already made, and for good reason.

How Persistence Works

Batch Saves During the Crawl

While a crawl runs, LibreCrawl writes results to SQLite in batches: every 50 URLs or every 30 seconds, whichever comes first. Batching matters here. Writing every single URL as it arrives would hammer the disk and slow the crawl down, while saving only at the end would defeat the entire point. The 50 URL / 30 second cadence means that even in the worst case, a crash costs you at most a few seconds of progress, not the whole run.

This happens automatically. There is no save button, no project setup step, no "remember to export before you close" ritual. Start a crawl and it is being persisted, full stop.

What gets saved is the full picture, not a summary. The URLs themselves with their crawl data, the link graph between them, and the SEO issues LibreCrawl found along the way all land in the database. That completeness is what makes everything later in this post possible: you cannot resume a crawl, reload one, or re-export one from a dataset that only kept the highlights.

Queue Checkpoints

Saving crawl results is only half the recovery problem. To resume a crawl, LibreCrawl also needs to know where it was: which URLs were still waiting in the queue and which had already been visited. Alongside the batch saves, LibreCrawl checkpoints exactly that, writing the pending queue and the visited set to the database.

This is the piece most people don't think about until they need it. Without the queue checkpoint, "resuming" a crawl would really mean re-crawling from the seed URL and re-discovering everything, wasting time and hammering the target server with requests it already answered. With it, a resumed crawl picks up the queue exactly as it stood and continues as if nothing happened.

Crash Recovery and Graceful Shutdown

When Things Go Wrong

Here is the scenario persistence was built for. You are 47,000 URLs into a crawl and the server process dies. Maybe the machine rebooted, maybe the OOM killer got hungry, maybe you tripped over the power cable. On restart, LibreCrawl notices the interrupted crawl and marks it as failed. That crawl then shows up in your dashboard with a resume option, and resuming continues from the checkpoint: same queue, same visited set, same collected data. You lose seconds, not hours.

When You Stop On Purpose

Not every interruption is a disaster. Sometimes you just need the machine back, or you want to continue a long crawl tomorrow. When you shut LibreCrawl down gracefully with Ctrl+C, it saves all active crawls before exiting and marks them as paused. Paused crawls sit in the dashboard waiting, and you resume them whenever you like. Overnight crawls that pick up where they left off in the morning are now a normal workflow instead of a gamble.

Crawl Statuses

With persistence in place, every crawl now carries an explicit status so you always know what state it is in:

• Running - actively crawling right now
• Paused - stopped gracefully, ready to resume
• Completed - finished normally
• Failed - interrupted by a crash, resumable from its checkpoint
• Stopped - manually stopped by you
• Archived - kept for reference but moved out of the active list

The distinction between failed, paused, and stopped sounds pedantic until you have a dashboard with a dozen crawls in it. At a glance you can tell which crawls need attention (failed), which are waiting on you (paused), and which are simply done. Archived keeps the list usable over the long run: crawls you want to hold onto for reference move out of the way without being deleted.

The Dashboard: Your Crawl History

Persistence would be pointless without a way to actually use the stored data, so LibreCrawl now ships with a dashboard at /dashboard that lists every crawl you have run. For each crawl you get four actions:

• Load - open a past crawl in the interface, restoring its URLs, links, and issues without sending a single new request
• Resume - continue a paused or failed crawl from its checkpoint
• Archive - tuck finished crawls away without deleting them
• Delete - remove a crawl and its data for good

Loading is the quiet workhorse here. Before persistence, answering a question about last month's crawl meant either digging through an old export or re-crawling the site. Now you load the crawl and the full dataset is back in the UI: every URL, every link, every issue, exactly as it was. If a client asks in March about something you crawled in January, the answer is one click away instead of a two-hour re-crawl.

History also changes how you verify fixes. Crawl a site, ship your changes, crawl again, and load the earlier crawl whenever you need to check what an issue looked like before. The before-and-after comparison that used to require carefully saved exports and a spreadsheet now just requires two entries in the dashboard.

Exports work on loaded crawls too. Load a historical crawl and export it to CSV, JSON, or XML just as you would a live one. For very large crawls, LibreCrawl uses a streaming export path that writes the file incrementally instead of building the entire export in memory, so exporting a huge crawl no longer risks taking the server down with it.

Content Duplication Detection

The Problem With Near-Duplicates

Exact duplicate detection is easy, and most crawlers do it: hash the content, compare hashes, done. But the duplicates that actually hurt sites in practice are rarely exact. They are the printer-friendly version of an article, the product page reachable under three different tracking parameters, the location pages that are 90 percent shared boilerplate with a city name swapped in. Byte-for-byte comparison sees all of these as unique pages. Search engines do not, and they quietly split your ranking signals across the variants.

The cost of this is real, even if it rarely shows up as a dramatic penalty. When five URLs serve essentially the same content, search engines pick one to index and rank, and it is not always the one you would have chosen. Internal links, external links, and user signals get spread across the variants instead of concentrating on a single strong page. Crawl budget gets spent re-fetching the same content under different addresses. None of it is catastrophic on its own; all of it is drag.

This is why duplicate content checks sit near the top of our technical SEO audit checklist, and why finding near-duplicates by hand is so miserable: on a site of any size, you would need to compare every page against every other page.

How LibreCrawl Detects Duplicates

So we automated it. At the end of each crawl, LibreCrawl runs a duplication analysis that compares page content across the entire site. Any pair of pages whose similarity exceeds the threshold gets flagged as a near-duplicate and reported as an SEO issue alongside everything else in the Issues tab.

The default threshold is 0.85, meaning pages that are 85 percent similar or more get flagged. In our testing that default catches real duplication problems without drowning you in false positives from pages that merely share a template. It is fully configurable, though. Auditing a site with heavy shared boilerplate? Raise the threshold to 0.9 or higher so only serious overlap surfaces. Hunting aggressively for thin variants? Drop it and see what appears.

What It Catches

The detector is built for exactly the near-duplicate patterns that slip past exact matching:

• Boilerplate-heavy pages - templated pages where the unique content is a thin layer over a large shared skeleton
• Printer views - print-friendly URLs serving the same article as the canonical page
• Parameter variants - the same page reachable at multiple URLs through sorting, filtering, or tracking parameters

Once flagged, the fixes are usually straightforward: canonical tags for parameter variants, noindex for printer views, and a content rewrite or consolidation for boilerplate-heavy pages. The hard part was always finding them. That part is now automatic.

Because the analysis runs against persisted crawl data, it fits naturally into the rest of the release: run a crawl today, load it next month, and the duplication findings are still there, ready to compare against a fresh crawl after your fixes ship.

What This Costs You

Nothing. As with everything in LibreCrawl, both features are free and MIT licensed. There is no "projects" paywall, no crawl history limited to premium plans, no per-seat pricing for the dashboard. Crawl history and duplicate detection are table stakes for a serious crawler in 2025, and we think they should be treated that way. If you are weighing LibreCrawl against commercial options, our Screaming Frog comparison covers where each tool earns its keep.

Conclusion

Database persistence is one of those features nobody gets excited about until the first time it saves them. The first crash that costs you seconds instead of hours, the first client question you answer by loading a two-month-old crawl instead of running a new one, and it quietly becomes something you refuse to work without. Duplication detection is the same kind of feature: invisible when things are clean, invaluable when a site is silently competing with itself.

Update to the latest version, run a crawl, and both features are simply on. No setup, no configuration, no migration step.

Key takeaways:

• Crawls persist to SQLite automatically, in batches of 50 URLs or every 30 seconds
• Queue checkpoints make crashed crawls resumable from exactly where they stopped
• Ctrl+C now pauses crawls gracefully instead of destroying them
• The /dashboard page lets you load, resume, archive, and delete past crawls, and exports work on loaded history
• Duplication detection flags near-duplicate pages above a configurable 0.85 similarity threshold as SEO issues