Crawl a React site with a plain HTTP crawler and you will often get back a page with no title, no headings, and a single empty div. The content exists, but only after JavaScript runs in a browser. This guide explains why that happens, how to tell when a site actually needs rendering, and how to crawl JavaScript-heavy websites accurately without wasting hours of crawl time.

Why Plain HTTP Crawlers See Empty Shells

A traditional crawler works the same way curl does. It sends an HTTP GET request, receives the raw HTML the server returns, parses that HTML for content and links, and moves on. This is fast and cheap. A single machine can fetch and parse hundreds of pages per minute this way, which is why every crawler defaults to it.

The problem is that on a client-side rendered site, the HTML the server returns is not the page. It is a loading stub. A typical client-side rendered document looks like this:

<!DOCTYPE html>
<html>
<head>
  <title>My App</title>
  <script src="/static/js/main.8f3a2c.js"></script>
</head>
<body>
  <div id="root"></div>
</body>
</html>

Everything the user sees, the product descriptions, the navigation, the internal links, the structured data, gets built by that JavaScript bundle after the browser downloads and executes it. An HTTP crawler never executes anything. It parses the markup above, finds zero content and zero links, and reports a technically successful crawl of a page that appears to be empty.

This produces a specific and recognizable failure pattern in crawl reports: hundreds of pages returning 200 status codes with missing titles, missing meta descriptions, zero word counts, and no outgoing internal links. If you have ever crawled a site and wondered why the crawler stopped after the homepage, this is usually why. The homepage HTML contained no anchor tags for the crawler to follow, so there was nothing to queue.

Rendering Strategies and What They Mean for Crawling

Not every JavaScript framework site has this problem. What matters is not which framework the site uses but where the HTML gets generated. There are four common approaches, and each one changes how you should crawl.

Client-Side Rendering (CSR)

The server sends a near-empty shell and the browser builds the entire page. This is the default behavior of a create-react-app build, a plain Vue CLI app, or a standard Angular application. CSR sites are the worst case for crawling: without a rendering engine you see essentially nothing. If the site you are auditing is pure CSR, you need JavaScript rendering, full stop.

Server-Side Rendering (SSR)

The server runs the JavaScript itself and sends fully-formed HTML on every request. The browser receives complete content immediately. From a crawler's perspective, an SSR site behaves like a traditional website. An HTTP crawl captures titles, content, and links correctly. Frameworks like Next.js and Nuxt do this out of the box when configured for SSR.

Static Site Generation (SSG)

The HTML for every page is generated once at build time and served as static files. Gatsby, Astro, and Next.js in static export mode work this way. Like SSR, static generation gives crawlers complete HTML without any rendering. These are the easiest sites to crawl, often faster than traditional CMS sites because there is no server processing per request.

Hydration

Most SSR and SSG sites also hydrate: the server sends complete HTML, then JavaScript loads and attaches interactivity on top of it. Hydration is mostly good news for crawling because the initial HTML is complete. The catch is that some sites hydrate incorrectly and the client-side code replaces or modifies content after load. If the raw HTML and the rendered DOM disagree, you want to know, because Google will index one and users will see the other. Crawling the site both ways (once without rendering, once with) is the fastest way to catch these mismatches.

How Googlebot Handles JavaScript

Google can render JavaScript, but the way it does so matters for how you interpret your own crawl data.

Googlebot processes pages in two waves. First, the crawler fetches the raw HTML and extracts whatever links and content it finds there. The page then enters a render queue, where a headless Chromium instance (the Web Rendering Service, kept evergreen with recent Chrome versions) eventually executes the JavaScript and re-processes the rendered result. The key word is eventually. Rendering is expensive at Google's scale too, and the render queue is processed as resources allow. For most pages the delay is short, but it can stretch longer for large sites or sites with limited crawl budget.

The practical consequences for SEO:

  • Links only present after rendering are discovered late. If your internal linking exists only in the rendered DOM, new pages take longer to be found and crawled.
  • Content only present after rendering can be indexed late or inconsistently. Time-sensitive content that depends on client-side rendering is at a real disadvantage.
  • Anything that fails during rendering fails silently. If your JavaScript errors out for Googlebot (blocked resources, timeouts, browser API assumptions), Google indexes the empty shell, and nothing in your analytics tells you.
  • Other search engines and most social crawlers render less reliably or not at all. Bing renders some JavaScript. Most link preview bots for social platforms and messaging apps read only the raw HTML.

This is why crawling your site with and without rendering is not academic. The unrendered crawl approximates what the first wave and non-Google bots see. The rendered crawl approximates what Google eventually indexes. Both views matter.

How to Detect Whether a Site Needs Rendering

Before configuring anything, spend five minutes figuring out whether the site actually needs rendering. Many sites that look like heavy JavaScript apps turn out to serve complete HTML, and rendering them would waste hours of crawl time for identical results.

The View-Source Test

Open the page in your browser, then view the page source (Ctrl+U in most browsers). This shows the raw HTML the server sent, before any JavaScript ran. Now search the source for a sentence of visible body content, not the title, actual paragraph text. If you can find it, the server is sending real content. If the source is a few dozen lines of script tags wrapped around an empty div, the site is client-side rendered.

Be careful to use view-source and not the browser's element inspector. The inspector shows the rendered DOM, which on a CSR site will always look complete because your browser already executed the JavaScript. Comparing the two views, source versus inspector, is itself the diagnostic: a large difference between them means rendering changes the page significantly.

The Disable-JavaScript Test

Disable JavaScript in your browser (in Chrome: DevTools, Ctrl+Shift+P, "Disable JavaScript") and reload the page. What remains is roughly what a non-rendering crawler sees. If the page still shows its content and navigation, HTTP crawling will work. If you get a blank page or a "please enable JavaScript" message, you need rendering.

The Curl Test

For a quick check from the command line:

curl -s https://example.com/some-page | grep -i "<h1"

If the H1 comes back with real text in it, the server is sending content. You can extend this to check for title tags, meta descriptions, or any distinctive body copy. Curl is also useful for spotting a subtler issue: some sites serve pre-rendered HTML to known bot user agents and a JavaScript shell to everyone else. Run the same curl with a Googlebot user agent string and compare the responses. If they differ substantially, the site is doing dynamic rendering, and you should crawl it both ways to see both versions.

Run these tests on several page types, not just the homepage. It is common for a site to server-render its marketing pages while the product catalog or search results are pure CSR. The homepage passing the test tells you nothing about the templates deeper in the site.

Framework Specifics: What to Expect

You can usually predict crawl behavior from the framework, though the detection tests above are always the final word.

React

A plain React single-page app (create-react-app or a similar Vite build) is client-side rendered and needs a rendering crawler. React itself has no opinion about SEO; everything depends on how it is deployed. React with Next.js in SSR or SSG mode serves complete HTML and crawls fine without rendering.

Vue

Same story. A standard Vue SPA mounts onto an empty element and needs rendering. Vue with Nuxt in server or static mode delivers full HTML. Vue sites are also common in a hybrid pattern where a traditional server-rendered site sprinkles Vue components into otherwise complete pages; those crawl fine without rendering because the core content is in the HTML.

Angular

Angular applications are client-side rendered by default and are among the most consistently empty pages you will see in raw HTML. Angular Universal adds server-side rendering, but in our experience it is deployed less often than Next or Nuxt are in their ecosystems. Assume an Angular site needs rendering until the view-source test proves otherwise.

Next.js and Nuxt

These SSR frameworks exist largely because of the SEO problems this guide covers, and they mostly solve them. A well-configured Next or Nuxt site serves complete HTML and can be crawled with plain HTTP at full speed. Two caveats. First, both frameworks allow per-page rendering modes, so a site can mix static pages with client-only pages; test more than one template. Second, hydration mismatches (server HTML that the client then rewrites) show up on these frameworks specifically, so a comparison crawl is still worth doing on important templates.

Common JavaScript SEO Failures

These are the issues we see most often when crawling JavaScript sites. Every one of them is invisible in a browser, because the browser runs the JavaScript, and every one of them hurts crawling and indexing.

Client-Side Routing Without Real URLs

Single-page apps intercept navigation and swap views without full page loads. Done correctly with the History API, every view has a unique, server-resolvable URL: requesting /products/blue-widget directly returns that page. Done incorrectly, the server only knows about the root URL, and requesting any deep URL directly returns a 404 or redirects to the homepage. Crawlers (and users sharing links) request URLs directly, so every route needs to resolve on its own. Test by pasting a deep URL into a fresh incognito window.

Hash-Based Routing

URLs like example.com/#/products/blue-widget use the URL fragment for routing. Everything after the # is never sent to the server and is ignored by search engines for indexing purposes, so to Google the entire application is one URL. Hash routing was a workaround from before the History API existed. If you find it on a site in 2025, migrating to path-based routing is one of the highest-impact recommendations you can make.

Content Behind Infinite Scroll and Lazy Loading

Content that loads only when the user scrolls does not exist in the DOM at render time. Googlebot does not scroll. It renders the page in a tall viewport, which triggers some lazy content, but items that require actual scroll events or repeated "load more" interactions never enter the DOM and never get indexed. The fix is paginated URLs alongside the infinite scroll (page 2 exists at ?page=2 with real links), so both users and crawlers have a path to everything. When auditing, compare the number of items in the rendered DOM against the number the category claims to have.

Links That Are Not Links

Crawlers discover URLs by extracting anchor tags with href attributes. Elements like <div onclick="navigate('/products')"> or <a href="#" onclick="..."> navigate fine for users but contain no crawlable URL. Google is explicit that it only follows links in anchor tags with resolvable hrefs. A site whose main navigation is built from click handlers can have thousands of pages that no crawler will ever discover. In a rendered crawl, these pages show up as orphans or do not show up at all; that discrepancy is your diagnostic.

Metadata Set Only Client-Side

Titles, meta descriptions, canonical tags, and robots directives injected by JavaScript are only seen by crawlers that render. Everything else, including most social preview bots, sees whatever placeholder the server sent, which is why sharing a page from a CSR site so often produces a preview titled "My App". Worse, a robots meta tag or canonical that differs between raw HTML and rendered DOM creates genuinely ambiguous signals for Google. Critical metadata belongs in the server response.

Crawling JavaScript Sites with LibreCrawl

LibreCrawl ships with full JavaScript rendering built on Playwright, the same browser automation framework used for modern end-to-end testing. Nothing about it is paywalled or limited; rendering is part of the free, open-source package.

Enabling and Configuring Rendering

Open Settings and enable JavaScript rendering. The options that matter:

  • Browser engine - Choose chromium, firefox, or webkit. Chromium is the sensible default since it is closest to what Googlebot uses. All engines run headless.
  • Wait time - How long LibreCrawl waits after the domcontentloaded event before capturing the page. The default is 3 seconds, which covers most apps. Slow apps that fetch data from APIs after load may need 5-7 seconds. If titles or content are missing from rendered results, raise this first.
  • Page timeout - Maximum time for a page to load before giving up, 30 seconds by default. Raise it for genuinely slow sites, but investigate pages that need more than 30 seconds; users will not wait that long either.
  • Viewport - Defaults to 1920x1080 and is configurable. Viewport size affects lazy loading behavior, so a taller viewport can pull more content into the DOM before capture.
  • Concurrency - LibreCrawl runs up to 3 concurrent browser pages by default. More pages mean faster crawls and more memory; each open browser page holds a full DOM in RAM.

Every row in your crawl results carries a javascript_rendered flag, so you always know which pages were captured through the browser and which came from plain HTTP. When you export results or query the API, that flag lets you separate the two populations cleanly instead of guessing.

The Two-Pass Workflow We Actually Use

Rendering is dramatically slower than HTTP crawling. Fetching and parsing raw HTML takes tens of milliseconds per page; rendering means launching a page in a real browser engine, executing scripts, waiting out the configured delay, and capturing the DOM, which multiplies per-page cost by 50-100x. Rendering an entire large site by default is how a two-hour audit becomes a two-day one. So do not start there.

  1. First pass: crawl without rendering. This is fast and maps the site as the server presents it: raw HTML titles, content, links, status codes.
  2. Review the results for rendering symptoms. Pages with 200 status but empty titles, near-zero word counts, or no outgoing links are your candidates. If the whole crawl died after a handful of URLs, the site is likely CSR throughout.
  3. Second pass: re-crawl with rendering enabled. If only a section of the site needs it (a catalog, a search area), limit the crawl scope to that section rather than re-rendering pages the first pass already captured correctly.
  4. Compare the two datasets. Differences between the raw and rendered crawls are themselves findings: content that only exists post-render, links only discoverable in the DOM, metadata that changes after hydration. These map directly to items in our technical SEO audit checklist.

For very large JavaScript sites, combine this workflow with sampling: render a representative slice of each template type rather than every URL. A rendered crawl of 500 pages across ten templates tells you almost everything a rendered crawl of 500,000 pages would, at a thousandth of the cost. Our guide to large-scale website crawling covers memory management and scoping strategies in depth.

Troubleshooting Rendered Crawls

When a rendered crawl misbehaves, the cause is usually one of a few things.

  • Blank titles and empty content with 200 status codes. If this appears in a non-rendered crawl, it is the classic sign that the site needs rendering; enable it and re-crawl. If it appears in a rendered crawl, the page probably had not finished building when LibreCrawl captured it. Raise the wait time from 3 seconds to 5-7 and try again.
  • Frequent timeouts. Pages hitting the 30 second timeout either are genuinely slow or are waiting on a resource that never resolves (a blocked third-party script is a common culprit). Raise the page timeout, but also spot-check the slow URLs manually; chronic timeouts are a site performance finding, not just a crawler nuisance.
  • Memory climbing during long crawls. Each concurrent browser page holds a full rendered DOM, so memory scales with concurrency. If your machine is under pressure, drop from 3 concurrent pages to 2 or 1 and let the crawl take longer instead of failing.
  • Rendered results differ from what you see in a browser. Check the viewport setting first, since lazy loading is viewport-dependent. Also confirm the site is not serving different content to headless traffic; the curl user-agent comparison from earlier applies here too.
  • Crawl works with chromium but the site claims a bug in another browser. Switch the engine to firefox or webkit and re-crawl the affected section. Being able to compare engines is occasionally the fastest way to confirm a browser-specific rendering problem.

Conclusion

JavaScript crawling is not complicated once you separate two questions: does this site put its content in the server response, and if not, what does it cost to render it? The detection tests take five minutes and prevent both failure modes, crawling a CSR site blind and getting garbage, or rendering an SSR site pointlessly and wasting a day.

Render when you need to, not by default. Crawl once over plain HTTP to map what the server sends, use that pass to find the sections that need a browser, then render those sections with realistic wait times. The differences between your two crawls are often the most valuable findings in the entire audit, because they are exactly the gaps between what search engines are promised and what they eventually get.

Key takeaways:

  • HTTP crawlers see only server-sent HTML; on CSR sites that is an empty shell
  • SSR and static sites (Next, Nuxt, Gatsby) usually crawl fine without rendering
  • Test with view-source, disabled JavaScript, and curl before enabling rendering
  • Watch for click-handler links, hash routing, lazy content, and client-only metadata
  • Crawl without JS first, then re-render only what needs it; rendering costs 50-100x more per page
  • In LibreCrawl, tune wait time (3s default), timeout (30s), and concurrency (3 pages) to the site

Crawl JavaScript Sites Free with LibreCrawl

LibreCrawl includes full Playwright-based JavaScript rendering with chromium, firefox, and webkit support. No paid tiers, no URL limits, no locked features.

Download LibreCrawl