In traditional Single Page Applications (SPAs) built with React, the client browser initially receives a bare-bones HTML document: an empty <div id="root"></div> container and a script reference to the JavaScript bundle. For end users, the UI renders entirely on the client (Client-Side Rendering, CSR). However, for search engines and social media crawlers, this architecture creates a fundamental indexing hurdle.
Googlebot utilizes a specialized subsystem called the Web Rendering Service (WRS). Indexing CSR applications in Google occurs in two distinct phases:
Furthermore, search bots like Yandex, Baidu, and Naver, alongside social media preview scrapers (OpenGraph crawlers for Telegram, Twitter/X, LinkedIn, Discord, and WhatsApp), feature severely constrained JavaScript runtimes or do not execute scripts at all. If fully rendered HTML is not served in the initial HTTP response, link previews fail to generate, and pages remain absent from the search index.
The Pre-rendering (Dynamic Rendering) architecture addresses this challenge without requiring a ground-up rewrite of the SPA codebase to full SSR frameworks like Next.js or Remix. A Reverse Proxy intercepts incoming requests: regular human users receive the standard CSR bundle, while verified search bot requests are transparently redirected to a headless browser pool (Puppeteer, Playwright, or Prerender.io) that renders and caches complete HTML snapshots of the requested routes.
At the core of pre-rendering is network-level traffic segregation at the Reverse Proxy, CDN, or Edge Worker layer. The edge evaluates the incoming User-Agent header and internal routing heuristics, routing traffic into one of two operational pipelines.
+-----------------------------------------------------------------------+
| Incoming HTTP Request |
+-----------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------+
| Reverse Proxy / CDN (Cloudflare / Nginx) |
| User-Agent Analysis & Pre-cache Evaluation |
+-----------------------------------------------------------------------+
| |
[Standard User] [Search Crawler / Bot]
| |
v v
+---------------------------+ +---------------------------------+
| Static Storage (S3 / CDN) | | HTML Snapshot Cache (Redis/S3) |
| Serves index.html + JS | +---------------------------------+
+---------------------------+ |
| +---------+---------+
v | |
+---------------------------+ [Cache Hit] [Cache Miss]
| Client Browser (CSR) | | |
| Hydration & JS Execution | v v
+---------------------------+ +----------------+ +----------------+
| Serve Cached | | Headless Pool |
| HTML (TTFB<80ms| | (Puppeteer) |
+----------------+ | DOM Render |
+----------------+
|
+---------+
v
[Write to Redis/S3]
|
v
[Stream HTML to Bot]Key architectural concepts:
The edge router examines the incoming User-Agent header against a verified crawler regex pattern. The edge router must append Vary: User-Agent so that downstream shared caches never serve pre-rendered snapshots to interactive users.
interface Env {
PRERENDER_SERVICE_URL: string;
PRERENDER_TOKEN: string;
CACHE_KV: KVNamespace;
}
const BOT_USER_AGENTS: readonly RegExp[] = [
/googlebot/i,
/yandex/i,
/bingbot/i,
/baiduspider/i,
/twitterbot/i,
/facebookexternalhit/i,
/linkedinbot/i,
/slackbot/i,
/telegrambot/i,
/whatsapp/i,
/chatgpt-user/i,
/gptbot/i,
/perplexitybot/i,
];
const IGNORED_EXTENSIONS: readonly string[] = [
'.js', '.css', '.xml', '.less', '.png', '.jpg', '.jpeg',
'.gif', '.pdf', '.svg', '.ico', '.woff', '.woff2', '.ttf',
];
function isBot(userAgent: string | null): boolean {
if (!userAgent) return false;
return BOT_USER_AGENTS.some((pattern) => pattern.test(userAgent));
}
function isStaticAsset(url: URL): boolean {
const pathname = url.pathname.toLowerCase();
return IGNORED_EXTENSIONS.some((ext) => pathname.endsWith(ext));
}
export default {
async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
const url = new URL(request.url);
const userAgent = request.headers.get('User-Agent');
// Bypass pre-rendering for static assets and standard human traffic
if (isStaticAsset(url) || !isBot(userAgent)) {
const response = await fetch(request);
const newHeaders = new Headers(response.headers);
newHeaders.append('Vary', 'User-Agent');
return new Response(response.body, {
status: response.status,
headers: newHeaders,
});
}
// Check for cached HTML snapshot in Edge KV
const cacheKey = `snapshot:${url.origin}${url.pathname}${url.search}`;
const cachedHtml = await env.CACHE_KV.get(cacheKey);
if (cachedHtml) {
return new Response(cachedHtml, {
status: 200,
headers: {
'Content-Type': 'text/html; charset=utf-8',
'X-Prerender-Cache': 'HIT',
'Vary': 'User-Agent',
'Cache-Control': 'public, max-age=3600, s-maxage=86400, stale-while-revalidate=604800',
},
});
}
// Route miss to Headless Pre-rendering cluster
const prerenderUrl = `${env.PRERENDER_SERVICE_URL}?url=${encodeURIComponent(url.toString())}`;
const prerenderResponse = await fetch(prerenderUrl, {
headers: {
'X-Prerender-Token': env.PRERENDER_TOKEN,
'User-Agent': userAgent || 'Prerender-Forwarder',
},
});
if (!prerenderResponse.ok) {
// Fallback to origin SPA server if prerender cluster fails
return fetch(request);
}
const html = await prerenderResponse.text();
// Asynchronously store snapshot in Edge KV
ctx.waitUntil(
env.CACHE_KV.put(cacheKey, html, {
expirationTtl: 86400, // 24 hours
})
);
return new Response(html, {
status: prerenderResponse.status,
headers: {
'Content-Type': 'text/html; charset=utf-8',
'X-Prerender-Cache': 'MISS',
'Vary': 'User-Agent',
'Cache-Control': 'public, max-age=3600, s-maxage=86400',
},
});
},
};To sustain TTFB < 300 ms, the headless browser process must be pre-warmed, and unneeded network requests (heavy images, external tracking scripts, custom font downloads) must be intercepted and aborted via the Chrome DevTools Protocol.
import puppeteer, { Browser, Page } from 'puppeteer';
import { createClient, RedisClientType } from 'redis';
interface RenderOptions {
url: string;
timeoutMs?: number;
waitForSelector?: string;
}
export class PrerenderCluster {
private browser: Browser | null = null;
private redis: RedisClientType;
private readonly blockedResourceTypes = new Set<string>([
'image',
'media',
'font',
'websocket',
]);
private readonly blockedDomains: readonly string[] = [
'google-analytics.com',
'googletagmanager.com',
'mc.yandex.ru',
'facebook.net',
'hotjar.com',
];
constructor(redisUrl: string) {
this.redis = createClient({ url: redisUrl });
}
public async initialize(): Promise<void> {
await this.redis.connect();
this.browser = await puppeteer.launch({
headless: true,
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-dev-shm-usage',
'--disable-accelerated-2d-canvas',
'--disable-gpu',
'--single-process',
'--no-zygote',
],
});
}
public async renderPage(options: RenderOptions): Promise<string> {
if (!this.browser) {
throw new Error('PrerenderCluster is not initialized');
}
const { url, timeoutMs = 10000 } = options;
const page: Page = await this.browser.newPage();
try {
// Network optimization: abort non-essential assets
await page.setRequestInterception(true);
page.on('request', (req) => {
const resourceType = req.resourceType();
const reqUrl = req.url().toLowerCase();
const isDomainBlocked = this.blockedDomains.some((d) => reqUrl.includes(d));
if (this.blockedResourceTypes.has(resourceType) || isDomainBlocked) {
req.abort();
} else {
req.continue();
}
});
// Emulate Googlebot desktop client
await page.setUserAgent(
'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
);
// Navigate and wait for network activity to subside
await page.goto(url, {
waitUntil: 'networkidle0',
timeout: timeoutMs,
});
// Await custom React readiness flag or DOM container
await page.waitForFunction(
() => Boolean(window.__PRERENDER_READY__) || document.querySelector('#root > *') !== null,
{ timeout: timeoutMs }
);
// Extract fully serialized DOM document
const html = await page.content();
return html;
} finally {
await page.close();
}
}
public async shutdown(): Promise<void> {
if (this.browser) {
await this.browser.close();
}
await this.redis.disconnect();
}
}
declare global {
interface Window {
__PRERENDER_READY__?: boolean;
}
}To prevent Chromium from capturing snapshots while skeleton loaders are active, the React application must dispatch a global signal window.__PRERENDER_READY__ = true strictly after all critical asynchronous API requests resolve and metadata tags mount into the DOM.
import { useEffect } from 'react';
interface PrerenderOptions {
isDataLoaded: boolean;
hasErrors?: boolean;
}
export function usePrerenderSignal({ isDataLoaded, hasErrors = false }: PrerenderOptions): void {
useEffect(() => {
if (typeof window === 'undefined') return;
if (isDataLoaded || hasErrors) {
// Allow React to finalize the commit phase and DOM paint
const rafId = requestAnimationFrame(() => {
window.__PRERENDER_READY__ = true;
});
return () => {
cancelAnimationFrame(rafId);
window.__PRERENDER_READY__ = false;
};
}
window.__PRERENDER_READY__ = false;
}, [isDataLoaded, hasErrors]);
}If the Reverse Proxy fails to emit Vary: User-Agent, intermediate CDN caches (Cloudflare, Fastly, Akamai) will cache the static HTML snapshot generated for a crawler and return it to subsequent human users. The interactive user receives a dead static HTML file with broken client hydration or missing interactive event listeners.
Solution: Ensure all edge routing rules enforce Vary: User-Agent on every HTTP response across both bot and human pipelines.
The headless browser finishes execution on the load or networkidle2 event while the page still renders an animated skeleton loader, because background fetch() requests are still in flight. The search bot indexes placeholder text like "Loading catalog..." instead of actual product listings.
Solution: Implement two-factor readiness: wait for content container presence combined with window.__PRERENDER_READY__ evaluated via page.waitForFunction().
Executing puppeteer.launch() per incoming request spawns a new OS-level process. Under a load of 50 requests/sec, the server exhausts its RAM, and orphaned child processes lock CPU cores.
Solution: Maintain a fixed warm pool of browser instances using pooling managers (generic-pool), isolate renders inside tabs (browser.newPage()) with guaranteed cleanup in finally blocks, and cycle browser instances every 500 render operations.
SPAs by default serve index.html with status 200 for all paths. When Chromium renders a 404 page inside React, the crawler still receives an HTTP 200 OK header with "Page Not Found" text — creating a disastrous Soft 404 that wastes crawl budget.
Solution: Inject <meta name="prerender-status-code" content="404"> inside React's NotFound view and configure your pre-rendering worker to parse this tag and emit the proper HTTP status code.
Reverse Proxy matches User-Agent against crawler database signatures. On match, the pre-render pipeline engages.
Checks for fresh snapshot matching the URL hash. On Cache Hit, serves pre-rendered HTML in under 60 ms.
On Cache Miss, dispatches URL to a warm worker in the Chromium pool, aborting media and tracking requests.
Page loads, React renders components, resolves promises, and asserts window.__PRERENDER_READY__ = true.
Extracts document outerHTML, stores it in Redis with configured TTL, and streams response to crawler.