Preview crawlers and JavaScript

Most link-preview crawlers fetch your HTML and parse it without running any JavaScript. So Open Graph tags added client-side — by a framework after hydration, by a tag manager, by any script — are frequently invisible to them, and the card gets built from whatever the raw HTML happened to contain.

This is the highest-impact failure in the whole subject, because it’s completely invisible in a browser. Your page looks correct in devtools, and the crawler saw something else.

Why they don’t run JavaScript

Running a JavaScript engine per fetched URL is expensive — a browser process, a render, a wait for network activity to settle, per link pasted anywhere on the platform. Parsing HTML for a handful of meta tags is cheap.

Some crawlers do render. Behaviour differs per platform and changes over time. Which means “does this one render JavaScript?” is not a question you want your previews to depend on.

The symptom

Distinctive once you know it:

• Devtools shows the tags present and correct.
• Any tool that renders the page shows a correct preview.
• Actual chat apps and social platforms show a wrong one — often your homepage’s title, some navigation text as the description, and either no image or an arbitrary one from the page.

That last part is the tell: the card looks like something guessed from the page rather than declared by it. Because that’s exactly what happened — the crawler found no tags and fell back to heuristics.

Confirming it

One command:

curl -s https://example.com/page/ | grep -i 'og:\|twitter:'

curl doesn’t run JavaScript, which makes it a good approximation of a crawler.

If the tags appear, they’re server-rendered and your problem is elsewhere. If they don’t, this is it.

Worth checking your framework’s output rather than assuming, because “server-side rendering” is sometimes partial — the initial HTML may contain the body and have the head assembled client-side, which is precisely the failure case.

Fixing it per situation

Static site generators — Astro, Eleventy, Hugo, Jekyll and similar. Tags go in the layout template and are in the built HTML by construction. Nothing to do beyond putting them there.

Server-rendered frameworks — Next, Nuxt, SvelteKit, Rails, Django. Use the framework’s head or metadata mechanism, which renders into the server response. The thing to avoid is setting tags in a client-side effect or lifecycle hook, which runs only in a browser.

Single-page applications with no server rendering. The hard case, because there’s genuinely one HTML file for every route and nothing route-specific in it. Options:

• Pre-render the shareable routes at build time, producing real HTML files with real tags. Best answer where the route list is known.
• Serve tags from the edge. A worker or middleware that intercepts requests, and for crawler requests returns HTML with the right tags. Practical, and it means maintaining a second rendering path.
• Accept generic previews. For an app whose deep links are mostly shared between authenticated users, one good site-level card may be a legitimate choice.

Tag managers. Don’t use one for Open Graph. Tag managers are client-side by nature; this is exactly the category of thing they can’t do.

Don’t serve crawlers different content

The tempting shortcut: detect the crawler’s user agent and serve it special HTML.

Two problems. Serving substantially different content to crawlers than to users is the definition of cloaking, and search engines treat it as a violation regardless of intent. And it’s fragile — user agent strings change, new platforms appear, and you’ve built a system that silently stops working.

Serving the same tags in the initial HTML for everyone is simpler and has none of these problems. If you do need edge-based tag injection for an SPA, inject the same tags a user would eventually get, not different ones.

While you’re there: check the fetch isn’t blocked

Related failures with the same symptom of an unexpectedly generic card:

robots.txt disallowing the crawler. Some preview crawlers respect it, and blocking them means no preview.

Bot protection. A WAF or DDoS-protection rule that challenges unknown user agents will block preview crawlers along with everything else. This one is common on sites that recently added protection.

Hotlink protection on images. The page previews fine, the image doesn’t load, because a referrer rule rejected a request with no referrer.

Geo-blocking. Crawlers fetch from wherever their servers are, which may not be a region you serve.

All produce “the card is wrong and my tags are right”, so they’re worth checking once the JavaScript question is settled.

The rule

Open Graph tags belong in the HTML your server sends, in the first response, with no JavaScript involved. Everything else in this area is an optimisation; this one is the difference between previews working and not.

And it’s cheap to verify — one curl, before you publish.