LinkFork

What a preview crawler actually reads

A link preview is built from a plain HTTP GET. Something fetches your URL with its own user agent, no cookies and no JavaScript engine, reads some number of bytes off the front of the response, parses the <head> for a dozen or so meta tags, and stops. Then it makes a second, separate request for whatever og:image pointed at. That is the whole pipeline, and every card you have ever seen is the output of it.

Knowing the shape of that request explains most preview failures better than any list of tags does, because the failures are nearly all about something not surviving one of those steps.

The request itself

It is an ordinary GET, and it is deliberately cheap:

  • Its own user agent. Each platform sends a documented string identifying its preview fetcher.
  • No cookies, no session, no referrer. It has never visited your site and is not logged in.
  • A short timeout. Seconds, not minutes. A slow origin simply produces no card.
  • A redirect limit. It will follow a hop or two; a long chain gets abandoned.
  • Accept: text/html. A response that isn’t HTML is usually discarded outright.

Anything your page needs in order to render correctly — a session, a consent choice, a geo lookup, a client-side render — is unavailable to it by construction.

It may not read the whole file

This is the part people are most often surprised by. A preview fetcher has no reason to download a 500 KB HTML document to extract fifteen meta tags, so many of them cap how much they read and parse whatever arrived. The cap is generous, but it is a cap.

The practical rule that falls out: put your Open Graph tags near the top of the <head>, above inline CSS, above inline JSON blobs, above anything large. A page that opens with 200 KB of inlined critical CSS and then declares og:title has arranged for a coin flip.

Same reason <meta charset> belongs in the first few hundred bytes. Declared late, the parser has already guessed an encoding, and the guess is what ends up in the card — which is where cards with ’ in the title come from.

What it looks for, in order

Parsers differ, but the precedence is broadly:

  1. og:title, og:description, og:image, og:url, og:type, og:site_name.
  2. twitter: tags, on the platforms that read them, as an override or a fallback.
  3. The plain <title> element and <meta name="description">.
  4. Heuristics: the first sizeable image in the body, some early text, the domain name.

Each level down is a worse card. When you see a preview showing navigation text as its description, you are looking at level four, and the usual cause is that the tags were never in the HTML the fetcher received — which on a JavaScript-rendered site is the default outcome rather than an edge case.

Then a second request, for the image

The image URL is resolved against the page — which is one of several good reasons to make og:image absolute — and fetched separately. Separate timeout, separate content-type check, separate opportunity to be blocked by a hotlink rule that the HTML fetch never triggered.

That second request is also where dimensions come from. An image is rarely just its URL anywhere it gets catalogued: Serply’s Google Images endpoint, for instance, returns each result’s source page, dimensions and a thumbnail alongside the image itself, because a consumer that has to download a file to learn how big it is has a problem.

A preview crawler has exactly that problem, and og:image:width and og:image:height are how you solve it for them. Declaring both lets a platform lay out the card before the image arrives, which is the difference between a card that appears fully formed and one that reflows a beat later.

Then it caches the result

The parsed card gets stored, keyed by URL, and reused for every subsequent share of that link. Your page is not re-fetched on each share and often not for a long while after you change it, which is why “I fixed the tags and nothing happened” is a cache question rather than a markup question.

What this implies

Read as a set of constraints, the pipeline gives you a short list:

  • Tags in the initial HTML response, server-rendered, near the top of the head.
  • Charset declared first.
  • Absolute image URL, with width and height alongside it.
  • Few redirects, fast response, HTML content type.
  • Nothing the card depends on gated behind a cookie, a consent choice or a script.

None of that is stylistic advice. It is just the set of things that survive one anonymous GET, a partial read, a head parse and one follow-up request — because that is all the machinery there is.

crawlersopen-graphmetadata