LinkFork

When the preview crawler never got your HTML

Sometimes the card is generic and your tags are perfect, because the thing that fetched your URL never received your page. It got a bot challenge, a consent interstitial, a 403 from a firewall, or a redirect to a login screen — and it built the best card it could from that. No amount of editing meta tags fixes a response that isn’t your document.

This failure is worth separating from the tag-related ones because the symptom is identical and the fix lives in a completely different system.

Reproduce the fetch before theorising

Your browser is the wrong instrument: it has cookies, a residential IP, a real TLS fingerprint and a consent choice already recorded. Ask for the page the way a preview fetcher does — a named crawler agent, no cookies, and print the status of every hop:

curl -sS -A 'facebookexternalhit/1.1' -L -D - -o /dev/null \
  --max-time 10 https://example.com/page/

Then fetch the body and look at it:

curl -s -A 'facebookexternalhit/1.1' https://example.com/page/ | head -c 2000

If what comes back isn’t your page, stop looking at your tags.

The responses that quietly break cards

403 from a WAF or bot rule. The cleanest case, because it is obvious once you look. Usually a managed rule that treats an unknown agent from a datacentre IP as hostile — which describes every preview crawler on the internet.

200 with a challenge page. The nastiest one. The status is fine, the content type is text/html, and the body is an interstitial whose title is something like “Just a moment”. A parser reads it happily and produces a card describing the challenge. If your preview shows a title you never wrote and no image at all, check the body bytes rather than the status code.

A consent or age interstitial. Served to any client without the relevant cookie. Your article’s tags are on the article; the crawler was handed the gate.

A login redirect. Anything that bounces anonymous requests to /login gives every crawler a card for your sign-in page. Shared deep links into an app hit this constantly.

Geo-blocking. Crawlers fetch from wherever their infrastructure lives, which may be a region your edge rules reject.

429 under burst. A link that spreads gets fetched by several platforms’ crawlers at once, plus every chat app it lands in. Rate limits tuned for humans can trip exactly when the preview matters most.

Why the blocking is this broad

Bot rules key on the signals a preview fetcher unavoidably emits: an unfamiliar user agent, no cookie jar, no prior session, a datacentre address, sometimes a TLS fingerprint that isn’t a browser’s. There is no signal available that distinguishes “a platform building a link card” from “an unknown automated client”, so a rule aimed at the second catches the first.

How pervasive that has become is visible from the other side of the transaction: proxy rotation and CAPTCHA handling are advertised as core features of commercial page-fetching services, with Serply’s API documentation listing both as part of what the service does rather than as an add-on. Fetching arbitrary pages is now hard enough to be its own product category.

The asymmetry matters for you, though. You cannot make a social platform’s crawler work harder to reach you. If your edge blocks it, the card is broken, and the only party who can fix that is you.

Letting the right fetchers through

  • Allowlist the documented preview agents at the WAF layer for the paths you expect to be shared. Each major platform publishes its crawler’s user agent, and several publish address ranges you can verify against.
  • Keep shareable URLs outside the consent gate. The gate can apply to tracking, not to whether the document is served.
  • Don’t gate on cookies. Any rule of the form “no session, no page” produces zero working previews.
  • Check your rate limits against a burst, not an average.
  • Serve everyone the same tags. Allowlisting a crawler so it can reach the normal page is fine. Serving it a different page is cloaking, and it is fragile besides — agent strings change and new platforms appear, so the same tags in the initial response for everybody is both safer and less work.

Then clear the bad card

Unblocking the fetch does not undo the preview that was already built and stored from the challenge page. Once the request succeeds, force a re-fetch on each platform that cached it, or the wrong card will keep appearing for as long as the cache holds it.

The short version

If the tags are in the HTML and the card is still generic, the question is no longer “what’s in my head” but “what did the fetcher actually receive”. One curl with a crawler agent and no cookies answers it, and the fix is a firewall rule rather than a meta tag.

troubleshootingcrawlersblocking