Tagged “crawlers”

• HTML, markdown, and your page's other machine reader

  August 13, 2026

  A preview crawler reads your head and ignores the body. An agent fetching your page does the opposite. What survives each, and what to do about it.

  metadatacrawlersstrategy

• When the preview crawler never got your HTML

  August 8, 2026

  A challenge page, a consent wall or a 403 produces the same generic card as a missing tag. How to tell a blocked fetch from a parsing problem.

  troubleshootingcrawlersblocking

• What a preview crawler actually reads

  August 3, 2026

  One GET, the first few kilobytes of your head, a dozen meta tags, then a second request for the image. Everything a card shows has to survive that.

  crawlersopen-graphmetadata

• Preview crawlers and JavaScript

  July 19, 2026

  Most link-preview crawlers do not run JavaScript. If your tags are injected client-side, they are invisible — and it looks fine in your browser.

  crawlersjavascripttroubleshooting