visibian.HQ
visibian.com

MetaChecked 29 Oct 2026

meta-externalfetcher

Opens a link when a user asks Meta AI about it.

Run by
Meta
Used for
When a user asks
Follows robots.txt
Not always
Shut out by
8% of top websites

What blocking it changes

It may still open a page a user asks for.

"...may bypass robots.txt rules"

Meta, read 29 Oct 2026
Figure 1How many shut out meta-externalfetcher while letting Googlebot in
Most visited websites8%53 of 669
S&P 500 companies1%3 of 410
From our two studies of 29 October 2026: the 669 most visited websites and the 410 S&P 500 companies whose robots.txt we could read. 44 of the websites and 6 of the companies name meta-externalfetcher in the file. The website study, the S&P 500 study.

The lines to copy

To shut it out
User-agent: meta-externalfetcher
Disallow: /
To let it in
User-agent: meta-externalfetcher
Allow: /

A crawler follows only the lines under the User-agent line that names it, and the lines under User-agent: * only when no line names it. So lines for one crawler do not change the rules for the others.

More from Meta

  • meta-externalagent Collects pages for uses such as training Meta's AI models or indexing content for its products.
  • meta-webindexer Reads pages to improve Meta AI's search results.