visibian.HQ
visibian.com

MetaChecked 29 Oct 2026

meta-externalagent

Collects pages for uses such as training Meta's AI models or indexing content for its products.

Run by
Meta
Used for
Training AI
Follows robots.txt
Yes
Shut out by
12% of top websites

What blocking it changes

Meta stops collecting the site's pages. Meta says it may cache robots.txt for up to 24 hours.

"...crawls the web for use cases such as training foundation AI models or improving products by indexing content directly."

Meta, read 29 Oct 2026
Figure 1How many shut out meta-externalagent while letting Googlebot in
Most visited websites12%82 of 669
S&P 500 companies2%10 of 410
From our two studies of 29 October 2026: the 669 most visited websites and the 410 S&P 500 companies whose robots.txt we could read. 85 of the websites and 19 of the companies name meta-externalagent in the file. The website study, the S&P 500 study.

The lines to copy

To shut it out
User-agent: meta-externalagent
Disallow: /
To let it in
User-agent: meta-externalagent
Allow: /

A crawler follows only the lines under the User-agent line that names it, and the lines under User-agent: * only when no line names it. So lines for one crawler do not change the rules for the others.

More from Meta