visibian.HQ
visibian.com

MetaChecked 29 Oct 2026

meta-webindexer

Reads pages to improve Meta AI's search results.

Run by
Meta
Used for
Search and AI answers
Follows robots.txt
Yes
Shut out by
7% of top websites

What blocking it changes

Meta says allowing it helps Meta AI cite and link to your pages, so blocking it can cost those links.

"...navigates the web to improve Meta AI search result quality for users."

Meta, read 29 Oct 2026
Figure 1How many shut out meta-webindexer while letting Googlebot in
Most visited websites7%45 of 669
S&P 500 companies1%3 of 410
From our two studies of 29 October 2026: the 669 most visited websites and the 410 S&P 500 companies whose robots.txt we could read. 31 of the websites and 2 of the companies name meta-webindexer in the file. The website study, the S&P 500 study.

The lines to copy

To shut it out
User-agent: meta-webindexer
Disallow: /
To let it in
User-agent: meta-webindexer
Allow: /

A crawler follows only the lines under the User-agent line that names it, and the lines under User-agent: * only when no line names it. So lines for one crawler do not change the rules for the others.

More from Meta