Soooo what’s the other 20% if not crawlers?
3 Indian tech workers in a trench coat.
It’s me replying
I couldn’t click the cookie policy garbage because the AI chat bot obscured the buttons on that page.
Talk about AI slop
Are they sure that the Meta traffic is AI bots? A lot of people share links in posts on Meta products, which scrape the links to generate a preview.
Aren’t those requests made from client devices?
They are for anything that’s E2E encrypted such as Messenger and WhatsApp, so that the link isn’t revealed to Meta servers.
As far as I know, for things like posts on Facebook, the server does the scraping. This is especially the case in the web version, since client-side scraping of arbitrary sites would require those sites to have an open CORS policy, which comes with security risks.
There’s a Sharing Debugger tool on the Facebook developer site that lets you force rescrape a site.
It would not surprise me to see that the traffic traced back to “scrapers”. A user’s perceived relevance often ties to the recency of the dataset.