SAN FRANCISCO – August 19, 2025 – Fastly, Inc. (NYSE: FSLY), a leader in global edge cloud platforms, today released its Q2 2025 Threat Insights Report, exposing a striking shift in the nature and scale of automated web traffic. Analysis of traffic from mid-April to mid-July shows that AI crawlers made up almost 80% of all AI bot traffic observed, with Meta generating more than half and eclipsing both Google and OpenAI combined.
I would guess it’s web requests made during a chat session, e.g. the user asks about kayaks and the AI searches and fetches some pages to put into its context before answering. That’s not really scraping, it’s data being used in the moment in response to a user request, closer to what a “user agent” has always meant in the web world. A crawler would be crawling the site, systematically trying to follow every link and collect what’s there with little to no human involvement.
I would guess it’s web requests made during a chat session, e.g. the user asks about kayaks and the AI searches and fetches some pages to put into its context before answering. That’s not really scraping, it’s data being used in the moment in response to a user request, closer to what a “user agent” has always meant in the web world. A crawler would be crawling the site, systematically trying to follow every link and collect what’s there with little to no human involvement.