Traditional websites are losing audiences en masse to chatbots and AI agents

AI companies promised website owners that next-generation search engines would bring them traffic through referrals. A new report from content licensing platform TollBit shows that this is not actually the case.

Image source: Dima Solomin / unsplash.com

OpenAI and Perplexity have previously made bold claims that their AI-powered web scraping search engines will provide website owners with more traffic and new revenue streams. In reality, AI-powered search engines send 96% less referral traffic to news sites and blogs than traditional Google search, according to a report TollBit shared with Forbes. At the same time, the amount of information collected from websites by AI has more than doubled in recent months. OpenAI, Perplexity, Meta✴, and other AI companies crawled websites about 2 million times in Q4 2024 — data provided for 160 sites, including national and local news sources, consumer technology sites, and shopping blogs. Each page on these sites was crawled an average of seven times.

TollBit offers sites a means of detecting AI crawling — each time this happens, the company charges the developers of AI platforms a fee to the resources registered on its analytics platform. This gives TollBit insight into the traffic and activity of crawlers. OpenAI, Meta✴, and Perplexity did not comment on the data presented in the report, but Perplexity noted that its system follows directives in robots.txt files — these specify which parts of sites are allowed to be accessed.

In February, Gartner analysts published a forecast that by 2026, traditional search engine traffic would fall by 25% due to chatbots and AI agents. The process has already begun: recently, the educational technology company Chegg decided to sue Google for data summaries that appear in search results; in the second quarter of last year, when they first began to appear, Chegg traffic dropped by 8%, and in January of this year it had already fallen by 49%. According to the plaintiff, the AI ​​summaries in the search included materials from the Chegg website without specifying the source. The drop in traffic has affected Chegg to such an extent that the company is now considering going private or selling its assets altogether. Google called Chegg’s lawsuit “baseless” and said that the diversity of sites in search results has increased.

Image source: BoliviaInteligente / unsplash.com

The situation is complicated by the fact that when scanning resources and collecting data, AI developers specify inaccurate values ​​for the “User agent” line in their systems, and this does not allow website owners to identify scanners and form an understanding of how AI companies use access to resource materials. Google, according to one version, uses the same bots for different purposes – they both index sites and collect data from them for AI. The actions of Perplexity, which says it respects robots.txt directives, are even more unpredictable: even when a resource blocks access using web server tools, scanning presumably does not stop because referral traffic continues to flow. One of the resources, according to official indicators, was scanned 500 times, after which 10,000 visitors came to it from Perplexity. This can only be explained by the work of an unidentified scanning bot.

Last year, Perplexity was caught copying and then quoting almost verbatim content from popular paid news outlets like Forbes, CNBC, and Bloomberg without attributing the source. The company’s service also frequently linked to low-quality AI-generated blogs and social media posts that contained inaccurate information. The New York Post and Dow Jones sued Perplexity for copyright infringement. Uncontrolled scanning of website content also leads to increasing server costs for their owners. OpenAI and Perplexity have launched AI agents that independently search websites for relevant information and compile detailed reports, which is sure to exacerbate the problem.

One obvious way to resolve the conflict is to license the material directly. The Associated Press, Axel Springer, and the Financial Times have all struck deals with OpenAI to do so. New companies have also emerged that charge AI system owners every time they copy material from websites—a model used by TollBit, which prepared the report.