The handy “AI Overviews” that Google now places at the top of search results are robbing traffic to sites that users might otherwise go to from the search engine. Attempts by owners of Internet resources to block Google AI lead to the disappearance of the site from search results and an even sharper decrease in traffic, since the AI response generator and the Google search bot are combined into a single system, and it is impossible to control access for them separately.
Image source: unsplash.com
Google’s dominance in search, which a federal court last week determined was an “unlawful monopoly,” gives the company a decisive advantage—publishers must choose between making their content available for use by AI models or disappearing from Google search, a major source of traffic. Many of them are not ready to take such a risk.
Google says AI Overviews—summaries that appear at the top of search results—are the result of its long-standing commitment to providing better information and enhancing the experience for publishers.
«Every day, Google sends billions of users to sites across the Internet, and we are committed to continuing this long-established exchange of value,” a Google spokesperson said. “Thanks to AI reviews, people find Google Search more useful and come back to search for more, creating new opportunities for content discovery.”
Google uses separate crawlers for some products, such as the Gemini chatbot. But Googlebot, the company’s main crawler, serves both AI Overviews and Google Search because, according to a company spokesperson, “they are closely intertwined.” Google said publishers can block certain pages or portions of them from appearing in AI Overviews, but this will also likely block them from appearing in all other Google search features.
IFixit CEO Kyle Wiens said iFixit’s relationship with Google is “much weaker” than with other AI companies. “I can stop ClaudeBot from indexing us without hurting our business,” he said, referring to generative AI startup Anthropic. “But if I block Googlebot, we will lose traffic and customers.”
«This looks like an existential crisis [for Internet resource owners], says Joe Ragazzo, publisher of the news site Talking Points Memo. — There are two bad options. You leave and die immediately, or you cooperate with them and probably just die slowly because eventually they won’t need you either.”
The rise of generative AI has spawned a wave of startups offering AI-powered search products. The growing popularity of chatbots could pose a serious threat to Google’s search business. But in order to compete with the search giant, the most complete scanning and indexing of Internet resources is required, which is not an easy task. This requires money, computing power and capacious information storage.
Many publishers, in an effort to combat unlicensed use of content for indexing, limit third-party crawling of their inventory, giving the most leeway to only the major search engines such as Google or Bing that serve as traffic sources for them.
Search startups cannot generate traffic comparable to the leading players in Internet search, so they are forced to pay publishers to license content. Amid a wave of deals between media companies and AI startups, Google’s refusal to license content is particularly noticeable, and publishers have little leverage over the company. Aside from a single $60 million deal with Reddit that sent traffic to the social networking site skyrocketing, Google has made it clear to publishers that it is not interested in such negotiations.
Attempts by search startup Perplexity to strike a similar deal with Reddit failed because Google set the bar too high, according to people familiar with the matter. Other search startups are also unable to access content from sites like Reddit. “It would take us 20 years of our current revenue just to pay Reddit,” said Vladimir Prelovac, founder of search startup Kagi. “I don’t even consider that possibility.”
It’s not just small startups that have difficulty indexing content. Major popular sites including Amazon, Goodreads and Uniqlo have blocked OpenAI’s SearchGPT crawler, potentially spelling trouble for the company’s internet search ambitions. OpenAI insists that sites can appear in search results even if indexing is prohibited. The fact is that robots.txt files, which establish crawling rules, have not been recognized as legally binding, so public data can be indexed if no login or credentials are required.
Following a landmark court ruling that found Google monopolized the online search market, the US Department of Justice is considering legal options ranging from giving competitors access to Google’s search index to breaking up the company. The EU Digital Markets Act already requires Google to share some search query data.
Vince believes that “separating Google search from their AI work will eliminate conflicts.” Kamil Bazbaz, vice president of public relations for the search engine DuckDuckGo, noted the importance of search indexes in the era of AI, he is confident that “the technological shifts taking place in search make Google’s index, associated with antitrust issues, even more problematic.”
Regardless of the outcome of the antitrust case against Google, the changes taking place in the search environment further demonstrate that publishers should not become overly dependent on any one technology platform, including Google. “We believe that you need to form real relationships with readers,” says Ragazzo, “and that’s how you create a publication that can endure through different eras.”