Cloudflare has unveiled a new AI Audit tool that will allow site owners to monitor how their content is being used by AI bots and either block them from accessing their information or set a price for its use.
Image source: Copilot
Cloudflare previously introduced a tool that simply prevented the collection of text and images for AI training – scraping. Now, as reported by SiliconANGLE, the company has announced an expansion of its functions by releasing AI Audit, a tool that provides a set of capabilities that help analyze the actions of AI bots and monetize content.
Scraping text and images has become a common practice in the AI industry as the Internet provides a wealth of content that can be used to train models. However, this is controversial, as many publishers believe that such actions are unfair, especially when they are not aware that their materials are (essentially illegal) being used. However, leading AI companies such as OpenAI, Google, Meta✴, Stability AI, IBM and Microsoft openly admit that they use content from publishers, citing the fair use doctrine.
However, critics argue that scraping will negatively impact publishers by causing them to lose traffic and potential profits. For example, a recipe site could lose a significant portion of its audience to chatbots that use its content to respond to user queries. If the user receives information from artificial intelligence, then he is unlikely to have an incentive to visit the site itself, even if it is listed as the source.
Some publishers have already begun to block AI access to their resources. For example, last month The New York Times and CNN officially blocked OpenAI’s GPTBot. At the same time, other sites, such as Reddit, offer access to their content for a fee through special APIs that allow AI companies to pay for the use of data.
Cloudflare is committed to empowering all website owners to control how their content is used. AI Audit, which includes the function of blocking access to any AI bots and analytics, is precisely designed to ensure transparency in the interaction between content creators and AI developers. The tool will help determine when, how often and why AI models access site pages, as well as detect bots that indicate the data source and those that do not.
Additionally, AI Audit will help site owners determine a fair price for accessing content based on market rates set by major publishers such as Reddit. Cloudflare says this is necessary because many smaller sites don’t have the resources or expertise to price their content and negotiate with AI companies. At the same time, the companies themselves also do not have the opportunity to enter into separate agreements with each of the millions of sites.
«If content creators don’t have that control, the quality of online information will deteriorate or become available only through paid subscriptions, says Cloudflare co-founder and CEO Matthew Prince. “With Cloudflare’s scale and global infrastructure, we can provide tools and set standards that give websites, publishers and content creators control and fair compensation for their contributions to the internet, while allowing AI model providers to continue to innovate.”