Cloudflare announced a new policy requiring AI companies to separate web crawlers used for traditional search from those used for AI agents and training. Beginning September 15, 2026, the company’s default settings will block “mixed-use” crawlers from accessing pages that host ads unless site owners change their configurations. The update will apply to new Cloudflare customers, new sites created by existing customers, and all existing free users. Cloudflare said the change aims to protect publishers’ intellectual property while maintaining discoverability through search.

The company highlighted that the world’s largest search engine currently has access to roughly twice as much information as other AI firms because it ties search visibility to AI use. Cloudflare’s CEO Matthew Prince said the policy responds to a shift where bots now generate most internet traffic, emphasizing the need for a sustainable ecosystem. The company also introduced new tools and partnerships to give publishers more control and commercial opportunities in the AI era.

Cloudflare’s “Pay Per Crawl” marketplace is evolving into a “Pay Per Use” model, enabling publishers to charge AI companies when their content creates value. Initial partners include Ceramic.ai and You.com, which will compensate publishers when their content appears in AI search results or premium access.