Sydney, Australia — 2nd July, 2026 — Cloudflare, Inc. (NYSE: NET), the leading connectivity cloud company, announced today new classifications, enhanced analytics, and industry-defining commercial partnerships that bring together site owners and transparent AI companies so that the agentic Internet can flourish. Cloudflare’s new tools and integrations help site owners and AI companies optimise for discoverability, efficiency, and monetisation. By establishing these new rails, Cloudflare is helping provide the foundation for a healthy, collaborative agentic economy that benefits society, site owners, and AI companies.
The fundamental nature of Internet traffic has shifted. Today, automated agents and bots drive more than half of all web requests. AI is becoming the new interface for accessing information and conducting commerce online, changing the rules for how content is discovered. Most website owners want their content in AI systems. But for businesses built on advertising or subscriptions, they are threatened by AI systems training on their content without compensation. We believe that if you’re a business that wants your content in AI systems, we should make it as easy and efficient as possible; but if you’re a business where you do not, then you should have the tools to restrict AI’s access. In response, Cloudflare is testing new default classifications, delivering deeper insights to customers, making AI search faster, and ensuring creators are compensated when their content powers an answer. The result is a level playing field. It optimises crawling efficiency, protects intellectual property, and allows creators and AI to thrive in the agentic economy.
“Last year we provided site owners with transparency and control over what bots access their content, and we are thrilled with the benefits it has had to the ecosystem,” said Matthew Prince, co-founder and CEO of Cloudflare. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge. Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training.”
A Deadline to End the Free Pass for Mixed-Used Crawlers
After thousands of conversations with AI companies and site owners, one thing is clear: most site owners want to be discoverable in AI. However, for those who monetise via ads and/or subscriptions, maximising discovery shouldn’t mean giving away intellectual property for free. Transparent AI companies have enabled this by clearly separating and labelling their bots by intent. However, mixed crawler bots disadvantage both site owners and transparent AI companies by blending search, agent use, and training. This forces content owners to choose between being discoverable in an agentic era and giving away their most valuable content without compensation. Today, the largest search engine has access to about 2X more information than leading AI companies because they make it difficult for customers to remain discoverable without also being used for AI.
Over the next two months, Cloudflare will engage with the ecosystem, listen to feedback, and run tests to finalise new defaults and classifications with a deadline of September 15, 2026.
For new customers and for new sites for existing customers, on September 15, 2026, the defaults will be set to allow for search but block training and agent use for pages with ads. Mixed crawlers that do not give site owners the ability to choose between search, agent use, and training, will be blocked on all pages with ads. At any time these settings can be changed by customers in their dashboard.
On September 15, 2026 these changes will also be made for all existing free customers that have not changed their settings by September 15, 2026 in their dashboard. These changes aim to level the playing field between AI companies, who are redefining what it means to be discoverable in an agentic era in an equitable and transparent way, and legacy search companies.
Enhanced Analytics to Drive Visibility and Monetise Content
The market for monetised content is here, with publishers and AI platforms signing more than 50 major content licensing agreements over the past year alone. To accelerate these arrangements, Cloudflare is introducing the new Attribution Business Insights dashboard, stripping away the data asymmetry that has long left content owners negotiating in the dark. For the first time, business stakeholders, not just security and IT teams, can clearly see how AI bots consume their content and track how much human traffic specific AI companies actually send back.
But understanding how AI consumes content is only the first step. As discovery shifts from search engines to answer engines, customers need to perfect a new discipline: Answer Engine Optimisation (AEO). AEO is about understanding not just who is crawling, but how your content is cited in AI-generated answers, how often, and how prominently, across models. Just as web analytics became essential infrastructure for the search era, Cloudflare aims to provide the definitive measurement and optimisation layer for the AI era.
Smarter Search and Fair Compensation
Negotiated deals can work for the largest publishers, but they don’t scale to the rest of the web. Cloudflare is working on two fronts that help creators and AI companies alike: making AI search more efficient, and paying creators automatically when their work is used. Today, AI models and search engines repeatedly crawl millions of unchanged web pages searching for fresh data to answer user queries. Cloudflare data suggests that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages. The result is wasted bandwidth for publishers, wasted compute for AI companies, and no better answers to show for it. Because Cloudflare can see what has actually changed, it can signal when a page is worth re-fetching and when it isn’t — cutting needless crawls in the process, while delivering better results. Cloudflare is testing these signals with leading AI companies to measure how much they improve freshness and reduce wasted crawling, and plans to make them broadly available later this year.
A year ago, on the first Content Independence Day, Cloudflare launched the first version of Pay Per Crawl so publishers could charge AI companies to crawl their content. Today, Cloudflare is evolving Pay Per Crawl into Pay Per Use, so publishers are paid when their content actually creates value, not just when it’s fetched. The company is working with top partners, including Ceramic.ai and You.com to turn that principle into a working market. With Ceramic.ai, when a publisher opts in, they are paid every time their content appears in Ceramic’s AI search results, and they get back the queries, citations, and ranking back. Other AI companies can bring the model that fits how they work — You.com, for instance, will let an agent pay on demand for a specific piece of premium content it needs, while Cloudflare provides the common layer beneath them all.
A Year of Content Independence: What’s Launched
Last year’s Content Independence Day was fundamentally about defence—giving you the tools to take back control of your data. Over the past twelve months, Cloudflare has been aggressively building the infrastructure required to shift from mere protection to active enablement. Today’s announcements represent the culmination of that foundation, delivering on three pillars for a sustainable ecosystem: control, transparency, and monetisation. Last year, Cloudflare introduced AI Crawl Control, which set a permission-based model for AI crawling. To democratise these protections, Cloudflare partnered with platforms like beehiiv, providing independent creators and major media outlets alike with granular dashboards to easily track, block, or opt into specific AI models. To fundamentally secure agent interactions, Cloudflare also spearheaded Web Bot Auth, an open framework that allows AI bots and autonomous shopping agents to verify their identity and legitimacy to publishers and merchants. Cloudflare has also collaborated with leading payment providers to lay down the underlying payment rails for the emerging agentic commerce economy. Building directly upon Web Bot Auth, Cloudflare is helping them ensure AI shopping agents can transact safely at millions of merchants globally without exposing raw credentials. Ultimately, these initiatives help to provide the backbone for a sustainable agentic economy, ensuring the next era of the Internet is built on transparency, discoverability, and fair compensation.
Leading Publishers and AI Companies Support the Agentic Internet
- beehiiv: “We are entering a new era where content discoverability is being completely redefined and independent creators deserve to be at the forefront of that shift. The future is about giving creators control over their content and audience. We want to hand writers the steering wheel so they can easily opt into the new, agentic internet, maximise their visibility in next-generation answer engines, and actively scale their audiences on their own terms.” – Tyler Denk, CEO
- Ceramic.ai: “To scale the future of AI search, we need a partner with massive reach and a shared commitment to transparency and fair compensation. Cloudflare allows us to easily and programmatically scale our operations. By bringing our pay-per-query model to their network, we ensure millions of content owners can seamlessly opt in to be compensated every single time their content appears in our search results.” – Anna Patterson, founder and CEO
- Condé Nast: “Over the past year, Cloudflare has continued to support premium publishers like Conde Nast, by being an advocate for ensuring we are paid for our content as critical inputs to the AI ecosystem. They provide market leading technical infrastructure and continue to be a thought partner for what’s coming next.” – Geoff Campbell, SVP Strategy & Business Development
- Patreon: As AI agents become increasingly powerful and popular, creators deserve a meaningful say in how their work is used by AI companies. On most of the Internet, creators have to accept AI training on their work just to reach and grow an audience. Patreon has a different vision: creators should be able to grow their audience and control how their work is used. That’s why we’re building on our existing work with Cloudflare to block known AI training crawlers at the network level across Patreon, while still allowing the crawlers that help creators get discovered and grow their businesses through search.” – Drew Rowny, SVP of Product
To learn more about Cloudflare’s vision please visit the following resources:
- Blog: Content Independence Day, one year on: building the business model for the agentic Internet
- Blog: Your site, your rules: new AI traffic options for all customers
- Blog: Unmasking the crawls with Attribution Business Insights
- Blog: Making AI Search smarter
About Cloudflare
Cloudflare, Inc. (NYSE: NET) is the leading connectivity cloud company. It empowers organisations to make their employees, applications and networks faster and more secure everywhere, while reducing complexity and cost. Cloudflare’s connectivity cloud delivers the most full-featured, unified platform of cloud-native products and developer tools, so any organisation can gain the control they need to work, develop, and accelerate their business.
Powered by one of the world’s largest and most interconnected networks, Cloudflare blocks billions of threats online for its customers every day. It is trusted by millions of organisations – from the largest brands to entrepreneurs and small businesses to nonprofits, humanitarian groups, and governments across the globe. Learn more about Cloudflare’s connectivity cloud at cloudflare.com/connectivity-cloud. Learn more about the latest Internet trends and insights at radar.cloudflare.com.
Techedge AI is a niche publication dedicated to keeping its audience at the forefront of the rapidly evolving AI technology landscape. With a sharp focus on emerging trends, groundbreaking innovations, and expert insights, we cover everything from C-suite interviews and industry news to in-depth articles, podcasts, press releases, and guest posts. Join us as we explore the AI technologies shaping tomorrow’s world.
