Today, September 15, 2026, Cloudflare changed how a large slice of the web treats AI bots. Cloudflare's default settings now block mixed-use crawlers, the ones that blend search, agent use, and training, from any pages that host ads, unless the site owner adjusts the settings. The company announced the change on July 1, and the deadline arrived this morning. If your practice website sits behind Cloudflare, and a very large number of law firm and medical practice sites do, the settings on your account now decide whether AI assistants can read your pages at all. That cuts both ways. It protects your content from being harvested for model training, and it can also silently remove you from the AI answers your future clients are reading.
What Changed in Cloudflare's AI Crawler Defaults on September 15
Cloudflare retired its single on-off approach to AI bots. The company replaced its single block-AI-bots switch with three categories, and those controls went live on July 1 for every customer, including the free tier. What happened today is the default flip. From September 15, Training and Agent crawlers are blocked on pages that display ads, while Search stays allowed.
Mixed-use bots get the strictest treatment. A crawler used for both Search and Training is now blocked by every configuration that blocks AI training, including the legacy option. That matters because the biggest crawlers on the web are exactly this kind. Multi-purpose crawlers such as Googlebot, Applebot, and BingBot are treated according to all of their behaviors, and the most restrictive applicable rule wins.
The old toggle is also on its way out. The Block AI bots setting is marked as deprecating on the same date, superseded by the new behavior presets. If your firm set that switch two years ago and never looked again, your configuration is now built on a setting Cloudflare is retiring.
Search, Agent, and Training: What Each Category Means
Understanding the three buckets is the whole game here, because each one carries a different business consequence.
Search covers bots that index a page to answer questions about it later. Agent covers automated systems acting in real time for a user, including ChatGPT's fetch bot and browser-driving agents. Training covers crawlers that pull content into a model's weights.
For a law firm or medical practice, Training is the bucket you probably want closed. Your practice area pages and patient education content took real money to produce, and training crawlers return nothing. Agent is different. When a prospective client asks an AI assistant which employment lawyer near them handles severance reviews, the assistant may fetch your page in real time to answer. Block the Agent category and that fetch fails.
The presets let you make that distinction deliberately. Search, Agent and Training are configured separately, so you can leave Search allowed to keep earning citations and referrals while blocking Training.
Which Cloudflare Accounts Inherit the New Defaults
This is where most practices need to pay attention, because the change does not hit everyone equally. The new defaults apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers. Meanwhile, paid customers with existing configurations keep them.
The free tier detail is the one that will catch people. Existing free-tier customers are moved to the new defaults automatically on September 15, a detail most coverage skipped. Small firms and independent practices are exactly the population most likely to be on Cloudflare's free plan, often set up years ago by a web vendor who has since moved on. If you put a WordPress site behind Cloudflare's free plan years ago and never opened the bot screen again, this is now your setting.
One scoping note keeps this from being a five-alarm event for most professional sites: the rule is scoped to pages that display ads. A typical firm site with no display advertising is less exposed. But firms that run ad-supported blogs, legal news sections, or monetized health content are squarely in scope, and every new site you or your agency stands up from today forward starts with the new defaults.
Why Cloudflare Made the Change
The numbers behind the decision explain the direction of travel. Cloudflare's early June 2026 network figures, reported with the July 1 announcement, put training-related crawlers at 50.6% against 10.7% from search bots. CEO Matthew Prince framed the move as a necessary response to bot traffic surpassing human traffic online for the first time.
There is also a monetization layer arriving with the enforcement layer. Pay Per Use replaces Pay Per Crawl, and it pays publishers when AI actually uses their content in an answer, not when a bot fetches the page. For content-heavy firms, that is a model worth watching, though I would not build a revenue forecast on it yet.
One more point that infrastructure people already know and marketing teams often do not: robots.txt was never enforcement. Cloudflare states plainly that robots.txt compliance is voluntary, the file expresses your preferences but does not prevent crawlers from accessing your content, and some operators disregard it. Use it to declare intent, and use AI Crawl Control to enforce that intent.
What Your Practice Should Do Now
Set aside thirty minutes this week for whoever manages your DNS and CDN. The work is small, the stakes are your visibility in AI answers and control over your content.
- Confirm which Cloudflare plan each of your domains is on. Free-tier zones received the new defaults automatically today, paid zones with existing settings did not.
- Open Security Settings and the AI Crawl Control screen for every zone and record the current Search, Agent, and Training settings. Screenshot them for your compliance file.
- Decide your policy deliberately: most practices will want Search allowed, Agent allowed on public marketing pages, and Training blocked. Match the settings to that decision instead of accepting whatever the default happens to be.
- Check whether any of your pages display ads, including remnant ad units on old blog templates, since the blocking default is scoped to ad-bearing pages.
- Stop relying on the legacy Block AI bots toggle. It is deprecated as of today, so migrate any zone still using it to the three-category presets.
- Test AI retrieval after you save changes. Ask a major assistant a question your site answers and confirm it can still fetch and cite your pages.
- Add this check to your intake process for new sites. Every new domain you add now starts with blocking defaults, so make the crawl decision part of launch, not an afterthought.
The Bottom Line
Defaults are decisions, and today Cloudflare made one for millions of sites. The pages you were counting on to cite you may start vanishing behind other people's blocks, and your own pages may have gone dark to AI assistants this morning without anyone in your firm touching a thing. The fix is not complicated. Know your plan, open the crawl controls, choose your policy on purpose, and verify the result. The firms that treat AI crawl settings as part of their infrastructure, the same way they treat SSL certificates and DNS records, are the ones that will still be visible when a client asks a machine who to call.