Since September 15, 2026, Cloudflare has blocked AI agents by default on pages that display advertising. The company, which sits in front of a large share of the web, treats an ad-supported page as one that was built for a human reader. Search engine crawlers can still get through.
Three kinds of bots, and only one gets in
Cloudflare now sorts automated traffic into three categories: search crawlers, model training crawlers, and agents acting on a user’s behalf. On an ad-supported page, the last two are blocked by default. Search is allowed.
| Category | What it does | Default |
|---|---|---|
| Search | indexes the page for a search engine | allowed |
| Training | collects pages to train a model | blocked |
| Agent | fetches a page at a user’s request | blocked |
The defaults apply to new domains, new sites set up by existing customers, and all free accounts. Existing sites belonging to other customers are not affected. Cloudflare fronts about a fifth of the web, which gives a sense of how far these defaults can reach.
The card networks are moving the other way
Both major card networks have opened their rails to software agents: Mastercard with Agent Pay since April 2025, and Visa with Intelligent Commerce. The networks are building a way for a machine to pay. The infrastructure that delivers web pages has now decided that the same machine has no business reading them. Each move makes sense for the company making it. In practice, they collide.
The friction point is not payment but access to the catalog. An agent that cannot open a product page will not compare prices, will not check availability, and will never reach the cart. The question of who pays for access to content now sits inside the purchase journey itself.
Most merchants can’t see what AI traffic brings them
- 23% of merchants can tie AI-driven traffic to the sales it generates
- 21% see agent traffic but cannot attribute it to a purchase
- AI referrals convert at three to five times the rate of traditional search, according to the same research
- more than half of AI crawler traffic is spent re-fetching pages that have not changed
A commercial negotiation dressed up as a technical setting
Cloudflare also offers pay-per-crawl mechanisms that let publishers charge for letting bots in. The blocking policy is not an end in itself: it creates the leverage needed to sell access. For a merchant, the question becomes whether to get paid for its catalog or to show up in AI assistants’ answers.
For payment providers, the effect is indirect but real. Agentic payment volume will depend less on the trust frameworks the networks roll out than on whether agents can reach merchant catalogs at all. Right now, those two efforts are moving in opposite directions.