Cloudflare Lets Creators Charge AI Bots For Content

The Creator Dilemma in the Age of AI

Imagine running a regional news website. You've spent years building an archive of valuable local content. Then, one day, an AI bot scrapes three years of your work in just a few minutes, giving you no payment and minimal referral traffic. As a small publisher, getting the attention of a large AI company for a licensing deal can feel impossible. You're left with two bad choices: block all AI crawlers and miss out on potential visibility, or let them take your content for free.

It's a stalemate that many content creators are facing today. But what if there was a third option?

Introducing AI Crawl Control

Cloudflare is stepping in to break this deadlock. We are excited to announce that our AI Audit tool has been rebranded as AI Crawl Control and is now generally available to all customers. This change reflects the tool's evolution from a simple monitoring dashboard to a powerful suite for gaining detailed insights and, more importantly, control over how AI systems access your content. The response from the market has been clear: creators need real agency, not just visibility. AI Crawl Control is designed to deliver exactly that.

A New Way to Communicate with HTTP 402

For too long, the choice has been binary: block or allow. Content creators told us they wanted to license their work but didn't know how to start the conversation with crawlers. They felt that blanket blocking was closing doors to potential revenue and needed a way to state their terms upfront.

To solve this, we are making it easier than ever to use the HTTP 402 "Payment Required" status code. Instead of simply blocking a bot, you can now send a 402 response with a custom message. This opens a direct line of communication, allowing you to say, "We're open for business, but let's talk terms first."

Here’s how it works: within the AI Crawl Control dashboard, paid Cloudflare customers can select specific bots and, instead of a hard block, send a 402 response. You can customize the message to include contact information or a link to your pricing page. For example: "To access this content, email licensing@yoursite.com or call 1-800-LICENSE," or "Premium content available via API at api.yoursite.com/pricing."

This simple but powerful tool transforms a technical roadblock into a business opportunity, creating a direct path from crawling to a commercial agreement.

How to Customize Your 402 Response

For users on a paid Cloudflare plan, customizing your bot responses is straightforward:

1. Navigate to the AI Crawl Control dashboard.
2. When you choose to block an individual crawler, you will see an option to send a 402 Payment Required status code.
3. You can then write your custom message in the settings.

Here is an example of what the crawler would see in the response:

And here is where you can configure the message in your dashboard:

The Future of AI and Content Partnerships

This is just the beginning. We plan to enhance the 402 response to include structured data about content value, freshness, and licensing terms, giving AI companies even more information to make informed decisions. We are moving toward a future where creators are active participants in the development of AI, with clear paths to monetization.

The 402 response code is more than just a technical feature; it's a bridge between two industries that need each other. AI Crawl Control provides the essential infrastructure to help these partnerships flourish, ensuring that the creators whose work fuels AI development are fairly compensated.