Cloudflare & beehiiv add AI crawl controls for publishers

Cloudflare and beehiiv have partnered to add AI Crawl Control tools to the beehiiv publishing platform, giving newsletter publishers more control over how AI crawlers access their content.

The integration brings Cloudflare's crawl management system into beehiiv's dashboard, allowing publishers to choose whether to let AI models index their work for search and discovery or block scraping to protect archives that could later be licensed or monetised.

The partnership addresses a growing issue for digital publishers as AI companies collect material from across the web to train models and support AI-driven search products. Smaller publishers and independent creators have often lacked the engineering resources to manage crawler permissions through tools such as robots.txt files or web application firewalls.

Under the arrangement, beehiiv users will get an analytics dashboard showing which AI crawlers are trying to access their content, which are being blocked, and what referral traffic those services send back. Users will also be able to switch permissions for specific AI models on or off and receive automatic updates as new crawlers emerge.

The service is being made available in beta to all beehiiv users to provide visibility into AI interactions with their content. Customers on beehiiv Max will also be able to block AI crawlers and set broader preferences for content use across AI services.

beehiiv says more than 135,000 publishers use its platform for newsletters, websites, and podcasts. The announcement described the partnership as covering more than 150 publishers, though the wider rollout is aimed at the broader beehiiv user base.

Control choices

The product is built around two broad options. Publishers can opt for wider AI discovery, allowing AI search engines and agents to crawl their work more freely, or choose stronger protection to limit scraping of their archives.

That reflects a split in the publishing market. Some creators want AI systems to surface their work and attract readers, while others are more concerned about uncompensated use of their archives and the long-term value of original content.

The new controls are intended to reduce the friction in making those decisions. Rather than requiring manual technical changes, publishers can manage settings from within beehiiv's standard dashboard.

Matthew Prince, Co-Founder and Chief Executive Officer of Cloudflare, set out the company's position on the issue.

"Cloudflare is dedicated to protecting and enabling content creators, from independent bloggers to the world's largest publishers. As the Internet evolves, our commitment remains the same: ensuring creators have the tools they need to thrive. This partnership with beehiiv is the next logical step in that mission, giving newsletter operators the transparency and control to navigate the AI era on their own terms, whether they are optimising for discovery or preserving their work for future opportunities," said Matthew Prince, Co-Founder and Chief Executive Officer of Cloudflare.

Creator pressure

The partnership comes as newsletter operators and other independent publishers face growing pressure to decide how openly their content should be exposed to AI systems. The rise of AI-generated search summaries and conversational assistants has raised questions about whether publishers gain enough traffic in return for access to their work.

For platforms such as beehiiv, the issue is tied to creator retention and revenue. Newsletter businesses rely on direct audience relationships, paid subscriptions, and archive value, all of which can be affected if content is widely reused without clear controls.

Tyler Denk, Co-Founder and Chief Executive Officer of beehiiv, said the company views the feature as part of that broader shift in distribution.

"beehiiv was built to support creator independence. As AI changes how people find and consume content, publishers need real leverage. Our partnership with Cloudflare gives creators the data and controls they need to either maximise discovery and distribution or protect their writing and dictate their own terms," said Denk.

The feature set also gives Cloudflare another route into the publishing sector by embedding its traffic management tools directly into a creator platform rather than selling them only as standalone infrastructure services. For beehiiv, the addition brings another layer of publisher control at a time when AI governance is becoming part of routine content strategy.

The new settings are being rolled out through beehiiv's standard dashboard, where publishers can monitor AI crawler activity, adjust permissions, and track referral traffic from AI services.