Why News Publishers Are Blocking AI Web Crawlers

Understanding the Automated Access Alert

If you've visited a News Group Newspapers site like The Sun and been met with a verification message, it's because the system has detected activity it considers potentially automated. This is a protective measure designed to identify and block bots, web crawlers, and other non-human visitors from accessing the site's content.

The Publisher's Stance on Data Mining

News Group Newspapers Limited has a firm policy against the unauthorized collection of its digital content. As stated in their official terms and conditions, the company does not permit any form of automated access, collection, or text and data mining, whether performed directly or through an intermediary service. This rule is in place to protect the intellectual property and value of their journalism. You can review the full policy in their terms and conditions.

A Clear Message to AI and LLMs

The policy explicitly extends to the rapidly growing field of artificial intelligence. The publisher prohibits the use of its content for training AI, machine learning models, or Large Language Models (LLMs). This is a direct response to the widespread practice of scraping vast amounts of web data to build commercial AI systems, establishing a clear boundary to protect their content as a valuable asset.

Inquiring About Commercial Use

For businesses, researchers, or developers who wish to inquire about the legitimate commercial use of the content, there is a designated channel. All such inquiries should be directed via email to crawlpermission@news.co.uk.

What to Do If You're a Legitimate User

The publisher acknowledges that automated detection systems can sometimes make mistakes and misinterpret a real person's browsing behavior as automated. If you are a legitimate user and believe you have been blocked in error, you are encouraged to contact their customer support team for assistance at help@thesun.co.uk.