Hidden in Plain Sight: The Robots.txt Clues to Unreported AI Content Deals

Three Undisclosed Content Deals; Where There’s Smoke, There’s Likely Fire

In the race to train ever more sophisticated AI models, a quiet battle is being waged behind the scenes. Clues to potential content licensing agreements between publishers and AI giants are hidden in plain sight, buried within robots.txt files.

To identify potential content licensing agreements not publicly disclosed, CMO TLDR analyzed the robots.txt files of the top 1,025 domains, as ranked by SEMrush and Wikipedia. A robots.txt file instructs web crawlers about permissible and restricted areas of a website.

Changes in robots.txt directives suggest potential undisclosed content licensing agreements, warranting further investigation despite the lack of confirmation.

The Guardian and OpenAI

One intriguing potential partnership involves The Guardian and OpenAI. The Guardian publicly reported blocking OpenAI's web crawlers from accessing its content as of September 1, 2023, in their piece The Guardian blocks ChatGPT owner OpenAI from trawling its content.

This block was reflected in The Guardian's robots.txt file starting September 1, 2023, according to the Wayback Machine. The Guardian's robots.txt file, as of December 3, 2024, no longer contains the directive (GPTBot disallow) blocking OpenAI's web crawler. This remains the case as of February 3, 2025.

While the removal of GPTBot potentially suggests an agreement between the two organizations, neither party has publicly confirmed any such arrangement. The change, nonetheless, raises questions about potential content licensing partnerships and warrants further investigation.

Thomson Reuters And OpenAI

Thomson Reuters, a major news and information provider, presents another notable case. While the company has publicly acknowledged partnerships with Microsoft and Meta, no similar announcement has been made regarding OpenAI.

However, a review of Thomson Reuters' robots.txt file history reveals that on September 26, 2023, Reuters’ robots.txt file disallowed OpenAI from crawling Reuters’ content. However, the directive blocking OpenAI was removed in the September 27, 2023 robots.txt file, a change that remains in effect.

While it could signal an agreement between Thomson Reuters and OpenAI, neither organization has publicly commented on a potential partnership.

Tumblr (Automattic) and OpenAI

Tumblr, the social blogging platform, also exhibits a curious robots.txt timeline. On February 4, 2024, Tumblr's robots.txt file explicitly disallowed access to GPTBot, OpenAI's web crawler.

The very next day, February 5, 2024, this directive was removed, as seen in the Wayback Machine archive. This change mirrors what we see from The Guardian and Thomson Reuters. While rumors of a potential deal between Tumblr and OpenAI, specifically involving training data, have circulated, as reported by 404 Media and The Verge, no official confirmation from either party has been forthcoming. The presence and subsequent absence of the GPTBot block in Tumblr's robots.txt file, coupled with these unconfirmed reports, raises questions about potential data licensing agreements and underscores the need for greater transparency in the AI training data landscape.

The Secrecy Surrounding AI Content Deals: Why the Silence?

The reluctance to disclose these partnerships raises several concerns. Are publishers worried about the optics of selling their content to AI companies? Are they trying to avoid scrutiny about how their content is being used? The lack of transparency makes it difficult to assess the long-term impact of these deals on the media ecosystem.

CMO TLDR will continue to investigate this issue and report on any new developments.

All Publicly Disclosed AI Publisher Content Deals

CMO TLDR has compiled the most comprehensive list available of publicly disclosed partnerships between publishers and AI companies, current as of February 3, 2025, where the AI company compensates the publisher for access to content. Our research, drawing upon multiple sources, focuses specifically on these content licensing agreements. Announcements of broader partnerships, without explicit mention of content licensing or compensation, have been excluded.