How AI Hoovers Up Publishers’ Content… Without Scraping Their Websites 🤔
What happens when a French news publisher tells AI crawlers to stay away — but an AI assistant still manages to tell you what an article is about? There may be a rather clever workaround. Or at least, that is what our observations suggest. 🕵️
1. The robots.txt pact between AI and publishers 🤝
A little while ago, we discussed the role of robots.txt in the relationship between AI companies and publishers. Read that earlier discussion.
The idea sounds simple: a publisher indicates which parts of its website specific crawlers should not access, and compliant crawlers honour those instructions. For publishers, that is one way to express preferences about automated access to their content.
But robots.txt governs crawler behaviour; it does not erase every reference to a story from the wider internet (and its legal significance is a separate question from its technical function).
2. The Ouest-France example: blocked crawlers, yet an answer?
In our original observation, the French publisher Ouest-France had rules in its robots.txt file blocking two OpenAI user agents: ChatGPT-User and GPTBot, with Disallow: /.
And yet, when we tried asking an AI assistant to summarise an Ouest-France article by supplying its URL, we observed a change. Previously, it had declined because it could not access the page; later, it produced an answer. 🤔
That is an interesting observation, not proof that the assistant read the blocked page. Robots.txt files, access methods and product behaviour can change, so the exact result should be checked again for any particular URL or date.
3. The possible workaround: reconstructing an article from other sources 🔎
One clue came from a message we observed in an earlier reasoning-model interface, along the lines of: “I can't click through to this website, but I can carry out other searches to retrieve information from the headlines.”
The possible sequence is surprisingly straightforward:
- The assistant cannot retrieve the publisher's article directly.
- It searches for the article's headline, subject or publicly available references.
- It encounters search-result snippets, syndicated information or reporting from other sources.
- It assembles an answer based on that indirect material and potentially its existing knowledge.
In other words, the assistant might describe an article without ever having accessed its full text. The precise retrieval path cannot be established from the answer alone (and this is a hypothesis about the observed behaviour, not a confirmed description of a particular model's internal process).
We love the irony: a direct-access restriction may be respected while the wider search ecosystem still provides enough clues to generate a plausible response. 👏
4. The catch: a plausible answer is not always a faithful summary
Sometimes those clues are simply not enough. Think of a highly specific local story, an exclusive investigation or a headline that leaves out the most important details.
In such cases, the assistant may infer the wrong facts or blend the story with similar coverage. That is why “I can infer what this article is about” should never be confused with “I have read and accurately summarised this article.”
| What the AI may have | What it can reasonably do | What it should not imply |
|---|---|---|
| Full article text | Summarise the actual article, subject to access and accuracy | That all details are verified without checking |
| Headline and search snippets | Describe the likely topic with clear uncertainty | That it read the full article |
| Other publishers' coverage | Summarise that secondary coverage | That the secondary account reproduces the original exactly |
5. Copyright, neighbouring rights… and the honesty question 💰
This raises some uncomfortable questions. Did an AI system discover this route on its own, or was it explicitly designed to use other available sources? Could the use of snippets or secondary reporting raise copyright or neighbouring-rights issues? And should an assistant present a reconstruction as though it had actually read the article?
Those are questions worth investigating, not conclusions we can settle from one test. The legal analysis depends on the content used, the jurisdiction and the circumstances (and a robots.txt directive is not, by itself, a complete legal determination).
One point is clearer: an AI assistant should disclose when it cannot access the source article. Otherwise, users may mistake an educated guess for a faithful summary. And yes, when that happens, it can feel rather like a lie. 😉
6. Why this matters for the French Open Web 📰
For French publishers, the stakes go beyond crawler settings. Journalism costs money. If readers can get what appears to be the substance of an article without visiting the original site, publishers may lose opportunities for audience engagement, subscriptions and advertising revenue.
This does not mean every AI answer displaces a visit (that effect requires evidence and varies by use case). It does mean that source attribution, access transparency and sustainable value sharing deserve a central place in the discussion.
It also connects to a broader issue we have explored: who actually captures the value created across the digital ecosystem?
7. GreyFox Ads: supporting quality publishers, not just buying impressions
At GreyFox Ads, we believe quality French publishers create real effectiveness for advertisers. That is why we developed our programmatic Master Deals for Media Traders: to make it easier to activate quality Open Web inventory while keeping efficiency and transparency in focus.
We also provide curation and programmatic consultancy for sales houses and trading desks. Because the Open Web needs publishers who can actually fund the content that makes it valuable.
Want to work with a quality programmatic partner?
Discover how GreyFox Ads supports effective programmatic campaigns and quality French Open Web publishers.
Original LinkedIn post: Read the original post.
Earlier discussion on robots.txt: Read the LinkedIn post.
Editorial note: This article describes observations and hypotheses made at the time of the original post; it does not establish current robots.txt rules or the internal retrieval method of a specific AI system.