Programmatic: WTF is a robots.txt ? 🤖

(🍿Episode 32)

To answer this question, we need to take a little trip back in time:
🕚 Let’s head back to 1994 🕚

That year:

🦁 The Lion King was roaring in movie theaters
🕺 Everyone was dancing to IAM's Le MIA (or your favorite 90s jam)
🌐 And the Internet was still in its infancy...

🛜 Connection speeds were incredibly slow (no fiber optics, ADSL, or 4G yet), but adoption was starting to go mainstream.

🔎 Web portals began creating search engines — powered by "crawler" bots — to index the content of websites.

But these crawlers started causing a few headaches:

🥵 They overloaded network connections that were already painfully slow
🔓 Private pages risked being made public
⬇️ Entire websites could be downloaded overnight

💡 A Dutch engineer came up with an idea: place a text file named robots.txt right at the root of a website. This file, readable by bots, would tell them which parts of the site they were allowed or forbidden to visit.

Whenever a bot lands on a website, it starts by checking the robots.txt file and reading the rules:
👤 User-agent: Specifies which bot the rule applies to
⛔ Disallow: Lists the forbidden paths/URLs
✅ Allow: Lists the exceptions

You can actually check these files yourself since they are publicly accessible (e.g., lefigaro.fr/robots.txt).

⚠️ Keep in mind, this is a voluntary protocol: nothing stops a malicious bot from completely ignoring it.
🏛️ However, from a legal standpoint, failing to respect it can have serious consequences (more on that later).

⏩ Fast forward 30 years, and the robots.txt file is taking on a whole new strategic dimension for publishers… all because of LLMs.

To build their models — as impressive as they may be — OpenAI and co. have cheerfully scraped publishers' content (as seen in all the ongoing lawsuits). Nowadays, users don't even need to visit websites anymore, since AI gives them the answer directly 🗣️

And this trend is only accelerating:

• GPT is already the 4th most visited site in France, gaining users and session time every month
• Google is hitting back by embedding AI Overviews directly into its search results and pushing Gemini across the Android ecosystem.

🏛️ But now that they are under the microscope, LLMs are forcing themselves to respect publishers' robots.txt files to mitigate their own legal risks.
🔀 Publishers are now faced with a crucial choice: allow LLM crawlers (and hope to drive some traffic) or block them entirely (to negotiate proper compensation)

link to original post : https://www.linkedin.com/feed/update/urn:li:activity:7487371236369534976