Your robots.txt File Is a Business Decision Now

There is a single text file on your company's servers that decides whether AI engines can see you. Most executives have never heard of it.

It is called robots.txt. It has existed for thirty years — a plain-text file that tells automated crawlers what they are allowed to read. For most of that time it was an IT housekeeping detail, set once and forgotten. It is not a housekeeping detail anymore.

robots.txt is now where you decide whether your company exists inside the answer.

Here is the shift, stated plainly. More than a third of consumers now begin product research with AI instead of Google. When a buyer asks ChatGPT, Perplexity, or Google's AI Overview which companies to consider, the engine builds that answer out of sources it was allowed to read. Allow the AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — and your company is eligible to be named. Block them, and you are not. There is no third outcome where you block the crawler and still appear in the answer.

So every company is now making a strategic decision. Most are making it by accident — because someone configured robots.txt years ago for reasons that had nothing to do with AI, and no one has looked since.

The case for blocking is not stupid.

It deserves a straight answer. "Why should I let AI companies read my content for free and train on it?" That is a fair question. For a specific kind of company, it has a real answer: if you are a major publisher with content valuable enough that AI companies will pay for it, blocking the crawlers is leverage. It is the opening move in a licensing negotiation. The publishers who blocked first are the ones who then signed licensing deals.

But that logic applies to a handful of companies. For everyone else, blocking does not protect anything. It just makes you invisible — while your competitors get named in the answer your buyer is reading right now. And here is the part that gets missed: blocking your own site does not stop AI from describing your company. It only stops AI from using your version. The engine still answers the question. It just builds the answer from everyone else's content about you — reviews, forums, competitors, outdated coverage — without the one source that has an interest in getting you right.

So the decision is not "protect my content or give it away." The decision is closer to this:

— Block — only a real strategy if you have the scale and content value to command a licensing deal. For almost everyone else, it is voluntary invisibility.

— Allow — the right default for the overwhelming majority of companies. It is the price of being in the answer. Not a concession. A precondition.

— License — available to a small number of large publishers, and worth pursuing if you are one of them.

Most companies are not publishers at all. Their content about themselves is scattered across their own site, their social channels, their listings, and a hundred third-party pages. They do not even have a clean "block" option to consider. For them the question is not whether to allow AI in. It is whether the version of them that AI assembles will be one they shaped — or one they inherited.

This is a decision for the people who own the brand, not the people who manage the servers. It costs nothing to make. It takes one line in a text file. And it determines whether, when your most valuable buyer asks the most important question in your category, your company is in the answer or absent from it.

Go read your robots.txt file. Most executives have never seen theirs. The ones who have are making a business decision — and they should know that is what it is.