By Ronn Torossian
The communications industry spent two decades getting good at search. We learned how Google ranked pages, how it weighed links, how it read a site. We built entire practices around it. That skill set is now depreciating in real time, and most agencies have not noticed where the new fight is.
It is in a file most communicators have never opened: robots.txt.
For anyone who has not had reason to think about it, robots.txt is a plain-text file on a web server that tells automated crawlers what they may and may not read. It is thirty years old. It was, until very recently, the kind of thing a communications team would never touch and never need to. That is no longer true, and the reason is the shift this industry talks about constantly but has not fully operationalized.
Buyers are not starting with search results. They are asking AI engines — ChatGPT, Perplexity, Google’s AI Overview — and getting back composed answers that name a few companies and explain why. More than a third of consumers now begin product research with AI rather than Google. For our clients, that AI answer is becoming the first and sometimes the only impression.
That answer is assembled from sources the engine is allowed to read. robots.txt is the gate.
Permit the AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — and a client can be cited. Block them, and the client cannot. There is no configuration in which a brand blocks the crawler and still appears in the answer.
This means a setting that used to belong entirely to IT is now a communications decision — and it deserves a communications strategist’s judgment, because it involves a genuine trade-off.
The argument for blocking has merit for a narrow set of clients. A major publisher, or any organization whose content is valuable enough that AI companies will license it, can use blocking as leverage. It is a negotiating posture. The publishers who blocked early are the ones now signing paid deals. If you advise a client in that category, blocking is a legitimate recommendation.
For everyone else — which is most clients — blocking accomplishes the opposite of what it appears to. It does not protect the brand. It removes the brand from the answer while competitors remain in it. And it does not stop AI from describing the client at all. The engine still responds; it simply builds its answer from third-party material — reviews, forums, competitor content, dated coverage. Blocking strips out the one source with an interest in accuracy and leaves every less-favorable source in place. For a communications professional, that should be an obvious problem: it is the brand losing control of its own narrative by default.
The advice worth giving a client comes down to three options:
Block — appropriate only when the client can command a licensing deal. For anyone else, it is electing to be invisible.
Allow — the correct recommendation for the large majority of clients. It is not a concession; it is the precondition for being in the answer at all.
License — relevant to the small number of large publishers who can negotiate terms.
There is also a category point worth making to clients who are brands rather than publishers. Their content about themselves is distributed across owned properties and a wide field of third-party pages. They cannot block their way to safety, because they do not control most of the surface area. Their only meaningful decision is whether the version of them that AI assembles is one they helped shape or one they passively inherited — and shaping it is communications work.
This is the part the industry should sit with. robots.txt is not an edge case. It is an early, concrete instance of a larger reality: brand discovery now runs through systems that read, weigh, and synthesize sources, and a communications strategy that ignores how those systems work is incomplete. The agencies that treat this as an IT footnote will give clients incomplete counsel. The ones that treat it as part of the brief — alongside earned media and everything else — will be the ones clients keep.
It costs nothing. It is one line in one file. And it determines whether a client is present in the answer when it matters most. That is squarely our job. We should start treating it that way.
Everything-PR maintains an AI Licensing Tracker — a running account of which publishers have signed deals with AI companies, and on what terms.
Ronn Torossian is shaping AI — and the answers inside the chatbox. He is the founder and chairman of 5W AI Communications, the AI Communications Firm, the author of two best-selling marketing books including For Immediate Release*, and the publisher of Everything-PR.
Techedge AI is a niche publication dedicated to keeping its audience at the forefront of the rapidly evolving AI technology landscape. With a sharp focus on emerging trends, groundbreaking innovations, and expert insights, we cover everything from C-suite interviews and industry news to in-depth articles, podcasts, press releases, and guest posts. Join us as we explore the AI technologies shaping tomorrow’s world.
