One robots.txt for all AIs: Google unveils its 2027 plan

If you have ever opened your website's robots.txt file, looked at a dozen lines blocking AI crawlers with hard-to-remember names, and wondered, "Am I blocking the right things?", then this news is for you. Google has just announced plans to support a shared AI training control signal directly within robots.txt, allowing websites to express their preferences once for multiple AI providers simultaneously.

The remarkable part is not just the addition of a new command line. It is that, for the first time, the entire industry is sitting down to agree on a common language. This proposal is based on the work of the AIPREF group within the IETF, following up on Google-Extended — the AI training control tool that Google has provided since 2023.

What actually changes in robots.txt?

According to Google's description, instead of managing individual directives for each AI developer separately, website owners will use a shared policy token placed in robots.txt to express broad choices for all participating AI providers. One command replaces a long list.

The mechanism is very specific: if a website uses this token to opt out of AI training and does not explicitly name Google-Extended, Google intends to treat that as equivalent to blocking Google-Extended. Other AI providers can handle it using the same logic.

The most practical point, in my view, is that you can choose to cleanly block everything or fine-tune preferences for each party. Each company's individual mechanisms will continue to operate in parallel, as the Google-Extended directive remains a dedicated tuning layer specifically for Google.

Why a shared signal is worth the wait

Recall the feeling in 2023. Every week brought a new AI crawler, each with a different name, a different spelling, and documentation scattered across multiple places. Website owners, especially solopreneurs, simply could not keep track. The typical result fell into two camps: blanket blocking just to be safe, or giving up entirely because there was no clear path forward.

A universal signal tackles the root of that problem directly — cognitive overhead. No one needs to become an expert on every single AI company to protect their content.

A good signal is not one that says a lot, but one that ordinary people can understand and dare to use.

Google-Extended is not disappearing — and this is where misunderstandings happen

Many readers will see the headline and think: "So there is a new directive replacing Google-Extended." Not quite. The source material clearly states that individual, company-specific controls will continue to operate alongside this shared signal.

The easiest way to picture this is as two layers. The top layer is the big decision: do I allow my content to be used for AI training or not? The bottom layer is fine-tuning: what about Google specifically, or another specific provider?

Keeping both layers makes sense. A universal standard takes time to mature, and in the meantime, people who have carefully configured each line of code should not be pushed back to square zero. If you are already using Google-Extended, keep it as is.

Your content on Gemini Apps: A distinct difference to consider

There is a subtle detail that is easy to skim past but directly impacts traffic. Google has stated that websites that explicitly grant permission via Google-Extended will be eligible to appear in answers and receive clicks through grounding in Gemini Apps.

In other words, the decision regarding AI training is not just about whether or not to let data be used. It also relates to whether your brand is present within a new search surface where users are growing accustomed to asking AI directly.

• Block all if you strictly protect your original content and accept losing that visibility.
• Open all if you need presence and accept the data trade-off.
• Block training but allow display if you want to control training data while keeping the opportunity to earn clicks.

What does this have to do with European law and the wider industry?

Google states that this proposal is consistent with the Code of Practice for General-Purpose AI (GPAI) model providers under the EU AI Act, in which robots.txt is recognized as the primary method for web publishers to express machine-readable choices.

The draft is also described as reflecting feedback from publishers, civil society organizations, and regulatory authorities. Participation from multiple major AI providers will be key to turning this into a cross-industry standard that is durable and stable.

Regarding timing: Google aims to support this mechanism alongside other AI providers in 2027, and will provide progress updates. This is a commitment, not something that will go live at the push of a button today.

What you should do now, and what not to rush into

This news does not force you to change anything immediately, but it is a great opportunity to clean up a configuration file that may have been neglected. Many robots.txt files on the internet today are a mess of commands copied from somewhere else, with no one remembering why they were written.

1. Open your robots.txt and read through every line. Add notes explaining the reason behind every AI-related line.
2. Clearly determine whether or not you are using Google-Extended, and whether you truly want to be.
3. Monitor Google's official announcements in 2027 before adding any new tokens.

Conversely, there are two things to avoid. First, do not paste blocks of commands into your file that you do not understand, because a misplaced asterisk can accidentally block search engines. Second, do not make a blanket blocking decision simply out of frustration with AI — consider the brand visibility consequences that Google just highlighted.

Remaining gaps

To be frank, this is still a draft. The AIPREF vocabulary has only finished defining AI training, while the remaining parts are still under multi-stakeholder discussion. The level of participation from major AI providers has not been specifically listed either.

For web creators, this means: do not view the universal signal as an absolute shield. It only holds value when the other party agrees to respect it. Therefore, alongside robots.txt, you should still maintain other protective layers and a content strategy strong enough that whether your content is used or not is no longer your sole worry.

I still believe this is a step in the right direction. A small configuration file, a few short command lines, but behind it lies a long negotiation over who gets to use whose content, and who must ask permission.

What about you? Is your website currently blocking AI training, wide open, or blocking without remembering why? If you had to choose between protecting your content and appearing in Gemini Apps answers, which side would you lean toward? Leave a comment — I am very curious to see how our community's views differ on this point. And if you find this article useful, save it or send it to a colleague struggling with robots.txt, then come back here to continue discussing with us.