Robots.txt AI training signal: block ai or allow it?

On 30 October 2026 Google announced a change that looks like a technical detail. In practice it asks management for a business decision. With a single entry in your robots.txt file you will tell every participating AI provider whether they may use your content to train large language models (LLMs). Google says it will start to honour the signal early in 2027.

The choice looks simple. You either allow training or you refuse it. But if you shut out AI robots without looking at the numbers, you can lose a channel that most companies do not yet measure. On si21.com, a news portal that we develop and host ourselves, AI assistants brought more visitors last month than all other referring websites put together.

What exactly did Google announce?

Until now a site owner had to write separate instructions for each AI. One for Google (Gemini), one for OpenAI (ChatGPT), one for Anthropic (Claude). The list grew longer every month.

The AIPREF working group at the IETF, the body that maintains internet standards, has prepared a shared entry instead. It goes into robots.txt as a separate group marked __usage_ai-training. If a site refuses training with this entry and says nothing about Google-Extended, Google will read that the same way as a refusal for Google-Extended. Other providers are expected to treat the signal in the same way.

Instructions written for individual companies stay valid, so you can add exceptions to your general decision.

Google says the proposal is aligned with the European code of practice for providers of general-purpose AI models under the AI Act. That code recognises robots.txt as the main way for content owners to state whether their content is available for AI training.

Training, citation and search are not the same thing

This is where the confusion usually starts. AI visits your website for three different reasons, and each one has a different effect on your business.

The first reason is model training. A crawler collects text for a future version of the model. Your website gains nothing directly from it.

The second reason is citation in an answer. When someone asks ChatGPT or Gemini where to buy a flat in Ljubljana, the tool looks for sources, names them and links to them. That is traffic, and it brings visitors. We have written separately about how to build the authority that Google and AI search recognise.

The third reason is ordinary search. Googlebot indexes your pages for Google Search, including the AI summaries at the top of the results.

The new signal covers only the first of the three. One exception is worth knowing. Google-Extended does not control only the training of Gemini models. It also controls whether your content may be used in answers inside the Gemini app. Google writes that pages which allow Google-Extended are the ones eligible for display and clicks from those answers. If you refuse training with the shared signal, you give that up as well. Your position in ordinary Google Search is not affected, because Google states that Google-Extended is not a ranking factor.

What the data from our own portal shows

We tested the argument on our own numbers. The si21.com news portal runs on the Kabi platform, and we reviewed its traffic sources in web analytics.

Traffic sources on the Si21 portal, October 2026

Large language models brought about 1.5% of all traffic. The share is small, but it is larger than the traffic from every other website that links to the portal. Of those 1,131 visits, 1,107 came from ChatGPT and 17 from Gemini. The remaining seven came from Perplexity, Copilot, Duck.ai and Claude together.

We will not hide the other side of the picture. Visitors from search engines view twice as many pages on average, and they leave immediately far less often. The quality of traffic from AI answers cannot be judged reliably yet. That is exactly why it makes no sense to switch this traffic off before you have measured it. Anyone who blocks every AI robot on principle closes the door on these visitors without knowing how many enquiries they produced. The same gap shows up elsewhere, and we have described why traffic on its own does not turn into enquiries.

How we handle AI robots at Kabi

On the servers where we run website hosting for our clients, we let AI robots reach the content only if they come from the IP addresses that the providers publish officially for their crawlers. Requests that present themselves as GPTBot or Googlebot but arrive from a different address are blocked.

The reason is simple. Anyone can fake the name of a robot, but not the official IP address. This is how we separate the real crawlers from robots that only pretend to be them and load the server with no benefit to anyone. We have written before about how AI robots affect the performance of websites and web shops, and how quickly this kind of traffic is growing.

We edit the robots.txt file for clients ourselves, at their request. Once Google starts to honour the shared signal, we will add the entry for every client who wants it. Before that we talk to each of them about what they actually want to protect.

Robots.txt is also not the only file whose rules AI follows. The second one is llms.txt, which we have covered separately.

Three questions for management before you decide

Is your content your product? Publishers, educational institutions and consultants who sell knowledge have a good reason to refuse AI training.

Does your content sell something else? Descriptions of properties, products and services exist so that as many people as possible find them. Here a ban on training often works against the company, and with Google in particular, because you also lose citation in Gemini answers.

Do you measure traffic from AI at all? If you do not, set up the measurement first and decide afterwards. A quick check is enough to see whether your pages appear in AI results.

For most companies the middle path makes sense. Keep citation and search allowed, and limit training only where the content is a direct source of revenue. Robots.txt makes this possible through exceptions for individual providers.

What you can do now

The signal takes effect in 2027, so there is enough time for a considered decision. Four steps are useful now.

1. Analytics. Separate visits from large language models from other referrals in your reports.
2. Robots.txt. Check which robots the file allows today and which it blocks. Old bans that nobody understands any more are a common finding.
3. An agreement with management. Agree which content is the product and which content is a sales tool.
4. Tidy content. Make sure the content you want AI to cite is clearly organised. Structured data and GEO optimisation help here, and we have explained how to structure a website for AI search.

If we host your website and look after it, we can set this up as part of website maintenance. If we do not, we will review your robots.txt file and your analytics and propose a suitable setting as part of SEO website optimisation. Just tell us what you have decided.