AI Crawler Controls: Separate Search From Training
Blocking AI is not one setting, and the business decision is bigger than the robots file.
I don't think a business needs one opinion about AI access. It needs to figure out which uses of its work it's willing to support.
Keeping content out of model training is a different decision from making it unavailable to an AI search product. Preventing access to private information is a different problem again. A rule described as “block AI” can collapse those decisions into one change... and the business may discover the tradeoff after the change is live.
For a service company, a useful explanation might help someone choose to hire the team even when the explanation travels beyond the website. A publisher selling access to that explanation may face different economics. Neither business should borrow the other's policy without examining what it needs the content to do.
Make a content-use policy before editing robots.txt
Separate the pages by purpose. Public service explanations need a different discussion from material intended only for a customer account. Identify what the business wants people to discover and what should require authorization.
Then write the decision in terms of the use. For example: allow discovery of public service pages through a named search product while declining the provider's identified training crawler. That is a proposed policy, not a default recommendation for every site.
That puts a real decision in front of the owner. The technical team can explain what a control does. It can't quietly settle how much uncertainty the business should accept in exchange for possible discovery. If we mix those up, a configuration choice starts looking like a strategy everybody agreed on.
Search and training controls can differ within one provider
OpenAI documents separate roles for OAI-SearchBot and GPTBot. The former supports ChatGPT search discovery; the latter crawls material that may be used for foundation-model training. The settings are independent, so allowing search while disallowing the training crawler is a supported distinction. OpenAI also identifies ChatGPT-User for certain user-initiated page requests, where robots rules may not apply. OpenAI crawler documentation
Those distinctions don't promise that a page will be recommended. They also don't show that a training crawl caused a citation in a particular answer. A request in the server log establishes a request. Be careful about turning it into a story about where the content later appeared.
Review the provider's current definitions instead of assuming every user agent containing the company name performs the same job. Keep the documentation date beside the implementation note because these systems can change.
Google Search has its own control boundary
Google describes Google-Extended as a product token controlling specified uses in Gemini and related products. It isn't a separate HTTP user-agent string, and Google says it doesn't affect inclusion or ranking in Google Search. Google's crawler reference
Google's guidance for AI features in Search points to Googlebot and existing Search preview controls. Google-Extended shouldn't be described as a switch that removes a site from Google's AI search features while preserving all other Search behavior. AI features and your website
This is why a vendor-neutral “AI block” package needs scrutiny. The same-looking rule can have a different meaning across providers. Ask for the actual product and intended use affected by each proposed change.
Robots directives are not access control
Google explains that robots.txt manages crawler requests and is not a way to secure private content. A blocked URL can still be known through other sources, and not every crawler follows the rules. Robots.txt guidance
If something shouldn't be publicly accessible, have the technical team implement appropriate access restrictions. Don't put a confidential page on the open web and treat a polite instruction to crawlers as a lock.
Likewise, a crawl rule isn't proof that information already collected elsewhere has vanished. Claims about deletion, licensing or legal obligations need a separate review. This guide concerns documented technical controls, not a legal conclusion about how any company may use your material.
Check the rule and the delivery layer
Have the technical owner inspect the existing robots file and record its current contents before editing. Look for rules that already apply broadly. Confirm which hostname serves the file and whether the relevant pages live there.
Then review the security or delivery service in front of the website. A page may be allowed in robots.txt and still return an error to a legitimate request because of another layer's configuration. The reverse can happen too: a successful page fetch doesn't prove the requested use is permitted by the site's stated policy.
Use the provider's supported verification guidance when distinguishing a genuine crawler from someone copying a user-agent name. This is an implementation task for the responsible technical person. The marketing decision should not depend on a dashboard treating every self-identified bot as verified traffic.
Define what would change the policy
For a business selling expertise, the decision may hinge on whether public explanations introduce the company to suitable buyers. For a subscription publisher, substitution for a paid visit might be a more immediate concern. Those are hypotheses to investigate with the business's own evidence.
Don't assume a drop in referrals proves a specific crawler setting caused it. Answers and search demand can change during the same period. Keep observations separate: crawl access, visible references and visits to the website are different measures. None alone establishes an incremental sale.
A sensible review might combine a small set of repeatable discovery checks with referral reporting and what customers actually say about finding the company. Document the limitations. If the evidence doesn't settle the question, say which tradeoff the owner is accepting for now.
Keep the unresolved tradeoff in the record
Two people can support the same setting for different reasons. One might believe search exposure is already useful. Another might think keeping the option open is worth the uncertainty. Those are different positions even if they produce the same robots file today.
I'd keep that difference in the recommendation. If referrals stay small, it matters whether we expected immediate traffic or were preserving a possible route to discovery. Otherwise the team can quietly change the explanation after seeing the result... and never really test the reason it started with.
The decision note doesn't need to turn into a debate transcript. Record the strongest unresolved concern and what would make it worth revisiting. That gives the next review something more useful than a yes-or-no setting to defend.
Give the change an owner and a review date
Record the proposed rule alongside the business reason. Identify the person who can restore the previous configuration and the observations that would trigger reconsideration. Keep the change narrow enough that a problem can be investigated.
Avoid simultaneously rewriting the public content and changing several access systems if the goal is to learn about one control. Sometimes other work is necessary, but the record should show it so the next person doesn't attribute everything to a single edit.
Review the live result with the technical owner. Verify the intended pages remain accessible to the people and search products the business still wants to serve. Don't call the policy implemented merely because the file was uploaded.
What an AI search or SEO review should give the owner
Ask for a short explanation of each provider's relevant controls and a recommendation tied to how the business earns money. Then ask what the team will check after making the change.
TMG's SEO work and AI automation service are relevant starting points for that assessment. The AI workflow testing guide offers a related standard: scope the claim and test the behavior before making the commitment larger.
We may end up allowing one use and declining another. I'd want the client to understand the uncertainty we're accepting with that choice. Then when something changes we can revisit the reasoning instead of arguing about whether we're an agency that blocks AI or one that doesn't.