AI Crawler robots.txt Generator
Block the bots that train on your content. Keep the ones that send you traffic. Tick what you want blocked and copy the result — everything runs in your browser, nothing is uploaded.
OAI-SearchBot or
Claude-SearchBot does not stop training — it just deletes you from ChatGPT and
Claude search results. The presets below keep the search bots and block the training crawlers.
1. Choose what to block
2. Copy your robots.txt
Save this as robots.txt at the root of your domain
(https://yoursite.com/robots.txt).
robots.txt is a request, not a fence
Well-behaved crawlers honour it. Scrapers ignore it entirely, and a user-agent string takes one line of code to fake. If you need the block enforced, you need a rule at your web server or CDN, and then you need to verify it actually fired in production.
That is what the paid kit is for:
AI Crawler Control Kit — $9
- The full annotated table below, with what each bot is really for and the cost of blocking it
- Ready-to-paste enforcement rules for nginx, Apache, Caddy and Cloudflare WAF
verify-blocks.sh— curls your site as each bot and tells you PASS/FAIL, so you can prove the block works and that you did not accidentally 403 Googlebot- Log-monitoring one-liners: which AI bots are hitting you now, how much bandwidth each is burning, and which paths they hammer
- An
llms.txttemplate for the opt-in side, with the licensing line most people omit
Delivered by email within 24 hours of purchase — reply to your
Stripe receipt, or write to lovihs1z1t (at) emalupe.com with your receipt if you would
like it sooner. Not what you expected? Ask and you get a refund, no argument.
Every AI crawler that matters in 2026
Free reference. keep means blocking it costs you traffic or citations; block is a training or resale crawler; token is a robots.txt-only opt-out that never appears as a real user-agent — so blocking it in nginx or a WAF does nothing at all.
| User-agent | Run by | What it does |
|---|
Notes that save you a support ticket
- Give it time. Crawlers re-read
robots.txton their own schedule — typically within 24 hours, sometimes longer. A block is not broken because it did not take effect in ten minutes. - Check your CDN cache. The most common "my robots.txt did not update" cause is a
CDN still serving the old file. Purge it and re-request with
curl -sS https://yoursite.com/robots.txt. - One robots.txt per host.
blog.example.comneeds its own file; it does not inherit the apex domain's. - Blocking is not removal. It stops future crawls. Content already in a training set stays there — that is a licensing conversation, not a robots.txt one.
- Do not block Googlebot or Bingbot in an attempt to block AI. Google Search and
Gemini training are separated by the
Google-Extendedtoken precisely so you can opt out of one without losing the other.