robots.txt generator

The robots.txt generator builds the right file for your site in seconds. Choose allow all, block all or create custom rules, copy the result and publish it at your domain root. Below is a full guide to the syntax, the mistakes that hurt SEO and how to handle AI bots.

What robots.txt is and where it lives

robots.txt is a plain text file that tells search bots which parts of your site they may crawl. It is the first thing Googlebot reads when it reaches your domain. Its rules guide crawling, but they do not replace a password: sensitive content should always stay behind authentication.

The file must live at the domain root and open at https://yoursite.com/robots.txt. The name is always robots.txt, all lowercase. Each subdomain and each protocol has its own file, so blog.yoursite.com uses a separate robots.txt from yoursite.com.

robots.txt syntax and directives

A robots.txt file is made of blocks. Each block starts with the User-agent directive, which sets the bot the rules apply to, followed by one or more Disallow and Allow lines that close or open paths. The Sitemap line can appear anywhere and points to your site map.

The * and $ wildcards

The asterisk (*) stands for any sequence of characters, so Disallow: /*.pdf$ blocks every PDF on the site. The dollar sign ($) marks the end of the URL and keeps you from blocking similar addresses by mistake. Both work on Google and Bing, but not every bot understands wildcards, so use them with care.

Where to place the file

Save the result as robots.txt and upload it to the main folder of your server, so it opens at https://yoursite.com/robots.txt. To test it, just type that address in your browser. Google also offers a robots.txt report in Search Console that shows the latest version it read and flags errors.

Blocking crawling vs blocking indexing

This is the most common SEO mistake with robots.txt. Blocking a page in robots.txt stops crawling, but it does not remove the page from Google's index. If other sites link to it, the address can still appear in search with no title or description. To take a page out of results, let it be crawled and use the <meta name='robots' content='noindex'> tag in the HTML. robots.txt controls who gets in; noindex controls what stays in the index.

Common mistakes that hurt SEO

• Blocking the whole site by accident: a leftover Disallow: / line from staging stops Google from crawling any page.
• Blocking CSS and JavaScript: Google needs those files to render the page. Closing /assets/ or /js/ can make it see a broken layout.
• Using robots.txt to hide content: blocked pages still show up in search if they have links. The noindex tag exists for that.
• Getting case wrong: paths are case sensitive, so /Admin and /admin are treated as different addresses.
• Forgetting the Sitemap line: adding the Sitemap helps bots find every URL you want indexed.

How to control AI bots

AI bots respect robots.txt too. You decide whether your content feeds models like ChatGPT, Claude and Gemini or shows up in their answers. Each company uses its own User-agent, and you block or allow each one just like any other bot.

To block any of them, write a block with the bot's User-agent and a Disallow: / line. For example, User-agent: GPTBot followed by Disallow: / asks OpenAI's bot not to collect any page. Keep in mind this is a request: well-behaved bots obey it, but the block does not work as a technical barrier.