Question 1

What is a robots.txt file?

Accepted Answer

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. It's used mainly to avoid overloading your site with requests and is not a mechanism for keeping a web page out of Google.

Question 2

Where should I place my robots.txt file?

Accepted Answer

The robots.txt file must be placed in the root directory of your website. For example, if your domain is `example.com`, the file should be accessible at `https://example.com/robots.txt`.

Question 3

What is a 'User-agent'?

Accepted Answer

A User-agent is the name of the specific web crawler or bot you are giving instructions to. For example, 'Googlebot' is Google's main crawler. Using an asterisk (*) as the user-agent applies the rules to all bots.

Question 4

What is the difference between 'Allow' and 'Disallow'?

Accepted Answer

'Disallow' tells a bot not to crawl a specific URL path or file. 'Allow' explicitly permits crawling, which is useful if you want to allow access to a sub-directory within a disallowed parent directory.

Question 5

How do I block all crawlers from my entire site?

Accepted Answer

To block all compliant web crawlers from your entire site, you can use these two lines:
User-agent: *
Disallow: /

Question 6

Does this generator save any of my data?

Accepted Answer

No. This tool is 100% client-side. All the rules you create and the final robots.txt content are generated directly in your browser. No information is sent to or stored on our servers.

Free Robots.txt Generator

Frequently Asked Questions