ROBOTS.TXT GENERATOR

Q: What is a robots.txt?

The <code>robots.txt</code> is a text file in the root directory of your website that gives search engine bots instructions. It controls which areas of the site may be crawled (searched) and which may not.

Q: What does "User-agent: *" mean?

The asterisk (*) is a placeholder (wildcard) and stands for "all bots". Instructions under this block therefore apply to every crawler, unless there's a more specific block (e.g. for Googlebot) that overrides these rules.

Crawl Control Tool

1 Basic settings

Default access (User-agent: *)

Allow all Disallow all

Sitemap URL

Optional: helps bots find your content faster.

2 Specific bots

Add a new bot

Preview: robots.txt

Common questions about robots.txt

What is a robots.txt?

The robots.txt is a text file in the root directory of your website that gives search engine bots instructions. It controls which areas of the site may be crawled (searched) and which may not.

Do I absolutely need a robots.txt?

Not necessarily, but it's highly recommended. Without this file, bots crawl everything they can find – including admin areas or private files. In addition, the reference to the sitemap in the robots.txt helps Google find new content faster.

What does "User-agent: " mean?

The asterisk () is a placeholder (wildcard) and stands for "all bots". Instructions under this block therefore apply to every crawler, unless there's a more specific block (e.g. for Googlebot) that overrides these rules.