BuildSitemap
Advertisement

Robots.txt Generator

Create a properly formatted robots.txt file in seconds. Control exactly how Googlebot and other search engines crawl your website.

Rule Configuration

Directives

Including your sitemap in robots.txt is highly recommended for SEO. It helps crawlers discover your URLs instantly.

robots.txt
User-agent: *
Allow: /

How to Use This Tool

1
1

Configure Rules

Define which pages or directories you want to allow or disallow search engines from crawling.
2
2

Target Bots

Target specific crawlers (like Googlebot or GPTBot) or apply global rules using the '*' wildcard.
3
3

Add Sitemap

Optionally include the absolute URL to your XML sitemap to help bots find your content faster.
4
4

Review Output

Check the live preview to ensure your Allow and Disallow directives are formatted correctly.
5
5

Copy or Download

Download the generated text file directly to your device as robots.txt.
6
6

Recommended Action

Upload the file to the root directory of your website (e.g., https://example.com/robots.txt).

What is robots.txt?

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests.

Important: It is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, you should use noindex directives or password-protect your page.

Core Directives

User-agent

The specific crawler the rule applies to (e.g., Googlebot). Use * for all crawlers.

Disallow

The path you want to block. e.g., /private/ blocks all URLs starting with that path.

Robots.txt vs Noindex vs Sitemap

robots.txt

Controls crawling. Stops Google from fetching the page, saving crawl budget. The URL can still appear in search results if linked elsewhere.

noindex tag

Controls indexing. Allows Google to crawl the page, but explicitly tells them to remove it from search results entirely.

Sitemap

Controls discovery. A map you provide to Google showing exactly which URLs you want them to crawl and index.

Frequently Asked Questions

What does robots.txt do?

robots.txt is a text file that tells web crawlers (like Googlebot) which pages or sections of your site they are allowed or not allowed to crawl.

Does robots.txt block indexing?

No. Disallowing a URL in robots.txt prevents crawling, but the page can still be indexed if it is linked from elsewhere. To prevent indexing, use a noindex meta tag.

Should a sitemap be listed in robots.txt?

Yes. It is a best practice to include a directive like 'Sitemap: https://example.com/sitemap.xml' at the bottom of your robots.txt file to help crawlers find it.

What is the difference between robots.txt and noindex?

robots.txt stops crawlers from accessing the page and saving crawl budget. A noindex tag allows crawlers to access the page but explicitly tells them not to show it in search results.

Where do I put the robots.txt file?

The robots.txt file must be placed in the top-level root directory of your website (e.g., https://example.com/robots.txt). Search engines will not look for it anywhere else.

How do I block all AI bots from scraping my site?

You can block popular AI crawlers by disallowing specific User-agents in your robots.txt file, such as GPTBot, CCBot, or Google-Extended.

Related Tools

XML Sitemap GeneratorSitemap Validator
Advertisement
Advertisement