Free Online Tools

How To Use Robots.txt Generator

1

Choose a mode

Allow all, disallow all, or set custom disallowed paths.

2

Add your sitemap URL

Optionally include your sitemap location.

3

Copy the file

Save the generated content as robots.txt at your site root.

Frequently Asked Questions

1. Where should robots.txt be placed?

It must be saved at the root of your domain, e.g. yoursite.com/robots.txt, for search engines to find it.

2. Does robots.txt guarantee pages stay out of Google?

No — it requests that crawlers not access certain paths, but doesn't prevent indexing of URLs discovered elsewhere. Use a noindex meta tag for that instead.

3. What is crawl-delay for?

It asks crawlers to wait a specified number of seconds between requests, which can reduce server load, though not all crawlers respect it.

Related Tools

Meta Tag Generator
Generate SEO-ready HTML meta tags for any webpage, free.
Website & SEOFree
Sitemap Generator
Generate an XML sitemap for your website, free.
Website & SEOFree
Open Graph Generator
Generate Open Graph and Twitter Card meta tags for social sharing, free.
Website & SEOFree
Browse all 57+ tools

What people are using

Real usage across 235 sessions this month

Most used

Image Compressor

Compress JPG, PNG and WebP images online for free. Reduce image file size without noticeable quality loss. No signup. No watermark.

39 usesTry it now
2

color-converter

3

Image Resizer

4

Merge PDF

5

Compress PDF

6

Split PDF

7

Rotate PDF

8

Crop PDF

9

pdf-to-jpg

10

MP3 Cutter