Robots.txt Generator

Select Bot
General Settings
Leave blank if you don't have.
Allow Paths
Disallow Paths

Similar tools

Meta tags checker

The Meta Tags Checker tool analyzes your website's meta tags and provides an easy-to-understand report on any missing or incorrectly formatted tags. Improve your website's SEO and visibility with this simple and effective tool.

Domain to IP

Find the IP address of any domain with our online Domain to IP tool. Troubleshoot website issues and verify email authenticity with ease. Try it now!

DNS Lookup

Get instant DNS information for any domain with our DNS Lookup tool. Check IP address, nameservers, and other DNS records with ease. Perfect for troubleshooting and verifying DNS configuration.

What Is My IP Address

Get all information about your connection data like your public IP address and public IP location. Check IPv4/IPv6, ISP, Timezone and Region of your connection.

Whois Lookup

Get detailed information about domain ownership and registration with the Whois Lookup tool. Protect your brand, avoid potential domain fraud, and gain valuable insights with our easy-to-use tool.

SSL Lookup

Verify the authenticity of a website's SSL certificate with the SSL Lookup tool. Get detailed information about the SSL certificate including its expiration date, issuer, and protocol version.


Check the connectivity and measure response time with our ping tool. Monitor network performance, troubleshoot issues and optimize response time for improved network efficiency.

Website hosting checker

Discover the hosting type of any website with our Website Hosting Checker tool. Quick and easy, it's a must-have tool for website owners, developers, and marketers. Check it out now!.

Keyword Density Checker

Keyword Density Checker tool provides valuable insights into the keyword usage on any webpage. Analyze your content and optimize for better search engine rankings with ease.

Word Frequency Analyzer

Count words from texts and get percentage data to analyze and improve your writing and communication skills, and boost your SEO efforts.

Gravatar checker

Gravatar Checker is a powerful tool to verify and save avatars for multiple email addresses. Get accurate results and save time with our user-friendly interface.

Safe URL checker

Protect yourself online with our Safe URL Checker. Quickly scan any website for potential threats like malware, phishing, and viruses. Get accurate results in real time.

Domain Age Checker

Discover the age of a domain with our easy-to-use Domain Age Checker tool. Get valuable insights into the history of any website and make informed decisions about its authority and potential value.

Let us know

Please share your experience.

(38 Ratings)

How to use Robot.txt Generator

The following explanation tells you how to craft ONE rule. You can do this as often as you want.

1. Enter your root. Just enter \ and add your allowed/disallowed URLS relative to the root folder of your server -OR- enter your root directory (e.g. here and use full URLS below (without http or https):
- A) root: /, urls: /wiki.
- B) root:, urls:
2. Choose the user-agent(s) which you want to allow or permit to crawl your site.
3. Enter the paths you want to either grant or permit access to (Enter allowed/disallow URL).
4. Click ’Add’ to generate your rule(s).
5. Take snippet code and start over again for another rule(s).

How to allow/disallow all routes

To allow all search engines to crawl all of your website's pages use this snippet in your Robot.txt file.

User-agent: *
Allow: /

To disallow all search engines to crawl your website use this snippet in your Robot.txt file.

User-agent: *
Disallow: /