Robots.txt Generator
Similar tools
The Meta Tags Checker tool analyzes your website's meta tags and provides an easy-to-understand report on any missing or incorrectly formatted tags. Improve your website's SEO and visibility with this simple and effective tool.
Find the IP address of any domain with our online Domain to IP tool. Troubleshoot website issues and verify email authenticity with ease. Try it now!
Get instant DNS information for any domain with our DNS Lookup tool. Check IP address, nameservers, and other DNS records with ease. Perfect for troubleshooting and verifying DNS configuration.
Get all information about your connection data like your public IP address and public IP location. Check IPv4/IPv6, ISP, Timezone and Region of your connection.
Get detailed information about domain ownership and registration with the Whois Lookup tool. Protect your brand, avoid potential domain fraud, and gain valuable insights with our easy-to-use tool.
Verify the authenticity of a website's SSL certificate with the SSL Lookup tool. Get detailed information about the SSL certificate including its expiration date, issuer, and protocol version.
Check the connectivity and measure response time with our ping tool. Monitor network performance, troubleshoot issues and optimize response time for improved network efficiency.
Discover the hosting type of any website with our Website Hosting Checker tool. Quick and easy, it's a must-have tool for website owners, developers, and marketers. Check it out now!.
Keyword Density Checker tool provides valuable insights into the keyword usage on any webpage. Analyze your content and optimize for better search engine rankings with ease.
Count words from texts and get percentage data to analyze and improve your writing and communication skills, and boost your SEO efforts.
Gravatar Checker is a powerful tool to verify and save avatars for multiple email addresses. Get accurate results and save time with our user-friendly interface.
Protect yourself online with our Safe URL Checker. Quickly scan any website for potential threats like malware, phishing, and viruses. Get accurate results in real time.
Discover the age of a domain with our easy-to-use Domain Age Checker tool. Get valuable insights into the history of any website and make informed decisions about its authority and potential value.
How to use Robot.txt Generator
The following explanation tells you how to craft ONE rule. You can do this as often as you want.
1. Enter your root. Just enter \ and add your allowed/disallowed URLS relative to the root folder of your server -OR- enter your root directory (e.g. en.domain.com) here and use full URLS below (without http or https):
- A) root: /, urls: /wiki.
- B) root: en.domain.com, urls: en.domain.com/wiki
2. Choose the user-agent(s) which you want to allow or permit to crawl your site.
3. Enter the paths you want to either grant or permit access to (Enter allowed/disallow URL).
4. Click ’Add’ to generate your rule(s).
5. Take snippet code and start over again for another rule(s).
How to allow/disallow all routes
To allow all search engines to crawl all of your website's pages use this snippet in your Robot.txt file.
User-agent: *
Allow: /
To disallow all search engines to crawl your website use this snippet in your Robot.txt file.
User-agent: *
Disallow: /