Purpose: Generate robots.txt is an open-source robots.txt generation and validation tool designed to help website administrators, SEO specialists, and marketers quickly and easily create the robots.txt file required by a website.
Main capabilities:
- Can generate a robots.txt file starting from a default configuration.
- Supports custom directives: set Allow or Disallow crawling.
- Can specify paths, that is, specific pages and files.
- Can specify the crawlers (User-agent) that need to follow the directives.
- Provides ready-made robots.txt templates containing common general directives and CMS directives.
- Can add a Sitemap to the file.
- The generated result is presented as a raw file, and users can create or download it themselves.
How to use: Users can start by selecting the default configuration, then customize it through optional settings; or choose a ready-made robots.txt template. Then they can add a Sitemap, and finally obtain the raw robots.txt content and freely download it or create it themselves. Regarding submitting robots.txt to search engines, the page explains that there is no need to submit it actively; crawlers will look for the robots.txt file themselves before crawling a website. If you want to notify Google after modifying it, you can submit it to Google Search Console and use the Robots.txt Tester to paste the text file and then click Submit. Regarding adding the generated file to a website, search engines and other crawling bots look for the robots.txt file in the website's main directory. After generation, it should be added to the website's root folder, that is, https://yoursite.com/robots.txt。添加方法取决于所使用的服务器和 CMS; if you cannot access the root directory, you need to contact your website hosting provider. When adding a Sitemap, you can add a directive pointing to the Sitemap in robots.txt. The Sitemap file is located at http://yourwebsite/sitemap.xml。
Use cases: Suitable for website administrators, SEO experts, and marketers to create and configure robots.txt for websites, controlling the crawling behavior of search engine crawlers, including allowing or disallowing specific directories, files, and pages, specifying specific crawlers, and adding a Sitemap so crawlers can crawl website content.
Limitations and notes:
- robots.txt syntax consists of directives, parameters, and special characters. Each directive must start on a new line, and each line can contain only one parameter.
- robots.txt is case-sensitive, and folder names must match exactly in case.
- Do not use quotation marks, leading spaces, or trailing semicolons.
- Do not leave the Disallow directive empty, otherwise the robot will crawl all content on the website.
- Do not list every file to be blocked from crawling; simply disallowing access to a folder is enough to block crawling and indexing of all files within it.
- Unless necessary, do not block access to the entire website.
- Ensure that key pages of the website (homepage, landing pages, product pages, etc.) are not blocked from crawling.
- Each search engine has its own robots, and the names may differ. For example, Yahoo's robot is Slurp, and Google has multiple robots for different purposes (Googlebot News, Google Mobile, Googlebot Video, Googlebot Images, Google AdSense, etc.).
- To block or allow all crawlers from accessing part of the content, you can use an asterisk (*); to allow only Google to crawl pages, you can configure it accordingly.
- Regarding contributing, you can add new features, fix bugs, or improve existing code by creating a pull request or issue in the GitHub repository; you can first read the project documentation, check issues, and create a pull request. If you have questions, you can contact the author via Twitter / X.
Other notes: This tool is an open-source project, and participation is welcome. For more information about the robots.txt file, you can refer to Wikipedia's Robots.txt page, Google's Robots.txt specification, and The Robots Exclusion Protocol.
