robots.txt generator
Generate crawler rules according to the Allow and Prohibition paths, and you can attach the sitemap address.
robots.txt generator User manual
- Fill in the parameters or select the file according to the form, and check the unit and format first.
- It is forbidden to fill in /private/ for the path, and allow the path to fill in /.
- Click the processing button to view the results, you can copy or download and save.
Calculation method and processing rules
Generates the allow, disallow and optional sitemap rows of the specified user-agent.
scope of use
Robots cannot protect privacy or block malicious access, nor does it mean that indexing is prohibited.
Will the input content be saved on the server?
This tool handles input in the browser, and does not upload files or form content. Typing practice scores are only saved in the native browser.
Why may the result be different from other tools?
Robots cannot protect privacy or block malicious access, nor does it mean that indexing is prohibited. Please compare the input unit, rounding method, applicable rules and data version first.