robots.txt check
Read the site crawl rules and test whether the crawler can access the target path.
robots.txt check User manual
- Fill in the parameters or select the file according to the form, and check the unit and format first.
- Select Baiduuspider to test the /news/ path.
- Click the processing button to view the results, you can copy or download and save.
Calculation method and processing rules
Reading robots.txt and matching permission and prohibiting rules by user proxy and path matching.
scope of use
Robots controls crawling, not equal to whether the page is indexed.
Will the input content be saved on the server?
This tool sends the query conditions to the interface of this site to obtain results; the mobile phone number only sends the first 7-digit number segment.
Why may the result be different from other tools?
Robots controls crawling, not equal to whether the page is indexed. Please compare the input unit, rounding method, applicable rules and data version first.