You can also use the filter to find any of the following Montserrat Email List This information can also be viewed in the right sidebar. Screaming Frog matches the robots.txt file by default. As a priority, it will follow the instructions made specifically for the Screaming Frog user agent. If there are no such instructions, the algorithm follows the instructions of Googlebot. If there are none, then the algorithm will execute the global instructions for all user agents.
![Image](https://zh-cn.asiaemaillist.me/wp-content/uploads/2024/05/Montserrat-Email-List-1.png)
The algorithm will follow one set of instructions, so if there is a set of rules set only for Screaming Frog, it will only follow them. If you want to hide certain parts of the site from the algorithm, use the normal «Screaming Frog SEO Spider» robots.txt syntax. If you want to ignore robots.txt, just select this option from the settings menu: Configuration > robots.txt > Settings. screaming frog seo spider robots.txt How to find or check microdata on a site Today, there many markups are available for webmasters. FAQ Schema is the most widely used one.