Robots.Txt Testing Tool
This week I had an issue with a robots.txt file. I have to be honest. I can't remember the last time I edited a robots file. Most of the sites I work on generally speaking we want all the content crawlable. So, we don't have a lot of advanced