Stack Snacks

Stack Snacks

Robots.Txt Testing Tool

John Siwicki's avatar
John Siwicki
Aug 02, 2024
∙ Paid

This week I had an issue with a robots.txt file. I have to be honest. I can't remember the last time I edited a robots file. Most of the sites I work on generally speaking we want all the content crawlable. So, we don't have a lot of advanced things set up. But, in this case. I needed to see what file Google had cached as well if one of my URLs was craw…

Keep reading with a 7-day free trial

Subscribe to Stack Snacks to keep reading this post and get 7 days of free access to the full post archives.

Already a paid subscriber? Sign in
© 2026 John Siwicki · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture