Get the guide by e-mail
What is inside
- The full reference entry "Robots.txt", as reviewed for 2026
- A step-by-step checklist you can work through on your own site
- A worksheet / template to fill in
- A table of tools for the job, free ones first
- The three most common mistakes and how to fix each
Contents
- What is robots.txt, and what does it control?
- When is a crawl restriction useful?
- Read the syntax and select the applicable rules
- Build a minimal ruleset with tested examples
- Publish the file at the correct origin
- Avoid indexing, security and rendering mistakes
- How do you test changes and investigate a blocked URL?
Prefer to read online? The entry is free too: Robots.txt.