Robots Exclusion Self Test
This test needs to have the crawler read the robots.txt (TODO).
- Excluded: A page the robot is
explicitly told not to visit.
- Excluded
Directory: A page several levels down a directory structure
the robot is told not to visit.
- Included: File to find.