Robots Exclusion Self Test

This test needs to have the crawler read the robots.txt (TODO).