Package | Description |
---|---|
org.archive.modules.net |
Modifier and Type | Field and Description |
---|---|
protected Robotstxt |
CustomRobotsPolicy.customRobotstxt |
static Robotstxt |
Robotstxt.NO_ROBOTS
empty, reusable instance for all sites providing no rules
|
protected Robotstxt |
CrawlServer.robotstxt |
Modifier and Type | Method and Description |
---|---|
Robotstxt |
CrawlServer.getRobotstxt() |
Modifier and Type | Method and Description |
---|---|
boolean |
IgnoreRobotsPolicy.allows(String userAgent,
CrawlURI curi,
Robotstxt robotstxt) |
boolean |
FirstNamedRobotsPolicy.allows(String userAgent,
CrawlURI curi,
Robotstxt robotstxt) |
boolean |
MostFavoredRobotsPolicy.allows(String userAgent,
CrawlURI curi,
Robotstxt robotstxt) |
boolean |
CustomRobotsPolicy.allows(String userAgent,
CrawlURI curi,
Robotstxt robotstxt) |
boolean |
ObeyRobotsPolicy.allows(String userAgent,
CrawlURI curi,
Robotstxt robotstxt) |
abstract boolean |
RobotsPolicy.allows(String userAgent,
CrawlURI curi,
Robotstxt robotstxt) |
Copyright © 2003-2014 Internet Archive. All Rights Reserved.