public class CustomRobotsPolicy extends RobotsPolicy
Modifier and Type | Field and Description |
---|---|
protected org.archive.io.ReadSource |
customRobots
textual alternate robots.txt rules to follow
|
protected Robotstxt |
customRobotstxt |
protected boolean |
obeyMetaRobotsNofollow
whether to obey the 'nofollow' directive in an HTML META ROBOTS element
|
STANDARD_POLICIES
Constructor and Description |
---|
CustomRobotsPolicy() |
Modifier and Type | Method and Description |
---|---|
boolean |
allows(String userAgent,
CrawlURI curi,
Robotstxt robotstxt) |
org.archive.io.ReadSource |
getCustomRobots() |
boolean |
isObeyMetaRobotsNofollow() |
boolean |
obeyMetaRobotsNofollow() |
void |
setCustomRobots(org.archive.io.ReadSource customRobots) |
void |
setObeyMetaRobotsNofollow(boolean obeyMetaRobotsNofollow) |
getPathQuery
protected Robotstxt customRobotstxt
protected org.archive.io.ReadSource customRobots
protected boolean obeyMetaRobotsNofollow
public org.archive.io.ReadSource getCustomRobots()
public void setCustomRobots(org.archive.io.ReadSource customRobots)
public boolean isObeyMetaRobotsNofollow()
public void setObeyMetaRobotsNofollow(boolean obeyMetaRobotsNofollow)
public boolean allows(String userAgent, CrawlURI curi, Robotstxt robotstxt)
allows
in class RobotsPolicy
public boolean obeyMetaRobotsNofollow()
obeyMetaRobotsNofollow
in class RobotsPolicy
Copyright © 2003-2014 Internet Archive. All Rights Reserved.