org.archive.modules.net

Class FirstNamedRobotsPolicy

java.lang.Object

org.archive.modules.net.RobotsPolicy

org.archive.modules.net.FirstNamedRobotsPolicy

public class FirstNamedRobotsPolicy
extends RobotsPolicy

Working from an ordered list of potential User-Agents, consisting of first the regularly-configured User-Agent and then those in the candidateUserAgents list, consider each potential agent in order. As soon as a matching (not '*' wildcard) set of directives is found, follow those. If none are followed, use the wildcard directives (if any). For example, is the usual User-Agent is 'A', and the candidateUserAgents are 'B' and 'C', any rules applying to 'A' will be followed if found. If not, rules applying to 'B' will be followed if found. If not, rules applying to 'C' will be followed if found. If not, wildcard User-Agent rules will be followed if found. Offers the option of adjusting the outgoing request to declare the User-Agent whose rules are being followed. (This option is the default and recommended.) (With an empty candidateUserAgents list, should behave same as the ObeyRobotsPolicy, but offers the a setting for obeyMetaRobotsNofollow.)

Contributor:

gojomo

Field Summary

Fields

Modifier and Type	Field and Description
protected List<String>	candidateUserAgents list of user-agents to try; if any are allowed, a URI will be crawled
protected boolean	obeyMetaRobotsNofollow whether to obey the 'nofollow' directive in an HTML META ROBOTS element
protected boolean	shouldMasquerade whether to adopt the user-agent that is allowed for the fetch

Fields inherited from class org.archive.modules.net.RobotsPolicy

STANDARD_POLICIES

Constructor Summary

Constructors

Constructor and Description
FirstNamedRobotsPolicy()

Method Summary

Methods

Modifier and Type	Method and Description
boolean	allows(String userAgent, CrawlURI curi, Robotstxt robotstxt)
List<String>	getCandidateUserAgents()
boolean	getShouldMasquerade()
boolean	isObeyMetaRobotsNofollow()
boolean	obeyMetaRobotsNofollow()
void	setCandidateUserAgents(List<String> candidateUserAgents)
void	setObeyMetaRobotsNofollow(boolean obeyMetaRobotsNofollow)
void	setShouldMasquerade(boolean shouldMasquerade)

Methods inherited from class org.archive.modules.net.RobotsPolicy

getPathQuery

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

candidateUserAgents

protected List<String> candidateUserAgents

list of user-agents to try; if any are allowed, a URI will be crawled

shouldMasquerade

protected boolean shouldMasquerade

whether to adopt the user-agent that is allowed for the fetch

obeyMetaRobotsNofollow

protected boolean obeyMetaRobotsNofollow

whether to obey the 'nofollow' directive in an HTML META ROBOTS element

Constructor Detail

FirstNamedRobotsPolicy

public FirstNamedRobotsPolicy()

Method Detail

getCandidateUserAgents

public List<String> getCandidateUserAgents()

setCandidateUserAgents

public void setCandidateUserAgents(List<String> candidateUserAgents)

getShouldMasquerade

public boolean getShouldMasquerade()

setShouldMasquerade

public void setShouldMasquerade(boolean shouldMasquerade)

isObeyMetaRobotsNofollow

public boolean isObeyMetaRobotsNofollow()

setObeyMetaRobotsNofollow

public void setObeyMetaRobotsNofollow(boolean obeyMetaRobotsNofollow)

allows

public boolean allows(String userAgent,
             CrawlURI curi,
             Robotstxt robotstxt)

Specified by:

allows in class RobotsPolicy

obeyMetaRobotsNofollow

public boolean obeyMetaRobotsNofollow()

Specified by:

obeyMetaRobotsNofollow in class RobotsPolicy