public class Robotstxt extends Object implements Serializable
Modifier and Type | Field and Description |
---|---|
protected Map<String,RobotsDirectives> |
agentsToDirectives |
protected boolean |
hasErrors |
protected static long |
MAX_SIZE |
protected LinkedList<String> |
namedUserAgents |
protected static RobotsDirectives |
NO_DIRECTIVES |
static Robotstxt |
NO_ROBOTS
empty, reusable instance for all sites providing no rules
|
protected RobotsDirectives |
wildcardDirectives |
Constructor and Description |
---|
Robotstxt() |
Robotstxt(BufferedReader reader) |
Robotstxt(org.archive.io.ReadSource customRobots) |
Modifier and Type | Method and Description |
---|---|
boolean |
allowsAll()
Does this policy effectively allow everything? (No
disallows or timing (crawl-delay) directives?)
|
static void |
autoregisterTo(AutoKryo kryo) |
RobotsDirectives |
getDirectivesFor(String userAgent)
Return directives to use for the given User-Agent, resorting to wildcard
rules or the default no-directives if necessary.
|
RobotsDirectives |
getDirectivesFor(String ua,
boolean useFallbacks)
Return the RobotsDirectives, if any, appropriate for the given User-Agent
string.
|
List<String> |
getNamedUserAgents() |
protected void |
initializeFromReader(BufferedReader reader) |
protected static final long MAX_SIZE
protected LinkedList<String> namedUserAgents
protected Map<String,RobotsDirectives> agentsToDirectives
protected RobotsDirectives wildcardDirectives
protected boolean hasErrors
protected static RobotsDirectives NO_DIRECTIVES
public static Robotstxt NO_ROBOTS
public Robotstxt()
public Robotstxt(BufferedReader reader) throws IOException
IOException
public Robotstxt(org.archive.io.ReadSource customRobots)
protected void initializeFromReader(BufferedReader reader) throws IOException
IOException
public boolean allowsAll()
public RobotsDirectives getDirectivesFor(String ua, boolean useFallbacks)
ua
- String User-Agent to lookupuseFallbacks
- if true, fall-back to wildcard directives or
default allow as neededpublic RobotsDirectives getDirectivesFor(String userAgent)
userAgent
- String User-Agent to lookuppublic static void autoregisterTo(AutoKryo kryo)
Copyright © 2003-2014 Internet Archive. All Rights Reserved.