Package | Description |
---|---|
org.archive.modules |
The beginnings of a refactored settings framework.
|
org.archive.modules.net |
Modifier and Type | Field and Description |
---|---|
protected Map<String,RobotsPolicy> |
CrawlMetadata.availableRobotsPolicies
Map of all available RobotsPolicies, by name, to choose from.
|
Modifier and Type | Method and Description |
---|---|
RobotsPolicy |
CrawlMetadata.getRobotsPolicy()
Get the currently-effective RobotsPolicy, as specified by the
string name and chosen from the full available map.
|
Modifier and Type | Method and Description |
---|---|
Map<String,RobotsPolicy> |
CrawlMetadata.getAvailableRobotsPolicies() |
Modifier and Type | Method and Description |
---|---|
void |
CrawlMetadata.setAvailableRobotsPolicies(Map<String,RobotsPolicy> policies) |
Modifier and Type | Class and Description |
---|---|
class |
CustomRobotsPolicy
Follow a custom-written robots policy, rather than the site's own declarations
Does not support overlays of different custom-robots; instead it is
recommended each custom policy be declared as a separate bean, with a
distinct name.
|
class |
FirstNamedRobotsPolicy
Working from an ordered list of potential User-Agents, consisting of first
the regularly-configured User-Agent and then those in the candidateUserAgents
list, consider each potential agent in order.
|
class |
IgnoreRobotsPolicy
Policy to ignore robots.
|
class |
MostFavoredRobotsPolicy
Follow a most-favored robots policy -- allowing an URL if either the
conventionally-configured User-Agent, or any of a number of alternate
User-Agents (from the candidateUserAgents list) would be allowed.
|
class |
ObeyRobotsPolicy
Classic obey-robots-as-declared policy.
|
Modifier and Type | Field and Description |
---|---|
static RobotsPolicy |
IgnoreRobotsPolicy.INSTANCE |
static RobotsPolicy |
ObeyRobotsPolicy.INSTANCE |
Modifier and Type | Field and Description |
---|---|
static Map<String,RobotsPolicy> |
RobotsPolicy.STANDARD_POLICIES |
Copyright © 2003-2014 Internet Archive. All Rights Reserved.