Package | Description |
---|---|
org.archive.crawler.frontier | |
org.archive.crawler.prefetch |
Modifier and Type | Class and Description |
---|---|
class |
AssignmentLevelSurtQueueAssignmentPolicy
Create a queueKey based on the SURT authority, reduced to the
public-suffix-plus-one domain (topmost assignable domain).
|
class |
BucketQueueAssignmentPolicy
Uses the target IPs as basis for queue-assignment,
distributing them over a fixed number of sub-queues.
|
class |
HostnameQueueAssignmentPolicy
QueueAssignmentPolicy based on the hostname:port evident in the given
CrawlURI.
|
class |
IPQueueAssignmentPolicy
Uses target IP as basis for queue-assignment, unless it is unavailable,
in which case it behaves as HostnameQueueAssignmentPolicy.
|
class |
SurtAuthorityQueueAssignmentPolicy
SurtAuthorityQueueAssignmentPolicy based on the surt form of hostname.
|
class |
URIAuthorityBasedQueueAssignmentPolicy
SurtAuthorityQueueAssignmentPolicy based on the surt form of hostname.
|
Modifier and Type | Method and Description |
---|---|
QueueAssignmentPolicy |
FrontierPreparer.getQueueAssignmentPolicy() |
Modifier and Type | Method and Description |
---|---|
void |
FrontierPreparer.setQueueAssignmentPolicy(QueueAssignmentPolicy policy) |
Copyright © 2003-2014 Internet Archive. All Rights Reserved.