Package | Description |
---|---|
org.archive.crawler.framework | |
org.archive.crawler.frontier | |
org.archive.crawler.prefetch | |
org.archive.modules.net |
Modifier and Type | Interface and Description |
---|---|
static interface |
Frontier.FrontierGroup
Generic interface representing the internal groupings
of a Frontier's URIs -- usually queues.
|
Modifier and Type | Class and Description |
---|---|
class |
BdbWorkQueue
One independent queue of items with the same 'classKey' (eg host).
|
class |
WorkQueue
A single queue of related URIs to visit, grouped by a classKey
(typically "hostname:port" or similar)
|
Modifier and Type | Method and Description |
---|---|
protected boolean |
QuotaEnforcer.checkQuotas(CrawlURI curi,
FetchStats.HasFetchStats hasStats,
int CAT)
Check all quotas for the given substats and category (server, host, or
group).
|
Modifier and Type | Class and Description |
---|---|
class |
CrawlHost
Represents a single remote "host".
|
class |
CrawlServer
Represents a single remote "server".
|
Copyright © 2003-2014 Internet Archive. All Rights Reserved.