Package | Description |
---|---|
org.archive.crawler.frontier |
Modifier and Type | Class and Description |
---|---|
class |
BdbFrontier
A Frontier using several BerkeleyDB JE Databases to hold its record of
known hosts (queues), and pending URIs.
|
Modifier and Type | Method and Description |
---|---|
protected abstract void |
WorkQueue.deleteItem(WorkQueueFrontier frontier,
CrawlURI item)
Removes the given item from the queue.
|
protected void |
BdbWorkQueue.deleteItem(WorkQueueFrontier frontier,
CrawlURI peekItem) |
long |
WorkQueue.deleteMatching(WorkQueueFrontier frontier,
String match)
Delete URIs matching the given pattern from this queue.
|
protected abstract long |
WorkQueue.deleteMatchingFromQueue(WorkQueueFrontier frontier,
String match)
Delete URIs matching the given pattern from this queue.
|
protected long |
BdbWorkQueue.deleteMatchingFromQueue(WorkQueueFrontier frontier,
String match) |
protected void |
WorkQueue.dequeue(WorkQueueFrontier frontier,
CrawlURI expected)
Remove the peekItem from the queue and adjusts the count.
|
protected long |
WorkQueue.enqueue(WorkQueueFrontier frontier,
CrawlURI curi)
Add the given CrawlURI, noting its addition in running count.
|
protected abstract void |
WorkQueue.insertItem(WorkQueueFrontier frontier,
CrawlURI curi,
boolean overwriteIfPresent)
Insert the given curi, whether it is already present or not.
|
protected void |
BdbWorkQueue.insertItem(WorkQueueFrontier frontier,
CrawlURI curi,
boolean overwriteIfPresent) |
CrawlURI |
WorkQueue.peek(WorkQueueFrontier frontier)
Return the topmost queue item -- and remember it,
such that even later higher-priority inserts don't
change it.
|
protected abstract CrawlURI |
WorkQueue.peekItem(WorkQueueFrontier frontier)
Returns first item from queue (does not delete)
|
protected CrawlURI |
BdbWorkQueue.peekItem(WorkQueueFrontier frontier) |
protected void |
WorkQueue.update(WorkQueueFrontier frontier,
CrawlURI curi)
Update the given CrawlURI, which should already be present.
|
Copyright © 2003-2014 Internet Archive. All Rights Reserved.