Package | Description |
---|---|
org.archive.crawler.frontier | |
org.archive.crawler.util |
Modifier and Type | Field and Description |
---|---|
protected UriUniqFilter |
WorkQueueFrontier.uriUniqFilter
The UriUniqFilter to use, tracking those UURIs which are
already in-process (or processed), and thus should not be
rescheduled.
|
Modifier and Type | Method and Description |
---|---|
UriUniqFilter |
WorkQueueFrontier.getUriUniqFilter() |
Modifier and Type | Method and Description |
---|---|
void |
WorkQueueFrontier.setUriUniqFilter(UriUniqFilter uriUniqFilter) |
Modifier and Type | Class and Description |
---|---|
class |
BdbUriUniqFilter
A BDB implementation of an AlreadySeen list.
|
class |
BloomUriUniqFilter
An implementation of an AlreadySeen list based on the MG4J BloomFilter.
|
class |
DiskFPMergeUriUniqFilter
Crude FPMergeUriUniqFilter using a disk data file of raw longs as the
overall FP record.
|
class |
FPMergeUriUniqFilter
UriUniqFilter based on merging FP arrays (in memory or from disk).
|
class |
FPUriUniqFilter
UriUniqFilter storing 64-bit UURI fingerprints, using an internal LongFPSet
instance.
|
class |
MemFPMergeUriUniqFilter
Crude all-in-memory FP-merging UriUniqFilter.
|
class |
MemUriUniqFilter
A purely in-memory UriUniqFilter based on a HashSet, which remembers
every full URI string it sees.
|
class |
NoopUriUniqFilter
A UriUniqFilter that doesn't actually provide any uniqueness
filter on presented items: all are passed through.
|
class |
SetBasedUriUniqFilter
UriUniqFilter based on an underlying UriSet (essentially a Set).
|
Copyright © 2003-2014 Internet Archive. All Rights Reserved.