org.archive.crawler.util

Class BloomUriUniqFilter

java.lang.Object

org.archive.crawler.util.SetBasedUriUniqFilter

org.archive.crawler.util.BloomUriUniqFilter

All Implemented Interfaces:

Serializable, UriUniqFilter, org.springframework.beans.factory.InitializingBean

public class BloomUriUniqFilter
extends SetBasedUriUniqFilter
implements Serializable, org.springframework.beans.factory.InitializingBean

An implementation of an AlreadySeen list based on the MG4J BloomFilter. This implementation performs adequately without blowing out the heap through to very large numbers of URIs. See AlreadySeen. It is inherent to Bloom filters that as they get 'saturated', their false-positive rate rises. The default parameters used by this class attempt to maintain a 1-in-4 million (1 in 2^22) false-positive chance through 125 million unique inserts, which creates a filter structure about 495MB in size. You may swap in an differently-configured BloomFilter class to alter these tradeoffs.

Version:

$Date$, $Revision$

Author:

gojomo

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.archive.crawler.datamodel.UriUniqFilter

UriUniqFilter.CrawlUriReceiver

Field Summary

Fields

Modifier and Type	Field and Description
protected BloomFilter	bloom

Fields inherited from class org.archive.crawler.util.SetBasedUriUniqFilter

duplicateCount, duplicatesAtLastSample, profileLog, receiver

Constructor Summary

Constructors

Constructor and Description
BloomUriUniqFilter() Default constructor

Method Summary

Methods

Modifier and Type	Method and Description
void	afterPropertiesSet() Initializer.
void	forget(String canonical, CrawlURI item) Forget item was seen
BloomFilter	getBloomFilter()
protected boolean	setAdd(CharSequence uri)
void	setBloomFilter(BloomFilter filter)
protected long	setCount()
protected boolean	setRemove(CharSequence uri)

Methods inherited from class org.archive.crawler.util.SetBasedUriUniqFilter

add, addForce, addNow, close, count, note, pending, profileLog, requestFlush, setDestination, setProfileLog

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

bloom

protected BloomFilter bloom

Constructor Detail

BloomUriUniqFilter

public BloomUriUniqFilter()

Default constructor

Method Detail

getBloomFilter

public BloomFilter getBloomFilter()

setBloomFilter

public void setBloomFilter(BloomFilter filter)

afterPropertiesSet

public void afterPropertiesSet()

Initializer.

Specified by:

afterPropertiesSet in interface org.springframework.beans.factory.InitializingBean

forget

public void forget(String canonical,
          CrawlURI item)

Description copied from interface: UriUniqFilter

Forget item was seen

Specified by:

forget in interface UriUniqFilter

Overrides:

forget in class SetBasedUriUniqFilter

Parameters:

canonical - Usually a canonicalized version of an URI. This is the key used doing lookups, forgets and insertions on the already included list.

item - item to add.

setAdd

protected boolean setAdd(CharSequence uri)

Specified by:

setAdd in class SetBasedUriUniqFilter

setCount

protected long setCount()

Specified by:

setCount in class SetBasedUriUniqFilter

setRemove

protected boolean setRemove(CharSequence uri)

Specified by:

setRemove in class SetBasedUriUniqFilter