org.archive.crawler.util

Class BdbUriUniqFilter

java.lang.Object

org.archive.crawler.util.SetBasedUriUniqFilter

org.archive.crawler.util.BdbUriUniqFilter

All Implemented Interfaces:

Serializable, Checkpointable, UriUniqFilter, org.springframework.beans.factory.BeanNameAware, org.springframework.beans.factory.DisposableBean, org.springframework.context.Lifecycle

public class BdbUriUniqFilter
extends SetBasedUriUniqFilter
implements org.springframework.context.Lifecycle, Checkpointable, org.springframework.beans.factory.BeanNameAware, org.springframework.beans.factory.DisposableBean

A BDB implementation of an AlreadySeen list. This implementation performs adequately without blowing out the heap. See AlreadySeen.

Makes keys that have URIs from same server close to each other. Mercator and 2.3.5 'Elminating Already-Visited URLs' in 'Mining the Web' by Soumen Chakrabarti talk of a two-level key with the first 24 bits a hash of the host plus port and with the last 40 as a hash of the path. Testing showed adoption of such a scheme halving lookup times (Tutilhis implementation actually concatenates scheme + host in first 24 bits and path + query in trailing 40 bits).

Version:

$Date$, $Revision$

Author:

stack

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.archive.crawler.datamodel.UriUniqFilter

UriUniqFilter.CrawlUriReceiver

Field Summary

Fields

Modifier and Type	Field and Description
protected com.sleepycat.je.Database	alreadySeen
protected BdbModule	bdb
protected String	beanName
protected AtomicLong	count
protected boolean	createdEnvironment
protected boolean	isRunning
protected long	lastCacheMiss
protected long	lastCacheMissDiff
protected Checkpoint	recoveryCheckpoint
protected com.sleepycat.je.DatabaseEntry	value
protected static com.sleepycat.je.DatabaseEntry	ZERO_LENGTH_ENTRY

Fields inherited from class org.archive.crawler.util.SetBasedUriUniqFilter

duplicateCount, duplicatesAtLastSample, profileLog, receiver

Constructor Summary

Constructors

Constructor and Description
BdbUriUniqFilter()
BdbUriUniqFilter(File bdbEnv) Constructor.
BdbUriUniqFilter(File bdbEnv, int cacheSizePercentage) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
protected static long	calcSchemeAuthorityKeyBytes(String url)
void	close() Close down any allocated resources.
static long	createKey(CharSequence uri) Create fingerprint.
void	destroy()
void	doCheckpoint(Checkpoint checkpointInProgress) Do the actual checkpoint.
void	finishCheckpoint(Checkpoint checkpointInProgress) Cleanup/unlock; need not complete for a checkpoint to be valid.
long	flush()
void	forgetAllSchemeAuthorityMatching(String url) Forget all entries that match the scheme+host+port of the given url, so that they can be crawled again if discovered again.
long	getCacheMisses()
protected BdbModule.BdbConfig	getDatabaseConfig()
long	getLastCacheMissDiff()
protected void	initialize(com.sleepycat.je.Database db) Method shared by constructors.
boolean	isRunning()
protected void	open(com.sleepycat.je.Database db)
void	reopen(com.sleepycat.je.Database db) Call after deserializing an instance of this class.
protected boolean	setAdd(CharSequence uri)
void	setBdbModule(BdbModule bdb)
void	setBeanName(String name)
protected long	setCount()
void	setRecoveryCheckpoint(Checkpoint recoveryCheckpoint) Used to inform a bean that it should restore its state from the given Checkpoint when launched (Lifecycle start()).
protected boolean	setRemove(CharSequence uri)
void	start()
void	startCheckpoint(Checkpoint checkpointInProgress) Note a checkpoint is about to begin.
void	stop()

Methods inherited from class org.archive.crawler.util.SetBasedUriUniqFilter

add, addForce, addNow, count, forget, note, pending, profileLog, requestFlush, setDestination, setProfileLog

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

createdEnvironment

protected boolean createdEnvironment

lastCacheMiss

protected long lastCacheMiss

lastCacheMissDiff

protected long lastCacheMissDiff

alreadySeen

protected transient com.sleepycat.je.Database alreadySeen

value

protected transient com.sleepycat.je.DatabaseEntry value

ZERO_LENGTH_ENTRY

protected static com.sleepycat.je.DatabaseEntry ZERO_LENGTH_ENTRY

count

protected AtomicLong count

bdb

protected BdbModule bdb

beanName

protected String beanName

isRunning

protected boolean isRunning

recoveryCheckpoint

protected Checkpoint recoveryCheckpoint

Constructor Detail

BdbUriUniqFilter

public BdbUriUniqFilter()

BdbUriUniqFilter

public BdbUriUniqFilter(File bdbEnv)
                 throws IOException

Constructor. Only used for testing; usually no-arg constructor is used, and environment provided by injected BdbModule.

Parameters:

bdbEnv - The directory that holds the bdb environment. Will make a database under here if doesn't already exit. Otherwise reopens any existing dbs.

Throws:

IOException

BdbUriUniqFilter

public BdbUriUniqFilter(File bdbEnv,
                int cacheSizePercentage)
                 throws IOException

Constructor. Only used for testing; usually no-arg constructor is used, and environment provided by injected BdbModule.

Parameters:

bdbEnv - The directory that holds the bdb environment. Will make a database under here if doesn't already exit. Otherwise reopens any existing dbs.

cacheSizePercentage - Percentage of JVM bdb allocates as its cache. Pass -1 to get default cache size.

Throws:

IOException

Method Detail

setBdbModule

public void setBdbModule(BdbModule bdb)

setBeanName

public void setBeanName(String name)

Specified by:

setBeanName in interface org.springframework.beans.factory.BeanNameAware

start

public void start()

Specified by:

start in interface org.springframework.context.Lifecycle

isRunning

public boolean isRunning()

Specified by:

isRunning in interface org.springframework.context.Lifecycle

stop

public void stop()

Specified by:

stop in interface org.springframework.context.Lifecycle

destroy

public void destroy()

Specified by:

destroy in interface org.springframework.beans.factory.DisposableBean

initialize

protected void initialize(com.sleepycat.je.Database db)
                   throws com.sleepycat.je.DatabaseException

Method shared by constructors.

Parameters:

env - Environment to use.

Throws:

com.sleepycat.je.DatabaseException

getDatabaseConfig

protected BdbModule.BdbConfig getDatabaseConfig()

Returns:

DatabaseConfig to use

reopen

public void reopen(com.sleepycat.je.Database db)
            throws com.sleepycat.je.DatabaseException

Call after deserializing an instance of this class. Will open the already seen in passed environment.

Parameters:

env - DB Environment to use.

Throws:

com.sleepycat.je.DatabaseException

open

protected void open(com.sleepycat.je.Database db)
             throws com.sleepycat.je.DatabaseException

Throws:

com.sleepycat.je.DatabaseException

close

public void close()

Description copied from interface: UriUniqFilter

Close down any allocated resources.

Specified by:

close in interface UriUniqFilter

Overrides:

close in class SetBasedUriUniqFilter

getCacheMisses

public long getCacheMisses()

getLastCacheMissDiff

public long getLastCacheMissDiff()

createKey

public static long createKey(CharSequence uri)

Create fingerprint. Pubic access so test code can access createKey.

Parameters:

uri - URI to fingerprint.

Returns:

Fingerprint of passed url.

calcSchemeAuthorityKeyBytes

protected static long calcSchemeAuthorityKeyBytes(String url)

setAdd

protected boolean setAdd(CharSequence uri)

Specified by:

setAdd in class SetBasedUriUniqFilter

setCount

protected long setCount()

Specified by:

setCount in class SetBasedUriUniqFilter

setRemove

protected boolean setRemove(CharSequence uri)

Specified by:

setRemove in class SetBasedUriUniqFilter

flush

public long flush()

startCheckpoint

public void startCheckpoint(Checkpoint checkpointInProgress)

Description copied from interface: Checkpointable

Note a checkpoint is about to begin. Most beans will ignore, but some can use this to wrap up tasks that shouldn't be half-done during a checkpoint (and hold a lock to prevent tasks from beginning during the checkpoint).

Specified by:

startCheckpoint in interface Checkpointable

Parameters:

checkpointInProgress - Checkpoint

doCheckpoint

public void doCheckpoint(Checkpoint checkpointInProgress)
                  throws IOException

Description copied from interface: Checkpointable

Do the actual checkpoint. Beans should ensure any state that they would need to recover gets saved in an appropriate place. A moderate amount of state may be saved as a JSONObject into the Checkpoint (which then keeps it in the checkpoint directory.) Larger amounts of state may be stored in a manner private to the bean, or via other collaborating beans (such as BdbModule) which checkpoint backgin database state.

Specified by:

doCheckpoint in interface Checkpointable

Parameters:

checkpointInProgress - Checkpoint

Throws:

IOException

finishCheckpoint

public void finishCheckpoint(Checkpoint checkpointInProgress)

Description copied from interface: Checkpointable

Cleanup/unlock; need not complete for a checkpoint to be valid.

Specified by:

finishCheckpoint in interface Checkpointable

Parameters:

checkpointInProgress - Checkpoint

setRecoveryCheckpoint

public void setRecoveryCheckpoint(Checkpoint recoveryCheckpoint)

Description copied from interface: Checkpointable

Used to inform a bean that it should restore its state from the given Checkpoint when launched (Lifecycle start()). May be autowired or configured after build, but before launch, for example via CheckpointService.setRecoveryCheckpointByName().

Specified by:

setRecoveryCheckpoint in interface Checkpointable

Parameters:

recoveryCheckpoint - Checkpoint

forgetAllSchemeAuthorityMatching

public void forgetAllSchemeAuthorityMatching(String url)

Forget all entries that match the scheme+host+port of the given url, so that they can be crawled again if discovered again. Expensive operation.

Because of the way keys are calculated, scheme+host+port is the only grouping of urls that is feasible to forget in bulk. See createKey(CharSequence)

WARNING: Value collisions in this 24-bit schemeAuthority part are going to be fairly common, by 'birthday problem' over 50% likely to show up with as few as 2^12 unique schemeAuthority strings. So the forgetting may forget other hosts.

Parameters:

url - whose scheme+host+port should be forgotten (remainder of url is ignored)