org.archive.crawler.frontier.precedence

Class PreloadedUriPrecedencePolicy

java.lang.Object

org.archive.crawler.frontier.precedence.UriPrecedencePolicy

org.archive.crawler.frontier.precedence.BaseUriPrecedencePolicy

org.archive.crawler.frontier.precedence.PreloadedUriPrecedencePolicy

All Implemented Interfaces:

Serializable, HasKeyedProperties, org.springframework.context.Lifecycle

public class PreloadedUriPrecedencePolicy
extends BaseUriPrecedencePolicy
implements org.springframework.context.Lifecycle

UriPrecedencePolicy which assigns URIs a precedence from a value that was preloaded for them into the uri-history database. NOTE: Because this is a Lifecycle bean requiring start and stop, it should not be instantiated as an anonymous inner bean. Rather, it should be a top-level named bean, then either autowired or placed-by- reference into the frontier.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected BdbModule	bdb
protected com.sleepycat.je.Database	historyDb
protected com.sleepycat.collections.StoredSortedMap<String,?>	store

Fields inherited from class org.archive.crawler.frontier.precedence.BaseUriPrecedencePolicy

kp

Constructor Summary

Constructors

Constructor and Description
PreloadedUriPrecedencePolicy()

Method Summary

Methods

Modifier and Type	Method and Description
protected int	calculatePrecedence(CrawlURI curi) Calculate the precedence value for the given URI.
UriPrecedencePolicy	getDefaultUriPrecedencePolicy()
boolean	isRunning()
protected void	mergePrior(CrawlURI curi) Merge any data from the Map stored in the URI-history store into the current instance.
void	setBdbModule(BdbModule bdb)
void	setDefaultUriPrecedencePolicy(UriPrecedencePolicy policy)
void	start()
void	stop()
void	uriScheduled(CrawlURI curi) Add a precedence value to the supplied CrawlURI, which is being scheduled onto a frontier queue for the first time.

Methods inherited from class org.archive.crawler.frontier.precedence.BaseUriPrecedencePolicy

getBasePrecedence, getKeyedProperties, setBasePrecedence

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

bdb

protected BdbModule bdb

store

protected com.sleepycat.collections.StoredSortedMap<String,?> store

historyDb

protected com.sleepycat.je.Database historyDb

Constructor Detail

PreloadedUriPrecedencePolicy

public PreloadedUriPrecedencePolicy()

Method Detail

getDefaultUriPrecedencePolicy

public UriPrecedencePolicy getDefaultUriPrecedencePolicy()

setDefaultUriPrecedencePolicy

public void setDefaultUriPrecedencePolicy(UriPrecedencePolicy policy)

setBdbModule

public void setBdbModule(BdbModule bdb)

start

public void start()

Specified by:

start in interface org.springframework.context.Lifecycle

isRunning

public boolean isRunning()

Specified by:

isRunning in interface org.springframework.context.Lifecycle

stop

public void stop()

Specified by:

stop in interface org.springframework.context.Lifecycle

uriScheduled

public void uriScheduled(CrawlURI curi)

Description copied from class: UriPrecedencePolicy

Add a precedence value to the supplied CrawlURI, which is being scheduled onto a frontier queue for the first time.

Overrides:

uriScheduled in class BaseUriPrecedencePolicy

Parameters:

curi - CrawlURI to assign a precedence value

calculatePrecedence

protected int calculatePrecedence(CrawlURI curi)

Description copied from class: BaseUriPrecedencePolicy

Calculate the precedence value for the given URI.

Overrides:

calculatePrecedence in class BaseUriPrecedencePolicy

Parameters:

curi - CrawlURI to evaluate

Returns:

int precedence for URI

mergePrior

protected void mergePrior(CrawlURI curi)

Merge any data from the Map stored in the URI-history store into the current instance. TODO: ensure compatibility with use of PersistLoadProcessor; suppress double-loading

Parameters:

curi - CrawlURI to receive prior state data