org.archive.crawler.prefetch

Class FrontierPreparer

java.lang.Object

org.archive.modules.Processor

org.archive.crawler.framework.Scoper

org.archive.crawler.prefetch.FrontierPreparer

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class FrontierPreparer
extends Scoper

Processor to preload URI with as much precalculated policy-based info as possible before it reaches frontier criticial sections. Frontiers also maintain a direct reference to this class, in case they need to perform remedial preparation for URIs that do not pass through this processor on the CandidateChain.

Contributor:

gojomo

Field Summary

Fields inherited from class org.archive.crawler.framework.Scoper

fileLogger, isRunning, loggerModule, scope

Fields inherited from class org.archive.modules.Processor

beanName, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
FrontierPreparer()

Method Summary

Methods

Modifier and Type	Method and Description
protected String	canonicalize(CrawlURI cauri) Canonicalize passed CrawlURI.
UriCanonicalizationPolicy	getCanonicalizationPolicy()
String	getClassKey(CrawlURI curi)
protected int	getCost(CrawlURI curi) Return the 'cost' of a CrawlURI (how much of its associated queue's budget it depletes upon attempted processing)
CostAssignmentPolicy	getCostAssignmentPolicy()
int	getPreferenceDepthHops()
int	getPreferenceEmbedHops()
QueueAssignmentPolicy	getQueueAssignmentPolicy()
protected int	getSchedulingDirective(CrawlURI curi) Calculate the coarse, original 'schedulingDirective' prioritization for the given CrawlURI
UriPrecedencePolicy	getUriPrecedencePolicy()
protected void	innerProcess(CrawlURI curi) Actually performs the process.
void	prepare(CrawlURI curi) Apply all configured policies to CrawlURI
void	setCanonicalizationPolicy(UriCanonicalizationPolicy policy)
void	setCostAssignmentPolicy(CostAssignmentPolicy policy)
void	setPreferenceDepthHops(int depth)
void	setPreferenceEmbedHops(int pref)
void	setQueueAssignmentPolicy(QueueAssignmentPolicy policy)
void	setUriPrecedencePolicy(UriPrecedencePolicy policy)
protected boolean	shouldProcess(CrawlURI uri) Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.crawler.framework.Scoper

getLoggerModule, getLogToFile, getScope, isInScope, isRunning, outOfScope, setLoggerModule, setLogToFile, setScope, start, stop

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FrontierPreparer

public FrontierPreparer()

Method Detail

getPreferenceDepthHops

public int getPreferenceDepthHops()

setPreferenceDepthHops

public void setPreferenceDepthHops(int depth)

getPreferenceEmbedHops

public int getPreferenceEmbedHops()

setPreferenceEmbedHops

public void setPreferenceEmbedHops(int pref)

getCanonicalizationPolicy

public UriCanonicalizationPolicy getCanonicalizationPolicy()

setCanonicalizationPolicy

public void setCanonicalizationPolicy(UriCanonicalizationPolicy policy)

getQueueAssignmentPolicy

public QueueAssignmentPolicy getQueueAssignmentPolicy()

setQueueAssignmentPolicy

public void setQueueAssignmentPolicy(QueueAssignmentPolicy policy)

getUriPrecedencePolicy

public UriPrecedencePolicy getUriPrecedencePolicy()

setUriPrecedencePolicy

public void setUriPrecedencePolicy(UriPrecedencePolicy policy)

getCostAssignmentPolicy

public CostAssignmentPolicy getCostAssignmentPolicy()

setCostAssignmentPolicy

public void setCostAssignmentPolicy(CostAssignmentPolicy policy)

shouldProcess

protected boolean shouldProcess(CrawlURI uri)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

uri - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI curi)

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

curi - the URI to process

prepare

public void prepare(CrawlURI curi)

Apply all configured policies to CrawlURI

Parameters:

curi - CrawlURI

getSchedulingDirective

protected int getSchedulingDirective(CrawlURI curi)

Calculate the coarse, original 'schedulingDirective' prioritization for the given CrawlURI

Parameters:

curi -

Returns:

canonicalize

protected String canonicalize(CrawlURI cauri)

Canonicalize passed CrawlURI. This method differs from #canonicalize(UURI) in that it takes a look at the CrawlURI context possibly overriding any canonicalization effect if it could make us miss content. If canonicalization produces an URL that was 'alreadyseen', but the entry in the 'alreadyseen' database did nothing but redirect to the current URL, we won't get the current URL; we'll think we've already see it. Examples would be archive.org redirecting to www.archive.org or the inverse, www.netarkivet.net redirecting to netarkivet.net (assuming stripWWW rule enabled).

Note, this method under circumstance sets the forceFetch flag.

Parameters:

cauri - CrawlURI to examine.

Returns:

Canonicalized cacuri.

getClassKey

public String getClassKey(CrawlURI curi)

Parameters:

cauri - CrawlURI we're to get a key for.

Returns:

a String token representing a queue

getCost

protected int getCost(CrawlURI curi)

Return the 'cost' of a CrawlURI (how much of its associated queue's budget it depletes upon attempted processing)

Parameters:

curi -

Returns:

the associated cost