org.archive.crawler.postprocessor

Class LinksScoper

java.lang.Object

org.archive.modules.Processor

org.archive.crawler.framework.Scoper

org.archive.crawler.postprocessor.LinksScoper

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

Deprecated.

Use CandidatesProcessor and CandidateChain/CandidateScoper instead

public class LinksScoper
extends Scoper

Determine which extracted links are within scope. TODO: To test scope, requires that Link be converted to a CrawlURI. Make it so don't have to make a CrawlURI to test if Link is in scope.

Since this scoper has to create CrawlURIs, no sense discarding them since later in the processing chain CrawlURIs rather than Links are whats needed scheduling extracted links w/ the Frontier (Frontier#schedule expects CrawlURI, not Link). This class replaces Links w/ the CrawlURI that wraps the Link in the CrawlURI.

Author:

gojomo, stack

Field Summary

Fields inherited from class org.archive.crawler.framework.Scoper

fileLogger, isRunning, loggerModule, scope

Fields inherited from class org.archive.modules.Processor

beanName, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
LinksScoper() Deprecated.

Method Summary

Methods

Modifier and Type	Method and Description
DecideRule	getLogRejectsRule() Deprecated.
int	getPreferenceDepthHops() Deprecated.
protected int	getSchedulingFor(CrawlURI curi, Link wref, int preferenceDepthHops) Deprecated. Determine scheduling for the curi.
boolean	getSeedsRedirectNewSeeds() Deprecated.
protected void	handlePrerequisite(CrawlURI curi) Deprecated. The CrawlURI has a prerequisite; apply scoping and update Link to CrawlURI in manner analogous to outlink handling.
protected void	innerProcess(CrawlURI puri) Deprecated. Actually performs the process.
protected void	outOfScope(CrawlURI caUri) Deprecated. Called when a CrawlURI is ruled out of scope.
void	setLogRejectsRule(DecideRule rule) Deprecated.
void	setPreferenceDepthHops(int depth) Deprecated.
void	setSeedsRedirectNewSeeds(boolean redirect) Deprecated.
protected boolean	shouldProcess(CrawlURI puri) Deprecated. Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.crawler.framework.Scoper

getLoggerModule, getLogToFile, getScope, isInScope, isRunning, setLoggerModule, setLogToFile, setScope, start, stop

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LinksScoper

public LinksScoper()

Deprecated.

Parameters:

name - Name of this filter.

Method Detail

getSeedsRedirectNewSeeds

public boolean getSeedsRedirectNewSeeds()

Deprecated.

setSeedsRedirectNewSeeds

public void setSeedsRedirectNewSeeds(boolean redirect)

Deprecated.

getLogRejectsRule

public DecideRule getLogRejectsRule()

Deprecated.

setLogRejectsRule

public void setLogRejectsRule(DecideRule rule)

Deprecated.

getPreferenceDepthHops

public int getPreferenceDepthHops()

Deprecated.

setPreferenceDepthHops

public void setPreferenceDepthHops(int depth)

Deprecated.

shouldProcess

protected boolean shouldProcess(CrawlURI puri)

Deprecated.

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

puri - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI puri)

Deprecated.

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

puri - the URI to process

handlePrerequisite

protected void handlePrerequisite(CrawlURI curi)

Deprecated.

The CrawlURI has a prerequisite; apply scoping and update Link to CrawlURI in manner analogous to outlink handling.

Parameters:

curi - CrawlURI with prereq to consider

outOfScope

protected void outOfScope(CrawlURI caUri)

Deprecated.

Description copied from class: Scoper

Called when a CrawlURI is ruled out of scope. Override if you don't want logs as coming from this class.

Overrides:

outOfScope in class Scoper

Parameters:

caUri - CrawlURI that is out of scope.

getSchedulingFor

protected int getSchedulingFor(CrawlURI curi,
                   Link wref,
                   int preferenceDepthHops)

Deprecated.

Determine scheduling for the curi. As with the LinksScoper in general, this only handles extracted links, seeds do not pass through here, but are given MEDIUM priority. Imports into the frontier similarly do not pass through here, but are given NORMAL priority.