org.archive.crawler.postprocessor

Class CandidatesProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.crawler.postprocessor.CandidatesProcessor

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class CandidatesProcessor
extends Processor

Processor which sends all candidate outlinks through the CandidateChain, scheduling those with non-negative status codes to the frontier. Also performs special handling for 'discovered seeds' -- URIs, as with redirects from seeds, that may deserve special treatment to expand the scope.

Field Summary

Fields

Modifier and Type	Field and Description
protected CandidateChain	candidateChain Candidate chain
protected Frontier	frontier The frontier to use.
protected CrawlerLoggerModule	loggerModule
protected SeedModule	seeds
protected static int	SEEDS_REDIRECT_NEW_SEEDS_MAX_HOPS
protected SheetOverlaysManager	sheetOverlaysManager

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
CandidatesProcessor() Usual no-argument constructor

Method Summary

Methods

Modifier and Type	Method and Description
protected boolean	checkForSeedPromotion(CrawlURI curi) Check if the URI needs special 'discovered seed' treatment.
CandidateChain	getCandidateChain()
Frontier	getFrontier()
CrawlerLoggerModule	getLoggerModule()
boolean	getProcessErrorOutlinks()
SeedModule	getSeeds()
boolean	getSeedsRedirectNewSeeds()
SheetOverlaysManager	getSheetOverlaysManager()
protected void	innerProcess(CrawlURI curi) Run candidates chain on each of (1) any prerequisite, if present; (2) any outCandidates, if present; (3) all outlinks, if appropriate
protected int	runCandidateChain(CrawlURI candidate, CrawlURI source) Run candidatesChain on a single candidate CrawlURI; if its reported status is nonnegative, schedule to frontier.
void	setCandidateChain(CandidateChain candidateChain)
void	setFrontier(Frontier frontier)
void	setLoggerModule(CrawlerLoggerModule loggerModule)
void	setProcessErrorOutlinks(boolean errorOutlinks)
void	setSeeds(SeedModule seeds)
void	setSeedsRedirectNewSeeds(boolean redirect)
void	setSheetOverlaysManager(SheetOverlaysManager sheetOverlaysManager)
protected boolean	shouldProcess(CrawlURI puri) Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

candidateChain

protected CandidateChain candidateChain

Candidate chain

frontier

protected Frontier frontier

The frontier to use.

loggerModule

protected CrawlerLoggerModule loggerModule

SEEDS_REDIRECT_NEW_SEEDS_MAX_HOPS

protected static final int SEEDS_REDIRECT_NEW_SEEDS_MAX_HOPS

See Also:

Constant Field Values

seeds

protected SeedModule seeds

sheetOverlaysManager

protected SheetOverlaysManager sheetOverlaysManager

Constructor Detail

CandidatesProcessor

public CandidatesProcessor()

Usual no-argument constructor

Method Detail

getCandidateChain

public CandidateChain getCandidateChain()

setCandidateChain

public void setCandidateChain(CandidateChain candidateChain)

getFrontier

public Frontier getFrontier()

setFrontier

public void setFrontier(Frontier frontier)

getLoggerModule

public CrawlerLoggerModule getLoggerModule()

setLoggerModule

public void setLoggerModule(CrawlerLoggerModule loggerModule)

getSeedsRedirectNewSeeds

public boolean getSeedsRedirectNewSeeds()

setSeedsRedirectNewSeeds

public void setSeedsRedirectNewSeeds(boolean redirect)

getProcessErrorOutlinks

public boolean getProcessErrorOutlinks()

setProcessErrorOutlinks

public void setProcessErrorOutlinks(boolean errorOutlinks)

getSeeds

public SeedModule getSeeds()

setSeeds

public void setSeeds(SeedModule seeds)

getSheetOverlaysManager

public SheetOverlaysManager getSheetOverlaysManager()

setSheetOverlaysManager

public void setSheetOverlaysManager(SheetOverlaysManager sheetOverlaysManager)

shouldProcess

protected boolean shouldProcess(CrawlURI puri)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

puri - the URI to test

Returns:

true if this processor should process that uri; false if not

runCandidateChain

protected int runCandidateChain(CrawlURI candidate,
                    CrawlURI source)
                         throws InterruptedException

Run candidatesChain on a single candidate CrawlURI; if its reported status is nonnegative, schedule to frontier. Also applies special handling of discovered URIs that by convention we want to treat as seeds (which then may be scheduled indirectly via addSeed).

Parameters:

candidate - CrawlURI to consider

source - CrawlURI from which candidate was discovered/derived

Returns:

candidate's status code at end of candidate chain execution

Throws:

InterruptedException

innerProcess

protected void innerProcess(CrawlURI curi)
                     throws InterruptedException

Run candidates chain on each of (1) any prerequisite, if present; (2) any outCandidates, if present; (3) all outlinks, if appropriate

Specified by:

innerProcess in class Processor

Parameters:

curi - the URI to process

Throws:

InterruptedException - if the thread is interrupted

See Also:

Processor.innerProcess(org.archive.modules.CrawlURI)

checkForSeedPromotion

protected boolean checkForSeedPromotion(CrawlURI curi)

Check if the URI needs special 'discovered seed' treatment.

Parameters:

curi -