org.archive.crawler.prefetch

Class CandidateScoper

java.lang.Object

org.archive.modules.Processor

org.archive.crawler.framework.Scoper

org.archive.crawler.prefetch.CandidateScoper

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class CandidateScoper
extends Scoper

Simple single-URI scoper, considers passed-in URI as candidate; sets fetchstatus negative and skips to end of processing if out-of-scope.

Contributor:

gojomo

Field Summary

Fields inherited from class org.archive.crawler.framework.Scoper

fileLogger, isRunning, loggerModule, scope

Fields inherited from class org.archive.modules.Processor

beanName, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
CandidateScoper()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	innerProcess(CrawlURI uri) Actually performs the process.
protected ProcessResult	innerProcessResult(CrawlURI curi)
protected boolean	shouldProcess(CrawlURI uri) Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.crawler.framework.Scoper

getLoggerModule, getLogToFile, getScope, isInScope, isRunning, outOfScope, setLoggerModule, setLogToFile, setScope, start, stop

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerRejectProcess, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CandidateScoper

public CandidateScoper()

Method Detail

innerProcessResult

protected ProcessResult innerProcessResult(CrawlURI curi)
                                    throws InterruptedException

Overrides:

innerProcessResult in class Processor

Throws:

InterruptedException

shouldProcess

protected boolean shouldProcess(CrawlURI uri)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

uri - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI uri)
                     throws InterruptedException

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

uri - the URI to process

Throws:

InterruptedException - if the thread is interrupted