org.archive.modules.recrawl

Class ContentDigestHistoryLoader

java.lang.Object

org.archive.modules.Processor

org.archive.modules.recrawl.ContentDigestHistoryLoader

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class ContentDigestHistoryLoader
extends Processor

Field Summary

Fields

Modifier and Type	Field and Description
protected AbstractContentDigestHistory	contentDigestHistory

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
ContentDigestHistoryLoader()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	innerProcess(CrawlURI curi) Actually performs the process.
void	setContentDigestHistory(AbstractContentDigestHistory contentDigestHistory)
protected boolean	shouldProcess(CrawlURI uri) Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

contentDigestHistory

protected AbstractContentDigestHistory contentDigestHistory

Constructor Detail

ContentDigestHistoryLoader

public ContentDigestHistoryLoader()

Method Detail

setContentDigestHistory

public void setContentDigestHistory(AbstractContentDigestHistory contentDigestHistory)

shouldProcess

protected boolean shouldProcess(CrawlURI uri)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

uri - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI curi)
                     throws InterruptedException

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

curi - the URI to process

Throws:

InterruptedException - if the thread is interrupted