org.archive.modules.recrawl

Class AbstractPersistProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.modules.recrawl.AbstractPersistProcessor

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

Direct Known Subclasses:

PersistProcessor

public abstract class AbstractPersistProcessor
extends Processor

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	onlyStoreIfWriteTagPresent

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
AbstractPersistProcessor()

Method Summary

Methods

Modifier and Type	Method and Description
boolean	getOnlyStoreIfWriteTagPresent()
protected boolean	hasWriteTag(CrawlURI uri)
void	setOnlyStoreIfWriteTagPresent(boolean onlyStoreIfWriteTagPresent)
protected boolean	shouldLoad(CrawlURI curi) Whether the current CrawlURI's state should be loaded
protected boolean	shouldStore(CrawlURI curi) Whether the current CrawlURI's state should be persisted (to log or direct to database)

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcess, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, shouldProcess, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

onlyStoreIfWriteTagPresent

protected boolean onlyStoreIfWriteTagPresent

See Also:

RecrawlAttributeConstants.A_WRITE_TAG

Constructor Detail

AbstractPersistProcessor

public AbstractPersistProcessor()

Method Detail

getOnlyStoreIfWriteTagPresent

public boolean getOnlyStoreIfWriteTagPresent()

setOnlyStoreIfWriteTagPresent

public void setOnlyStoreIfWriteTagPresent(boolean onlyStoreIfWriteTagPresent)

shouldStore

protected boolean shouldStore(CrawlURI curi)

Whether the current CrawlURI's state should be persisted (to log or direct to database)

Parameters:

curi - CrawlURI

Returns:

true if state should be stored; false to skip persistence

hasWriteTag

protected boolean hasWriteTag(CrawlURI uri)

Parameters:

curi -

Returns:

true if curi has WRITE_TAG in the latest fetch history (i.e. this crawl).

shouldLoad

protected boolean shouldLoad(CrawlURI curi)

Whether the current CrawlURI's state should be loaded

Parameters:

curi - CrawlURI

Returns:

true if state should be loaded; false to skip loading