org.archive.modules.recrawl

Class PersistLogProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.modules.recrawl.AbstractPersistProcessor

org.archive.modules.recrawl.PersistProcessor

org.archive.modules.recrawl.PersistLogProcessor

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class PersistLogProcessor
extends PersistProcessor
implements Checkpointable, org.springframework.context.Lifecycle

Log CrawlURI attributes from latest fetch for consultation by a later recrawl. Log must be imported into alternate data structure in order to be consulted.

Version:

$Date: 2006-09-25 20:19:54 +0000 (Mon, 25 Sep 2006) $, $Revision: 4654 $

Author:

gojomo

Field Summary

Fields

Modifier and Type	Field and Description
protected CrawlerJournal	log
protected ConfigPath	logFile

Fields inherited from class org.archive.modules.recrawl.PersistProcessor

HISTORY_DB_CONFIG, URI_HISTORY_DBNAME

Fields inherited from class org.archive.modules.recrawl.AbstractPersistProcessor

onlyStoreIfWriteTagPresent

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
PersistLogProcessor()

Method Summary

Methods

Modifier and Type	Method and Description
void	doCheckpoint(Checkpoint checkpointInProgress) Do the actual checkpoint.
void	finishCheckpoint(Checkpoint checkpointInProgress) Cleanup/unlock; need not complete for a checkpoint to be valid.
ConfigPath	getLogFile()
protected void	innerProcess(CrawlURI curi) Actually performs the process.
boolean	isRunning()
void	setLogFile(ConfigPath path)
protected boolean	shouldProcess(CrawlURI uri) Determines whether the given uri should be processed by this processor.
void	start()
void	startCheckpoint(Checkpoint checkpointInProgress) Note a checkpoint is about to begin.
void	stop()

Methods inherited from class org.archive.modules.recrawl.PersistProcessor

copyPersistSourceToHistoryMap, copyPersistSourceToHistoryMap, main, persistKeyFor, persistKeyFor, populatePersistEnv, setupCopyEnvironment, setupCopyEnvironment

Methods inherited from class org.archive.modules.recrawl.AbstractPersistProcessor

getOnlyStoreIfWriteTagPresent, hasWriteTag, setOnlyStoreIfWriteTagPresent, shouldLoad, shouldStore

Methods inherited from class org.archive.modules.Processor

flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.archive.checkpointing.Checkpointable

setRecoveryCheckpoint

Field Detail

log

protected CrawlerJournal log

logFile

protected ConfigPath logFile

Constructor Detail

PersistLogProcessor

public PersistLogProcessor()

Method Detail

getLogFile

public ConfigPath getLogFile()

setLogFile

public void setLogFile(ConfigPath path)

start

public void start()

Specified by:

start in interface org.springframework.context.Lifecycle

Overrides:

start in class Processor

isRunning

public boolean isRunning()

Specified by:

isRunning in interface org.springframework.context.Lifecycle

Overrides:

isRunning in class Processor

stop

public void stop()

Specified by:

stop in interface org.springframework.context.Lifecycle

Overrides:

stop in class Processor

innerProcess

protected void innerProcess(CrawlURI curi)

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

curi - the URI to process

startCheckpoint

public void startCheckpoint(Checkpoint checkpointInProgress)

Description copied from interface: Checkpointable

Note a checkpoint is about to begin. Most beans will ignore, but some can use this to wrap up tasks that shouldn't be half-done during a checkpoint (and hold a lock to prevent tasks from beginning during the checkpoint).

Specified by:

startCheckpoint in interface Checkpointable

Overrides:

startCheckpoint in class Processor

Parameters:

checkpointInProgress - Checkpoint

doCheckpoint

public void doCheckpoint(Checkpoint checkpointInProgress)
                  throws IOException

Description copied from interface: Checkpointable

Do the actual checkpoint. Beans should ensure any state that they would need to recover gets saved in an appropriate place. A moderate amount of state may be saved as a JSONObject into the Checkpoint (which then keeps it in the checkpoint directory.) Larger amounts of state may be stored in a manner private to the bean, or via other collaborating beans (such as BdbModule) which checkpoint backgin database state.

Specified by:

doCheckpoint in interface Checkpointable

Overrides:

doCheckpoint in class Processor

Parameters:

checkpointInProgress - Checkpoint

Throws:

IOException

finishCheckpoint

public void finishCheckpoint(Checkpoint checkpointInProgress)

Description copied from interface: Checkpointable

Cleanup/unlock; need not complete for a checkpoint to be valid.

Specified by:

finishCheckpoint in interface Checkpointable

Overrides:

finishCheckpoint in class Processor

Parameters:

checkpointInProgress - Checkpoint

shouldProcess

protected boolean shouldProcess(CrawlURI uri)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

uri - the URI to test

Returns:

true if this processor should process that uri; false if not