org.archive.crawler.postprocessor

Class ReschedulingProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.crawler.postprocessor.ReschedulingProcessor

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class ReschedulingProcessor
extends Processor

The most simple forced-rescheduling step possible: use a local setting (perhaps overlaid to vary based on the URI) to set an exact future reschedule time, as a delay from now. Unless the reschedulDelaySeconds value is changed from its default, URIs are not rescheduled.

Version:

$Date: 2009-11-16 22:10:42 -0800 (Mon, 16 Nov 2009) $, $Revision: 6665 $

Author:

gojomo

Field Summary

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
ReschedulingProcessor()

Method Summary

Methods

Modifier and Type	Method and Description
long	getRescheduleDelaySeconds()
protected void	innerProcess(CrawlURI curi) Actually performs the process.
void	setRescheduleDelaySeconds(long rescheduleDelaySeconds)
protected boolean	shouldProcess(CrawlURI curi) Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ReschedulingProcessor

public ReschedulingProcessor()

Method Detail

getRescheduleDelaySeconds

public long getRescheduleDelaySeconds()

setRescheduleDelaySeconds

public void setRescheduleDelaySeconds(long rescheduleDelaySeconds)

shouldProcess

protected boolean shouldProcess(CrawlURI curi)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

curi - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI curi)

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

curi - the URI to process