org.archive.crawler.prefetch

Class RuntimeLimitEnforcer

java.lang.Object

org.archive.modules.Processor

org.archive.crawler.prefetch.RuntimeLimitEnforcer

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class RuntimeLimitEnforcer
extends Processor

A processor to enforce runtime limits on crawls.

This processor extends and improves on the 'max-time' capability of Heritrix. Essentially, the 'Terminate job' option functions the same way as 'max-time'. The processor however also enables pausing when the runtime is exceeded and the blocking of all URIs.

1. Pause job - Pauses the crawl. A change (increase) to the runtime duration will make it pausible to resume the crawl. Attempts to resume the crawl without modifying the run time will cause it to be immediately paused again.
2. Terminate job - Terminates the job. Equivalent to using the max-time setting on the CrawlController.
3. Block URIs - Blocks each URI with an -5002 (blocked by custom processor) fetch status code. This will cause all the URIs queued to wind up in the crawl.log.

The processor allows variable runtime based on host (or other override/refinement criteria) however using such overrides only makes sense when using 'Block URIs' as pause and terminate will have global impact once encountered anywhere.

Author:

Kristinn Sigurðsson

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RuntimeLimitEnforcer.Operation The action that the processor takes once the runtime has elapsed.

Field Summary

Fields

Modifier and Type	Field and Description
protected CrawlController	controller
protected RuntimeLimitEnforcer.Operation	expirationOperation The action that the processor takes once the runtime has elapsed.
protected static Logger	logger
protected long	runtimeSeconds The amount of time, in seconds, that the crawl will be allowed to run before this processor performs it's 'end operation.'
protected StatisticsTracker	statisticsTracker

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
RuntimeLimitEnforcer()

Method Summary

Methods

Modifier and Type	Method and Description
CrawlController	getCrawlController()
RuntimeLimitEnforcer.Operation	getExpirationOperation()
long	getRuntimeSeconds()
StatisticsTracker	getStatisticsTracker()
protected void	innerProcess(CrawlURI curi) Actually performs the process.
protected ProcessResult	innerProcessResult(CrawlURI curi)
void	setCrawlController(CrawlController controller)
void	setExpirationOperation(RuntimeLimitEnforcer.Operation op)
void	setRuntimeSeconds(long secs)
void	setStatisticsTracker(StatisticsTracker statisticsTracker)
protected boolean	shouldProcess(CrawlURI puri) Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

protected static Logger logger

runtimeSeconds

protected long runtimeSeconds

The amount of time, in seconds, that the crawl will be allowed to run before this processor performs it's 'end operation.'

expirationOperation

protected RuntimeLimitEnforcer.Operation expirationOperation

The action that the processor takes once the runtime has elapsed.

Operation: Pause job - Pauses the crawl. A change (increase) to the runtime duration will make it pausible to resume the crawl. Attempts to resume the crawl without modifying the run time will cause it to be immediately paused again.

Operation: Terminate job - Terminates the job. Equivalent to using the max-time setting on the CrawlController.

Operation: Block URIs - Blocks each URI with an -5002 (blocked by custom processor) fetch status code. This will cause all the URIs queued to wind up in the crawl.log.

controller

protected CrawlController controller

statisticsTracker

protected StatisticsTracker statisticsTracker

Constructor Detail

RuntimeLimitEnforcer

public RuntimeLimitEnforcer()

Method Detail

getRuntimeSeconds

public long getRuntimeSeconds()

setRuntimeSeconds

public void setRuntimeSeconds(long secs)

getExpirationOperation

public RuntimeLimitEnforcer.Operation getExpirationOperation()

setExpirationOperation

public void setExpirationOperation(RuntimeLimitEnforcer.Operation op)

getCrawlController

public CrawlController getCrawlController()

setCrawlController

public void setCrawlController(CrawlController controller)

getStatisticsTracker

public StatisticsTracker getStatisticsTracker()

setStatisticsTracker

public void setStatisticsTracker(StatisticsTracker statisticsTracker)

shouldProcess

protected boolean shouldProcess(CrawlURI puri)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

puri - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI curi)

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

curi - the URI to process

innerProcessResult

protected ProcessResult innerProcessResult(CrawlURI curi)
                                    throws InterruptedException

Overrides:

innerProcessResult in class Processor

Throws:

InterruptedException