org.archive.crawler.postprocessor

Class LowDiskPauseProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.crawler.postprocessor.LowDiskPauseProcessor

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

Deprecated.

Is highly system dependant. Use DiskSpaceMonitor instead.

@Deprecated
public class LowDiskPauseProcessor
extends Processor

Processor module which uses 'df -k', where available and with the expected output format (on Linux), to monitor available disk space and pause the crawl if free space on monitored filesystems falls below certain thresholds.

Field Summary

Fields

Modifier and Type	Field and Description
static Pattern	AVAILABLE_EXTRACTOR Deprecated.
protected int	contentSinceCheck Deprecated.
protected CrawlController	controller Deprecated.
protected List<String>	monitorMounts Deprecated. List of filessystem mounts whose 'available' space should be monitored via 'df' (if available).
protected int	pauseThresholdKb Deprecated. When available space on any monitored mounts falls below this threshold, the crawl will be paused.
protected int	recheckThresholdKb Deprecated. Available space via 'df' is rechecked after every increment of this much content (uncompressed) is observed.
static Pattern	VALID_DF_OUTPUT Deprecated.

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
LowDiskPauseProcessor() Deprecated.

Method Summary

Methods

Modifier and Type	Method and Description
CrawlController	getCrawlController() Deprecated.
List<String>	getMonitorMounts() Deprecated.
int	getPauseThresholdKb() Deprecated.
int	getRecheckThresholdKb() Deprecated.
protected void	innerProcess(CrawlURI uri) Deprecated. Actually performs the process.
protected ProcessResult	innerProcessResult(CrawlURI curi) Deprecated. Notes a CrawlURI's content size in its running tally.
void	setCrawlController(CrawlController controller) Deprecated.
void	setMonitorMounts(List<String> monitorMounts) Deprecated.
void	setPauseThresholdKb(int pauseThresholdKb) Deprecated.
void	setRecheckThresholdKb(int recheckThresholdKb) Deprecated.
protected boolean	shouldProcess(CrawlURI curi) Deprecated. Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

controller

protected CrawlController controller

Deprecated.

monitorMounts

protected List<String> monitorMounts

Deprecated.

List of filessystem mounts whose 'available' space should be monitored via 'df' (if available).

pauseThresholdKb

protected int pauseThresholdKb

Deprecated.

When available space on any monitored mounts falls below this threshold, the crawl will be paused.

recheckThresholdKb

protected int recheckThresholdKb

Deprecated.

Available space via 'df' is rechecked after every increment of this much content (uncompressed) is observed.

contentSinceCheck

protected int contentSinceCheck

Deprecated.

VALID_DF_OUTPUT

public static final Pattern VALID_DF_OUTPUT

Deprecated.

AVAILABLE_EXTRACTOR

public static final Pattern AVAILABLE_EXTRACTOR

Deprecated.

Constructor Detail

LowDiskPauseProcessor

public LowDiskPauseProcessor()

Deprecated.

Parameters:

name - Name of this writer.

Method Detail

getCrawlController

public CrawlController getCrawlController()

Deprecated.

setCrawlController

public void setCrawlController(CrawlController controller)

Deprecated.

getMonitorMounts

public List<String> getMonitorMounts()

Deprecated.

setMonitorMounts

public void setMonitorMounts(List<String> monitorMounts)

Deprecated.

getPauseThresholdKb

public int getPauseThresholdKb()

Deprecated.

setPauseThresholdKb

public void setPauseThresholdKb(int pauseThresholdKb)

Deprecated.

getRecheckThresholdKb

public int getRecheckThresholdKb()

Deprecated.

setRecheckThresholdKb

public void setRecheckThresholdKb(int recheckThresholdKb)

Deprecated.

shouldProcess

protected boolean shouldProcess(CrawlURI curi)

Deprecated.

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

curi - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI uri)

Deprecated.

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

uri - the URI to process

innerProcessResult

protected ProcessResult innerProcessResult(CrawlURI curi)

Deprecated.

Notes a CrawlURI's content size in its running tally. If the recheck increment of content has passed through since the last available-space check, checks available space and pauses the crawl if any monitored mounts are below the configured threshold.

Overrides:

innerProcessResult in class Processor

Parameters:

curi - CrawlURI to process.