org.archive.crawler.postprocessor

Class DispositionProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.crawler.postprocessor.DispositionProcessor

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class DispositionProcessor
extends Processor

A step, late in the processing of a CrawlURI, for marking-up the CrawlURI with values to affect frontier disposition, and updating information that may have been affected by the fetch. This includes robots info and other stats. (Formerly called CrawlStateUpdater, when it did less.)

Version:

$Date$, $Revision$

Author:

gojomo

Field Summary

Fields

Modifier and Type	Field and Description
protected CrawlMetadata	metadata Auto-discovered module providing configured (or overridden) User-Agent value and RobotsHonoringPolicy
protected ServerCache	serverCache

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
DispositionProcessor()

Method Summary

Methods

Modifier and Type	Method and Description
float	getDelayFactor()
boolean	getForceRetire()
int	getMaxDelayMs()
int	getMaxPerHostBandwidthUsageKbSec()
CrawlMetadata	getMetadata()
int	getMinDelayMs()
int	getRespectCrawlDelayUpToSeconds()
ServerCache	getServerCache()
protected void	innerProcess(CrawlURI puri) Actually performs the process.
protected long	politenessDelayFor(CrawlURI curi) Update any scheduling structures with the new information in this CrawlURI.
void	setDelayFactor(float factor)
void	setForceRetire(boolean force)
void	setMaxDelayMs(int maxDelay)
void	setMaxPerHostBandwidthUsageKbSec(int max)
void	setMetadata(CrawlMetadata provider)
void	setMinDelayMs(int minDelay)
void	setRespectCrawlDelayUpToSeconds(int respect)
void	setServerCache(ServerCache serverCache)
protected boolean	shouldProcess(CrawlURI puri) Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

serverCache

protected ServerCache serverCache

metadata

protected CrawlMetadata metadata

Auto-discovered module providing configured (or overridden) User-Agent value and RobotsHonoringPolicy

Constructor Detail

DispositionProcessor

public DispositionProcessor()

Method Detail

getServerCache

public ServerCache getServerCache()

setServerCache

public void setServerCache(ServerCache serverCache)

getDelayFactor

public float getDelayFactor()

setDelayFactor

public void setDelayFactor(float factor)

getMinDelayMs

public int getMinDelayMs()

setMinDelayMs

public void setMinDelayMs(int minDelay)

getRespectCrawlDelayUpToSeconds

public int getRespectCrawlDelayUpToSeconds()

setRespectCrawlDelayUpToSeconds

public void setRespectCrawlDelayUpToSeconds(int respect)

getMaxDelayMs

public int getMaxDelayMs()

setMaxDelayMs

public void setMaxDelayMs(int maxDelay)

getMaxPerHostBandwidthUsageKbSec

public int getMaxPerHostBandwidthUsageKbSec()

setMaxPerHostBandwidthUsageKbSec

public void setMaxPerHostBandwidthUsageKbSec(int max)

getForceRetire

public boolean getForceRetire()

setForceRetire

public void setForceRetire(boolean force)

getMetadata

public CrawlMetadata getMetadata()

setMetadata

public void setMetadata(CrawlMetadata provider)

shouldProcess

protected boolean shouldProcess(CrawlURI puri)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

puri - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI puri)

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

puri - the URI to process

politenessDelayFor

protected long politenessDelayFor(CrawlURI curi)

Update any scheduling structures with the new information in this CrawlURI. Chiefly means make necessary arrangements for no other URIs at the same host to be visited within the appropriate politeness window.

Parameters:

curi - The CrawlURI

Returns:

millisecond politeness delay