org.archive.modules.extractor

Class HTTPContentDigest

java.lang.Object

org.archive.modules.Processor

org.archive.modules.extractor.HTTPContentDigest

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class HTTPContentDigest
extends Processor

A processor for calculating custom HTTP content digests in place of the default (if any) computed by the HTTP fetcher processors.

This processor allows the user to specify a regular expression called strip-reg-expr. Any segment of a document (text only, binary files will be skipped) that matches this regular expression will by rewritten with the blank character (character 32 in the ANSI character set) for the purpose of the digest this has no effect on the document for subsequent processing or archiving.

NOTE: Content digest only accounts for the document body, not headers.

The operator will also be able to specify a maximum length for documents being evaluated by this processors. Documents exceeding that length will be ignored.

To further discriminate by file type or URL, an operator should use the override and refinement options.

It is generally recommended that this recalculation only be performed when absolutely needed (because of stripping data that changes automatically each time the URL is fetched) as this is an expensive operation. NOTE: This processor may open a ReplayCharSequence from the CrawlURI's Recorder, without closing that ReplayCharSequence, to allow reuse by later processors in sequence. In the usual (Heritrix) case, a call after all processing to the Recorder's endReplays() method ensures timely close of any reused ReplayCharSequences. Reuse of this processor elsewhere should ensure a similar cleanup call to Recorder.endReplays() occurs.

Author:

Kristinn Sigurdsson

Field Summary

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
HTTPContentDigest() Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
long	getMaxSizeToDigest()
String	getStripRegex()
protected void	innerProcess(CrawlURI curi) Actually performs the process.
void	setMaxSizeToDigest(long threshold)
void	setStripRegex(String regex)
protected boolean	shouldProcess(CrawlURI uri) Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HTTPContentDigest

public HTTPContentDigest()

Constructor.

Method Detail

getStripRegex

public String getStripRegex()

setStripRegex

public void setStripRegex(String regex)

getMaxSizeToDigest

public long getMaxSizeToDigest()

setMaxSizeToDigest

public void setMaxSizeToDigest(long threshold)

shouldProcess

protected boolean shouldProcess(CrawlURI uri)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

uri - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI curi)
                     throws InterruptedException

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

curi - the URI to process

Throws:

InterruptedException - if the thread is interrupted