org.archive.modules.writer

Class ARCWriterProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.modules.writer.WriterPoolProcessor

org.archive.modules.writer.ARCWriterProcessor

All Implemented Interfaces:

Checkpointable, org.archive.io.WriterPoolSettings, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class ARCWriterProcessor
extends WriterPoolProcessor

Processor module for writing the results of successful fetches (and perhaps someday, certain kinds of network failures) to the Internet Archive ARC file format. Assumption is that there is only one of these ARCWriterProcessors per Heritrix instance.

Author:

Parker Thompson

Field Summary

Fields inherited from class org.archive.modules.writer.WriterPoolProcessor

ANNOTATION_UNWRITTEN, compress, directory, frequentFlushes, maxFileSizeBytes, maxTotalBytesToWrite, maxWaitForIdleMs, poolMaxActive, prefix, serverCache, skipIdenticalDigests, startNewFilesOnCheckpoint, storePaths, template, writeBufferSize

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
ARCWriterProcessor()

Method Summary

Methods

Modifier and Type	Method and Description
long	getDefaultMaxFileSize()
List<ConfigPath>	getDefaultStorePaths()
List<String>	getMetadata()
protected ProcessResult	innerProcessResult(CrawlURI puri) Writes a CrawlURI and its associated data to store file.
protected void	setupPool(AtomicInteger serialNo) Set up pool of files.
protected ProcessResult	write(CrawlURI curi, long recordLength, InputStream in, String ip)

Methods inherited from class org.archive.modules.writer.WriterPoolProcessor

calcOutputDirs, checkBytesWritten, copyForwardWriteTagIfDupe, doCheckpoint, fromCheckpointJson, getCompress, getDirectory, getFrequentFlushes, getHostAddress, getMaxFileSizeBytes, getMaxTotalBytesToWrite, getMaxWaitForIdleMs, getMetadataProvider, getPool, getPoolMaxActive, getPrefix, getSerialNo, getServerCache, getSkipIdenticalDigests, getStartNewFilesOnCheckpoint, getStorePaths, getTemplate, getTotalBytesWritten, getWriteBufferSize, innerProcess, innerRejectProcess, setCompress, setDirectory, setFrequentFlushes, setMaxFileSizeBytes, setMaxTotalBytesToWrite, setMaxWaitForIdleMs, setMetadataProvider, setPool, setPoolMaxActive, setPrefix, setServerCache, setSkipIdenticalDigests, setStartNewFilesOnCheckpoint, setStorePaths, setTemplate, setTotalBytesWritten, setWriteBufferSize, shouldProcess, shouldWrite, start, stop, toCheckpointJson

Methods inherited from class org.archive.modules.Processor

finishCheckpoint, flattenVia, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.springframework.context.Lifecycle

isRunning

Methods inherited from interface org.archive.checkpointing.Checkpointable

finishCheckpoint, setRecoveryCheckpoint, startCheckpoint

Constructor Detail

ARCWriterProcessor

public ARCWriterProcessor()

Method Detail

getDefaultMaxFileSize

public long getDefaultMaxFileSize()

Specified by:

getDefaultMaxFileSize in class WriterPoolProcessor

getDefaultStorePaths

public List<ConfigPath> getDefaultStorePaths()

Specified by:

getDefaultStorePaths in class WriterPoolProcessor

setupPool

protected void setupPool(AtomicInteger serialNo)

Description copied from class: WriterPoolProcessor

Set up pool of files.

Specified by:

setupPool in class WriterPoolProcessor

innerProcessResult

protected ProcessResult innerProcessResult(CrawlURI puri)

Writes a CrawlURI and its associated data to store file. Currently this method understands the following uri types: dns, http, and https.

Specified by:

innerProcessResult in class WriterPoolProcessor

Parameters:

curi - CrawlURI to process.

write

protected ProcessResult write(CrawlURI curi,
                  long recordLength,
                  InputStream in,
                  String ip)
                       throws IOException

Throws:

IOException

getMetadata

public List<String> getMetadata()

Specified by:

getMetadata in interface org.archive.io.WriterPoolSettings

Specified by:

getMetadata in class WriterPoolProcessor