org.archive.modules.fetcher

Class FetchFTP

java.lang.Object

org.archive.modules.Processor

org.archive.modules.fetcher.FetchFTP

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class FetchFTP
extends Processor

Fetches documents and directory listings using FTP. This class will also try to extract FTP "links" from directory listings. For this class to archive a directory listing, the remote FTP server must support the NLIST command. Most modern FTP servers should.

Author:

pjack

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	FetchFTP.SocketFactoryWithTimeout A SocketFactory much like DefaultSocketFactory, except that the createSocket() methods that open connections support a connect timeout.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	digestAlgorithm Which algorithm (for example MD5 or SHA-1) to use to perform an on-the-fly digest hash of retrieved content-bodies.
protected FetchFTP.SocketFactoryWithTimeout	socketFactory

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
FetchFTP() Constructs a new FetchFTP.

Method Summary

Methods

Modifier and Type	Method and Description
String	getDigestAlgorithm()
boolean	getDigestContent()
boolean	getExtractFromDirs() Returns the extract.from.dirs attribute for this FetchFTP and the given curi.
boolean	getExtractParent() Returns the extract.parent attribute for this FetchFTP and the given curi.
int	getMaxFetchKBSec()
long	getMaxLengthBytes()
String	getPassword()
int	getSoTimeoutMs()
int	getTimeoutSeconds()
String	getUsername()
protected void	innerProcess(CrawlURI curi) Processes the given URI.
void	setDigestAlgorithm(String digestAlgorithm)
void	setDigestContent(boolean digest)
void	setExtractFromDirs(boolean extractFromDirs)
void	setExtractParent(boolean extractParent)
void	setMaxFetchKBSec(int rate)
void	setMaxLengthBytes(long timeout)
void	setPassword(String pw)
void	setSoTimeoutMs(int timeout)
void	setTimeoutSeconds(int timeout)
void	setUsername(String username)
protected boolean	shouldProcess(CrawlURI curi) Determines whether the given uri should be processed by this processor.

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

digestAlgorithm

protected String digestAlgorithm

Which algorithm (for example MD5 or SHA-1) to use to perform an on-the-fly digest hash of retrieved content-bodies.

socketFactory

protected FetchFTP.SocketFactoryWithTimeout socketFactory

Constructor Detail

FetchFTP

public FetchFTP()

Constructs a new FetchFTP.

Method Detail

getUsername

public String getUsername()

setUsername

public void setUsername(String username)

getPassword

public String getPassword()

setPassword

public void setPassword(String pw)

getExtractFromDirs

public boolean getExtractFromDirs()

Returns the extract.from.dirs attribute for this FetchFTP and the given curi.

Parameters:

curi - the curi whose attribute to return

Returns:

that curi's extract.from.dirs

setExtractFromDirs

public void setExtractFromDirs(boolean extractFromDirs)

getExtractParent

public boolean getExtractParent()

Returns the extract.parent attribute for this FetchFTP and the given curi.

Parameters:

curi - the curi whose attribute to return

Returns:

that curi's extract-parent

setExtractParent

public void setExtractParent(boolean extractParent)

getDigestContent

public boolean getDigestContent()

setDigestContent

public void setDigestContent(boolean digest)

getDigestAlgorithm

public String getDigestAlgorithm()

setDigestAlgorithm

public void setDigestAlgorithm(String digestAlgorithm)

getMaxLengthBytes

public long getMaxLengthBytes()

setMaxLengthBytes

public void setMaxLengthBytes(long timeout)

getMaxFetchKBSec

public int getMaxFetchKBSec()

setMaxFetchKBSec

public void setMaxFetchKBSec(int rate)

getTimeoutSeconds

public int getTimeoutSeconds()

setTimeoutSeconds

public void setTimeoutSeconds(int timeout)

getSoTimeoutMs

public int getSoTimeoutMs()

setSoTimeoutMs

public void setSoTimeoutMs(int timeout)

shouldProcess

protected boolean shouldProcess(CrawlURI curi)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

curi - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI curi)
                     throws InterruptedException

Processes the given URI. If the given URI is not an FTP URI, then this method does nothing. Otherwise an attempt is made to connect to the FTP server.

If the connection is successful, an attempt will be made to CD to the path specified in the URI. If the remote CD command succeeds, then it is assumed that the URI represents a directory. If the CD command fails, then it is assumed that the URI represents a file.

For directories, the directory listing will be fetched using the FTP LIST command, and saved to the HttpRecorder. If the extract.from.dirs attribute is set to true, then the files in the fetched list will be added to the curi as extracted FTP links. (It was easier to do that here, rather than writing a separate FTPExtractor.)

For files, the file will be fetched using the FTP RETR command, and saved to the HttpRecorder.

All file transfers (including directory listings) occur using Binary mode transfer. Also, the local passive transfer mode is always used, to play well with firewalls.

Specified by:

innerProcess in class Processor

Parameters:

curi - the curi to process

Throws:

InterruptedException - if the thread is interrupted during processing