FetchWhois (Heritrix 3 3.2.0 API)

java.lang.Object
- org.archive.modules.Processor
- - org.archive.modules.fetcher.FetchWhois

All Implemented Interfaces:

Checkpointable, CoreAttributeConstants, FetchStatusCodes, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle
```
public class FetchWhois
extends Processor
implements CoreAttributeConstants, FetchStatusCodes, org.springframework.context.Lifecycle
```
WHOIS Fetcher (RFC 3912). If this fetcher is enabled, Heritrix will attempt WHOIS lookups on the topmost assigned domain and the IP address of each URL.
WHOIS URIs

There is no pre-existing, canonical specification for WHOIS URIs. What follows is the the format that Heritrix uses, which we propose for general use.

Syntax in ABNF as used in RFC 3986 Uniform Resource Identifier (URI): Generic Syntax:

whoisurl = "whois:" [ "//" host [ ":" port ] "/" ] whoisquery

whoisquery is a url-encoded string. In ABNF, whoisquery = 1*pchar where pchar is defined in RFC 3986. host and port also as defined in RFC 3986.

To resolve a WHOIS URI which specifies host[:port], open a TCP connection to the host at the specified port (default 43), send the query (whoisquery, url-decoded) followed by CRLF, and read the response until the server closes the connection. For more details see RFC 3912.

Resolution of a "serverless" WHOIS URI, which does not specify host[:port], is implementation-dependent.

Serverless WHOIS URIs in Heritrix

For each non-WHOIS URI processed which has an authority, FetchWhois adds 1 or 2 serverless WHOIS URIs to the CrawlURI's outlinks. These are "whois:{ipAddress}" and, if the authority includes a hostname, "whois:{topLevelDomain}". See addWhoisLinks(CrawlURI).

Heritrix resolves serverless WHOIS URIs by first querying an initial server, then following referrals to other servers. In pseudocode:
```
 if query is an IPv4 address
     resolve whois://DEFAULT_IP_WHOIS_SERVER/whoisquery
```
else let domainSuffix = part of query after the last '.' (or the whole query if no '.'), url-encoded resolve whois://ULTRA_SUFFIX_WHOIS_SERVER/domainSuffix while last response refers to another server, i.e. matches regex WHOIS_SERVER_REGEX if we have a special query formatting rule for this whois server, apply it - see specialQueryTemplates resolve whois://referralServer/whoisquery
See deferOrFinishGeneric(CrawlURI, String)

Contributor:: nlevitt

Nested Classes
Modifier and Type	Class and Description
`protected static class`	`FetchWhois.UrlStatus`

Fields
Modifier and Type	Field and Description
`protected BdbModule`	`bdb`
`protected static String`	`DEFAULT_IP_WHOIS_SERVER`
`static String`	`IP_ADDRESS_REGEX`
`protected ServerCache`	`serverCache`
`protected Map<String,String>`	`specialQueryTemplates`
`protected static String`	`ULTRA_SUFFIX_WHOIS_SERVER`
`protected static String`	`WHOIS_SERVER_REGEX`

Constructors
Constructor and Description
`FetchWhois()`

Methods
Modifier and Type	Method and Description
`protected void`	`addWhoisLink(CrawlURI curi, String query)`
`protected void`	`addWhoisLinks(CrawlURI curi)` Adds outlinks to whois:{domain} and whois:{ipAddress}
`protected ProcessResult`	`deferOrFinishGeneric(CrawlURI curi, String domainOrIp)`
`protected void`	`fetch(CrawlURI curi, String whoisServer, String whoisQuery)`
`ServerCache`	`getServerCache()`
`int`	`getSoTimeoutMs()`
`protected String`	`getWhoisQuery(CrawlURI curi)`
`protected String`	`getWhoisServer(CrawlURI curi)`
`protected void`	`innerProcess(CrawlURI uri)` Actually performs the process.
`protected ProcessResult`	`innerProcessResult(CrawlURI curi)`
`boolean`	`isRunning()`
`protected String`	`makeWhoisUrl(String server, String principal)`
`void`	`setBdbModule(BdbModule bdb)`
`void`	`setServerCache(ServerCache serverCache)`
`void`	`setSoTimeoutMs(int timeout)`
`void`	`setSpecialQueryTemplates(Map<String,String> m)`
`protected boolean`	`shouldProcess(CrawlURI uri)` Determines whether the given uri should be processed by this processor.
`void`	`start()`
`void`	`stop()`

- Field Detail
  - IP_ADDRESS_REGEX
```
public static final String IP_ADDRESS_REGEX
```
    See Also:
    Constant Field Values
  - DEFAULT_IP_WHOIS_SERVER
```
protected static final String DEFAULT_IP_WHOIS_SERVER
```
    See Also:
    Constant Field Values
  - ULTRA_SUFFIX_WHOIS_SERVER
```
protected static final String ULTRA_SUFFIX_WHOIS_SERVER
```
    See Also:
    Constant Field Values
  - WHOIS_SERVER_REGEX
```
protected static String WHOIS_SERVER_REGEX
```
  - bdb
```
protected BdbModule bdb
```
  - specialQueryTemplates
```
protected Map<String,String> specialQueryTemplates
```
  - serverCache
```
protected ServerCache serverCache
```
- Constructor Detail
  - FetchWhois
```
public FetchWhois()
```
- Method Detail
  - setBdbModule
```
public void setBdbModule(BdbModule bdb)
```
  - setSpecialQueryTemplates
```
public void setSpecialQueryTemplates(Map<String,String> m)
```
  - getSoTimeoutMs
```
public int getSoTimeoutMs()
```
  - setSoTimeoutMs
```
public void setSoTimeoutMs(int timeout)
```
  - start
```
public void start()
```
    Specified by:
    
    start in interface org.springframework.context.Lifecycle
    
    Overrides:
    
    start in class Processor
  - isRunning
```
public boolean isRunning()
```
    Specified by:
    
    isRunning in interface org.springframework.context.Lifecycle
    
    Overrides:
    
    isRunning in class Processor
  - stop
```
public void stop()
```
    Specified by:
    
    stop in interface org.springframework.context.Lifecycle
    
    Overrides:
    
    stop in class Processor
  - innerProcessResult
```
protected ProcessResult innerProcessResult(CrawlURI curi)
                                    throws InterruptedException
```
    Overrides:
    
    innerProcessResult in class Processor
    
    Throws:
    
    InterruptedException
  - deferOrFinishGeneric
```
protected ProcessResult deferOrFinishGeneric(CrawlURI curi,
                                 String domainOrIp)
```
  - makeWhoisUrl
```
protected String makeWhoisUrl(String server,
                  String principal)
```
  - fetch
```
protected void fetch(CrawlURI curi,
         String whoisServer,
         String whoisQuery)
```
  - getWhoisQuery
```
protected String getWhoisQuery(CrawlURI curi)
```
  - getWhoisServer
```
protected String getWhoisServer(CrawlURI curi)
```
  - shouldProcess
```
protected boolean shouldProcess(CrawlURI uri)
```
    Description copied from class: Processor
    
    Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.
    
    Specified by:
    
    shouldProcess in class Processor
    
    Parameters:
    uri - the URI to test
    
    Returns:
    true if this processor should process that uri; false if not
  - getServerCache
```
public ServerCache getServerCache()
```
  - setServerCache
```
public void setServerCache(ServerCache serverCache)
```
  - addWhoisLink
```
protected void addWhoisLink(CrawlURI curi,
                String query)
```
  - addWhoisLinks
```
protected void addWhoisLinks(CrawlURI curi)
                      throws InterruptedException
```
    Adds outlinks to whois:{domain} and whois:{ipAddress}
    
    Throws:
    
    InterruptedException
  - innerProcess
```
protected void innerProcess(CrawlURI uri)
                     throws InterruptedException
```
    Description copied from class: Processor
    
    Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.
    
    Specified by:
    
    innerProcess in class Processor
    
    Parameters:
    uri - the URI to process
    
    Throws:
    
    InterruptedException - if the thread is interrupted

Class FetchWhois

WHOIS URIs

Serverless WHOIS URIs in Heritrix

Nested Class Summary

Field Summary

Fields inherited from class org.archive.modules.Processor

Fields inherited from interface org.archive.modules.CoreAttributeConstants

Fields inherited from interface org.archive.modules.fetcher.FetchStatusCodes

Constructor Summary

Method Summary

Methods inherited from class org.archive.modules.Processor

Methods inherited from class java.lang.Object

Field Detail

IP_ADDRESS_REGEX

DEFAULT_IP_WHOIS_SERVER

ULTRA_SUFFIX_WHOIS_SERVER

WHOIS_SERVER_REGEX

bdb

specialQueryTemplates

serverCache

Constructor Detail

FetchWhois

Method Detail

setBdbModule

setSpecialQueryTemplates

getSoTimeoutMs

setSoTimeoutMs

start

isRunning

stop

innerProcessResult

deferOrFinishGeneric

makeWhoisUrl

fetch

getWhoisQuery

getWhoisServer

shouldProcess

getServerCache

setServerCache

addWhoisLink

addWhoisLinks

innerProcess