Package | Description |
---|---|
org.archive.crawler.processor |
Modifier and Type | Class and Description |
---|---|
class |
HashCrawlMapper
Maps URIs to one of N crawler names by applying a hash to the
URI's (possibly-transformed) classKey.
|
class |
LexicalCrawlMapper
A simple crawl splitter/mapper, dividing up CrawlURIs/CrawlURIs
between crawlers by diverting some range of URIs to local log files
(which can then be imported to other crawlers).
|
Copyright © 2003-2014 Internet Archive. All Rights Reserved.