Class | Description |
---|---|
CandidateScoper |
Simple single-URI scoper, considers passed-in URI as candidate; sets
fetchstatus negative and skips to end of processing if out-of-scope.
|
FrontierPreparer |
Processor to preload URI with as much precalculated policy-based
info as possible before it reaches frontier criticial sections.
|
PreconditionEnforcer |
Ensures the preconditions for a fetch -- such as DNS lookup
or acquiring and respecting a robots.txt policy -- are
satisfied before a URI is passed to subsequent stages.
|
Preselector |
If set to recheck the crawl's scope, gives a yes/no on whether
a CrawlURI should be processed at all.
|
QuotaEnforcer |
A simple quota enforcer.
|
RuntimeLimitEnforcer |
A processor to enforce runtime limits on crawls.
|
Enum | Description |
---|---|
RuntimeLimitEnforcer.Operation |
The action that the processor takes once the runtime has elapsed.
|
Copyright © 2003-2014 Internet Archive. All Rights Reserved.