Package org.archive.crawler.deciderules

Provides classes for a simple decision rules framework.

See: Description

Class Summary

Class	Description
ClassKeyMatchesRegexDecideRule	Rule applies configured decision to any CrawlURI class key -- i.e.

Package org.archive.crawler.deciderules Description

Provides classes for a simple decision rules framework.

Each 'step' in a decision rule set which can affect an objects ultimate fate is called a DecideRule. Each DecideRule renders a decision (possibly neutral) on the passed objects fate.

Possible decisions are:

• ACCEPT means the object is ruled-in for further processing
• REJECT means the object is ruled-out for further processing
• PASS means this particular DecideRule has no opinion

As previously outlined, each DecideRule is applied in turn; the last one to express a non-PASS preference wins.

For example, if the rules are:

• AcceptDecideRule -- ACCEPTs all (establishing a default)
• TooManyHopsDecideRule(max-hops=3) -- REJECTS all with hopsPath.length()>3, PASSes otherwise
• PrerequisiteAcceptDecideRule -- ACCEPTs any with 'P' as last hop, PASSes otherwise (this allows 'LLL's which need a 'LLLP' prerequisite a chance to complete)

Then, you have a crawl that will go 3 hops (of any type) from the seeds, with a special affordance to get prerequisites of 3-hop items (which may be 4 "hops" out)

To allow this style of decision processing to be plugged into the existing Filter and Scope slots:

• There's a DecidingFilter which takes an (ordered) map of DecideRules
• There's a DecidingScope which takes the same

See NewScopingModel for background.