org.archive.modules.deciderules.surt

Class SurtPrefixedDecideRule

java.lang.Object

org.archive.modules.deciderules.DecideRule

org.archive.modules.deciderules.PredicatedDecideRule

org.archive.modules.deciderules.surt.SurtPrefixedDecideRule

All Implemented Interfaces:

Serializable, EventListener, Checkpointable, SeedListener, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.ApplicationListener<org.springframework.context.ApplicationEvent>

Direct Known Subclasses:

NotSurtPrefixedDecideRule, OnDomainsDecideRule, OnHostsDecideRule

public class SurtPrefixedDecideRule
extends PredicatedDecideRule
implements SeedListener, org.springframework.context.ApplicationListener<org.springframework.context.ApplicationEvent>, Checkpointable, org.springframework.beans.factory.BeanNameAware

Rule applies configured decision to any URIs that, when expressed in SURT form, begin with one of the prefixes in the configured set. The set can be filled with SURT prefixes implied or listed in the seeds file, or another external file. The "also-check-via" option to implement "one hop off" scoping derives from a contribution by Shifra Raffel of the California Digital Library.

Author:

gojomo

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected String	beanName
protected Checkpoint	recoveryCheckpoint
protected SeedModule	seeds
protected boolean	seedsAsSurtPrefixes Should seeds also be interpreted as SURT prefixes.
protected org.archive.util.SurtPrefixSet	surtPrefixes
protected ConfigFile	surtsDumpFile Dump file to save SURT prefixes actually used: Useful debugging SURTs.
protected org.archive.io.ReadSource	surtsSource Text from which to infer SURT prefixes.

Fields inherited from class org.archive.modules.deciderules.DecideRule

comment, kp

Constructor Summary

Constructors

Constructor and Description
SurtPrefixedDecideRule()

Method Summary

Methods

Modifier and Type	Method and Description
void	addedSeed(CrawlURI curi) If appropriate, convert seed notification into prefix-addition.
protected void	buildSurtPrefixSet() Construct the set of prefixes to use, from the seed list ( which may include both URIs and '+'-prefixed directives).
void	concludedSeedBatch()
void	doCheckpoint(Checkpoint checkpointInProgress) Do the actual checkpoint.
protected void	dumpSurtPrefixSet() Dump the current prefixes in use to configured dump file (if any)
protected boolean	evaluate(CrawlURI uri) Evaluate whether given object's URI is covered by the SURT prefix set
void	finishCheckpoint(Checkpoint checkpointInProgress) Cleanup/unlock; need not complete for a checkpoint to be valid.
boolean	getAlsoCheckVia()
SeedModule	getSeeds()
boolean	getSeedsAsSurtPrefixes()
ConfigFile	getSurtsDumpFile()
org.archive.io.ReadSource	getSurtsSource()
ConfigFile	getSurtsSourceFile() Deprecated. redundant now that we have surtsSource
boolean	nonseedLine(String line) Consider nonseed lines as possible SURT prefix directives.
void	onApplicationEvent(org.springframework.context.ApplicationEvent event)
protected String	prefixFrom(String uri)
protected void	readPrefixes()
void	setAlsoCheckVia(boolean checkVia)
void	setBeanName(String name)
void	setRecoveryCheckpoint(Checkpoint recoveryCheckpoint) Used to inform a bean that it should restore its state from the given Checkpoint when launched (Lifecycle start()).
void	setSeeds(SeedModule seeds)
void	setSeedsAsSurtPrefixes(boolean seedsAsSurtPrefixes)
void	setSurtsDumpFile(ConfigFile cp)
void	setSurtsSource(org.archive.io.ReadSource surtsSource)
void	setSurtsSourceFile(ConfigFile cp) Deprecated.
void	startCheckpoint(Checkpoint checkpointInProgress) Note a checkpoint is about to begin.

Methods inherited from class org.archive.modules.deciderules.PredicatedDecideRule

getDecision, innerDecide, setDecision

Methods inherited from class org.archive.modules.deciderules.DecideRule

accepts, decisionFor, getComment, getEnabled, getKeyedProperties, onlyDecision, setComment, setEnabled

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

surtsSource

protected org.archive.io.ReadSource surtsSource

Text from which to infer SURT prefixes. Any URLs will be converted to the implied SURT prefix, and literal SURT prefixes may be listed on lines beginning with a '+' character.

seedsAsSurtPrefixes

protected boolean seedsAsSurtPrefixes

Should seeds also be interpreted as SURT prefixes.

surtsDumpFile

protected ConfigFile surtsDumpFile

Dump file to save SURT prefixes actually used: Useful debugging SURTs.

seeds

protected SeedModule seeds

surtPrefixes

protected org.archive.util.SurtPrefixSet surtPrefixes

beanName

protected String beanName

recoveryCheckpoint

protected Checkpoint recoveryCheckpoint

Constructor Detail

SurtPrefixedDecideRule

public SurtPrefixedDecideRule()

Method Detail

getSurtsSourceFile

public ConfigFile getSurtsSourceFile()

Deprecated. redundant now that we have surtsSource

Source file from which to infer SURT prefixes. Any URLs in file will be converted to the implied SURT prefix, and literal SURT prefixes may be listed on lines beginning with a '+' character.

setSurtsSourceFile

public void setSurtsSourceFile(ConfigFile cp)

Deprecated.

getSurtsSource

public org.archive.io.ReadSource getSurtsSource()

setSurtsSource

public void setSurtsSource(org.archive.io.ReadSource surtsSource)

getSeedsAsSurtPrefixes

public boolean getSeedsAsSurtPrefixes()

setSeedsAsSurtPrefixes

public void setSeedsAsSurtPrefixes(boolean seedsAsSurtPrefixes)

getSurtsDumpFile

public ConfigFile getSurtsDumpFile()

setSurtsDumpFile

public void setSurtsDumpFile(ConfigFile cp)

getAlsoCheckVia

public boolean getAlsoCheckVia()

setAlsoCheckVia

public void setAlsoCheckVia(boolean checkVia)

getSeeds

public SeedModule getSeeds()

setSeeds

public void setSeeds(SeedModule seeds)

concludedSeedBatch

public void concludedSeedBatch()

Specified by:

concludedSeedBatch in interface SeedListener

evaluate

protected boolean evaluate(CrawlURI uri)

Evaluate whether given object's URI is covered by the SURT prefix set

Specified by:

evaluate in class PredicatedDecideRule

Parameters:

object - Item to evaluate.

Returns:

true if item, as SURT form URI, is prefixed by an item in the set

readPrefixes

protected void readPrefixes()

dumpSurtPrefixSet

protected void dumpSurtPrefixSet()

Dump the current prefixes in use to configured dump file (if any)

buildSurtPrefixSet

protected void buildSurtPrefixSet()

Construct the set of prefixes to use, from the seed list ( which may include both URIs and '+'-prefixed directives).

addedSeed

public void addedSeed(CrawlURI curi)

If appropriate, convert seed notification into prefix-addition.

Specified by:

addedSeed in interface SeedListener

See Also:

SeedListener.addedSeed(org.archive.modules.CrawlURI)

prefixFrom

protected String prefixFrom(String uri)

nonseedLine

public boolean nonseedLine(String line)

Consider nonseed lines as possible SURT prefix directives. If we are ACCEPTing URIs, expect '+' as the add-SURT directive. If we are REJECTing URIs, expect '-' as the add-SURT directive. TODO: perhaps, make directive configurable, so many SeedListener SurtPrefixedDecideRules can all listen for directives meant only for them.

Specified by:

nonseedLine in interface SeedListener

See Also:

SeedListener.nonseedLine(java.lang.String)

onApplicationEvent

public void onApplicationEvent(org.springframework.context.ApplicationEvent event)

Specified by:

onApplicationEvent in interface org.springframework.context.ApplicationListener<org.springframework.context.ApplicationEvent>

setBeanName

public void setBeanName(String name)

Specified by:

setBeanName in interface org.springframework.beans.factory.BeanNameAware

startCheckpoint

public void startCheckpoint(Checkpoint checkpointInProgress)

Description copied from interface: Checkpointable

Note a checkpoint is about to begin. Most beans will ignore, but some can use this to wrap up tasks that shouldn't be half-done during a checkpoint (and hold a lock to prevent tasks from beginning during the checkpoint).

Specified by:

startCheckpoint in interface Checkpointable

Parameters:

checkpointInProgress - Checkpoint

doCheckpoint

public void doCheckpoint(Checkpoint checkpointInProgress)
                  throws IOException

Description copied from interface: Checkpointable

Do the actual checkpoint. Beans should ensure any state that they would need to recover gets saved in an appropriate place. A moderate amount of state may be saved as a JSONObject into the Checkpoint (which then keeps it in the checkpoint directory.) Larger amounts of state may be stored in a manner private to the bean, or via other collaborating beans (such as BdbModule) which checkpoint backgin database state.

Specified by:

doCheckpoint in interface Checkpointable

Parameters:

checkpointInProgress - Checkpoint

Throws:

IOException

finishCheckpoint

public void finishCheckpoint(Checkpoint checkpointInProgress)

Description copied from interface: Checkpointable

Cleanup/unlock; need not complete for a checkpoint to be valid.

Specified by:

finishCheckpoint in interface Checkpointable

Parameters:

checkpointInProgress - Checkpoint

setRecoveryCheckpoint

public void setRecoveryCheckpoint(Checkpoint recoveryCheckpoint)

Description copied from interface: Checkpointable

Used to inform a bean that it should restore its state from the given Checkpoint when launched (Lifecycle start()). May be autowired or configured after build, but before launch, for example via CheckpointService.setRecoveryCheckpointByName().

Specified by:

setRecoveryCheckpoint in interface Checkpointable

Parameters:

recoveryCheckpoint - Checkpoint