org.archive.crawler.frontier

Class BdbMultipleWorkQueues

java.lang.Object

org.archive.crawler.frontier.BdbMultipleWorkQueues

public class BdbMultipleWorkQueues
extends Object

A BerkeleyDB-database-backed structure for holding ordered groupings of CrawlURIs. Reading the groupings from specific per-grouping (per-classKey/per-Host) starting points allows this to act as a collection of independent queues.

For how the bdb keys are made, see calculateInsertKey(CrawlURI).

TODO: refactor, improve naming.

Author:

gojomo

Constructor Summary

Constructors

Constructor and Description
BdbMultipleWorkQueues(com.sleepycat.je.Database db, com.sleepycat.bind.serial.StoredClassCatalog classCatalog) Create the multi queue in the given environment.

Method Summary

Methods

Modifier and Type	Method and Description
void	addCap(byte[] origin) Add a dummy 'cap' entry at the given insertion key.
protected static com.sleepycat.je.DatabaseEntry	calculateInsertKey(CrawlURI curi) Calculate the insertKey that places a CrawlURI in the desired spot.
protected static byte[]	calculateOriginKey(String classKey) Calculate the 'origin' key for a virtual queue of items with the given classKey.
void	close() clean up
void	delete(CrawlURI item) Delete the given CrawlURI from persistent store.
long	deleteMatchingFromQueue(String match, String queue, com.sleepycat.je.DatabaseEntry headKey) Delete all CrawlURIs matching the given expression.
protected void	forAllPendingDo(org.apache.commons.collections.Closure c) Utility method to perform action for all pending CrawlURI instances.
CrawlURI	get(com.sleepycat.je.DatabaseEntry headKey) Get the next nearest item after the given key.
protected com.sleepycat.je.DatabaseEntry	getFirstKey()
CompositeData	getFrom(String m, int maxMatches, Pattern pattern, boolean verbose)
protected com.sleepycat.je.OperationStatus	getNextNearestItem(com.sleepycat.je.DatabaseEntry headKey, com.sleepycat.je.DatabaseEntry result)
protected static String	insertKeyToString(com.sleepycat.je.DatabaseEntry holderKey)
void	put(CrawlURI curi, boolean overwriteIfPresent) Put the given CrawlURI in at the appropriate place.
protected void	sync() Method used by BdbFrontier during checkpointing.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BdbMultipleWorkQueues

public BdbMultipleWorkQueues(com.sleepycat.je.Database db,
                     com.sleepycat.bind.serial.StoredClassCatalog classCatalog)
                      throws com.sleepycat.je.DatabaseException

Create the multi queue in the given environment.

Parameters:

env - bdb environment to use

classCatalog - Class catalog to use.

recycle - True if we are to reuse db content if any.

Throws:

com.sleepycat.je.DatabaseException

Method Detail

deleteMatchingFromQueue

public long deleteMatchingFromQueue(String match,
                           String queue,
                           com.sleepycat.je.DatabaseEntry headKey)
                             throws com.sleepycat.je.DatabaseException

Delete all CrawlURIs matching the given expression.

Parameters:

match -

queue -

headKey -

Returns:

count of deleted items

Throws:

com.sleepycat.je.DatabaseException

com.sleepycat.je.DatabaseException

getFrom

public CompositeData getFrom(String m,
                    int maxMatches,
                    Pattern pattern,
                    boolean verbose)
                      throws com.sleepycat.je.DatabaseException

Parameters:

m - marker or null to start with first entry

maxMatches -

Returns:

list of matches starting from marker position

Throws:

com.sleepycat.je.DatabaseException

getFirstKey

protected com.sleepycat.je.DatabaseEntry getFirstKey()
                                              throws com.sleepycat.je.DatabaseException

Returns:

the key to the first item in the database

Throws:

com.sleepycat.je.DatabaseException

get

public CrawlURI get(com.sleepycat.je.DatabaseEntry headKey)
             throws com.sleepycat.je.DatabaseException

Get the next nearest item after the given key. Relies on external discipline -- we'll look at the queues count of how many items it has -- to avoid asking for something from a range where there are no associated items -- otherwise could get first item of next 'queue' by mistake.

TODO: hold within a queue's range

Parameters:

headKey - Key prefix that demarks the beginning of the range in pendingUrisDB we're interested in.

Returns:

CrawlURI.

Throws:

com.sleepycat.je.DatabaseException

getNextNearestItem

protected com.sleepycat.je.OperationStatus getNextNearestItem(com.sleepycat.je.DatabaseEntry headKey,
                                                  com.sleepycat.je.DatabaseEntry result)
                                                       throws com.sleepycat.je.DatabaseException

Throws:

com.sleepycat.je.DatabaseException

put

public void put(CrawlURI curi,
       boolean overwriteIfPresent)
         throws com.sleepycat.je.DatabaseException

Put the given CrawlURI in at the appropriate place.

Parameters:

curi -

Throws:

com.sleepycat.je.DatabaseException

calculateOriginKey

protected static byte[] calculateOriginKey(String classKey)

Calculate the 'origin' key for a virtual queue of items with the given classKey. This origin key will be a prefix of the keys for all items in the queue.

Parameters:

classKey - String key to derive origin byte key from

Returns:

a byte array key

calculateInsertKey

protected static com.sleepycat.je.DatabaseEntry calculateInsertKey(CrawlURI curi)

Calculate the insertKey that places a CrawlURI in the desired spot. First bytes are always classKey (usu. host) based -- ensuring grouping by host -- terminated by a zero byte. Then 8 bytes of data ensuring desired ordering within that 'queue' are used. The first byte of these 8 is priority -- allowing 'immediate' and 'soon' items to sort above regular. Next 1 byte is 'precedence'. Last 6 bytes are ordinal serial number, ensuring earlier-discovered URIs sort before later. NOTE: Dangers here are: (1) priorities or precedences over 2^7 (signed byte comparison) (2) ordinals over 2^48 Package access & static for testing purposes.

Parameters:

curi -

Returns:

a DatabaseEntry key for the CrawlURI

insertKeyToString

protected static String insertKeyToString(com.sleepycat.je.DatabaseEntry holderKey)

delete

public void delete(CrawlURI item)
            throws com.sleepycat.je.DatabaseException

Delete the given CrawlURI from persistent store. Requires the key under which it was stored be available.

Parameters:

item -

Throws:

com.sleepycat.je.DatabaseException

sync

protected void sync()

Method used by BdbFrontier during checkpointing.

The backing bdbje database has been marked deferred write so we save on writes to disk. Means no guarantees disk will have whats in memory unless a sync is called (Calling sync on the bdbje Environment is not sufficent).

Package access only because only Frontiers of this package would ever need access.

See Also:

Deferred Write Databases

close

public void close()

clean up

addCap

public void addCap(byte[] origin)

Add a dummy 'cap' entry at the given insertion key. Prevents 'seeks' to queue heads from holding lock on last item of 'preceding' queue. See: http://sourceforge.net/tracker/index.php?func=detail&aid=1262665&group_id=73833&atid=539102

Parameters:

origin - key at which to insert the cap

forAllPendingDo

protected void forAllPendingDo(org.apache.commons.collections.Closure c)
                        throws com.sleepycat.je.DatabaseException

Utility method to perform action for all pending CrawlURI instances.

Parameters:

c - Closure action to perform

Throws:

com.sleepycat.je.DatabaseException