public class DiskFPMergeUriUniqFilter extends FPMergeUriUniqFilter
Modifier and Type | Class and Description |
---|---|
class |
DiskFPMergeUriUniqFilter.DataFileLongIterator |
FPMergeUriUniqFilter.PendingItem
UriUniqFilter.CrawlUriReceiver
Modifier and Type | Field and Description |
---|---|
protected long |
count |
protected File |
currentFps |
protected long |
newCount |
protected DataOutputStream |
newFps |
protected File |
newFpsFile |
protected DataInputStream |
oldFps |
protected File |
scratchDir |
DEFAULT_MAX_PENDING, FLUSH_DELAY_FACTOR, maxPending, mergeDupAtLast, mergeDuplicateCount, nextFlushAllowableAfter, pendDupAtLast, pendDuplicateCount, pendingSet, profileLog, quickCache, quickDupAtLast, quickDuplicateCount, receiver
Constructor and Description |
---|
DiskFPMergeUriUniqFilter(File scratchDir) |
Modifier and Type | Method and Description |
---|---|
protected void |
addNewFp(long fp)
Add an FP (which may be an old or new FP) to the new complete
list.
|
protected it.unimi.dsi.fastutil.longs.LongIterator |
beginFpMerge()
Begin merging pending candidates with complete list.
|
long |
count() |
protected void |
finishFpMerge()
Complete the merge of candidate and previously-known FPs (closing
files/iterators as appropriate).
|
add, addForce, addNow, close, createFp, flush, forget, note, pend, pending, profileLog, requestFlush, setDestination, setMaxPending, setProfileLog
protected long count
protected File scratchDir
protected File currentFps
protected File newFpsFile
protected DataOutputStream newFps
protected long newCount
protected DataInputStream oldFps
public DiskFPMergeUriUniqFilter(File scratchDir)
protected it.unimi.dsi.fastutil.longs.LongIterator beginFpMerge()
FPMergeUriUniqFilter
beginFpMerge
in class FPMergeUriUniqFilter
protected void addNewFp(long fp)
FPMergeUriUniqFilter
addNewFp
in class FPMergeUriUniqFilter
fp
- the FP to addprotected void finishFpMerge()
FPMergeUriUniqFilter
finishFpMerge
in class FPMergeUriUniqFilter
public long count()
Copyright © 2003-2014 Internet Archive. All Rights Reserved.