CrawlController (Heritrix 3 3.2.0 API)

java.lang.Object
- org.archive.crawler.framework.CrawlController

All Implemented Interfaces:

Serializable, Checkpointable, org.springframework.context.ApplicationContextAware, org.springframework.context.Lifecycle
```
public class CrawlController
extends Object
implements Serializable, org.springframework.context.Lifecycle, org.springframework.context.ApplicationContextAware, Checkpointable
```
CrawlController collects all the classes which cooperate to perform a crawl and provides a high-level interface to the running crawl. As the "global context" for a crawl, subcomponents will often reach each other through the CrawlController.

See Also:
Serialized Form
Contributor:

gojomo

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class CrawlController.State

static class CrawlController.StopCompleteEvent

Nested Classes
Modifier and Type	Class and Description
`static class`	`CrawlController.State`
`static class`	`CrawlController.StopCompleteEvent`

Field Summary

Fields
Modifier and Type	Field and Description
`protected AlertThreadGroup`	`alertThreadGroup`
`protected org.springframework.context.support.AbstractApplicationContext`	`appCtx`
`protected CandidateChain`	`candidateChain` Candidate chain
`protected DispositionChain`	`dispositionChain` Disposition chain
`protected FetchChain`	`fetchChain` Fetch chain
`protected Frontier`	`frontier` The frontier to use for the crawl.
`protected boolean`	`hasStarted`
`protected boolean`	`isRunning`
`protected boolean`	`isStopComplete`
`protected CrawlerLoggerModule`	`loggerModule`
`protected int`	`maxToeThreads` Maximum number of threads processing URIs at the same time.
`protected CrawlMetadata`	`metadata`
`protected boolean`	`pauseAtStart` whether to pause at crawl start
`protected int`	`recorderInBufferBytes` Size in bytes of in-memory buffer to record inbound traffic.
`protected int`	`recorderOutBufferBytes` Size in bytes of in-memory buffer to record outbound traffic.
`protected Checkpoint`	`recoveryCheckpoint`
`protected boolean`	`runWhileEmpty` whether to keep running (without pause or finish) when frontier is empty
`protected ConfigPath`	`scratchDir` Scratch directory for temporary overflow-to-disk
`protected SeedModule`	`seeds`
`protected ServerCache`	`serverCache`
`protected StatisticsTracker`	`statisticsTracker` Statistics tracking modules.

Constructor Summary

Constructors
Constructor and Description

CrawlController()

Constructors
Constructor and Description
`CrawlController()`

Method Summary

Methods
Modifier and Type	Method and Description
`boolean`	`atFinish()` Evaluate if the crawl should stop because it is finished, without actually stopping the crawl.
`void`	`beginCrawlStop()` Start the process of stopping the crawl.
`protected void`	`completePause()`
`protected void`	`completeStop()` Called when the last toethread exits.
`void`	`doCheckpoint(Checkpoint checkpointInProgress)` Do the actual checkpoint.
`void`	`finishCheckpoint(Checkpoint checkpointInProgress)` Cleanup/unlock; need not complete for a checkpoint to be valid.
`void`	`freeReserveMemory()`
`int`	`getActiveToeCount()`
`CandidateChain`	`getCandidateChain()`
`CrawlStatus`	`getCrawlExitStatus()`
`DispositionChain`	`getDispositionChain()`
`FetchChain`	`getFetchChain()`
`Frontier`	`getFrontier()`
`String`	`getFrontierReportShort()`
`CrawlerLoggerModule`	`getLoggerModule()`
`int`	`getMaxToeThreads()`
`CrawlMetadata`	`getMetadata()`
`boolean`	`getPauseAtStart()`
`int`	`getRecorderInBufferBytes()`
`int`	`getRecorderOutBufferBytes()`
`boolean`	`getRunWhileEmpty()`
`ConfigPath`	`getScratchDir()`
`SeedModule`	`getSeeds()`
`ServerCache`	`getServerCache()`
`Object`	`getState()`
`StatisticsTracker`	`getStatisticsTracker()`
`int`	`getToeCount()`
`ToePool`	`getToePool()`
`String`	`getToeThreadReport()`
`String`	`getToeThreadReportShort()`
`Map<String,Object>`	`getToeThreadReportShortData()`
`boolean`	`hasStarted()`
`boolean`	`isActive()` Is this crawl actively able/trying to crawl? Includes both states RUNNING and EMPTY.
`boolean`	`isFinished()`
`boolean`	`isPaused()` Tell if the controller is paused
`boolean`	`isPausing()`
`boolean`	`isRunning()`
`boolean`	`isStopComplete()`
`void`	`killThread(int threadNumber, boolean replace)` Kills a thread.
`void`	`logProgressStatistics(String msg)` Log to the progress statistics log.
`void`	`noteFrontierState(Frontier.State reachedState)` Receive notification from the frontier, in the frontier's own manager thread, that the frontier has reached a new state.
`void`	`requestCrawlPause()` Stop the crawl temporarly.
`void`	`requestCrawlResume()` Resume crawl from paused state
`void`	`requestCrawlStart()` Operator requested crawl begin
`void`	`requestCrawlStop()` Operator requested for crawl to stop.
`void`	`requestCrawlStop(CrawlStatus message)` Operator requested for crawl to stop.
`protected void`	`sendCrawlStateChangeEvent(CrawlController.State newState, CrawlStatus status)` Send crawl change event to all listeners.
`void`	`setApplicationContext(org.springframework.context.ApplicationContext applicationContext)`
`void`	`setCandidateChain(CandidateChain candidateChain)`
`void`	`setDispositionChain(DispositionChain dispositionChain)`
`void`	`setFetchChain(FetchChain fetchChain)`
`void`	`setFrontier(Frontier frontier)`
`void`	`setLoggerModule(CrawlerLoggerModule loggerModule)`
`void`	`setMaxToeThreads(int maxToeThreads)`
`void`	`setMetadata(CrawlMetadata provider)`
`void`	`setPauseAtStart(boolean pauseAtStart)`
`void`	`setRecorderInBufferBytes(int recorderInBufferBytes)`
`void`	`setRecorderOutBufferBytes(int recorderOutBufferBytes)`
`void`	`setRecoveryCheckpoint(Checkpoint recoveryCheckpoint)` Used to inform a bean that it should restore its state from the given Checkpoint when launched (Lifecycle start()).
`void`	`setRunWhileEmpty(boolean runWhileEmpty)`
`void`	`setScratchDir(ConfigPath scratchDir)`
`void`	`setSeeds(SeedModule seeds)`
`void`	`setServerCache(ServerCache serverCache)`
`void`	`setStatisticsTracker(StatisticsTracker statisticsTracker)`
`protected void`	`setupToePool()`
`void`	`start()`
`void`	`startCheckpoint(Checkpoint checkpointInProgress)` Note a checkpoint is about to begin.
`void`	`stop()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - appCtx
```
protected org.springframework.context.support.AbstractApplicationContext appCtx
```
  - metadata
```
protected CrawlMetadata metadata
```
  - serverCache
```
protected ServerCache serverCache
```
  - frontier
```
protected Frontier frontier
```
    The frontier to use for the crawl.
  - scratchDir
```
protected ConfigPath scratchDir
```
    Scratch directory for temporary overflow-to-disk
  - statisticsTracker
```
protected StatisticsTracker statisticsTracker
```
    Statistics tracking modules. Any number of specialized statistics trackers that monitor a crawl and write logs, reports and/or provide information to the user interface.
  - seeds
```
protected SeedModule seeds
```
  - fetchChain
```
protected FetchChain fetchChain
```
    Fetch chain
  - dispositionChain
```
protected DispositionChain dispositionChain
```
    Disposition chain
  - candidateChain
```
protected CandidateChain candidateChain
```
    Candidate chain
  - maxToeThreads
```
protected int maxToeThreads
```
    Maximum number of threads processing URIs at the same time.
  - runWhileEmpty
```
protected boolean runWhileEmpty
```
    whether to keep running (without pause or finish) when frontier is empty
  - pauseAtStart
```
protected boolean pauseAtStart
```
    whether to pause at crawl start
  - recorderOutBufferBytes
```
protected int recorderOutBufferBytes
```
    Size in bytes of in-memory buffer to record outbound traffic. One such buffer is reserved for every ToeThread.
  - recorderInBufferBytes
```
protected int recorderInBufferBytes
```
    Size in bytes of in-memory buffer to record inbound traffic. One such buffer is reserved for every ToeThread.
  - loggerModule
```
protected CrawlerLoggerModule loggerModule
```
  - alertThreadGroup
```
protected transient AlertThreadGroup alertThreadGroup
```
  - isRunning
```
protected boolean isRunning
```
  - hasStarted
```
protected boolean hasStarted
```
  - isStopComplete
```
protected boolean isStopComplete
```
  - recoveryCheckpoint
```
protected Checkpoint recoveryCheckpoint
```
- Constructor Detail
  - CrawlController
```
public CrawlController()
```
- Method Detail
  - setApplicationContext
```
public void setApplicationContext(org.springframework.context.ApplicationContext applicationContext)
                           throws org.springframework.beans.BeansException
```
    Specified by:
    
    setApplicationContext in interface org.springframework.context.ApplicationContextAware
    
    Throws:
    
    org.springframework.beans.BeansException
  - getMetadata
```
public CrawlMetadata getMetadata()
```
  - setMetadata
```
public void setMetadata(CrawlMetadata provider)
```
  - getServerCache
```
public ServerCache getServerCache()
```
  - setServerCache
```
public void setServerCache(ServerCache serverCache)
```
  - getFrontier
```
public Frontier getFrontier()
```
  - setFrontier
```
public void setFrontier(Frontier frontier)
```
  - getScratchDir
```
public ConfigPath getScratchDir()
```
  - setScratchDir
```
public void setScratchDir(ConfigPath scratchDir)
```
  - getStatisticsTracker
```
public StatisticsTracker getStatisticsTracker()
```
  - setStatisticsTracker
```
public void setStatisticsTracker(StatisticsTracker statisticsTracker)
```
  - getSeeds
```
public SeedModule getSeeds()
```
  - setSeeds
```
public void setSeeds(SeedModule seeds)
```
  - getFetchChain
```
public FetchChain getFetchChain()
```
  - setFetchChain
```
public void setFetchChain(FetchChain fetchChain)
```
  - getDispositionChain
```
public DispositionChain getDispositionChain()
```
  - setDispositionChain
```
public void setDispositionChain(DispositionChain dispositionChain)
```
  - getCandidateChain
```
public CandidateChain getCandidateChain()
```
  - setCandidateChain
```
public void setCandidateChain(CandidateChain candidateChain)
```
  - getMaxToeThreads
```
public int getMaxToeThreads()
```
  - setMaxToeThreads
```
public void setMaxToeThreads(int maxToeThreads)
```
  - getRunWhileEmpty
```
public boolean getRunWhileEmpty()
```
  - setRunWhileEmpty
```
public void setRunWhileEmpty(boolean runWhileEmpty)
```
  - getPauseAtStart
```
public boolean getPauseAtStart()
```
  - setPauseAtStart
```
public void setPauseAtStart(boolean pauseAtStart)
```
  - getRecorderOutBufferBytes
```
public int getRecorderOutBufferBytes()
```
  - setRecorderOutBufferBytes
```
public void setRecorderOutBufferBytes(int recorderOutBufferBytes)
```
  - getRecorderInBufferBytes
```
public int getRecorderInBufferBytes()
```
  - setRecorderInBufferBytes
```
public void setRecorderInBufferBytes(int recorderInBufferBytes)
```
  - getLoggerModule
```
public CrawlerLoggerModule getLoggerModule()
```
  - setLoggerModule
```
public void setLoggerModule(CrawlerLoggerModule loggerModule)
```
  - start
```
public void start()
```
    Specified by:
    
    start in interface org.springframework.context.Lifecycle
  - isRunning
```
public boolean isRunning()
```
    Specified by:
    
    isRunning in interface org.springframework.context.Lifecycle
  - stop
```
public void stop()
```
    Specified by:
    
    stop in interface org.springframework.context.Lifecycle
  - sendCrawlStateChangeEvent
```
protected void sendCrawlStateChangeEvent(CrawlController.State newState,
                             CrawlStatus status)
```
    Send crawl change event to all listeners.
    
    Parameters:
    newState - State change we're to tell listeners' about.
    message - Message on state change.
  - hasStarted
```
public boolean hasStarted()
```
  - isStopComplete
```
public boolean isStopComplete()
```
  - requestCrawlStart
```
public void requestCrawlStart()
```
    Operator requested crawl begin
  - completeStop
```
protected void completeStop()
```
    Called when the last toethread exits.
  - completePause
```
protected void completePause()
```
  - requestCrawlStop
```
public void requestCrawlStop()
```
    Operator requested for crawl to stop.
  - requestCrawlStop
```
public void requestCrawlStop(CrawlStatus message)
```
    Operator requested for crawl to stop.
    
    Parameters:
    message -
  - beginCrawlStop
```
public void beginCrawlStop()
```
    Start the process of stopping the crawl.
  - requestCrawlPause
```
public void requestCrawlPause()
```
    Stop the crawl temporarly.
  - isPaused
```
public boolean isPaused()
```
    Tell if the controller is paused
    
    Returns:
    true if paused
  - isPausing
```
public boolean isPausing()
```
  - isActive
```
public boolean isActive()
```
    Is this crawl actively able/trying to crawl? Includes both states RUNNING and EMPTY.
    
    Returns:
  - isFinished
```
public boolean isFinished()
```
  - requestCrawlResume
```
public void requestCrawlResume()
```
    Resume crawl from paused state
  - getActiveToeCount
```
public int getActiveToeCount()
```
    Returns:
    Active toe thread count.
  - setupToePool
```
protected void setupToePool()
```
  - getToeCount
```
public int getToeCount()
```
    Returns:
    The number of ToeThreads
    See Also:
    ToePool.getToeCount()
  - getToePool
```
public ToePool getToePool()
```
    Returns:
    The ToePool
  - killThread
```
public void killThread(int threadNumber,
              boolean replace)
```
    Kills a thread. For details see ToePool.killThread(int, boolean).
    
    Parameters:
    threadNumber - Thread to kill.
    replace - Should thread be replaced.
    See Also:
    ToePool.killThread(int, boolean)
  - atFinish
```
public boolean atFinish()
```
    Evaluate if the crawl should stop because it is finished, without actually stopping the crawl.
    
    Returns:
    true if crawl is at a finish-possible state
  - freeReserveMemory
```
public void freeReserveMemory()
```
  - logProgressStatistics
```
public void logProgressStatistics(String msg)
```
    Log to the progress statistics log.
    
    Parameters:
    msg - Message to write the progress statistics log.
  - getState
```
public Object getState()
```
    Returns:
    CrawlController state.
  - getCrawlExitStatus
```
public CrawlStatus getCrawlExitStatus()
```
  - getToeThreadReport
```
public String getToeThreadReport()
```
  - getToeThreadReportShort
```
public String getToeThreadReportShort()
```
  - getToeThreadReportShortData
```
public Map<String,Object> getToeThreadReportShortData()
```
  - getFrontierReportShort
```
public String getFrontierReportShort()
```
  - noteFrontierState
```
public void noteFrontierState(Frontier.State reachedState)
```
    Receive notification from the frontier, in the frontier's own manager thread, that the frontier has reached a new state.
    
    Parameters:
    reachedState - the state the frontier has reached
  - startCheckpoint
```
public void startCheckpoint(Checkpoint checkpointInProgress)
```
    Description copied from interface: Checkpointable
    
    Note a checkpoint is about to begin. Most beans will ignore, but some can use this to wrap up tasks that shouldn't be half-done during a checkpoint (and hold a lock to prevent tasks from beginning during the checkpoint).
    
    Specified by:
    
    startCheckpoint in interface Checkpointable
    
    Parameters:
    checkpointInProgress - Checkpoint
  - doCheckpoint
```
public void doCheckpoint(Checkpoint checkpointInProgress)
                  throws IOException
```
    Description copied from interface: Checkpointable
    
    Do the actual checkpoint. Beans should ensure any state that they would need to recover gets saved in an appropriate place. A moderate amount of state may be saved as a JSONObject into the Checkpoint (which then keeps it in the checkpoint directory.) Larger amounts of state may be stored in a manner private to the bean, or via other collaborating beans (such as BdbModule) which checkpoint backgin database state.
    
    Specified by:
    
    doCheckpoint in interface Checkpointable
    
    Parameters:
    checkpointInProgress - Checkpoint
    
    Throws:
    
    IOException
  - finishCheckpoint
```
public void finishCheckpoint(Checkpoint checkpointInProgress)
```
    Description copied from interface: Checkpointable
    
    Cleanup/unlock; need not complete for a checkpoint to be valid.
    
    Specified by:
    
    finishCheckpoint in interface Checkpointable
    
    Parameters:
    checkpointInProgress - Checkpoint
  - setRecoveryCheckpoint
```
public void setRecoveryCheckpoint(Checkpoint recoveryCheckpoint)
```
    Description copied from interface: Checkpointable
    
    Used to inform a bean that it should restore its state from the given Checkpoint when launched (Lifecycle start()). May be autowired or configured after build, but before launch, for example via CheckpointService.setRecoveryCheckpointByName().
    
    Specified by:
    
    setRecoveryCheckpoint in interface Checkpointable
    
    Parameters:
    recoveryCheckpoint - Checkpoint

Class CrawlController

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

appCtx

metadata

serverCache

frontier

scratchDir

statisticsTracker

seeds

fetchChain

dispositionChain

candidateChain

maxToeThreads

runWhileEmpty

pauseAtStart

recorderOutBufferBytes

recorderInBufferBytes

loggerModule

alertThreadGroup

isRunning

hasStarted

isStopComplete

recoveryCheckpoint

Constructor Detail

CrawlController

Method Detail

setApplicationContext

getMetadata

setMetadata

getServerCache

setServerCache

getFrontier

setFrontier

getScratchDir

setScratchDir

getStatisticsTracker

setStatisticsTracker

getSeeds

setSeeds

getFetchChain

setFetchChain

getDispositionChain

setDispositionChain

getCandidateChain

setCandidateChain

getMaxToeThreads

setMaxToeThreads

getRunWhileEmpty

setRunWhileEmpty

getPauseAtStart

setPauseAtStart

getRecorderOutBufferBytes

setRecorderOutBufferBytes

getRecorderInBufferBytes

setRecorderInBufferBytes

getLoggerModule

setLoggerModule

start

isRunning

stop

sendCrawlStateChangeEvent

hasStarted

isStopComplete

requestCrawlStart

completeStop

completePause

requestCrawlStop

requestCrawlStop

beginCrawlStop

requestCrawlPause

isPaused

isPausing

isActive

isFinished

requestCrawlResume