CrawlJob (Heritrix 3 3.2.0 API)

java.lang.Object
- org.archive.crawler.framework.CrawlJob

All Implemented Interfaces:

Comparable<CrawlJob>, EventListener, org.springframework.context.ApplicationListener<org.springframework.context.ApplicationEvent>
```
public class CrawlJob
extends Object
implements Comparable<CrawlJob>, org.springframework.context.ApplicationListener<org.springframework.context.ApplicationEvent>
```
CrawlJob represents a crawl configuration, including its configuration files, instantiated/running ApplicationContext, and disk output, potentially across multiple runs. CrawlJob provides convenience methods for an administrative interface to assemble, launch, monitor, and manage crawls.

Contributor:

gojomo

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

class CrawlJob.JobLogFormatter
Formatter for job.log

Nested Classes
Modifier and Type	Class and Description
`class`	`CrawlJob.JobLogFormatter` Formatter for job.log

Field Summary

Fields
Modifier and Type	Field and Description
`protected PathSharingContext`	`ac`
`protected AlertThreadGroup`	`alertThreadGroup`
`protected Handler`	`currentLaunchJobLogHandler`
`protected boolean`	`isLaunchInfoPartial`
`protected Logger`	`jobLogger`
`protected org.joda.time.DateTime`	`lastLaunch`
`protected int`	`launchCount`
`protected boolean`	`needTeardown`
`protected File`	`primaryConfig`
`protected org.joda.time.DateTime`	`xmlOkAt`

Constructor Summary

Constructors
Constructor and Description

CrawlJob(File cxml)

Constructors
Constructor and Description
`CrawlJob(File cxml)`

Method Summary

Methods
Modifier and Type	Method and Description
`protected void`	`beansException(org.springframework.beans.BeansException be)` Report a BeansException during instantiation; report chain in reverse order (so root cause is first); ignore non-BeansExceptions or messages without a useful compact message.
`void`	`checkXML()` Is the primary XML config minimally well-formed?
`int`	`compareTo(CrawlJob o)` Sort for reverse-chronological listing.
`protected void`	`doTeardown()`
`String`	`elapsedReport()`
`Map<String,Object>`	`elapsedReportData()`
`String`	`frontierReport()`
`Map<String,Object>`	`frontierReportData()`
`int`	`getAlertCount()`
`Object`	`getBeanpathTarget(String beanPath)` Utility method for getting a bean or any other object addressable with a 'bean path' -- a property-path string (with dots and []indexes) starting with a bean name.
`CheckpointService`	`getCheckpointService()` Return the configured Checkpointer instance, if there is exactly one, otherwise null.
`Map<String,ConfigPath>`	`getConfigPaths()` Return all known ConfigPaths, as an aid to viewing or editting.
`CrawlController`	`getCrawlController()`
`protected Document`	`getDomDocument(File f)` Read a file to a DOM Document; return null if this isn't possible for any reason.
`List<File>`	`getImportedConfigs(File xml)` Return all config files included via 'import' statements in the primary config (or other included configs).
`PathSharingContext`	`getJobContext()`
`File`	`getJobDir()`
`File`	`getJobLog()`
`Logger`	`getJobLogger()` Get a logger to a distinguished file, job.log in the job's directory, into which job-specific events may be reported.
`String`	`getJobStatusDescription()`
`long`	`getLastActivityTime()`
`org.joda.time.DateTime`	`getLastLaunch()`
`int`	`getLaunchCount()`
`File`	`getPrimaryConfig()`
`String`	`getShortName()`
`protected StatisticsTracker`	`getStats()`
`boolean`	`hasApplicationContext()`
`boolean`	`hasValidApplicationContext()` Did the ApplicationContext self-validate? return true if validation passed without errors
`void`	`instantiateContainer()` Can the configuration yield an assembled ApplicationContext?
`boolean`	`isLaunchable()` Is it reasonable to offer a launch button
`boolean`	`isLaunchInfoPartial()`
`boolean`	`isPausable()`
`boolean`	`isProfile()` Is this job a 'profile' (or template), meaning it may be editted or copied to another jobs, but should not be launched.
`boolean`	`isRunning()`
`boolean`	`isUnpausable()`
`boolean`	`isXmlOk()` Is the primary config file legal XML?
`String`	`jobDirRelativePath(File f)` Compute a path relative to the job directory for all contained files, or null if the File is not inside the job directory.
`void`	`launch()` Launch a crawl into 'running' status, assembling if necessary.
`Object`	`loadReport()`
`Map<String,Number>`	`loadReportData()`
`void`	`onApplicationEvent(org.springframework.context.ApplicationEvent event)` Log note of all ApplicationEvents.
`Object`	`rateReport()`
`Map<String,Number>`	`rateReportData()`
`protected void`	`scanJobLog()` Refresh knowledge of total launched and last launch by scanning the job.log.
`protected String`	`shortMessage(org.springframework.beans.BeansException ex)` Return a short useful message for common BeansExceptions.
`String`	`sizeTotalsReport()`
`Map<String,Long>`	`sizeTotalsReportData()`
`protected void`	`startContext()` Start the context, catching and reporting any BeansExceptions.
`boolean`	`teardown()` Ensure a fresh start for any configuration changes or relaunches, by stopping and discarding an existing ApplicationContext.
`void`	`terminate()`
`String`	`threadReport()`
`Map<String,Object>`	`threadReportData()`
`String`	`uriTotalsReport()`
`Map<String,Long>`	`uriTotalsReportData()`
`void`	`validateConfiguration()` Does the assembled ApplicationContext self-validate? Any failures are reported as WARNING log events in the job log.
`void`	`writeHtmlTo(PrintWriter pw)`
`void`	`writeHtmlTo(PrintWriter pw, String uriPrefix)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - primaryConfig
```
protected File primaryConfig
```
  - ac
```
protected PathSharingContext ac
```
  - launchCount
```
protected int launchCount
```
  - isLaunchInfoPartial
```
protected boolean isLaunchInfoPartial
```
  - lastLaunch
```
protected org.joda.time.DateTime lastLaunch
```
  - alertThreadGroup
```
protected AlertThreadGroup alertThreadGroup
```
  - xmlOkAt
```
protected org.joda.time.DateTime xmlOkAt
```
  - jobLogger
```
protected Logger jobLogger
```
  - currentLaunchJobLogHandler
```
protected transient Handler currentLaunchJobLogHandler
```
  - needTeardown
```
protected boolean needTeardown
```
- Constructor Detail
  - CrawlJob
```
public CrawlJob(File cxml)
```
- Method Detail
  - getPrimaryConfig
```
public File getPrimaryConfig()
```
  - getJobDir
```
public File getJobDir()
```
  - getShortName
```
public String getShortName()
```
  - getJobLog
```
public File getJobLog()
```
  - getJobContext
```
public PathSharingContext getJobContext()
```
  - isLaunchInfoPartial
```
public boolean isLaunchInfoPartial()
```
  - getJobLogger
```
public Logger getJobLogger()
```
    Get a logger to a distinguished file, job.log in the job's directory, into which job-specific events may be reported.
    
    Returns:
    Logger writing to the job-specific log
  - getLastLaunch
```
public org.joda.time.DateTime getLastLaunch()
```
  - getLaunchCount
```
public int getLaunchCount()
```
  - scanJobLog
```
protected void scanJobLog()
```
    Refresh knowledge of total launched and last launch by scanning the job.log.
  - isProfile
```
public boolean isProfile()
```
    Is this job a 'profile' (or template), meaning it may be editted or copied to another jobs, but should not be launched. Profiles are marked with the convention that their short name (job directory name) begins "profile-".
    
    Returns:
    true if this job is a 'profile'
  - writeHtmlTo
```
public void writeHtmlTo(PrintWriter pw)
```
  - writeHtmlTo
```
public void writeHtmlTo(PrintWriter pw,
               String uriPrefix)
```
  - checkXML
```
public void checkXML()
```
    Is the primary XML config minimally well-formed?
  - getDomDocument
```
protected Document getDomDocument(File f)
```
    Read a file to a DOM Document; return null if this isn't possible for any reason.
    
    Parameters:
    f - File of XML
    
    Returns:
    org.w3c.dom.Document or null if problems encountered
  - isXmlOk
```
public boolean isXmlOk()
```
    Is the primary config file legal XML?
    
    Returns:
    true if the primary configuration file passed XML testing
  - instantiateContainer
```
public void instantiateContainer()
```
    Can the configuration yield an assembled ApplicationContext?
  - beansException
```
protected void beansException(org.springframework.beans.BeansException be)
```
    Report a BeansException during instantiation; report chain in reverse order (so root cause is first); ignore non-BeansExceptions or messages without a useful compact message.
    
    Parameters:
    be - BeansException
  - shortMessage
```
protected String shortMessage(org.springframework.beans.BeansException ex)
```
    Return a short useful message for common BeansExceptions.
    
    Parameters:
    ex - BeansException
    
    Returns:
    String short descriptive message
  - hasApplicationContext
```
public boolean hasApplicationContext()
```
  - validateConfiguration
```
public void validateConfiguration()
```
    Does the assembled ApplicationContext self-validate? Any failures are reported as WARNING log events in the job log. TODO: make these severe?
  - hasValidApplicationContext
```
public boolean hasValidApplicationContext()
```
    Did the ApplicationContext self-validate? return true if validation passed without errors
  - launch
```
public void launch()
```
    Launch a crawl into 'running' status, assembling if necessary. (Note the crawl may have been configured to start in a 'paused' state.)
  - startContext
```
protected void startContext()
```
    Start the context, catching and reporting any BeansExceptions.
  - compareTo
```
public int compareTo(CrawlJob o)
```
    Sort for reverse-chronological listing.
    
    Specified by:
    
    compareTo in interface Comparable<CrawlJob>
    
    See Also:
    Comparable.compareTo(java.lang.Object)
  - getLastActivityTime
```
public long getLastActivityTime()
```
  - isRunning
```
public boolean isRunning()
```
  - getCrawlController
```
public CrawlController getCrawlController()
```
  - isPausable
```
public boolean isPausable()
```
  - isUnpausable
```
public boolean isUnpausable()
```
  - getCheckpointService
```
public CheckpointService getCheckpointService()
```
    Return the configured Checkpointer instance, if there is exactly one, otherwise null.
    
    Returns:
    Checkpointer
  - teardown
```
public boolean teardown()
```
    Ensure a fresh start for any configuration changes or relaunches, by stopping and discarding an existing ApplicationContext.
    
    Returns:
    true if teardown is complete when method returns, false if still in progress
  - doTeardown
```
protected void doTeardown()
```
  - getImportedConfigs
```
public List<File> getImportedConfigs(File xml)
```
    Return all config files included via 'import' statements in the primary config (or other included configs).
    
    Parameters:
    xml - File to examine
    
    Returns:
    List of all transitively-imported Files
  - getConfigPaths
```
public Map<String,ConfigPath> getConfigPaths()
```
    Return all known ConfigPaths, as an aid to viewing or editting.
    
    Returns:
    all ConfigPaths known to the ApplicationContext, in a map by name, or an empty map if no ApplicationContext
  - jobDirRelativePath
```
public String jobDirRelativePath(File f)
```
    Compute a path relative to the job directory for all contained files, or null if the File is not inside the job directory.
    
    Parameters:
    f - File
    
    Returns:
    path relative to the job directory, or null if File not inside job dir
  - onApplicationEvent
```
public void onApplicationEvent(org.springframework.context.ApplicationEvent event)
```
    Log note of all ApplicationEvents.
    
    Specified by:
    
    onApplicationEvent in interface org.springframework.context.ApplicationListener<org.springframework.context.ApplicationEvent>
    
    See Also:
    ApplicationListener.onApplicationEvent(org.springframework.context.ApplicationEvent)
  - isLaunchable
```
public boolean isLaunchable()
```
    Is it reasonable to offer a launch button
    
    Returns:
    true if launchable
  - getAlertCount
```
public int getAlertCount()
```
  - getStats
```
protected StatisticsTracker getStats()
```
  - rateReportData
```
public Map<String,Number> rateReportData()
```
  - rateReport
```
public Object rateReport()
```
  - loadReportData
```
public Map<String,Number> loadReportData()
```
  - loadReport
```
public Object loadReport()
```
  - uriTotalsReportData
```
public Map<String,Long> uriTotalsReportData()
```
  - uriTotalsReport
```
public String uriTotalsReport()
```
  - sizeTotalsReportData
```
public Map<String,Long> sizeTotalsReportData()
```
  - sizeTotalsReport
```
public String sizeTotalsReport()
```
  - elapsedReportData
```
public Map<String,Object> elapsedReportData()
```
  - elapsedReport
```
public String elapsedReport()
```
  - threadReportData
```
public Map<String,Object> threadReportData()
```
  - threadReport
```
public String threadReport()
```
  - frontierReportData
```
public Map<String,Object> frontierReportData()
```
  - frontierReport
```
public String frontierReport()
```
  - terminate
```
public void terminate()
```
  - getBeanpathTarget
```
public Object getBeanpathTarget(String beanPath)
```
    Utility method for getting a bean or any other object addressable with a 'bean path' -- a property-path string (with dots and []indexes) starting with a bean name. TODO: move elsewhere? on the appContext? a util class?
    
    Parameters:
    beanPath - String 'property-path' with bean name as first segment
    
    Returns:
    Object targeted by beanPath, or null if nont
  - getJobStatusDescription
```
public String getJobStatusDescription()
```

Class CrawlJob

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

primaryConfig

ac

launchCount

isLaunchInfoPartial

lastLaunch

alertThreadGroup

xmlOkAt

jobLogger

currentLaunchJobLogHandler

needTeardown

Constructor Detail

CrawlJob

Method Detail

getPrimaryConfig

getJobDir

getShortName

getJobLog

getJobContext

isLaunchInfoPartial

getJobLogger

getLastLaunch

getLaunchCount

scanJobLog

isProfile

writeHtmlTo

writeHtmlTo

checkXML

getDomDocument

isXmlOk

instantiateContainer

beansException

shortMessage

hasApplicationContext

validateConfiguration

hasValidApplicationContext

launch

startContext

compareTo

getLastActivityTime

isRunning

getCrawlController

isPausable

isUnpausable

getCheckpointService

teardown

doTeardown

getImportedConfigs

getConfigPaths

jobDirRelativePath

onApplicationEvent

isLaunchable

getAlertCount

getStats

rateReportData

rateReport

loadReportData

loadReport

uriTotalsReportData

uriTotalsReport

sizeTotalsReportData

sizeTotalsReport

elapsedReportData

elapsedReport

threadReportData

threadReport

frontierReportData

frontierReport

terminate

getBeanpathTarget

getJobStatusDescription