org.archive.modules.forms

Class FormLoginProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.modules.forms.FormLoginProcessor

All Implemented Interfaces:

Checkpointable, HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class FormLoginProcessor
extends Processor
implements Checkpointable

A step, post-ExtractorHTMLForms, where a followup CrawlURI to attempt a form submission may be synthesized.

If an HTMLForm was previously discovered (by ExtractorHTMLForms), and that form appears to be a login form, and at the very least the loginUsername setting is non-empty, and the current (NOT 'action') URI fits under a configured SURT prefix, then a submission CrawlURI will be composed.

This submission CrawlURI will be added to the current URI's outCandidates, and prefilled with settings for a POST and input values that are a merging of: (a) original discovered in-page values; (b) the 'loginUsername' into the first plausible text/email-type input field; (c) the 'loginPassword' into the first password-type input field.

Typically the settings 'applicableSurtPrefix', 'loginUsername', and 'loginPassword' would be set in an overlay sheet and only applied to one or more sites (by SURT prefix), rather than set globally. An example minimal set of beans to add to CXML could look like:

 <bean id='formLoginFields' class='org.archive.spring.Sheet'>
  <property name='map'>
   <map>
    <entry key='formFiller.loginUsername' value='EXAMPLE_USERNAME'/>
    <entry key='formFiller.loginPassword' value='EXAMPLE_PASSWORD'/>
   </map>
  </property>
 </bean>

 <bean class='org.archive.crawler.spring.SurtPrefixesSheetAssociation'>
  <property name='surtPrefixes'>
   <list>
    <value>http://(net,example,www,)/</value>
    <value>http://(com,example,</value>
   </list>
  </property>
  <property name='targetSheetNames'>
   <list>
    <value>formLoginFields</value>
   </list>
  </property>
 </bean>
 
 

(Remember: https URIs are always collapsed to http form before overlay-surt-prefix comparisons, so surtPrefixes in the above association should always be in http form, even if the actual target URIs are https.)

Finally, while there is not yet support for testing if the submitted CrawlURI succeeded, this processor keeps track of a count of FORMS seen that are eligible for attempts, and attempts made (for now, just once), per 'formTrackingDomain' (which is either the applicableSurtPrefix or the form-origin-URI trimmed to its pathless root in SURT form). This is also added to the submission URI for logging to the resulting WARC 'response' record.

Version:

$Date$, $Revision$

Author:

gojomo

Field Summary

Fields

Modifier and Type	Field and Description
protected UriErrorLoggerModule	loggerModule

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
FormLoginProcessor()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	createFormSubmissionAttempt(CrawlURI curi, HTMLForm templateForm, String formProvince)
protected void	fromCheckpointJson(org.json.JSONObject json) Restore internal state from JSONObject stored at earlier checkpoint-time.
String	getApplicableSurtPrefix()
protected String	getFormProvince(CrawlURI curi) Get the 'form province' - either the configured (applicableSurtPrefix) or inferred (full current server) range of URIs that is considered covered by one form login
UriErrorLoggerModule	getLoggerModule()
String	getLoginPassword()
String	getLoginUsername()
protected void	innerProcess(CrawlURI curi) Actually performs the process.
void	setApplicableSurtPrefix(String applicableSurtPrefix)
void	setLoggerModule(UriErrorLoggerModule loggerModule)
void	setLoginPassword(String loginPassword)
void	setLoginUsername(String loginUsername)
protected boolean	shouldProcess(CrawlURI curi) Determines whether the given uri should be processed by this processor.
protected String	submitStatusFor(String formProvince)
protected org.json.JSONObject	toCheckpointJson() Return a JSONObject of current stat that can be consulted on recovery to restore necessary values.
protected String	warcHeaderFor(String formProvince)

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.archive.checkpointing.Checkpointable

doCheckpoint, finishCheckpoint, setRecoveryCheckpoint, startCheckpoint

Field Detail

loggerModule

protected transient UriErrorLoggerModule loggerModule

Constructor Detail

FormLoginProcessor

public FormLoginProcessor()

Method Detail

getApplicableSurtPrefix

public String getApplicableSurtPrefix()

setApplicableSurtPrefix

public void setApplicableSurtPrefix(String applicableSurtPrefix)

getLoginUsername

public String getLoginUsername()

setLoginUsername

public void setLoginUsername(String loginUsername)

getLoginPassword

public String getLoginPassword()

setLoginPassword

public void setLoginPassword(String loginPassword)

getLoggerModule

public UriErrorLoggerModule getLoggerModule()

setLoggerModule

public void setLoggerModule(UriErrorLoggerModule loggerModule)

shouldProcess

protected boolean shouldProcess(CrawlURI curi)

Description copied from class: Processor

Determines whether the given uri should be processed by this processor. For instance, a processor that only works on HTML content might reject the URI if its content type is not "text/html", if its content length is zero, and so on.

Specified by:

shouldProcess in class Processor

Parameters:

curi - the URI to test

Returns:

true if this processor should process that uri; false if not

innerProcess

protected void innerProcess(CrawlURI curi)

Description copied from class: Processor

Actually performs the process. By the time this method is invoked, it is known that the given URI passes the #ENABLED, the #DECIDE_RULES and the #shouldProcess(ProcessorURI) tests.

Specified by:

innerProcess in class Processor

Parameters:

curi - the URI to process

getFormProvince

protected String getFormProvince(CrawlURI curi)

Get the 'form province' - either the configured (applicableSurtPrefix) or inferred (full current server) range of URIs that is considered covered by one form login

Parameters:

curi -

Returns:

createFormSubmissionAttempt

protected void createFormSubmissionAttempt(CrawlURI curi,
                               HTMLForm templateForm,
                               String formProvince)

warcHeaderFor

protected String warcHeaderFor(String formProvince)

submitStatusFor

protected String submitStatusFor(String formProvince)

toCheckpointJson

protected org.json.JSONObject toCheckpointJson()
                                        throws org.json.JSONException

Description copied from class: Processor

Return a JSONObject of current stat that can be consulted on recovery to restore necessary values.

Overrides:

toCheckpointJson in class Processor

Returns:

JSONObject

Throws:

org.json.JSONException

fromCheckpointJson

protected void fromCheckpointJson(org.json.JSONObject json)
                           throws org.json.JSONException

Description copied from class: Processor

Restore internal state from JSONObject stored at earlier checkpoint-time.

Overrides:

fromCheckpointJson in class Processor

Parameters:

json - JSONObject

Throws:

org.json.JSONException