Package org.apache.nutch.parsefilter.naivebayes

Class NaiveBayesParseFilter

java.lang.Object

org.apache.nutch.parsefilter.naivebayes.NaiveBayesParseFilter

All Implemented Interfaces:

Configurable, HtmlParseFilter, Pluggable

public class NaiveBayesParseFilter
extends Object
implements HtmlParseFilter

Html Parse filter that classifies the outlinks from the parseresult as relevant or irrelevant based on the parseText's relevancy (using a training file where you can give positive and negative example texts see the description of parsefilter.naivebayes.trainfile) and if found irrelevant it gives the link a second chance if it contains any of the words from the list given in parsefilter.naivebayes.wordlist. CAUTION: Set the parser.timeout to -1 or a bigger value than 30, when using this classifier.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	DICTFILE_MODELFILTER
static final String	TRAINFILE_MODELFILTER

Fields inherited from interface org.apache.nutch.parse.HtmlParseFilter

X_POINT_ID

Constructor Summary

Constructors

Constructor	Description
NaiveBayesParseFilter()

Method Summary

Modifier and Type	Method	Description
boolean	classify(String text)
boolean	containsWord(String url, ArrayList<String> wordlist)
ParseResult	filter(Content content, ParseResult parseResult, HTMLMetaTags metaTags, DocumentFragment doc)	Adds metadata or otherwise modifies a parse of HTML content, given the DOM tree of a page.
boolean	filterParse(String text)
boolean	filterUrl(String url)
Configuration	getConf()
void	setConf(Configuration conf)
void	train()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

TRAINFILE_MODELFILTER

public static final String TRAINFILE_MODELFILTER

See Also:

• Constant Field Values

DICTFILE_MODELFILTER

public static final String DICTFILE_MODELFILTER

See Also:

• Constant Field Values

Constructor Details

NaiveBayesParseFilter

public NaiveBayesParseFilter()

Method Details

filterParse

public boolean filterParse(String text)

filterUrl

public boolean filterUrl(String url)

classify

public boolean classify(String text)
                 throws IOException

Throws:

IOException

train

public void train()
           throws Exception

Throws:

Exception

containsWord

public boolean containsWord(String url,
 ArrayList<String> wordlist)

setConf

public void setConf(Configuration conf)

Specified by:

setConf in interface Configurable

getConf

public Configuration getConf()

Specified by:

getConf in interface Configurable

filter

public ParseResult filter(Content content,
 ParseResult parseResult,
 HTMLMetaTags metaTags,
 DocumentFragment doc)

Description copied from interface: HtmlParseFilter

Adds metadata or otherwise modifies a parse of HTML content, given the DOM tree of a page.

Specified by:

filter in interface HtmlParseFilter

Parameters:

content - the Content for a given response

parseResult - the result of running on or more Parser's on the content.

metaTags - a populated HTMLMetaTags object

doc - a DocumentFragment (DOM) which can be processed in the filtering process.

Returns:

a filtered ParseResult

See Also:

• Parser.getParse(Content)