Package org.apache.nutch.crawl

Class CrawlDbFilter

java.lang.Object

org.apache.hadoop.mapreduce.Mapper<Text,CrawlDatum,Text,CrawlDatum>

org.apache.nutch.crawl.CrawlDbFilter

public class CrawlDbFilter
extends Mapper<Text,CrawlDatum,Text,CrawlDatum>

This class provides a way to separate the URL normalization and filtering steps from the rest of CrawlDb manipulation code.

Author:

Andrzej Bialecki

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Mapper

Mapper.Context

Field Summary

Fields

Modifier and Type	Field	Description
static final String	URL_FILTERING
static final String	URL_NORMALIZING
static final String	URL_NORMALIZING_SCOPE

Constructor Summary

Constructors

Constructor	Description
CrawlDbFilter()

Method Summary

Modifier and Type	Method	Description
void	map(Text key, CrawlDatum value, Mapper<Text,CrawlDatum,Text,CrawlDatum>.Context context)
void	setup(Mapper<Text,CrawlDatum,Text,CrawlDatum>.Context context)

Methods inherited from class org.apache.hadoop.mapreduce.Mapper

cleanup, run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

URL_FILTERING

public static final String URL_FILTERING

See Also:

• Constant Field Values

URL_NORMALIZING

public static final String URL_NORMALIZING

See Also:

• Constant Field Values

URL_NORMALIZING_SCOPE

public static final String URL_NORMALIZING_SCOPE

See Also:

• Constant Field Values

Constructor Details

CrawlDbFilter

public CrawlDbFilter()

Method Details

setup

public void setup(Mapper<Text,CrawlDatum,Text,CrawlDatum>.Context context)

Overrides:

setup in class Mapper<Text,CrawlDatum,Text,CrawlDatum>

map

public void map(Text key,
 CrawlDatum value,
 Mapper<Text,CrawlDatum,Text,CrawlDatum>.Context context)
         throws IOException,
InterruptedException

Overrides:

map in class Mapper<Text,CrawlDatum,Text,CrawlDatum>

Throws:

IOException

InterruptedException