Package org.apache.nutch.crawl

Class URLPartitioner

java.lang.Object

org.apache.hadoop.mapreduce.Partitioner<Text,Writable>

org.apache.nutch.crawl.URLPartitioner

All Implemented Interfaces:

Configurable

public class URLPartitioner
extends Partitioner<Text,Writable>
implements Configurable

Partition urls by host, domain name or IP depending on the value of the parameter 'partition.url.mode' which can be 'byHost', 'byDomain' or 'byIP'

Field Summary

Fields

Modifier and Type	Field	Description
static final String	PARTITION_MODE_DOMAIN
static final String	PARTITION_MODE_HOST
static final String	PARTITION_MODE_IP
static final String	PARTITION_MODE_KEY

Constructor Summary

Constructors

Constructor	Description
URLPartitioner()

Method Summary

Modifier and Type	Method	Description
Configuration	getConf()
int	getPartition(Text key, Writable value, int numReduceTasks)	Hash by host or domain name or IP address.
void	setConf(Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

PARTITION_MODE_KEY

public static final String PARTITION_MODE_KEY

See Also:

• Constant Field Values

PARTITION_MODE_HOST

public static final String PARTITION_MODE_HOST

See Also:

• Constant Field Values

PARTITION_MODE_DOMAIN

public static final String PARTITION_MODE_DOMAIN

See Also:

• Constant Field Values

PARTITION_MODE_IP

public static final String PARTITION_MODE_IP

See Also:

• Constant Field Values

Constructor Details

URLPartitioner

public URLPartitioner()

Method Details

setConf

public void setConf(Configuration conf)

Specified by:

setConf in interface Configurable

getConf

public Configuration getConf()

Specified by:

getConf in interface Configurable

getPartition

public int getPartition(Text key,
 Writable value,
 int numReduceTasks)

Hash by host or domain name or IP address.

Specified by:

getPartition in class Partitioner<Text,Writable>