Package org.apache.nutch.hostdb

Class UpdateHostDbReducer

java.lang.Object

org.apache.hadoop.mapreduce.Reducer<Text,NutchWritable,Text,HostDatum>

org.apache.nutch.hostdb.UpdateHostDbReducer

public class UpdateHostDbReducer
extends Reducer<Text,NutchWritable,Text,HostDatum>

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Reducer

Reducer.Context

Field Summary

Fields

Modifier and Type	Field	Description
protected static boolean	checkAny
protected static boolean	checkFailed
protected static boolean	checkKnown
protected static boolean	checkNew
protected static CrawlDatumProcessor[]	crawlDatumProcessors
protected ThreadPoolExecutor	executor
protected static boolean	force
protected static long	now
protected static String[]	numericFields
protected static Text[]	numericFieldWritables
protected Integer	numResolverThreads
protected static int[]	percentiles
protected static Integer	purgeFailedHostsThreshold
protected BlockingQueue<Runnable>	queue
protected static Integer	recheckInterval
protected ResolverThread	resolverThread
protected Counter	skippedNotEligibleCounter
protected static String[]	stringFields
protected static Text[]	stringFieldWritables
protected Counter	totalHostsCounter
protected static long	urlLimit
protected Counter	urlLimitNotReachedCounter

Constructor Summary

Constructors

Constructor	Description
UpdateHostDbReducer()

Method Summary

Modifier and Type	Method	Description
void	cleanup(Reducer<Text,NutchWritable,Text,HostDatum>.Context context)	Shut down all running threads and wait for completion.
protected boolean	isEligibleForCheck(HostDatum datum)	Determines whether a record is eligible for recheck.
void	reduce(Text key, Iterable<NutchWritable> values, Reducer<Text,NutchWritable,Text,HostDatum>.Context context)
void	setup(Reducer<Text,NutchWritable,Text,HostDatum>.Context context)	Configures the thread pool and prestarts all resolver threads.
protected boolean	shouldCheck(HostDatum datum)	Determines whether a record should be checked.

Methods inherited from class org.apache.hadoop.mapreduce.Reducer

run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

resolverThread

protected ResolverThread resolverThread

numResolverThreads

protected Integer numResolverThreads

purgeFailedHostsThreshold

protected static Integer purgeFailedHostsThreshold

recheckInterval

protected static Integer recheckInterval

checkFailed

protected static boolean checkFailed

checkNew

protected static boolean checkNew

checkKnown

protected static boolean checkKnown

checkAny

protected static boolean checkAny

force

protected static boolean force

urlLimit

protected static long urlLimit

now

protected static long now

numericFields

protected static String[] numericFields

stringFields

protected static String[] stringFields

percentiles

protected static int[] percentiles

numericFieldWritables

protected static Text[] numericFieldWritables

stringFieldWritables

protected static Text[] stringFieldWritables

crawlDatumProcessors

protected static CrawlDatumProcessor[] crawlDatumProcessors

queue

protected BlockingQueue<Runnable> queue

executor

protected ThreadPoolExecutor executor

urlLimitNotReachedCounter

protected Counter urlLimitNotReachedCounter

totalHostsCounter

protected Counter totalHostsCounter

skippedNotEligibleCounter

protected Counter skippedNotEligibleCounter

Constructor Details

UpdateHostDbReducer

public UpdateHostDbReducer()

Method Details

setup

public void setup(Reducer<Text,NutchWritable,Text,HostDatum>.Context context)

Configures the thread pool and prestarts all resolver threads.

Overrides:

setup in class Reducer<Text,NutchWritable,Text,HostDatum>

reduce

public void reduce(Text key,
 Iterable<NutchWritable> values,
 Reducer<Text,NutchWritable,Text,HostDatum>.Context context)
            throws IOException,
InterruptedException

Overrides:

reduce in class Reducer<Text,NutchWritable,Text,HostDatum>

Throws:

IOException

InterruptedException

shouldCheck

protected boolean shouldCheck(HostDatum datum)

Determines whether a record should be checked.

Parameters:

datum - a HostDatum to check for eligibility

Returns:

true if it should be checked, false otherwise

isEligibleForCheck

protected boolean isEligibleForCheck(HostDatum datum)

Determines whether a record is eligible for recheck.

Parameters:

datum - a HostDatum to check for eligibility

Returns:

true if eligible for recheck, false otherwise

cleanup

public void cleanup(Reducer<Text,NutchWritable,Text,HostDatum>.Context context)

Shut down all running threads and wait for completion.

Overrides:

cleanup in class Reducer<Text,NutchWritable,Text,HostDatum>