Package org.apache.nutch.hostdb

# Class UpdateHostDbReducer

java.lang.Object
org.apache.hadoop.mapreduce.Reducer<Text,NutchWritable,Text,HostDatum>
org.apache.nutch.hostdb.UpdateHostDbReducer

public class UpdateHostDbReducer extends Reducer<Text,NutchWritable,Text,HostDatum>

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Reducer

  `Reducer.Context`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected static boolean`
  `checkAny`
  `protected static boolean`
  `checkFailed`
  `protected static boolean`
  `checkKnown`
  `protected static boolean`
  `checkNew`
  `protected static CrawlDatumProcessor[]`
  `crawlDatumProcessors`
  `protected ThreadPoolExecutor`
  `executor`
  `protected static boolean`
  `force`
  `protected static long`
  `now`
  `protected static String[]`
  `numericFields`
  `protected static Text[]`
  `numericFieldWritables`
  `protected Integer`
  `numResolverThreads`
  `protected static int[]`
  `percentiles`
  `protected static Integer`
  `purgeFailedHostsThreshold`
  `protected BlockingQueue<Runnable>`
  `queue`
  `protected static Integer`
  `recheckInterval`
  `protected ResolverThread`
  `resolverThread`
  `protected Counter`
  `skippedNotEligibleCounter`
  `protected static String[]`
  `stringFields`
  `protected static Text[]`
  `stringFieldWritables`
  `protected Counter`
  `totalHostsCounter`
  `protected static long`
  `urlLimit`
  `protected Counter`
  `urlLimitNotReachedCounter`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `UpdateHostDbReducer()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `cleanup(Reducer<Text,NutchWritable,Text,HostDatum>.Context context)`
  Shut down all running threads and wait for completion.
  `protected boolean`
  `isEligibleForCheck(HostDatum datum)`
  Determines whether a record is eligible for recheck.
  `void`
  `reduce(Text key, Iterable<NutchWritable> values, Reducer<Text,NutchWritable,Text,HostDatum>.Context context)`
  `void`
  `setup(Reducer<Text,NutchWritable,Text,HostDatum>.Context context)`
  Configures the thread pool and prestarts all resolver threads.
  `protected boolean`
  `shouldCheck(HostDatum datum)`
  Determines whether a record should be checked.

  ### Methods inherited from class org.apache.hadoop.mapreduce.Reducer

  `run`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### resolverThread

    protected ResolverThread resolverThread

  - ### numResolverThreads

    protected Integer numResolverThreads

  - ### purgeFailedHostsThreshold

    protected static Integer purgeFailedHostsThreshold

  - ### recheckInterval

    protected static Integer recheckInterval

  - ### checkFailed

    protected static boolean checkFailed

  - ### checkNew

    protected static boolean checkNew

  - ### checkKnown

    protected static boolean checkKnown

  - ### checkAny

    protected static boolean checkAny

  - ### force

    protected static boolean force

  - ### urlLimit

    protected static long urlLimit

  - ### now

    protected static long now

  - ### numericFields

    protected static String[] numericFields

  - ### stringFields

    protected static String[] stringFields

  - ### percentiles

    protected static int[] percentiles

  - ### numericFieldWritables

    protected static Text[] numericFieldWritables

  - ### stringFieldWritables

    protected static Text[] stringFieldWritables

  - ### crawlDatumProcessors

    protected static CrawlDatumProcessor[] crawlDatumProcessors

  - ### queue

    protected BlockingQueue<Runnable> queue

  - ### executor

    protected ThreadPoolExecutor executor

  - ### urlLimitNotReachedCounter

    protected Counter urlLimitNotReachedCounter

  - ### totalHostsCounter

    protected Counter totalHostsCounter

  - ### skippedNotEligibleCounter

    protected Counter skippedNotEligibleCounter

- ## Constructor Details

  - ### UpdateHostDbReducer

    public UpdateHostDbReducer()

- ## Method Details

  - ### setup

    public void setup(Reducer<Text,NutchWritable,Text,HostDatum>.Context context)
    Configures the thread pool and prestarts all resolver threads.
    Overrides:
      `setup` in class `Reducer<Text,NutchWritable,Text,HostDatum>`

  - ### reduce

    public void reduce(Text key, Iterable<NutchWritable> values, Reducer<Text,NutchWritable,Text,HostDatum>.Context context) throws IOException, InterruptedException
    Overrides:
      `reduce` in class `Reducer<Text,NutchWritable,Text,HostDatum>`
    Throws:
      `IOException`
      `InterruptedException`

  - ### shouldCheck

    protected boolean shouldCheck(HostDatum datum)
    Determines whether a record should be checked.
    Parameters:
      `datum` - a `HostDatum` to check for eligibility
    Returns:
      true if it should be checked, false otherwise

  - ### isEligibleForCheck

    protected boolean isEligibleForCheck(HostDatum datum)
    Determines whether a record is eligible for recheck.
    Parameters:
      `datum` - a `HostDatum` to check for eligibility
    Returns:
      true if eligible for recheck, false otherwise

  - ### cleanup

    public void cleanup(Reducer<Text,NutchWritable,Text,HostDatum>.Context context)
    Shut down all running threads and wait for completion.
    Overrides:
      `cleanup` in class `Reducer<Text,NutchWritable,Text,HostDatum>`