Package org.apache.nutch.hostdb

# Class UpdateHostDbMapper

java.lang.Object
org.apache.hadoop.mapreduce.Mapper<Text,Writable,Text,NutchWritable>
org.apache.nutch.hostdb.UpdateHostDbMapper

public class UpdateHostDbMapper extends Mapper<Text,Writable,Text,NutchWritable>
Mapper ingesting HostDB and CrawlDB entries. Additionally it can also read host score info from a plain text key/value file generated by the Webgraph's NodeDumper tool.

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Mapper

  `Mapper.Context`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected String[]`
  `args`
  `protected String`
  `buffer`
  `protected CrawlDatum`
  `crawlDatum`
  `protected ErrorTracker`
  `errorTracker`
  `protected boolean`
  `filter`
  `protected Counter`
  `filteredRecordsCounter`
  `protected URLFilters`
  `filters`
  `protected Text`
  `host`
  `protected HostDatum`
  `hostDatum`
  `protected boolean`
  `normalize`
  `protected URLNormalizers`
  `normalizers`
  `protected boolean`
  `readingCrawlDb`
  `protected String`
  `reprUrl`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `UpdateHostDbMapper()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected String`
  `filterNormalize(String hostName)`
  Filters and or normalizes the input hostname by applying the configured URL filters and normalizers the URL "http://hostname/".
  `void`
  `map(Text key, Writable value, Mapper<Text,Writable,Text,NutchWritable>.Context context)`
  Mapper ingesting records from the HostDB, CrawlDB and plaintext host scores file.
  `void`
  `setup(Mapper<Text,Writable,Text,NutchWritable>.Context context)`

  ### Methods inherited from class org.apache.hadoop.mapreduce.Mapper

  `cleanup, run`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### host

    protected Text host

  - ### hostDatum

    protected HostDatum hostDatum

  - ### crawlDatum

    protected CrawlDatum crawlDatum

  - ### reprUrl

    protected String reprUrl

  - ### buffer

    protected String buffer

  - ### args

    protected String[] args

  - ### filter

    protected boolean filter

  - ### normalize

    protected boolean normalize

  - ### readingCrawlDb

    protected boolean readingCrawlDb

  - ### filters

    protected URLFilters filters

  - ### normalizers

    protected URLNormalizers normalizers

  - ### filteredRecordsCounter

    protected Counter filteredRecordsCounter

  - ### errorTracker

    protected ErrorTracker errorTracker

- ## Constructor Details

  - ### UpdateHostDbMapper

    public UpdateHostDbMapper()

- ## Method Details

  - ### setup

    public void setup(Mapper<Text,Writable,Text,NutchWritable>.Context context)
    Overrides:
      `setup` in class `Mapper<Text,Writable,Text,NutchWritable>`

  - ### filterNormalize

    protected String filterNormalize(String hostName)
    Filters and or normalizes the input hostname by applying the configured URL filters and normalizers the URL "http://hostname/".
    Parameters:
      `hostName` - the input hostname
    Returns:
      the normalized hostname, or null if the URL is excluded by URL filters or failed to be normalized converted

  - ### map

    public void map(Text key, Writable value, Mapper<Text,Writable,Text,NutchWritable>.Context context) throws IOException, InterruptedException
    Mapper ingesting records from the HostDB, CrawlDB and plaintext host scores file. Statistics and scores are passed on.
    Overrides:
      `map` in class `Mapper<Text,Writable,Text,NutchWritable>`
    Parameters:
      `key` - record `Text` key
      `value` - associated `Writable` object
      `context` - `Reducer.Context` for writing custom counters and output.
    Throws:
      `IOException`
      `InterruptedException`