Package org.apache.nutch.crawl

# Class LinkDbMerger

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.nutch.crawl.LinkDbMerger

All Implemented Interfaces:
  `Configurable`, `Tool`

public class LinkDbMerger extends Configured implements Tool
This tool merges several LinkDb-s into one, optionally filtering URLs through the current URLFilters, to skip prohibited URLs and links.

It's possible to use this tool just for filtering - in that case only one LinkDb should be specified in arguments.

If more than one LinkDb contains information about the same URL, all inlinks are accumulated, but only at most `linkdb.max.inlinks` inlinks will ever be added.

If activated, URLFilters will be applied to both the target URLs and to any incoming link URL. If a target URL is prohibited, all inlinks to that target will be removed, including the target URL. If some of incoming links are prohibited, only they will be removed, and they won't count when checking the above-mentioned maximum limit.

Author:
  Andrzej Bialecki

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `LinkDbMerger.LinkDbMergeReducer`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `LinkDbMerger()`
  `LinkDbMerger(Configuration conf)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static Job`
  `createMergeJob(Configuration config, Path linkDb, boolean normalize, boolean filter)`
  `static void`
  `main(String[] args)`
  Run the job
  `void`
  `merge(Path output, Path[] dbs, boolean normalize, boolean filter)`
  `int`
  `run(String[] args)`

  ### Methods inherited from class org.apache.hadoop.conf.Configured

  `getConf, setConf`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

  ### Methods inherited from interface org.apache.hadoop.conf.Configurable

  `getConf, setConf`

- ## Constructor Details

  - ### LinkDbMerger

    public LinkDbMerger()

  - ### LinkDbMerger

    public LinkDbMerger(Configuration conf)

- ## Method Details

  - ### merge

    public void merge(Path output, Path[] dbs, boolean normalize, boolean filter) throws Exception
    Throws:
      `Exception`

  - ### createMergeJob

    public static Job createMergeJob(Configuration config, Path linkDb, boolean normalize, boolean filter) throws IOException
    Throws:
      `IOException`

  - ### main

    public static void main(String[] args) throws Exception
    Run the job
    Parameters:
      `args` - input arguments for the job
    Throws:
      `Exception` - if there is an error running the job

  - ### run

    public int run(String[] args) throws Exception
    Specified by:
      `run` in interface `Tool`
    Throws:
      `Exception`