Package org.apache.nutch.net

# Class URLNormalizers

java.lang.Object
org.apache.nutch.net.URLNormalizers

public final class URLNormalizers extends Object
This class uses a "chained filter" pattern to run defined normalizers. Different lists of normalizers may be defined for different "scopes", or contexts where they are used (note however that they need to be activated first through `plugin.include` property).

There is one global scope defined by default, which consists of all active normalizers. The order in which these normalizers are executed may be defined in "urlnormalizer.order" property, which lists space-separated implementation classes (if this property is missing normalizers will be run in random order). If there are more normalizers activated than explicitly named on this list, the remaining ones will be run in random order after the ones specified on the list are executed.

You can define a set of contexts (or scopes) in which normalizers may be called. Each scope can have its own list of normalizers (defined in "urlnormalizer.scope.<scope_name>" property) and its own order (defined in "urlnormalizer.order.<scope_name>" property). If any of these properties are missing, default settings are used for the global scope.

In case no normalizers are required for any given scope, a `org.apache.nutch.net.urlnormalizer.pass.PassURLNormalizer` should be used.

Each normalizer may further select among many configurations, depending on the scope in which it is called, because the scope name is passed as a parameter to each normalizer. You can also use the same normalizer for many scopes.

Several scopes have been defined, and various Nutch tools will attempt using scope-specific normalizers first (and fall back to default config if scope-specific configuration is missing).

Normalizers may be run several times, to ensure that modifications introduced by normalizers at the end of the list can be further reduced by normalizers executed at the beginning. By default this loop is executed just once - if you want to ensure that all possible combinations have been applied you may want to run this loop up to the number of activated normalizers. This loop count can be configured through `urlnormalizer.loop.count` property. As soon as the url is unchanged the loop will stop and return the result.

Author:
  Andrzej Bialecki

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `SCOPE_CRAWLDB`
  Scope used when updating the CrawlDb with new URLs.
  `static final String`
  `SCOPE_DEFAULT`
  Default scope.
  `static final String`
  `SCOPE_FETCHER`
  Scope used by `Fetcher` when processing redirect URLs.
  `static final String`
  `SCOPE_GENERATE_HOST_COUNT`
  Scope used by `Generator`.
  `static final String`
  `SCOPE_INDEXER`
  Scope used when indexing URLs.
  `static final String`
  `SCOPE_INJECT`
  Scope used by `Injector`.
  `static final String`
  `SCOPE_LINKDB`
  Scope used when updating the LinkDb with new URLs.
  `static final String`
  `SCOPE_OUTLINK`
  Scope used when constructing new `Outlink` instances.
  `static final String`
  `SCOPE_PARTITION`
  Scope used by `URLPartitioner`.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `URLNormalizers(Configuration conf, String scope)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `String`
  `normalize(String urlString, String scope)`
  Normalize

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### SCOPE_DEFAULT

    public static final String SCOPE_DEFAULT
    Default scope. If no scope properties are defined then the configuration for this scope will be used.
    See Also:
      - Constant Field Values

  - ### SCOPE_PARTITION

    public static final String SCOPE_PARTITION
    Scope used by `URLPartitioner`.
    See Also:
      - Constant Field Values

  - ### SCOPE_GENERATE_HOST_COUNT

    public static final String SCOPE_GENERATE_HOST_COUNT
    Scope used by `Generator`.
    See Also:
      - Constant Field Values

  - ### SCOPE_FETCHER

    public static final String SCOPE_FETCHER
    Scope used by `Fetcher` when processing redirect URLs.
    See Also:
      - Constant Field Values

  - ### SCOPE_CRAWLDB

    public static final String SCOPE_CRAWLDB
    Scope used when updating the CrawlDb with new URLs.
    See Also:
      - Constant Field Values

  - ### SCOPE_LINKDB

    public static final String SCOPE_LINKDB
    Scope used when updating the LinkDb with new URLs.
    See Also:
      - Constant Field Values

  - ### SCOPE_INJECT

    public static final String SCOPE_INJECT
    Scope used by `Injector`.
    See Also:
      - Constant Field Values

  - ### SCOPE_OUTLINK

    public static final String SCOPE_OUTLINK
    Scope used when constructing new `Outlink` instances.
    See Also:
      - Constant Field Values

  - ### SCOPE_INDEXER

    public static final String SCOPE_INDEXER
    Scope used when indexing URLs.
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### URLNormalizers

    public URLNormalizers(Configuration conf, String scope)

- ## Method Details

  - ### normalize

    public String normalize(String urlString, String scope) throws MalformedURLException
    Normalize
    Parameters:
      `urlString` - The URL string to normalize.
      `scope` - The given scope.
    Returns:
      A normalized String, using the given `scope`
    Throws:
      `MalformedURLException` - If the given URL string is malformed.