Package org.apache.nutch.net.urlnormalizer.basic

# Class BasicURLNormalizer

java.lang.Object
org.apache.nutch.net.urlnormalizer.basic.BasicURLNormalizer

All Implemented Interfaces:
  `Configurable`, `URLNormalizer`

public class BasicURLNormalizer extends Object implements URLNormalizer
Converts URLs to a normal form:

- remove dot segments in path: `/./` or `/../`
- remove default ports, e.g. 80 for protocol `http://`
- normalize percent-encoding in URL paths
- normalize the host name if it is an Internationalized Domain Name (IDN) to ASCII or Unicode, depending on the configuration properties `urlnormalizer.basic.host.idn` and `urlnormalizer.basic.host.idna2008`
- remove a trailing dot in the host name (if the property `urlnormalizer.basic.host.trim-trailing-dot` is true)

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `NORM_HOST_IDN`
  `static final String`
  `NORM_HOST_IDNA_2008`
  `static final String`
  `NORM_HOST_TRIM_TRAILING_DOT`

  ### Fields inherited from interface org.apache.nutch.net.URLNormalizer

  `X_POINT_ID`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `BasicURLNormalizer()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `Configuration`
  `getConf()`
  `static void`
  `main(String[] args)`
  `String`
  `normalize(String urlString, String scope)`
  `void`
  `setConf(Configuration conf)`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### NORM_HOST_IDN

    public static final String NORM_HOST_IDN
    See Also:
      - Constant Field Values

  - ### NORM_HOST_IDNA_2008

    public static final String NORM_HOST_IDNA_2008
    See Also:
      - Constant Field Values

  - ### NORM_HOST_TRIM_TRAILING_DOT

    public static final String NORM_HOST_TRIM_TRAILING_DOT
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### BasicURLNormalizer

    public BasicURLNormalizer()

- ## Method Details

  - ### getConf

    public Configuration getConf()
    Specified by:
      `getConf` in interface `Configurable`

  - ### setConf

    public void setConf(Configuration conf)
    Specified by:
      `setConf` in interface `Configurable`

  - ### normalize

    public String normalize(String urlString, String scope) throws MalformedURLException
    Specified by:
      `normalize` in interface `URLNormalizer`
    Throws:
      `MalformedURLException`

  - ### main

    public static void main(String[] args) throws IOException
    Throws:
      `IOException`