Package org.apache.nutch.net.urlnormalizer.regex

# Class RegexURLNormalizer

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer

All Implemented Interfaces:
  `Configurable`, `URLNormalizer`

public class RegexURLNormalizer extends Configured implements URLNormalizer
Allows users to do regex substitutions on all/any URLs that are encountered, which is useful for stripping session IDs from URLs.

This class uses the `urlnormalizer.regex.file` property. It should be set to the file name of an xml file which should contain the patterns and substitutions to be done on encountered URLs.

This class also supports different rules depending on the scope. Please see the javadoc in `URLNormalizers` for more details.

- ## Field Summary

  ### Fields inherited from interface org.apache.nutch.net.URLNormalizer

  `X_POINT_ID`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `RegexURLNormalizer()`
  The default constructor which is called from UrlNormalizerFactory (normalizerClass.newInstance()) in method: getNormalizer()*
  `RegexURLNormalizer(Configuration conf)`
  `RegexURLNormalizer(Configuration conf, String filename)`
  Constructor which can be passed the configuration file name, so it doesn't look in other configuration files for it.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `HashMap<String,List<org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer.Rule>>`
  `getScopedRules()`
  `static void`
  `main(String[] args)`
  Spits out patterns and substitutions that are in the configuration file.
  `String`
  `normalize(String urlString, String scope)`
  `String`
  `regexNormalize(String urlString, String scope)`
  This function does the replacements by iterating through all the regex patterns.
  `void`
  `setConf(Configuration conf)`

  ### Methods inherited from class org.apache.hadoop.conf.Configured

  `getConf`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

  ### Methods inherited from interface org.apache.hadoop.conf.Configurable

  `getConf`

- ## Constructor Details

  - ### RegexURLNormalizer

    public RegexURLNormalizer()
    The default constructor which is called from UrlNormalizerFactory (normalizerClass.newInstance()) in method: getNormalizer()*

  - ### RegexURLNormalizer

    public RegexURLNormalizer(Configuration conf)

  - ### RegexURLNormalizer

    public RegexURLNormalizer(Configuration conf, String filename) throws IOException, PatternSyntaxException
    Constructor which can be passed the configuration file name, so it doesn't look in other configuration files for it.
    Parameters:
      `conf` - A populated `Configuration`
      `filename` - A specific configuration file
    Throws:
      `IOException` - if there is an error locatingf the specified input file
      `PatternSyntaxException` - If there is an error whilst interpreting rule patterns.

- ## Method Details

  - ### getScopedRules

    public HashMap<String,List<org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer.Rule>> getScopedRules()

  - ### setConf

    public void setConf(Configuration conf)
    Specified by:
      `setConf` in interface `Configurable`
    Overrides:
      `setConf` in class `Configured`

  - ### regexNormalize

    public String regexNormalize(String urlString, String scope)
    This function does the replacements by iterating through all the regex patterns. It accepts a string url as input and returns the altered string.
    Parameters:
      `urlString` - A url string to process
      `scope` - The identifier for a specific scoped rule
    Returns:
      The altered string

  - ### normalize

    public String normalize(String urlString, String scope) throws MalformedURLException
    Specified by:
      `normalize` in interface `URLNormalizer`
    Throws:
      `MalformedURLException`

  - ### main

    public static void main(String[] args) throws PatternSyntaxException, IOException
    Spits out patterns and substitutions that are in the configuration file.
    Parameters:
      `args` - accepts one argument which is a scope
    Throws:
      `IOException` - Can be thrown by `normalize(String, String)`
      `PatternSyntaxException` - If there is an error with the provided scope rule pattern.