Package org.apache.nutch.net.urlnormalizer.ajax

Class AjaxURLNormalizer

java.lang.Object
org.apache.nutch.net.urlnormalizer.ajax.AjaxURLNormalizer
All Implemented Interfaces:
Configurable, URLNormalizer
public class AjaxURLNormalizer extends Object implements URLNormalizer
URLNormalizer capable of dealing with AJAX URL's. Use the following regex filter to prevent escaped fragments from being fetched. ^(.*)\?.*_escaped_fragment_

  • Field Details

    • AJAX_URL_PART

      public static String AJAX_URL_PART

    • ESCAPED_URL_PART

      public static String ESCAPED_URL_PART

  • Constructor Details

    • AjaxURLNormalizer

      public AjaxURLNormalizer()
      Default constructor.

  • Method Details

    • normalize

      public String normalize(String urlString, String scope) throws MalformedURLException
      Attempts to normalize the input URL string
      Specified by:
      normalize in interface URLNormalizer
      Parameters:
      urlString - a String to process
      scope - used when indexing URLs
      Returns:
      String
      Throws:
      MalformedURLException - if the urlString is malformed

    • normalizeHashedFragment

      protected String normalizeHashedFragment(String urlString) throws MalformedURLException
      Returns a normalized input URL. #! querystrings are transformed to a _escaped_fragment_ form.
      Parameters:
      urlString - a String to process
      Returns:
      String
      Throws:
      MalformedURLException - if the urlString is malformed

    • normalizeEscapedFragment

      protected String normalizeEscapedFragment(String urlString) throws MalformedURLException
      Returns a normalized input URL. _escaped_fragment_ querystrings are transformed to a #! form.
      Parameters:
      urlString - a String to process
      Returns:
      String
      Throws:
      MalformedURLException - if the urlString is malformed

    • unescape

      protected String unescape(String fragmentPart)
      Unescape some exotic characters in the fragment part
      Parameters:
      fragmentPart - a String to process
      Returns:
      String

    • escape

      protected String escape(String fragmentPart)
      Escape some exotic characters in the fragment part
      Parameters:
      fragmentPart - a String to process
      Returns:
      String

    • setConf

      public void setConf(Configuration conf)
      Specified by:
      setConf in interface Configurable
      Parameters:
      conf - a populated Configuration

    • getConf

      public Configuration getConf()
      Specified by:
      getConf in interface Configurable
      Returns:
      Configuration