Package org.apache.nutch.urlfilter.fast

# Class FastURLFilter

java.lang.Object
org.apache.nutch.urlfilter.fast.FastURLFilter

All Implemented Interfaces:
  `Configurable`, `URLFilter`, `Pluggable`

public class FastURLFilter extends Object implements URLFilter
Filters URLs based on a file of regular expressions using host/domains matching first. The default policy is to accept a URL if no matches are found. Rule Format:

     Host www.example.org
       DenyPath /path/to/be/excluded
       DenyPath /some/other/path/excluded

     # Deny everything from *.example.com and example.com
     Domain example.com
       DenyPath .*

     Domain example.org
       DenyPathQuery /resource/.*?action=exclude

`Host` rules are evaluated before `Domain` rules. For `Host` rules the entire host name of a URL must match while the domain names in `Domain` rules are considered as matches if the domain is a suffix of the host name (consisting of complete host name parts). Shorter domain suffixes are checked first, a single dot "`.`" as "domain name" can be used to specify global rules applied to every URL. E.g., for "www.example.com" the rules given above are looked up in the following order:

1. check "www.example.com" whether host-based rules exist and whether one of them matches
2. check "www.example.com" for domain-based rules
3. check "example.com" for domain-based rules
4. check "com" for domain-based rules
5. check for global rules ("`Domain .`")

The first matching rule will reject the URL and no further rules are checked. If no rule matches the URL is accepted. URLs without a host name (e.g., `file:/path/file.txt` are checked for global rules only. URLs which fail to be parsed as `URL` are always rejected. For rules either the URL path (`DenyPath`) or path and query (`DenyPathQuery`) are checked whether the given `Java Regular expression` is found (see `Matcher.find()`) in the URL path (and query). Rules are applied in the order of their definition. For better performance, regular expressions which are simpler/faster or match more URLs should be defined earlier. Comments in the rule file start with the `#` character and reach until the end of the line. The rules file is defined via the property `urlfilter.fast.file`, the default name is `fast-urlfilter.txt`. In addition, it can filter based on the length of the whole URL, its path element or its query element. See `urlfilter.fast.url.*` configurations.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `FastURLFilter.DenyAllRule`
  Rule for `DenyPath .*` or `DenyPath .?`
  `static class`
  `FastURLFilter.DenyPathQueryRule`
  `static class`
  `FastURLFilter.DenyPathRule`
  `static class`
  `FastURLFilter.Rule`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected static final org.slf4j.Logger`
  `LOG`
  `static final String`
  `URLFILTER_FAST_FILE`
  `static final String`
  `URLFILTER_FAST_MAX_LENGTH`
  `static final String`
  `URLFILTER_FAST_PATH_MAX_LENGTH`
  `static final String`
  `URLFILTER_FAST_QUERY_MAX_LENGTH`

  ### Fields inherited from interface org.apache.nutch.net.URLFilter

  `X_POINT_ID`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `FastURLFilter()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `String`
  `filter(String url)`
  Interface for a filter that transforms a URL: it can pass the original URL through or "delete" the URL by returning null
  `Configuration`
  `getConf()`
  `void`
  `reloadRules()`
  `void`
  `setConf(Configuration conf)`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### LOG

    protected static final org.slf4j.Logger LOG

  - ### URLFILTER_FAST_FILE

    public static final String URLFILTER_FAST_FILE
    See Also:
      - Constant Field Values

  - ### URLFILTER_FAST_MAX_LENGTH

    public static final String URLFILTER_FAST_MAX_LENGTH
    See Also:
      - Constant Field Values

  - ### URLFILTER_FAST_PATH_MAX_LENGTH

    public static final String URLFILTER_FAST_PATH_MAX_LENGTH
    See Also:
      - Constant Field Values

  - ### URLFILTER_FAST_QUERY_MAX_LENGTH

    public static final String URLFILTER_FAST_QUERY_MAX_LENGTH
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### FastURLFilter

    public FastURLFilter()

- ## Method Details

  - ### setConf

    public void setConf(Configuration conf)
    Specified by:
      `setConf` in interface `Configurable`

  - ### getConf

    public Configuration getConf()
    Specified by:
      `getConf` in interface `Configurable`

  - ### filter

    public String filter(String url)
    Description copied from interface: `URLFilter`
    Interface for a filter that transforms a URL: it can pass the original URL through or "delete" the URL by returning null
    Specified by:
      `filter` in interface `URLFilter`
    Parameters:
      `url` - the URL string the filter is applied on
    Returns:
      the original URL string if the URL is accepted by the filter or null in case the URL is rejected

  - ### reloadRules

    public void reloadRules() throws IOException
    Throws:
      `IOException`