Package org.apache.nutch.urlfilter.ignoreexempt

# Class ExemptionUrlFilter

java.lang.Object
org.apache.nutch.urlfilter.api.RegexURLFilterBase
org.apache.nutch.urlfilter.regex.RegexURLFilter
org.apache.nutch.urlfilter.ignoreexempt.ExemptionUrlFilter

All Implemented Interfaces:
  `Configurable`, `URLExemptionFilter`, `URLFilter`, `Pluggable`

public class ExemptionUrlFilter extends RegexURLFilter implements URLExemptionFilter
This implementation of `URLExemptionFilter` uses regex configuration to check if URL is eligible for exemption from the `db.ignore.external.links` configuration property. When this filter is enabled, the external urls will be checked against configured sequence of regex rules.

The exemption rule file defaults to `db-ignore-external-exemptions.txt` in the classpath but can be overridden using the configuration property `db.ignore.external.exemptions.file`.

The exemption rules are specified in plain text file where each line is a rule. The format is same same as `regex-urlfilter.txt`. Each non-comment, non-blank line contains a regular expression prefixed by + or -. The first matching pattern in the file determines whether a URL is exempted or ignored. If no pattern matches, the URL is ignored.

Since:
  Feb 10, 2016
Version:
  1
See Also:
  - `URLExemptionFilter`
  - `RegexURLFilter`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE`

  ### Fields inherited from class org.apache.nutch.urlfilter.regex.RegexURLFilter

  `URLFILTER_REGEX_FILE, URLFILTER_REGEX_RULES`

  ### Fields inherited from class org.apache.nutch.urlfilter.api.RegexURLFilterBase

  `hasHostDomainRules`

  ### Fields inherited from interface org.apache.nutch.net.URLExemptionFilter

  `X_POINT_ID`

  ### Fields inherited from interface org.apache.nutch.net.URLFilter

  `X_POINT_ID`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `ExemptionUrlFilter()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `boolean`
  `filter(String fromUrl, String toUrl)`
  Checks if toUrl is exempted when the ignore external is enabled
  `List<Pattern>`
  `getExemptions()`
  `protected Reader`
  `getRulesReader(Configuration conf)`
  Gets reader for regex rules
  `static void`
  `main(String[] args)`

  ### Methods inherited from class org.apache.nutch.urlfilter.regex.RegexURLFilter

  `createRule, createRule`

  ### Methods inherited from class org.apache.nutch.urlfilter.api.RegexURLFilterBase

  `filter, getConf, main, setConf`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

  ### Methods inherited from interface org.apache.hadoop.conf.Configurable

  `getConf, setConf`

- ## Field Details

  - ### DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE

    public static final String DB_IGNORE_EXTERNAL_EXEMPTIONS_FILE
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### ExemptionUrlFilter

    public ExemptionUrlFilter()

- ## Method Details

  - ### getExemptions

    public List<Pattern> getExemptions()

  - ### filter

    public boolean filter(String fromUrl, String toUrl)
    Description copied from interface: `URLExemptionFilter`
    Checks if toUrl is exempted when the ignore external is enabled
    Specified by:
      `filter` in interface `URLExemptionFilter`
    Parameters:
      `fromUrl` - : the source url which generated the outlink
      `toUrl` - : the destination url which needs to be checked for exemption
    Returns:
      true when toUrl is exempted from dbIgnore

  - ### getRulesReader

    protected Reader getRulesReader(Configuration conf) throws IOException
    Gets reader for regex rules
    Overrides:
      `getRulesReader` in class `RegexURLFilter`
    Parameters:
      `conf` - is the current configuration.
    Returns:
      the name of the resource containing the rules to use.
    Throws:
      `IOException` - if there is a fatal error obtaining the `Reader`

  - ### main

    public static void main(String[] args)