Package org.apache.nutch.parse

# Class ParserChecker

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.nutch.util.AbstractChecker
org.apache.nutch.parse.ParserChecker

All Implemented Interfaces:
  `Configurable`, `Tool`

public class ParserChecker extends AbstractChecker
Parser checker, useful for testing parser. It also accurately reports possible fetching and parsing failures and presents protocol status signals to aid debugging. The tool enables us to retrieve the following data from any url:

1. `contentType`: The URL `Content` type.
2. `signature`: Digest is used to identify pages (like unique ID) and is used to remove duplicates during the dedup procedure. It is calculated using `MD5Signature` or `TextProfileSignature`.
3. `Version`: From `ParseData`.
4. `Status`: From `ParseData`.
5. `Title`: of the URL
6. `Outlinks`: associated with the URL
7. `Content Metadata`: such as X-AspNet-Version, Date, Content-length, servedBy, Content-Type, Cache-Control, etc.
8. `Parse Metadata`: such as CharEncodingForConversion, OriginalCharEncoding, language, etc.
9. `ParseText`: The page parse text which varies in length depdnecing on `content.length` configuration.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected boolean`
  `checkRobotsTxt`
  `protected boolean`
  `dumpText`
  `protected boolean`
  `followRedirects`
  `protected String`
  `forceAsContentType`
  `protected HashMap<String,String>`
  `metadata`
  `protected URLNormalizers`
  `normalizers`

  ### Fields inherited from class org.apache.nutch.util.AbstractChecker

  `keepClientCnxOpen, stdin, tcpPort, usage`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `ParserChecker()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static void`
  `main(String[] args)`
  `protected int`
  `process(String url, StringBuilder output)`
  `int`
  `run(String[] args)`

  ### Methods inherited from class org.apache.nutch.util.AbstractChecker

  `getProtocolOutput, parseArgs, processSingle, processStdin, processTCP, run`

  ### Methods inherited from class org.apache.hadoop.conf.Configured

  `getConf, setConf`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

  ### Methods inherited from interface org.apache.hadoop.conf.Configurable

  `getConf, setConf`

- ## Field Details

  - ### normalizers

    protected URLNormalizers normalizers

  - ### dumpText

    protected boolean dumpText

  - ### followRedirects

    protected boolean followRedirects

  - ### checkRobotsTxt

    protected boolean checkRobotsTxt

  - ### metadata

    protected HashMap<String,String> metadata

  - ### forceAsContentType

    protected String forceAsContentType

- ## Constructor Details

  - ### ParserChecker

    public ParserChecker()

- ## Method Details

  - ### run

    public int run(String[] args) throws Exception
    Throws:
      `Exception`

  - ### process

    protected int process(String url, StringBuilder output) throws Exception
    Specified by:
      `process` in class `AbstractChecker`
    Throws:
      `Exception`

  - ### main

    public static void main(String[] args) throws Exception
    Throws:
      `Exception`