Package org.apache.nutch.parse

# Class ParseSegment

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.nutch.util.NutchTool
org.apache.nutch.parse.ParseSegment

All Implemented Interfaces:
  `Configurable`, `Tool`

public class ParseSegment extends NutchTool implements Tool

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `ParseSegment.ParseSegmentMapper`
  `static class`
  `ParseSegment.ParseSegmentPartitioner`
  Sends LATENCY_KEY to partition 0 so one reducer merges all TDigests.
  `static class`
  `ParseSegment.ParseSegmentReducer`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `DELETE_FAILED_PARSE`
  Configuration property to delete documents that failed to be parsed.
  `static final String`
  `SKIP_TRUNCATED`

  ### Fields inherited from class org.apache.nutch.util.NutchTool

  `currentJob, currentJobNum, numJobs, results, status`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `ParseSegment()`
  `ParseSegment(Configuration conf)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static boolean`
  `isTruncated(Content content)`
  Checks if the page's content is truncated.
  `static void`
  `main(String[] args)`
  `void`
  `parse(Path segment)`
  `int`
  `run(String[] args)`
  `Map<String,Object>`
  `run(Map<String,Object> args, String crawlId)`
  Programmatic entry point for `NutchTool` callers.

  ### Methods inherited from class org.apache.nutch.util.NutchTool

  `getProgress, getStatus, killJob, setConf, stopJob`

  ### Methods inherited from class org.apache.hadoop.conf.Configured

  `getConf`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

  ### Methods inherited from interface org.apache.hadoop.conf.Configurable

  `getConf, setConf`

- ## Field Details

  - ### SKIP_TRUNCATED

    public static final String SKIP_TRUNCATED
    See Also:
      - Constant Field Values

  - ### DELETE_FAILED_PARSE

    public static final String DELETE_FAILED_PARSE
    Configuration property to delete documents that failed to be parsed. Also used in Fetcher and Indexer.
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### ParseSegment

    public ParseSegment()

  - ### ParseSegment

    public ParseSegment(Configuration conf)

- ## Method Details

  - ### isTruncated

    public static boolean isTruncated(Content content)
    Checks if the page's content is truncated.
    Parameters:
      `content` - the response `Content`
    Returns:
      If the page is truncated `true`. When it is not, or when it couldn't be determined, `false`.

  - ### parse

    public void parse(Path segment) throws IOException, InterruptedException, ClassNotFoundException
    Throws:
      `IOException`
      `InterruptedException`
      `ClassNotFoundException`

  - ### main

    public static void main(String[] args) throws Exception
    Throws:
      `Exception`

  - ### run

    public int run(String[] args) throws Exception
    Specified by:
      `run` in interface `Tool`
    Throws:
      `Exception`

  - ### run

    public Map<String,Object> run(Map<String,Object> args, String crawlId) throws Exception
    Programmatic entry point for `NutchTool` callers.
    Specified by:
      `run` in class `NutchTool`
    Parameters:
      `args` - a `Map` of arguments to be run with the tool
      `crawlId` - a crawl identifier to associate with the tool invocation
    Returns:
      Map results object if tool executes successfully otherwise null
    Throws:
      `Exception` - if there is an error during the tool execution