Package org.apache.nutch.tools.arc

# Class ArcSegmentCreator

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.nutch.tools.arc.ArcSegmentCreator

All Implemented Interfaces:
  `Configurable`, `Tool`

public class ArcSegmentCreator extends Configured implements Tool

The `ArcSegmentCreator` is a replacement for fetcher that will take arc files as input and produce a nutch segment as output.

Arc files are tars of compressed gzips which are produced by both the internet archive project and the grub distributed crawler project.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `ArcSegmentCreator.ArcSegmentCreatorMapper`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `URL_VERSION`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `ArcSegmentCreator()`
  `ArcSegmentCreator(Configuration conf)`
  Constructor that sets the job configuration.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `close()`
  `void`
  `createSegments(Path arcFiles, Path segmentsOutDir)`
  Creates the arc files to segments job.
  `static String`
  `generateSegmentName()`
  Generates a random name for the segments.
  `static void`
  `main(String[] args)`
  `int`
  `run(String[] args)`

  ### Methods inherited from class org.apache.hadoop.conf.Configured

  `getConf, setConf`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

  ### Methods inherited from interface org.apache.hadoop.conf.Configurable

  `getConf, setConf`

- ## Field Details

  - ### URL_VERSION

    public static final String URL_VERSION
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### ArcSegmentCreator

    public ArcSegmentCreator()

  - ### ArcSegmentCreator

    public ArcSegmentCreator(Configuration conf)
    Constructor that sets the job configuration.
    Parameters:
      `conf` - a populated `Configuration`

- ## Method Details

  - ### generateSegmentName

    public static String generateSegmentName()
    Generates a random name for the segments.
    Returns:
      The generated segment name.

  - ### close

    public void close()

  - ### createSegments

    public void createSegments(Path arcFiles, Path segmentsOutDir) throws IOException, InterruptedException, ClassNotFoundException
    Creates the arc files to segments job.
    Parameters:
      `arcFiles` - The path to the directory holding the arc files
      `segmentsOutDir` - The output directory for writing the segments
    Throws:
      `IOException` - If an IO error occurs while running the job.
      `InterruptedException` - if this `Job` is interrupted
      `ClassNotFoundException` - if there is an error locating a class during runtime

  - ### main

    public static void main(String[] args) throws Exception
    Throws:
      `Exception`

  - ### run

    public int run(String[] args) throws Exception
    Specified by:
      `run` in interface `Tool`
    Throws:
      `Exception`