Package org.apache.nutch.tools.arc

Class ArcSegmentCreator

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.nutch.tools.arc.ArcSegmentCreator

All Implemented Interfaces:

Configurable, Tool

public class ArcSegmentCreator
extends Configured
implements Tool

The ArcSegmentCreator is a replacement for fetcher that will take arc files as input and produce a nutch segment as output.

Arc files are tars of compressed gzips which are produced by both the internet archive project and the grub distributed crawler project.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ArcSegmentCreator.ArcSegmentCreatorMapper

Field Summary

Fields

Modifier and Type	Field	Description
static final String	URL_VERSION

Constructor Summary

Constructors

Constructor	Description
ArcSegmentCreator()
ArcSegmentCreator(Configuration conf)	Constructor that sets the job configuration.

Method Summary

Modifier and Type	Method	Description
void	close()
void	createSegments(Path arcFiles, Path segmentsOutDir)	Creates the arc files to segments job.
static String	generateSegmentName()	Generates a random name for the segments.
static void	main(String[] args)
int	run(String[] args)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Details

URL_VERSION

public static final String URL_VERSION

See Also:

• Constant Field Values

Constructor Details

ArcSegmentCreator

public ArcSegmentCreator()

ArcSegmentCreator

public ArcSegmentCreator(Configuration conf)

Constructor that sets the job configuration.

Parameters:

conf - a populated Configuration

Method Details

generateSegmentName

public static String generateSegmentName()

Generates a random name for the segments.

Returns:

The generated segment name.

close

public void close()

createSegments

public void createSegments(Path arcFiles,
 Path segmentsOutDir)
                    throws IOException,
InterruptedException,
ClassNotFoundException

Creates the arc files to segments job.

Parameters:

arcFiles - The path to the directory holding the arc files

segmentsOutDir - The output directory for writing the segments

Throws:

IOException - If an IO error occurs while running the job.

InterruptedException - if this Job is interrupted

ClassNotFoundException - if there is an error locating a class during runtime

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface Tool

Throws:

Exception