Package org.apache.nutch.tools.arc

# Class ArcRecordReader

java.lang.Object
org.apache.hadoop.mapreduce.RecordReader<Text,BytesWritable>
org.apache.nutch.tools.arc.ArcRecordReader

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public class ArcRecordReader extends RecordReader<Text,BytesWritable>
The `ArchRecordReader` class provides a record reader which reads records from arc files. Arc files are essentially tars of gzips. Each record in an arc file is a compressed gzip. Multiple records are concatenated together to form a complete arc. For more information on the arc file format

See Also:
  - ArcFileFormat. Arc files are used by the Internet Archive and grub projects.
  - archive.org
  - grub.org

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected Configuration`
  `conf`
  `protected long`
  `fileLen`
  `protected FSDataInputStream`
  `in`
  `protected long`
  `pos`
  `protected long`
  `splitEnd`
  `protected long`
  `splitLen`
  `protected long`
  `splitStart`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `ArcRecordReader(Configuration conf, FileSplit split)`
  Constructor that sets the configuration and file split.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `close()`
  Closes the record reader resources.
  `Text`
  `createKey()`
  Creates a new instance of the `Text` object for the key.
  `BytesWritable`
  `createValue()`
  Creates a new instance of the `BytesWritable` object for the key
  `Text`
  `getCurrentKey()`
  `BytesWritable`
  `getCurrentValue()`
  `long`
  `getPos()`
  Returns the current position in the file.
  `float`
  `getProgress()`
  Returns the percentage of progress in processing the file.
  `void`
  `initialize(InputSplit split, TaskAttemptContext context)`
  `static boolean`
  `isMagic(byte[] input)`
  Returns true if the byte array passed matches the gzip header magic number.
  `boolean`
  `next(Text key, BytesWritable value)`
  Returns true if the next record in the split is read into the key and value pair.
  `boolean`
  `nextKeyValue()`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### conf

    protected Configuration conf

  - ### splitStart

    protected long splitStart

  - ### pos

    protected long pos

  - ### splitEnd

    protected long splitEnd

  - ### splitLen

    protected long splitLen

  - ### fileLen

    protected long fileLen

  - ### in

    protected FSDataInputStream in

- ## Constructor Details

  - ### ArcRecordReader

    public ArcRecordReader(Configuration conf, FileSplit split) throws IOException
    Constructor that sets the configuration and file split.
    Parameters:
      `conf` - The job configuration.
      `split` - The file split to read from.
    Throws:
      `IOException` - If an IO error occurs while initializing file split.

- ## Method Details

  - ### isMagic

    public static boolean isMagic(byte[] input)

    Returns true if the byte array passed matches the gzip header magic number.

    Parameters:
      `input` - The byte array to check.
    Returns:
      True if the byte array matches the gzip header magic number.

  - ### close

    public void close() throws IOException
    Closes the record reader resources.
    Specified by:
      `close` in interface `AutoCloseable`
    Specified by:
      `close` in interface `Closeable`
    Specified by:
      `close` in class `RecordReader<Text,BytesWritable>`
    Throws:
      `IOException`

  - ### createKey

    public Text createKey()
    Creates a new instance of the `Text` object for the key.
    Returns:
      `Text`

  - ### createValue

    public BytesWritable createValue()
    Creates a new instance of the `BytesWritable` object for the key
    Returns:
      `BytesWritable`

  - ### getPos

    public long getPos() throws IOException
    Returns the current position in the file.
    Returns:
      The long of the current position in the file.
    Throws:
      `IOException` - if there is a fatal I/O error reading the position within the `FSDataInputStream`

  - ### getProgress

    public float getProgress() throws IOException
    Returns the percentage of progress in processing the file. This will be represented as a float from 0 to 1 with 1 being 100% completed.
    Specified by:
      `getProgress` in class `RecordReader<Text,BytesWritable>`
    Returns:
      The percentage of progress as a float from 0 to 1.
    Throws:
      `IOException`

  - ### getCurrentValue

    public BytesWritable getCurrentValue()
    Specified by:
      `getCurrentValue` in class `RecordReader<Text,BytesWritable>`

  - ### getCurrentKey

    public Text getCurrentKey()
    Specified by:
      `getCurrentKey` in class `RecordReader<Text,BytesWritable>`

  - ### nextKeyValue

    public boolean nextKeyValue()
    Specified by:
      `nextKeyValue` in class `RecordReader<Text,BytesWritable>`

  - ### initialize

    public void initialize(InputSplit split, TaskAttemptContext context)
    Specified by:
      `initialize` in class `RecordReader<Text,BytesWritable>`

  - ### next

    public boolean next(Text key, BytesWritable value) throws IOException

    Returns true if the next record in the split is read into the key and value pair. The key will be the arc record header and the values will be the raw content bytes of the arc record.

    Parameters:
      `key` - The record key
      `value` - The record value
    Returns:
      True if the next record is read.
    Throws:
      `IOException` - If an error occurs while reading the record value.