Package org.apache.nutch.tools

Class FileDumper

java.lang.Object
org.apache.nutch.tools.FileDumper
public class FileDumper extends Object
The file dumper tool enables one to reverse generate the raw content from Nutch segment data directories.

The tool has a number of immediate uses:

  1. one can see what a page looked like at the time it was crawled
  2. one can see different media types acquired as part of the crawl
  3. it enables us to see webpages before we augment them with additional metadata, this can be handy for providing a provenance trail for your crawl data.

Upon successful completion the tool displays a very convenient JSON snippet detailing the mimetype classifications and the counts of documents which fall into those classifications. An example is as follows: 

 
 INFO: File Types:
   TOTAL Stats:
    [
     {"mimeType":"application/xml","count":"19"}
     {"mimeType":"image/png","count":"47"}
     {"mimeType":"image/jpeg","count":"141"}
     {"mimeType":"image/vnd.microsoft.icon","count":"4"}
     {"mimeType":"text/plain","count":"89"}
     {"mimeType":"video/quicktime","count":"2"}
     {"mimeType":"image/gif","count":"63"}
     {"mimeType":"application/xhtml+xml","count":"1670"}
     {"mimeType":"application/octet-stream","count":"40"}
     {"mimeType":"text/html","count":"1863"}
   ]

   FILTER Stats:
   [
     {"mimeType":"image/png","count":"47"}
     {"mimeType":"image/jpeg","count":"141"}
     {"mimeType":"image/vnd.microsoft.icon","count":"4"}
     {"mimeType":"video/quicktime","count":"2"}
     {"mimeType":"image/gif","count":"63"}
   ]

In the case above, the tool would have been run with the -mimeType image/png image/jpeg image/vnd.microsoft.icon video/quicktime image/gif flag and corresponding values activated.

  • Constructor Summary

    Constructors
    Constructor
    Description
    FileDumper()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    dump(File outputDir, File segmentRootDir, String[] mimeTypes, boolean flatDir, boolean mimeTypeStats, boolean reverseURLDump)
    Dumps the reverse engineered raw content from the provided segment directories if a parent directory contains more than one segment, otherwise a single segment can be passed as an argument.
    static void
    main(String[] args)
    Main method for invoking this tool

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • FileDumper

      public FileDumper()

  • Method Details

    • dump

      public void dump(File outputDir, File segmentRootDir, String[] mimeTypes, boolean flatDir, boolean mimeTypeStats, boolean reverseURLDump) throws Exception
      Dumps the reverse engineered raw content from the provided segment directories if a parent directory contains more than one segment, otherwise a single segment can be passed as an argument.
      Parameters:
      outputDir - the directory you wish to dump the raw content to. This directory will be created.
      segmentRootDir - a directory containing one or more segments.
      mimeTypes - an array of mime types we have to dump, all others will be filtered out.
      flatDir - a boolean flag specifying whether the output directory should contain only files instead of using nested directories to prevent naming conflicts.
      mimeTypeStats - a flag indicating whether mimetype stats should be displayed instead of dumping files.
      reverseURLDump - whether to reverse the URLs when they are written to disk
      Throws:
      Exception - if there is a fatal error dumping files to disk

    • main

      public static void main(String[] args) throws Exception
      Main method for invoking this tool
      Parameters:
      args - 1) output directory (which will be created) to host the raw data and 2) a directory containing one or more segments.
      Throws:
      Exception - if there is a fatal error running this tool