Package org.apache.nutch.protocol.file

Class File

java.lang.Object
org.apache.nutch.protocol.file.File
All Implemented Interfaces:
Configurable, Pluggable, Protocol
public class File extends Object implements Protocol
This class is a protocol plugin used for file: scheme. It creates FileResponse object and gets the content of the url from it. Configurable parameters are file.content.limit and file.crawl.parent in nutch-default.xml defined under "file properties" section.

  • Field Details

    • LOG

      protected static final org.slf4j.Logger LOG

  • Constructor Details

    • File

      public File()

  • Method Details

    • setConf

      public void setConf(Configuration conf)
      Set the Configuration object
      Specified by:
      setConf in interface Configurable

    • getConf

      public Configuration getConf()
      Get the Configuration object
      Specified by:
      getConf in interface Configurable

    • setMaxContentLength

      public void setMaxContentLength(int maxContentLength)
      Set the length after at which content is truncated.
      Parameters:
      maxContentLength - max content in bytes

    • getProtocolOutput

      public ProtocolOutput getProtocolOutput(Text url, CrawlDatum datum)
      Creates a FileResponse object corresponding to the url and return a ProtocolOutput object as per the content received
      Specified by:
      getProtocolOutput in interface Protocol
      Parameters:
      url - Text containing the url
      datum - The CrawlDatum object corresponding to the url
      Returns:
      ProtocolOutput object for the content of the file indicated by url

    • main

      public static void main(String[] args) throws Exception
      Quick way for running this class. Useful for debugging.
      Parameters:
      args - run with no args to print help
      Throws:
      Exception - if there is a fatal error running this class with the given input

    • getRobotRules

      public crawlercommons.robots.BaseRobotRules getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)
      No robots parsing is done for file protocol. So this returns a set of empty rules which will allow every url.
      Specified by:
      getRobotRules in interface Protocol
      Parameters:
      url - URL to check
      datum - page datum
      robotsTxtContent - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response Content is appended to the passed list. If null is passed nothing is stored.
      Returns:
      robot rules (specific for this URL or default), never null

    • getRobotRules

      public crawlercommons.robots.BaseRobotRules getRobotRules(URL url, CrawlDatum datum, List<Content> robotsTxtContent)
      No robots parsing is done for file protocol. So this returns a set of empty rules which will allow every url.
      Specified by:
      getRobotRules in interface Protocol
      Parameters:
      url - URL to check
      datum - page datum
      robotsTxtContent - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response Content is appended to the passed list. If null is passed nothing is stored.
      Returns:
      robot rules (specific for this URL or default), never null