Package org.apache.nutch.protocol.file

# Class File

java.lang.Object
org.apache.nutch.protocol.file.File

All Implemented Interfaces:
  `Configurable`, `Pluggable`, `Protocol`

public class File extends Object implements Protocol
This class is a protocol plugin used for file: scheme. It creates `FileResponse` object and gets the content of the url from it. Configurable parameters are `file.content.limit` and `file.crawl.parent` in nutch-default.xml defined under "file properties" section.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected static final org.slf4j.Logger`
  `LOG`

  ### Fields inherited from interface org.apache.nutch.protocol.Protocol

  `X_POINT_ID`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `File()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `Configuration`
  `getConf()`
  Get the `Configuration` object
  `ProtocolOutput`
  `getProtocolOutput(Text url, CrawlDatum datum)`
  Creates a `FileResponse` object corresponding to the url and return a `ProtocolOutput` object as per the content received
  `crawlercommons.robots.BaseRobotRules`
  `getRobotRules(URL url, CrawlDatum datum, List<Content> robotsTxtContent)`
  No robots parsing is done for file protocol.
  `crawlercommons.robots.BaseRobotRules`
  `getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)`
  No robots parsing is done for file protocol.
  `static void`
  `main(String[] args)`
  Quick way for running this class.
  `void`
  `setConf(Configuration conf)`
  Set the `Configuration` object
  `void`
  `setMaxContentLength(int maxContentLength)`
  Set the length after at which content is truncated.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

  ### Methods inherited from interface org.apache.nutch.protocol.Protocol

  `resolveUrl`

- ## Field Details

  - ### LOG

    protected static final org.slf4j.Logger LOG

- ## Constructor Details

  - ### File

    public File()

- ## Method Details

  - ### setConf

    public void setConf(Configuration conf)
    Set the `Configuration` object
    Specified by:
      `setConf` in interface `Configurable`

  - ### getConf

    public Configuration getConf()
    Get the `Configuration` object
    Specified by:
      `getConf` in interface `Configurable`

  - ### setMaxContentLength

    public void setMaxContentLength(int maxContentLength)
    Set the length after at which content is truncated.
    Parameters:
      `maxContentLength` - max content in bytes

  - ### getProtocolOutput

    public ProtocolOutput getProtocolOutput(Text url, CrawlDatum datum)
    Creates a `FileResponse` object corresponding to the url and return a `ProtocolOutput` object as per the content received
    Specified by:
      `getProtocolOutput` in interface `Protocol`
    Parameters:
      `url` - Text containing the url
      `datum` - The CrawlDatum object corresponding to the url
    Returns:
      `ProtocolOutput` object for the content of the file indicated by url

  - ### main

    public static void main(String[] args) throws Exception
    Quick way for running this class. Useful for debugging.
    Parameters:
      `args` - run with no args to print help
    Throws:
      `Exception` - if there is a fatal error running this class with the given input

  - ### getRobotRules

    public crawlercommons.robots.BaseRobotRules getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)
    No robots parsing is done for file protocol. So this returns a set of empty rules which will allow every url.
    Specified by:
      `getRobotRules` in interface `Protocol`
    Parameters:
      `url` - URL to check
      `datum` - page datum
      `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
    Returns:
      robot rules (specific for this URL or default), never null

  - ### getRobotRules

    public crawlercommons.robots.BaseRobotRules getRobotRules(URL url, CrawlDatum datum, List<Content> robotsTxtContent)
    No robots parsing is done for file protocol. So this returns a set of empty rules which will allow every url.
    Specified by:
      `getRobotRules` in interface `Protocol`
    Parameters:
      `url` - URL to check
      `datum` - page datum
      `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
    Returns:
      robot rules (specific for this URL or default), never null