Package org.apache.nutch.protocol.ftp

# Class Ftp

java.lang.Object
org.apache.nutch.protocol.ftp.Ftp

All Implemented Interfaces:
  `Configurable`, `Pluggable`, `Protocol`

public class Ftp extends Object implements Protocol
This class is a protocol plugin used for ftp: scheme. It creates `FtpResponse` object and gets the content of the url from it. Configurable parameters are `ftp.username`, `ftp.password`, `ftp.content.limit`, `ftp.timeout`, `ftp.server.timeout`, `ftp.password`, `ftp.keep.connection` and `ftp.follow.talk` . For details see "FTP properties" section in `nutch-default.xml`.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected static final org.slf4j.Logger`
  `LOG`

  ### Fields inherited from interface org.apache.nutch.protocol.Protocol

  `X_POINT_ID`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `Ftp()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected void`
  `finalize()`
  `int`
  `getBufferSize()`
  `Configuration`
  `getConf()`
  Get the `Configuration` object
  `ProtocolOutput`
  `getProtocolOutput(Text url, CrawlDatum datum)`
  Creates a `FtpResponse` object corresponding to the url and returns a `ProtocolOutput` object as per the content received
  `crawlercommons.robots.BaseRobotRules`
  `getRobotRules(URL url, CrawlDatum datum, List<Content> robotsTxtContent)`
  Get the robots rules for a given url
  `crawlercommons.robots.BaseRobotRules`
  `getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)`
  Get the robots rules for a given url
  `static void`
  `main(String[] args)`
  For debugging.
  `void`
  `setConf(Configuration conf)`
  Set the `Configuration` object
  `void`
  `setFollowTalk(boolean followTalk)`
  Set followTalk i.e.
  `void`
  `setKeepConnection(boolean keepConnection)`
  Whether to keep ftp connection.
  `void`
  `setMaxContentLength(int length)`
  Set the length after at which content is truncated.
  `void`
  `setTimeout(int to)`
  Set the timeout.

  ### Methods inherited from class java.lang.Object

  `clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

  ### Methods inherited from interface org.apache.nutch.protocol.Protocol

  `resolveUrl`

- ## Field Details

  - ### LOG

    protected static final org.slf4j.Logger LOG

- ## Constructor Details

  - ### Ftp

    public Ftp()

- ## Method Details

  - ### setTimeout

    public void setTimeout(int to)
    Set the timeout.
    Parameters:
      `to` - a maximum timeout in milliseconds

  - ### setMaxContentLength

    public void setMaxContentLength(int length)
    Set the length after at which content is truncated.
    Parameters:
      `length` - max content length in bytes

  - ### setFollowTalk

    public void setFollowTalk(boolean followTalk)
    Set followTalk i.e. to log dialogue between our client and remote server. Useful for debugging.
    Parameters:
      `followTalk` - if true will follow, false by default

  - ### setKeepConnection

    public void setKeepConnection(boolean keepConnection)
    Whether to keep ftp connection. Useful if crawling same host again and again. When set to true, it avoids connection, login and dir list parser setup for subsequent URLs. If it is set to true, however, you must make sure (roughly): (1) ftp.timeout is less than ftp.server.timeout (2) ftp.timeout is larger than (fetcher.threads.fetch * fetcher.server.delay) Otherwise there will be too many "delete client because idled too long" messages in thread logs.
    Parameters:
      `keepConnection` - if true we will keep the connection, false by default

  - ### getProtocolOutput

    public ProtocolOutput getProtocolOutput(Text url, CrawlDatum datum)
    Creates a `FtpResponse` object corresponding to the url and returns a `ProtocolOutput` object as per the content received
    Specified by:
      `getProtocolOutput` in interface `Protocol`
    Parameters:
      `url` - Text containing the ftp url
      `datum` - The CrawlDatum object corresponding to the url
    Returns:
      `ProtocolOutput` object for the url

  - ### finalize

    protected void finalize()
    Overrides:
      `finalize` in class `Object`

  - ### main

    public static void main(String[] args) throws Exception
    For debugging.
    Parameters:
      `args` - run with no args for help
    Throws:
      `Exception` - if there is an error running this program

  - ### setConf

    public void setConf(Configuration conf)
    Set the `Configuration` object
    Specified by:
      `setConf` in interface `Configurable`

  - ### getConf

    public Configuration getConf()
    Get the `Configuration` object
    Specified by:
      `getConf` in interface `Configurable`

  - ### getRobotRules

    public crawlercommons.robots.BaseRobotRules getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)
    Get the robots rules for a given url
    Specified by:
      `getRobotRules` in interface `Protocol`
    Parameters:
      `url` - URL to check
      `datum` - page datum
      `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
    Returns:
      robot rules (specific for this URL or default), never null

  - ### getRobotRules

    public crawlercommons.robots.BaseRobotRules getRobotRules(URL url, CrawlDatum datum, List<Content> robotsTxtContent)
    Get the robots rules for a given url
    Specified by:
      `getRobotRules` in interface `Protocol`
    Parameters:
      `url` - URL to check
      `datum` - page datum
      `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
    Returns:
      robot rules (specific for this URL or default), never null

  - ### getBufferSize

    public int getBufferSize()