Package org.apache.nutch.protocol.ftp

# Class FtpRobotRulesParser

java.lang.Object
org.apache.nutch.protocol.RobotRulesParser
org.apache.nutch.protocol.ftp.FtpRobotRulesParser

All Implemented Interfaces:
  `Configurable`, `Tool`

public class FtpRobotRulesParser extends RobotRulesParser
This class is used for parsing robots for urls belonging to FTP protocol. It extends the generic `RobotRulesParser` class and contains Ftp protocol specific implementation for obtaining the robots file.

- ## Field Summary

  ### Fields inherited from class org.apache.nutch.protocol.RobotRulesParser

  `agentNames, allowList, CACHE, conf, DEFER_VISIT_RULES, EMPTY_RULES, FORBID_ALL_RULES, maxNumRedirects`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `FtpRobotRulesParser(Configuration conf)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `crawlercommons.robots.BaseRobotRules`
  `getRobotRulesSet(Protocol ftp, URL url, List<Content> robotsTxtContent)`
  The hosts for which the caching of robots rules is yet to be done, it sends a Ftp request to the host corresponding to the `URL` passed, gets robots file, parses the rules and caches the rules object to avoid re-work in future.

  ### Methods inherited from class org.apache.nutch.protocol.RobotRulesParser

  `getConf, getRobotRulesSet, isAllowListed, main, parseRules, parseRules, run, setConf`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### FtpRobotRulesParser

    public FtpRobotRulesParser(Configuration conf)

- ## Method Details

  - ### getRobotRulesSet

    public crawlercommons.robots.BaseRobotRules getRobotRulesSet(Protocol ftp, URL url, List<Content> robotsTxtContent)
    The hosts for which the caching of robots rules is yet to be done, it sends a Ftp request to the host corresponding to the `URL` passed, gets robots file, parses the rules and caches the rules object to avoid re-work in future.
    Specified by:
      `getRobotRulesSet` in class `RobotRulesParser`
    Parameters:
      `ftp` - The `Protocol` object
      `url` - URL
      `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
    Returns:
      robotRules A `BaseRobotRules` object for the rules