Package org.apache.nutch.protocol

# Interface Protocol

All Superinterfaces:
  `Configurable`, `Pluggable`

All Known Implementing Classes:
  `File`, `Ftp`, `Http`, `Http`, `Http`, `Http`, `Http`, `HttpBase`, `OkHttp`

public interface Protocol extends Pluggable, Configurable
A retriever of url content. Implemented by protocol extensions.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `X_POINT_ID`
  The name of the extension point.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `ProtocolOutput`
  `getProtocolOutput(Text url, CrawlDatum datum)`
  Get the `ProtocolOutput` for a given url and crawldatum
  `default crawlercommons.robots.BaseRobotRules`
  `getRobotRules(URL url, CrawlDatum datum, List<Content> robotsTxtContent)`
  Retrieve robot rules applicable for this URL.
  `crawlercommons.robots.BaseRobotRules`
  `getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)`
  Retrieve robot rules applicable for this URL.
  `default URL`
  `resolveUrl(URL base, String relative)`
  Resolve a relative URL against a base URL using the protocol's URL library.

  ### Methods inherited from interface org.apache.hadoop.conf.Configurable

  `getConf, setConf`

- ## Field Details

  - ### X_POINT_ID

    static final String X_POINT_ID
    The name of the extension point.

- ## Method Details

  - ### getProtocolOutput

    ProtocolOutput getProtocolOutput(Text url, CrawlDatum datum)
    Get the `ProtocolOutput` for a given url and crawldatum
    Parameters:
      `url` - canonical url
      `datum` - associated `CrawlDatum`
    Returns:
      the `ProtocolOutput`

  - ### resolveUrl

    default URL resolveUrl(URL base, String relative) throws MalformedURLException
    Resolve a relative URL against a base URL using the protocol's URL library.
    Parameters:
      `base` - the base URL the relative URL is resolved against
      `relative` - the relative URL string (typically a Location: header value)
    Returns:
      resolved absolute URL
    Throws:
      `MalformedURLException` - if the URL is malformed

  - ### getRobotRules

    crawlercommons.robots.BaseRobotRules getRobotRules(Text url, CrawlDatum datum, List<Content> robotsTxtContent)
    Retrieve robot rules applicable for this URL.
    Parameters:
      `url` - URL to check
      `datum` - page datum
      `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
    Returns:
      robot rules (specific for this URL or default), never null

  - ### getRobotRules

    default crawlercommons.robots.BaseRobotRules getRobotRules(URL url, CrawlDatum datum, List<Content> robotsTxtContent)
    Retrieve robot rules applicable for this URL.
    Parameters:
      `url` - URL to check
      `datum` - page datum
      `robotsTxtContent` - container to store responses when fetching the robots.txt file for debugging or archival purposes. Instead of a robots.txt file, it may include redirects or an error page (404, etc.). Response `Content` is appended to the passed list. If null is passed nothing is stored.
    Returns:
      robot rules (specific for this URL or default), never null