Package org.apache.nutch.protocol.httpclient

# Class Http

java.lang.Object
org.apache.nutch.protocol.http.api.HttpBase
org.apache.nutch.protocol.httpclient.Http

All Implemented Interfaces:
  `Configurable`, `Pluggable`, `Protocol`

public class Http extends HttpBase

This class is a protocol plugin that configures an HTTP client for Basic, Digest and NTLM authentication schemes for web server as well as proxy server. It takes care of HTTPS protocol as well as cookies in a single fetch session.

Documentation can be found on the Nutch HttpAuthenticationSchemes wiki page.

The original description of the motivation to support HttpPostAuthentication is also included on the Nutch wiki. Additionally HttpPostAuthentication development is documented at the NUTCH-827 Jira issue.

Author:
  Susam Pal

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected static final org.slf4j.Logger`
  `LOG`

  ### Fields inherited from class org.apache.nutch.protocol.http.api.HttpBase

  `accept, acceptCharset, acceptLanguage, BUFFER_SIZE, COOKIE, enableCookieHeader, enableIfModifiedsinceHeader, maxContent, maxCrawlDelay, maxDuration, partialAsTruncated, proxyException, proxyHost, proxyPort, proxyType, RESPONSE_TIME, responseTime, storeHttpHeaders, storeHttpRequest, storeIPAddress, storeProtocolVersions, timeout, tlsCheckCertificate, tlsPreferredCipherSuites, tlsPreferredProtocols, useHttp11, useHttp2, useProxy, userAgent`

  ### Fields inherited from interface org.apache.nutch.protocol.Protocol

  `X_POINT_ID`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `Http()`
  Constructs this plugin.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected Response`
  `getResponse(URL url, CrawlDatum datum, boolean redirect)`
  Fetches the `url` with a configured HTTP client and gets the response.
  `static void`
  `main(String[] args)`
  Main method.
  `void`
  `setConf(Configuration conf)`
  Reads the configuration from the Nutch configuration files and sets the configuration.

  ### Methods inherited from class org.apache.nutch.protocol.http.api.HttpBase

  `getAccept, getAcceptCharset, getAcceptLanguage, getConf, getCookie, getMaxContent, getMaxDuration, getProtocolOutput, getProxyHost, getProxyPort, getRobotRules, getRobotRules, getTimeout, getTlsPreferredCipherSuites, getTlsPreferredProtocols, getUseHttp11, getUserAgent, isCookieEnabled, isIfModifiedSinceEnabled, isProxyException, isStoreHttpHeaders, isStoreHttpRequest, isStoreIPAddress, isStorePartialAsTruncated, isTlsCheckCertificates, logConf, main, processDeflateEncoded, processGzipEncoded, useProxy, useProxy, useProxy`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

  ### Methods inherited from interface org.apache.nutch.protocol.Protocol

  `resolveUrl`

- ## Field Details

  - ### LOG

    protected static final org.slf4j.Logger LOG

- ## Constructor Details

  - ### Http

    public Http()
    Constructs this plugin.

- ## Method Details

  - ### setConf

    public void setConf(Configuration conf)
    Reads the configuration from the Nutch configuration files and sets the configuration.
    Specified by:
      `setConf` in interface `Configurable`
    Overrides:
      `setConf` in class `HttpBase`
    Parameters:
      `conf` - Configuration

  - ### main

    public static void main(String[] args) throws Exception
    Main method.
    Parameters:
      `args` - Command line arguments
    Throws:
      `Exception` - if a fatal error is encountered whilst running the program

  - ### getResponse

    protected Response getResponse(URL url, CrawlDatum datum, boolean redirect) throws ProtocolException, IOException
    Fetches the `url` with a configured HTTP client and gets the response.
    Specified by:
      `getResponse` in class `HttpBase`
    Parameters:
      `url` - URL to be fetched
      `datum` - Crawl data
      `redirect` - Follow redirects if and only if true
    Returns:
      HTTP response
    Throws:
      `ProtocolException`
      `IOException`