Package org.apache.nutch.protocol.okhttp

Class OkHttp

java.lang.Object

org.apache.nutch.protocol.http.api.HttpBase

org.apache.nutch.protocol.okhttp.OkHttp

All Implemented Interfaces:

Configurable, Pluggable, Protocol

public class OkHttp
extends HttpBase

Field Summary

Fields

Modifier and Type	Field	Description
protected static final org.slf4j.Logger	LOG

Fields inherited from class org.apache.nutch.protocol.http.api.HttpBase

accept, acceptCharset, acceptLanguage, BUFFER_SIZE, COOKIE, enableCookieHeader, enableIfModifiedsinceHeader, maxContent, maxCrawlDelay, maxDuration, partialAsTruncated, proxyException, proxyHost, proxyPort, proxyType, RESPONSE_TIME, responseTime, storeHttpHeaders, storeHttpRequest, storeIPAddress, storeProtocolVersions, timeout, tlsCheckCertificate, tlsPreferredCipherSuites, tlsPreferredProtocols, useHttp11, useHttp2, useProxy, userAgent

Fields inherited from interface org.apache.nutch.protocol.Protocol

X_POINT_ID

Constructor Summary

Constructors

Constructor	Description
OkHttp()

Method Summary

Modifier and Type	Method	Description
protected okhttp3.OkHttpClient	getClient(URL url)	Distribute hosts over clients by host name
protected List<String[]>	getCustomRequestHeaders()
protected Response	getResponse(URL url, CrawlDatum datum, boolean redirect)
static void	main(String[] args)
URL	resolveUrl(URL base, String relative)	Resolve a relative URL using OkHttp's HttpUrl parser, which is more lenient than Java's URL (handles malformed protocol-relative slashes such as https:////host/path, IDN→punycode, host case normalization, etc.).
void	setConf(Configuration conf)

Methods inherited from class org.apache.nutch.protocol.http.api.HttpBase

getAccept, getAcceptCharset, getAcceptLanguage, getConf, getCookie, getMaxContent, getMaxDuration, getProtocolOutput, getProxyHost, getProxyPort, getRobotRules, getRobotRules, getTimeout, getTlsPreferredCipherSuites, getTlsPreferredProtocols, getUseHttp11, getUserAgent, isCookieEnabled, isIfModifiedSinceEnabled, isProxyException, isStoreHttpHeaders, isStoreHttpRequest, isStoreIPAddress, isStorePartialAsTruncated, isTlsCheckCertificates, logConf, main, processDeflateEncoded, processGzipEncoded, useProxy, useProxy, useProxy

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOG

protected static final org.slf4j.Logger LOG

Constructor Details

OkHttp

public OkHttp()

Method Details

setConf

public void setConf(Configuration conf)

Specified by:

setConf in interface Configurable

Overrides:

setConf in class HttpBase

getCustomRequestHeaders

protected List<String[]> getCustomRequestHeaders()

getClient

protected okhttp3.OkHttpClient getClient(URL url)

Distribute hosts over clients by host name

Parameters:

url - URL to fetch

Returns:

client responsible to fetch the given URL

getResponse

protected Response getResponse(URL url,
 CrawlDatum datum,
 boolean redirect)
                        throws ProtocolException,
IOException

Specified by:

getResponse in class HttpBase

Throws:

ProtocolException

IOException

resolveUrl

public URL resolveUrl(URL base,
 String relative)
               throws MalformedURLException

Resolve a relative URL using OkHttp's HttpUrl parser, which is more lenient than Java's URL (handles malformed protocol-relative slashes such as https:////host/path, IDN→punycode, host case normalization, etc.). Falls back to Java URL if HttpUrl cannot parse.

Parameters:

base - the base URL the relative URL is resolved against

relative - the relative URL string (typically a Location: header value)

Returns:

resolved absolute URL

Throws:

MalformedURLException - if the URL is malformed

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception