Package org.apache.nutch.tools

Class CommonCrawlFormatWARC

java.lang.Object

org.apache.nutch.tools.AbstractCommonCrawlFormat

org.apache.nutch.tools.CommonCrawlFormatWARC

All Implemented Interfaces:

Closeable, AutoCloseable, CommonCrawlFormat

public class CommonCrawlFormatWARC
extends AbstractCommonCrawlFormat

Field Summary

Fields

Modifier and Type	Field	Description
static final String	MAX_WARC_FILE_SIZE
static final String	TEMPLATE

Fields inherited from class org.apache.nutch.tools.AbstractCommonCrawlFormat

conf, content, inLinks, jsonArray, keyPrefix, LOG, metadata, reverseKey, reverseKeyValue, simpleDateFormat, url

Constructor Summary

Constructors

Constructor	Description
CommonCrawlFormatWARC(String url, Content content, Metadata metadata, Configuration nutchConf, CommonCrawlConfig config, ParseData parseData)
CommonCrawlFormatWARC(Configuration nutchConf, CommonCrawlConfig config)

Method Summary

Modifier and Type	Method	Description
void	close()	Optional method that could be implemented if the actual format needs some close procedure.
protected void	closeArray(String key, boolean nested, boolean newline)
protected void	closeObject(String key)
protected String	generateJson()
String	getJsonData()	Get a string representation of the JSON structure of the URL content.
String	getJsonData(String url, Content content, Metadata metadata, ParseData parseData)	Returns a string representation of the JSON structure of the URL content.
protected void	startArray(String key, boolean nested, boolean newline)
protected void	startObject(String key)
protected void	writeArrayValue(String value)
protected void	writeKeyNull(String key)
protected void	writeKeyValue(String key, String value)
protected URI	writeRequest(URI id)
protected URI	writeResponse()

Methods inherited from class org.apache.nutch.tools.AbstractCommonCrawlFormat

getImported, getInLinks, getJsonData, getKey, getMethod, getRequestAccept, getRequestAcceptEncoding, getRequestAcceptLanguage, getRequestContactEmail, getRequestContactName, getRequestHostAddress, getRequestHostName, getRequestRobots, getRequestSoftware, getRequestUserAgent, getResponseAddress, getResponseContent, getResponseContentEncoding, getResponseContentType, getResponseDate, getResponseHostName, getResponseServer, getResponseStatus, getTimestamp, getUrl, setInLinks

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

MAX_WARC_FILE_SIZE

public static final String MAX_WARC_FILE_SIZE

See Also:

• Constant Field Values

TEMPLATE

public static final String TEMPLATE

See Also:

• Constant Field Values

Constructor Details

CommonCrawlFormatWARC

public CommonCrawlFormatWARC(Configuration nutchConf,
 CommonCrawlConfig config)
                      throws IOException

Throws:

IOException

CommonCrawlFormatWARC

public CommonCrawlFormatWARC(String url,
 Content content,
 Metadata metadata,
 Configuration nutchConf,
 CommonCrawlConfig config,
 ParseData parseData)
                      throws IOException

Throws:

IOException

Method Details

getJsonData

public String getJsonData(String url,
 Content content,
 Metadata metadata,
 ParseData parseData)
                   throws IOException

Description copied from interface: CommonCrawlFormat

Returns a string representation of the JSON structure of the URL content. Takes into consideration the Content, Metadata and ParseData.

Specified by:

getJsonData in interface CommonCrawlFormat

Overrides:

getJsonData in class AbstractCommonCrawlFormat

Parameters:

url - the canonical url

content - url Content

metadata - url Metadata

parseData - url ParseData

Returns:

the JSON URL content string

Throws:

IOException - if there is a fatal I/O error obtaining JSON data

getJsonData

public String getJsonData()
                   throws IOException

Description copied from interface: CommonCrawlFormat

Get a string representation of the JSON structure of the URL content.

Specified by:

getJsonData in interface CommonCrawlFormat

Overrides:

getJsonData in class AbstractCommonCrawlFormat

Returns:

the JSON URL content string

Throws:

IOException - if there is a fatal I/O error obtaining JSON data

writeResponse

protected URI writeResponse()
                     throws IOException,
ParseException

Throws:

IOException

ParseException

writeRequest

protected URI writeRequest(URI id)
                    throws IOException,
ParseException

Throws:

IOException

ParseException

generateJson

protected String generateJson()
                       throws IOException

Specified by:

generateJson in class AbstractCommonCrawlFormat

Throws:

IOException

writeKeyValue

protected void writeKeyValue(String key,
 String value)
                      throws IOException

Specified by:

writeKeyValue in class AbstractCommonCrawlFormat

Throws:

IOException

writeKeyNull

protected void writeKeyNull(String key)
                     throws IOException

Specified by:

writeKeyNull in class AbstractCommonCrawlFormat

Throws:

IOException

startArray

protected void startArray(String key,
 boolean nested,
 boolean newline)
                   throws IOException

Specified by:

startArray in class AbstractCommonCrawlFormat

Throws:

IOException

closeArray

protected void closeArray(String key,
 boolean nested,
 boolean newline)
                   throws IOException

Specified by:

closeArray in class AbstractCommonCrawlFormat

Throws:

IOException

writeArrayValue

protected void writeArrayValue(String value)
                        throws IOException

Specified by:

writeArrayValue in class AbstractCommonCrawlFormat

Throws:

IOException

startObject

protected void startObject(String key)
                    throws IOException

Specified by:

startObject in class AbstractCommonCrawlFormat

Throws:

IOException

closeObject

protected void closeObject(String key)
                    throws IOException

Specified by:

closeObject in class AbstractCommonCrawlFormat

Throws:

IOException

close

public void close()

Description copied from interface: CommonCrawlFormat

Optional method that could be implemented if the actual format needs some close procedure.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in interface CommonCrawlFormat

Overrides:

close in class AbstractCommonCrawlFormat