Package org.apache.nutch.tools

Class AbstractCommonCrawlFormat

java.lang.Object
org.apache.nutch.tools.AbstractCommonCrawlFormat
All Implemented Interfaces:
Closeable, AutoCloseable, CommonCrawlFormat
Direct Known Subclasses:
CommonCrawlFormatJackson, CommonCrawlFormatJettinson, CommonCrawlFormatSimple, CommonCrawlFormatWARC
public abstract class AbstractCommonCrawlFormat extends Object implements CommonCrawlFormat
Abstract class that implements { @see org.apache.nutch.tools.CommonCrawlFormat } interface.

  • Field Details

    • LOG

      protected static final org.slf4j.Logger LOG

    • url

      protected String url

    • content

      protected Content content

    • metadata

      protected Metadata metadata

    • conf

      protected Configuration conf

    • keyPrefix

      protected String keyPrefix

    • simpleDateFormat

      protected boolean simpleDateFormat

    • jsonArray

      protected boolean jsonArray

    • reverseKey

      protected boolean reverseKey

    • reverseKeyValue

      protected String reverseKeyValue

  • Constructor Details

  • Method Details

    • getJsonData

      public String getJsonData(String url, Content content, Metadata metadata) throws IOException
      Description copied from interface: CommonCrawlFormat
      Returns a string representation of the JSON structure of the URL content. Takes into consideration both the Content and Metadata
      Specified by:
      getJsonData in interface CommonCrawlFormat
      Parameters:
      url - the canonical url
      content - url Content
      metadata - url Metadata
      Returns:
      the JSON URL content string
      Throws:
      IOException - if there is a fatal I/O error obtaining JSON data

    • getJsonData

      public String getJsonData(String url, Content content, Metadata metadata, ParseData parseData) throws IOException
      Description copied from interface: CommonCrawlFormat
      Returns a string representation of the JSON structure of the URL content. Takes into consideration the Content, Metadata and ParseData.
      Specified by:
      getJsonData in interface CommonCrawlFormat
      Parameters:
      url - the canonical url
      content - url Content
      metadata - url Metadata
      parseData - url ParseData
      Returns:
      the JSON URL content string
      Throws:
      IOException - if there is a fatal I/O error obtaining JSON data

    • getJsonData

      public String getJsonData() throws IOException
      Description copied from interface: CommonCrawlFormat
      Get a string representation of the JSON structure of the URL content.
      Specified by:
      getJsonData in interface CommonCrawlFormat
      Returns:
      the JSON URL content string
      Throws:
      IOException - if there is a fatal I/O error obtaining JSON data

    • writeKeyValue

      protected abstract void writeKeyValue(String key, String value) throws IOException
      Throws:
      IOException

    • writeKeyNull

      protected abstract void writeKeyNull(String key) throws IOException
      Throws:
      IOException

    • startArray

      protected abstract void startArray(String key, boolean nested, boolean newline) throws IOException
      Throws:
      IOException

    • closeArray

      protected abstract void closeArray(String key, boolean nested, boolean newline) throws IOException
      Throws:
      IOException

    • writeArrayValue

      protected abstract void writeArrayValue(String value) throws IOException
      Throws:
      IOException

    • startObject

      protected abstract void startObject(String key) throws IOException
      Throws:
      IOException

    • closeObject

      protected abstract void closeObject(String key) throws IOException
      Throws:
      IOException

    • generateJson

      protected abstract String generateJson() throws IOException
      Throws:
      IOException

    • getUrl

      protected String getUrl()

    • getTimestamp

      protected String getTimestamp()

    • getMethod

      protected String getMethod()

    • getRequestHostName

      protected String getRequestHostName()

    • getRequestHostAddress

      protected String getRequestHostAddress()

    • getRequestSoftware

      protected String getRequestSoftware()

    • getRequestRobots

      protected String getRequestRobots()

    • getRequestContactName

      protected String getRequestContactName()

    • getRequestContactEmail

      protected String getRequestContactEmail()

    • getRequestAccept

      protected String getRequestAccept()

    • getRequestAcceptEncoding

      protected String getRequestAcceptEncoding()

    • getRequestAcceptLanguage

      protected String getRequestAcceptLanguage()

    • getRequestUserAgent

      protected String getRequestUserAgent()

    • getResponseStatus

      protected String getResponseStatus()

    • getResponseHostName

      protected String getResponseHostName()

    • getResponseAddress

      protected String getResponseAddress()

    • getResponseContentEncoding

      protected String getResponseContentEncoding()

    • getResponseContentType

      protected String getResponseContentType()

    • getInLinks

      public List<String> getInLinks()
      Description copied from interface: CommonCrawlFormat
      gets set of inlinks
      Specified by:
      getInLinks in interface CommonCrawlFormat
      Returns:
      gets inlinks of this document

    • setInLinks

      public void setInLinks(List<String> inLinks)
      Description copied from interface: CommonCrawlFormat
      sets inlinks of this document
      Specified by:
      setInLinks in interface CommonCrawlFormat
      Parameters:
      inLinks - list of inlinks

    • getResponseDate

      protected String getResponseDate()

    • getResponseServer

      protected String getResponseServer()

    • getResponseContent

      protected String getResponseContent()

    • getKey

      protected String getKey()

    • getImported

      protected String getImported()

    • close

      public void close()
      Description copied from interface: CommonCrawlFormat
      Optional method that could be implemented if the actual format needs some close procedure.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Specified by:
      close in interface CommonCrawlFormat