Package org.apache.nutch.tools

# Class WARCUtils

java.lang.Object
org.apache.nutch.tools.WARCUtils

public class WARCUtils extends Object

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `COLONSP`
  `static final String`
  `CONFORMS_TO`
  `static final String`
  `CRLF`
  `static final String`
  `FORMAT`
  `static final org.archive.uid.UUIDGenerator`
  `generator`
  `static final String`
  `HOSTNAME`
  `static final String`
  `HTTP_HEADER_FROM`
  `static final String`
  `HTTP_HEADER_USER_AGENT`
  `static final String`
  `IP`
  `static final String`
  `OPERATOR`
  `protected static final Pattern`
  `PROBLEMATIC_HEADERS`
  `static final String`
  `ROBOTS`
  `static final String`
  `SOFTWARE`
  `protected static final String`
  `X_HIDE_HEADER`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `WARCUtils()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static final org.archive.io.warc.WARCRecordInfo`
  `docToMetadata(NutchDocument doc)`
  `static final String`
  `fixHttpHeaders(String headers, int contentLength)`
  Modify verbatim HTTP response headers: fix, remove or replace headers `Content-Length`, `Content-Encoding` and `Transfer-Encoding` which may confuse WARC readers.
  `static final String`
  `getAgentString(String name, String version, String description, String URL, String email)`
  `static final String`
  `getHostname(Configuration conf)`
  `static final String`
  `getIPAddress(Configuration conf)`
  `static final org.archive.util.anvl.ANVLRecord`
  `getWARCInfoContent(Configuration conf)`
  `static final byte[]`
  `toByteArray(org.archive.format.http.HttpHeaders headers)`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### SOFTWARE

    public static final String SOFTWARE
    See Also:
      - Constant Field Values

  - ### HTTP_HEADER_FROM

    public static final String HTTP_HEADER_FROM
    See Also:
      - Constant Field Values

  - ### HTTP_HEADER_USER_AGENT

    public static final String HTTP_HEADER_USER_AGENT
    See Also:
      - Constant Field Values

  - ### HOSTNAME

    public static final String HOSTNAME
    See Also:
      - Constant Field Values

  - ### ROBOTS

    public static final String ROBOTS
    See Also:
      - Constant Field Values

  - ### OPERATOR

    public static final String OPERATOR
    See Also:
      - Constant Field Values

  - ### FORMAT

    public static final String FORMAT
    See Also:
      - Constant Field Values

  - ### CONFORMS_TO

    public static final String CONFORMS_TO
    See Also:
      - Constant Field Values

  - ### IP

    public static final String IP
    See Also:
      - Constant Field Values

  - ### generator

    public static final org.archive.uid.UUIDGenerator generator

  - ### CRLF

    public static final String CRLF
    See Also:
      - Constant Field Values

  - ### COLONSP

    public static final String COLONSP
    See Also:
      - Constant Field Values

  - ### PROBLEMATIC_HEADERS

    protected static final Pattern PROBLEMATIC_HEADERS

  - ### X_HIDE_HEADER

    protected static final String X_HIDE_HEADER
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### WARCUtils

    public WARCUtils()

- ## Method Details

  - ### getWARCInfoContent

    public static final org.archive.util.anvl.ANVLRecord getWARCInfoContent(Configuration conf)

  - ### getHostname

    public static final String getHostname(Configuration conf) throws UnknownHostException
    Throws:
      `UnknownHostException`

  - ### getIPAddress

    public static final String getIPAddress(Configuration conf) throws UnknownHostException
    Throws:
      `UnknownHostException`

  - ### toByteArray

    public static final byte[] toByteArray(org.archive.format.http.HttpHeaders headers) throws IOException
    Throws:
      `IOException`

  - ### getAgentString

    public static final String getAgentString(String name, String version, String description, String URL, String email)

  - ### docToMetadata

    public static final org.archive.io.warc.WARCRecordInfo docToMetadata(NutchDocument doc) throws UnsupportedEncodingException
    Throws:
      `UnsupportedEncodingException`

  - ### fixHttpHeaders

    public static final String fixHttpHeaders(String headers, int contentLength)
    Modify verbatim HTTP response headers: fix, remove or replace headers `Content-Length`, `Content-Encoding` and `Transfer-Encoding` which may confuse WARC readers. Ensure that returned header end with a single empty line (`\r\n\r\n`).
    Parameters:
      `headers` - HTTP 1.1 or 1.0 response header string, CR-LF-separated lines, first line is status line
      `contentLength` - Effective uncompressed and unchunked length of content
    Returns:
      safe HTTP response header