Package org.apache.nutch.parse.html

# Class DOMContentUtils

java.lang.Object
org.apache.nutch.parse.html.DOMContentUtils

public class DOMContentUtils extends Object
A collection of methods for extracting content from DOM trees. This class holds a few utility methods for pulling content out of DOM nodes, such as getOutlinks, getText, etc.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `DOMContentUtils.LinkParams`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `DOMContentUtils(Configuration conf)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `String`
  `getBase(Node node)`
  If Node contains a BASE tag then it's HREF is returned.
  `void`
  `getOutlinks(URL base, ArrayList<Outlink> outlinks, Node node)`
  This method finds all anchors below the supplied DOM `node`, and creates appropriate `Outlink` records for each (relative to the supplied `base` URL), and adds them to the `outlinks` `ArrayList`.
  `void`
  `getText(StringBuffer sb, Node node)`
  This is a convinience method, equivalent to `getText(sb, node, false)`.
  `boolean`
  `getText(StringBuffer sb, Node node, boolean abortOnNestedAnchors)`
  This method takes a `StringBuffer` and a DOM `Node`, and will append all the content text found beneath the DOM node to the `StringBuffer`.
  `boolean`
  `getTitle(StringBuffer sb, Node node)`
  This method takes a `StringBuffer` and a DOM `Node`, and will append the content text found beneath the first `title` node to the `StringBuffer`.
  `void`
  `setConf(Configuration conf)`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### DOMContentUtils

    public DOMContentUtils(Configuration conf)

- ## Method Details

  - ### setConf

    public void setConf(Configuration conf)

  - ### getText

    public boolean getText(StringBuffer sb, Node node, boolean abortOnNestedAnchors)
    This method takes a `StringBuffer` and a DOM `Node`, and will append all the content text found beneath the DOM node to the `StringBuffer`.

    If `abortOnNestedAnchors` is true, DOM traversal will be aborted and the `StringBuffer` will not contain any text encountered after a nested anchor is found.

    Parameters:
      `sb` - a `StringBuffer` used to store content text found beneath the DOM node... if any exists
      `node` - a DOM `Node` to check for content text
      `abortOnNestedAnchors` - true to abort if nested anchors are encountered, false otherwise
    Returns:
      true if nested anchors were found

  - ### getText

    public void getText(StringBuffer sb, Node node)
    This is a convinience method, equivalent to `getText(sb, node, false)`.
    Parameters:
      `sb` - a `StringBuffer` used to store content text found beneath the DOM node... if any exists
      `node` - a DOM `Node` to check for content text

  - ### getTitle

    public boolean getTitle(StringBuffer sb, Node node)
    This method takes a `StringBuffer` and a DOM `Node`, and will append the content text found beneath the first `title` node to the `StringBuffer`.
    Parameters:
      `sb` - a `StringBuffer` used to store content text found beneath the DOM node... if any exists
      `node` - a DOM `Node` to check for content text
    Returns:
      true if a title node was found, false otherwise

  - ### getBase

    public String getBase(Node node)
    If Node contains a BASE tag then it's HREF is returned.
    Parameters:
      `node` - a DOM `Node` to check for a BASE tag
    Returns:
      HREF if one exists

  - ### getOutlinks

    public void getOutlinks(URL base, ArrayList<Outlink> outlinks, Node node)
    This method finds all anchors below the supplied DOM `node`, and creates appropriate `Outlink` records for each (relative to the supplied `base` URL), and adds them to the `outlinks` `ArrayList`.

    Links without inner structure (tags, text, etc) are discarded, as are links which contain only single nested links and empty text nodes (this is a common DOM-fixup artifact, at least with nekohtml).

    Parameters:
      `base` - the canonical `URL`
      `outlinks` - the `ArrayList` of `Outlink`'s associated with the base URL
      `node` - a `Node` under which to discover anchors