Package org.apache.nutch.fetcher

# Class FetchItemQueues

java.lang.Object
org.apache.nutch.fetcher.FetchItemQueues

public class FetchItemQueues extends Object
A collection of queues that keeps track of the total number of items, and provides items eligible for fetching from any queue.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `DEFAULT_ID`
  `static final String`
  `QUEUE_MODE_DOMAIN`
  `static final String`
  `QUEUE_MODE_HOST`
  `static final String`
  `QUEUE_MODE_IP`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `FetchItemQueues(Configuration conf)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `org.apache.nutch.fetcher.FetchItemQueues.QueuingStatus`
  `addFetchItem(Text url, CrawlDatum datum)`
  `org.apache.nutch.fetcher.FetchItemQueues.QueuingStatus`
  `addFetchItem(FetchItem it)`
  `int`
  `checkExceptionThreshold(String queueid)`
  Increment the exception counter of a queue in case of an exception e.g.
  `int`
  `checkExceptionThreshold(String queueid, int maxExceptions, long delay)`
  Increment the exception counter of a queue in case of an exception e.g.
  `protected static String`
  `checkQueueMode(String queueMode)`
  Check whether queue mode is valid, fall-back to default mode if not.
  `int`
  `checkTimelimit()`
  `void`
  `dump()`
  `int`
  `emptyQueues()`
  `void`
  `finishFetchItem(FetchItem it)`
  `void`
  `finishFetchItem(FetchItem it, boolean asap)`
  `FetchItem`
  `getFetchItem()`
  `FetchItemQueue`
  `getFetchItemQueue(String id)`
  `int`
  `getQueueCount()`
  `int`
  `getQueueCountMaxExceptions()`
  `int`
  `getTotalSize()`
  `boolean`
  `redirectIsQueuedRecently(Text redirUrl)`
  `void`
  `setTimeoutReached()`
  Signal that the hard timeout is reached because new fetches / requests where made during half of the MapReduce task timeout (`mapreduce.task.timeout`, default value: 10 minutes).
  `boolean`
  `timelimitExceeded()`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### DEFAULT_ID

    public static final String DEFAULT_ID
    See Also:
      - Constant Field Values

  - ### QUEUE_MODE_HOST

    public static final String QUEUE_MODE_HOST
    See Also:
      - Constant Field Values

  - ### QUEUE_MODE_DOMAIN

    public static final String QUEUE_MODE_DOMAIN
    See Also:
      - Constant Field Values

  - ### QUEUE_MODE_IP

    public static final String QUEUE_MODE_IP
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### FetchItemQueues

    public FetchItemQueues(Configuration conf)

- ## Method Details

  - ### checkQueueMode

    protected static String checkQueueMode(String queueMode)
    Check whether queue mode is valid, fall-back to default mode if not.
    Parameters:
      `queueMode` - queue mode to check
    Returns:
      valid queue mode or default

  - ### getTotalSize

    public int getTotalSize()

  - ### getQueueCount

    public int getQueueCount()

  - ### getQueueCountMaxExceptions

    public int getQueueCountMaxExceptions()

  - ### addFetchItem

    public org.apache.nutch.fetcher.FetchItemQueues.QueuingStatus addFetchItem(Text url, CrawlDatum datum)

  - ### addFetchItem

    public org.apache.nutch.fetcher.FetchItemQueues.QueuingStatus addFetchItem(FetchItem it)

  - ### finishFetchItem

    public void finishFetchItem(FetchItem it)

  - ### finishFetchItem

    public void finishFetchItem(FetchItem it, boolean asap)

  - ### getFetchItemQueue

    public FetchItemQueue getFetchItemQueue(String id)

  - ### getFetchItem

    public FetchItem getFetchItem()

  - ### timelimitExceeded

    public boolean timelimitExceeded()
    Returns:
      true if the fetcher timelimit is defined and has been exceeded (`fetcher.timelimit.mins` minutes after fetching started)

  - ### checkTimelimit

    public int checkTimelimit()

  - ### setTimeoutReached

    public void setTimeoutReached()
    Signal that the hard timeout is reached because new fetches / requests where made during half of the MapReduce task timeout (`mapreduce.task.timeout`, default value: 10 minutes). In order to avoid that the task timeout is hit and the fetcher job is failed, we stop the fetching now. See also the property `fetcher.threads.timeout.divisor`.

  - ### emptyQueues

    public int emptyQueues()

  - ### checkExceptionThreshold

    public int checkExceptionThreshold(String queueid, int maxExceptions, long delay)
    Increment the exception counter of a queue in case of an exception e.g. timeout; when higher than a given threshold simply empty the queue. The next fetch is delayed if specified by the param `delay` or configured by the property `fetcher.exceptions.per.queue.delay`.
    Parameters:
      `queueid` - a queue identifier to locate and check
      `maxExceptions` - custom-defined number of max. exceptions - if negative the value of the property `fetcher.max.exceptions.per.queue` is used.
      `delay` - a custom-defined time span in milliseconds to delay the next fetch in addition to the delay defined for the given queue. If a negative value is passed the delay is chosen by `fetcher.exceptions.per.queue.delay`
    Returns:
      number of purged items

  - ### checkExceptionThreshold

    public int checkExceptionThreshold(String queueid)
    Increment the exception counter of a queue in case of an exception e.g. timeout; when higher than a given threshold simply empty the queue.
    Parameters:
      `queueid` - queue identifier to locate and check
    Returns:
      number of purged items
    See Also:
      - `checkExceptionThreshold(String, int, long)`

  - ### redirectIsQueuedRecently

    public boolean redirectIsQueuedRecently(Text redirUrl)
    Parameters:
      `redirUrl` - redirect target
    Returns:
      true if redirects are deduplicated and redirUrl has been queued recently

  - ### dump

    public void dump()