Package org.apache.nutch.scoring.opic

# Class OPICScoringFilter

java.lang.Object
org.apache.nutch.scoring.opic.OPICScoringFilter

All Implemented Interfaces:
  `Configurable`, `Pluggable`, `ScoringFilter`

public class OPICScoringFilter extends Object implements ScoringFilter
This plugin implements a variant of an Online Page Importance Computation (OPIC) score, described in this paper: Abiteboul, Serge and Preda, Mihai and Cobena, Gregory (2003), Adaptive On-Line Page Importance Computation.

Author:
  Andrzej Bialecki

- ## Field Summary

  ### Fields inherited from interface org.apache.nutch.scoring.ScoringFilter

  `X_POINT_ID`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `OPICScoringFilter()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `CrawlDatum`
  `distributeScoreToOutlinks(Text fromUrl, ParseData parseData, Collection<Map.Entry<Text,CrawlDatum>> targets, CrawlDatum adjust, int allCount)`
  Get a float value from Fetcher.SCORE_KEY, divide it by the number of outlinks and apply.
  `float`
  `generatorSortValue(Text url, CrawlDatum datum, float initSort)`
  Use `CrawlDatum.getScore()`.
  `Configuration`
  `getConf()`
  `float`
  `indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore)`
  Dampen the boost value by scorePower.
  `void`
  `initialScore(Text url, CrawlDatum datum)`
  Set to 0.0f (unknown value) - inlink contributions will bring it to a correct level.
  `void`
  `injectedScore(Text url, CrawlDatum datum)`
  Set an initial score for newly injected pages.
  `void`
  `passScoreAfterParsing(Text url, Content content, Parse parse)`
  Copy the value from Content metadata under Fetcher.SCORE_KEY to parseData.
  `void`
  `passScoreBeforeParsing(Text url, CrawlDatum datum, Content content)`
  Store a float value of CrawlDatum.getScore() under Fetcher.SCORE_KEY.
  `void`
  `setConf(Configuration conf)`
  `void`
  `updateDbScore(Text url, CrawlDatum old, CrawlDatum datum, List<CrawlDatum> inlinked)`
  Increase the score by a sum of inlinked scores.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

  ### Methods inherited from interface org.apache.nutch.scoring.ScoringFilter

  `orphanedScore`

- ## Constructor Details

  - ### OPICScoringFilter

    public OPICScoringFilter()

- ## Method Details

  - ### getConf

    public Configuration getConf()
    Specified by:
      `getConf` in interface `Configurable`

  - ### setConf

    public void setConf(Configuration conf)
    Specified by:
      `setConf` in interface `Configurable`

  - ### injectedScore

    public void injectedScore(Text url, CrawlDatum datum) throws ScoringFilterException
    Description copied from interface: `ScoringFilter`
    Set an initial score for newly injected pages. Note: newly injected pages may have no inlinks, so filter implementations may wish to set this score to a non-zero value, to give newly injected pages some initial credit.
    Specified by:
      `injectedScore` in interface `ScoringFilter`
    Parameters:
      `url` - url of the page
      `datum` - new datum. Filters will modify it in-place.
    Throws:
      `ScoringFilterException` - if there is a fatal error setting an initial score for newly injected pages

  - ### initialScore

    public void initialScore(Text url, CrawlDatum datum) throws ScoringFilterException
    Set to 0.0f (unknown value) - inlink contributions will bring it to a correct level. Newly discovered pages have at least one inlink.
    Specified by:
      `initialScore` in interface `ScoringFilter`
    Parameters:
      `url` - url of the page
      `datum` - new datum. Filters will modify it in-place.
    Throws:
      `ScoringFilterException` - if there is a fatal error setting an initial score for newly discovered pages

  - ### generatorSortValue

    public float generatorSortValue(Text url, CrawlDatum datum, float initSort) throws ScoringFilterException
    Use `CrawlDatum.getScore()`.
    Specified by:
      `generatorSortValue` in interface `ScoringFilter`
    Parameters:
      `url` - url of the page
      `datum` - page's datum, should not be modified
      `initSort` - initial sort value, or a value from previous filters in chain
    Returns:
      a sort value for use in sorting and selecting the top N scoring pages during fetchlist generation
    Throws:
      `ScoringFilterException` - if there is a fatal error preparing the sort value

  - ### updateDbScore

    public void updateDbScore(Text url, CrawlDatum old, CrawlDatum datum, List<CrawlDatum> inlinked) throws ScoringFilterException
    Increase the score by a sum of inlinked scores.
    Specified by:
      `updateDbScore` in interface `ScoringFilter`
    Parameters:
      `url` - url of the page
      `old` - original datum, with original score. May be null if this is a newly discovered page. If not null, filters should use score values from this parameter as the starting values - the `datum` parameter may contain values that are no longer valid, if other updates occurred between generation and this update.
      `datum` - the new datum, with the original score saved at the time when fetchlist was generated. Filters should update this in-place, and it will be saved in the crawldb.
      `inlinked` - (partial) list of CrawlDatum-s (with their scores) from links pointing to this page, found in the current update batch.
    Throws:
      `ScoringFilterException` - there is a fatal error calculating a new score of `CrawlDatum` during CrawlDb update

  - ### passScoreBeforeParsing

    public void passScoreBeforeParsing(Text url, CrawlDatum datum, Content content)
    Store a float value of CrawlDatum.getScore() under Fetcher.SCORE_KEY.
    Specified by:
      `passScoreBeforeParsing` in interface `ScoringFilter`
    Parameters:
      `url` - url of the page
      `datum` - source datum. NOTE: modifications to this value are not persisted.
      `content` - instance of content. Implementations may modify this in-place, primarily by setting some metadata properties.

  - ### passScoreAfterParsing

    public void passScoreAfterParsing(Text url, Content content, Parse parse)
    Copy the value from Content metadata under Fetcher.SCORE_KEY to parseData.
    Specified by:
      `passScoreAfterParsing` in interface `ScoringFilter`
    Parameters:
      `url` - page url
      `content` - original content. NOTE: modifications to this value are not persisted.
      `parse` - target instance to copy the score information to. Implementations may modify this in-place, primarily by setting some metadata properties.

  - ### distributeScoreToOutlinks

    public CrawlDatum distributeScoreToOutlinks(Text fromUrl, ParseData parseData, Collection<Map.Entry<Text,CrawlDatum>> targets, CrawlDatum adjust, int allCount) throws ScoringFilterException
    Get a float value from Fetcher.SCORE_KEY, divide it by the number of outlinks and apply.
    Specified by:
      `distributeScoreToOutlinks` in interface `ScoringFilter`
    Parameters:
      `fromUrl` - url of the source page
      `parseData` - ParseData instance, which stores relevant score value(s) in its metadata. NOTE: filters may modify this in-place, all changes will be persisted.
      `targets` - <url, CrawlDatum> pairs. NOTE: filters can modify this in-place, all changes will be persisted.
      `adjust` - a CrawlDatum instance, initially null, which implementations may use to pass adjustment values to the original CrawlDatum. When creating this instance, set its status to `CrawlDatum.STATUS_LINKED`.
      `allCount` - number of all collected outlinks from the source page
    Returns:
      if needed, implementations may return an instance of CrawlDatum, with status `CrawlDatum.STATUS_LINKED`, which contains adjustments to be applied to the original CrawlDatum score(s) and metadata. This can be null if not needed.
    Throws:
      `ScoringFilterException` - there is a fatal error distributing score data from the current page to all of its outlinks

  - ### indexerScore

    public float indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore) throws ScoringFilterException
    Dampen the boost value by scorePower.
    Specified by:
      `indexerScore` in interface `ScoringFilter`
    Parameters:
      `url` - url of the page
      `doc` - indexed document. NOTE: this already contains all information collected by indexing filters. Implementations may modify this instance, in order to store/remove some information.
      `dbDatum` - current page from CrawlDb. NOTE:
      - changes made to this instance are not persisted
      - may be null if indexing is done without CrawlDb or if the segment is generated not from the CrawlDb (via FreeGenerator).
      `fetchDatum` - datum from FetcherOutput (containing among others the fetching status)
      `parse` - parsing result. NOTE: changes made to this instance are not persisted.
      `inlinks` - current inlinks from LinkDb. NOTE: changes made to this instance are not persisted.
      `initScore` - initial boost value for the indexed document.
    Returns:
      boost value for the indexed document. This value is passed as an argument to the next scoring filter in chain. NOTE: implementations may also express other scoring strategies by modifying the indexed document directly.
    Throws:
      `ScoringFilterException` - if there is a fatal error whilst calculating the indexed document score/boost