Package org.apache.nutch.scoring.similarity.util

# Class LuceneTokenizer

java.lang.Object
org.apache.nutch.scoring.similarity.util.LuceneTokenizer

public class LuceneTokenizer extends Object

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static enum`
  `LuceneTokenizer.TokenizerType`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, boolean useStopFilter, LuceneAnalyzerUtil.StemFilterType stemFilterType)`
  Creates a tokenizer based on param values
  `LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, List<String> stopWords, boolean addToDefault, LuceneAnalyzerUtil.StemFilterType stemFilterType)`
  Creates a tokenizer based on param values
  `LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, LuceneAnalyzerUtil.StemFilterType stemFilterType, int mingram, int maxgram)`
  Creates a tokenizer for the ngram model based on param values

- ## Method Summary

  Modifier and Type
  Method
  Description
  `org.apache.lucene.analysis.TokenStream`
  `getTokenStream()`
  get the tokenStream created by `Tokenizer`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### LuceneTokenizer

    public LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, boolean useStopFilter, LuceneAnalyzerUtil.StemFilterType stemFilterType)
    Creates a tokenizer based on param values
    Parameters:
      `content` - - The text to tokenize
      `tokenizer` - - the type of tokenizer to use CLASSIC or DEFAULT
      `useStopFilter` - - if set to true the token stream will be filtered using default Lucene stopset
      `stemFilterType` - a preferred `LuceneAnalyzerUtil.StemFilterType` to use. Can be one of `LuceneAnalyzerUtil.StemFilterType.PORTERSTEM_FILTER`, `LuceneAnalyzerUtil.StemFilterType.ENGLISHMINIMALSTEM_FILTER`, or `LuceneAnalyzerUtil.StemFilterType.NONE`

  - ### LuceneTokenizer

    public LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, List<String> stopWords, boolean addToDefault, LuceneAnalyzerUtil.StemFilterType stemFilterType)
    Creates a tokenizer based on param values
    Parameters:
      `content` - - The text to tokenize
      `tokenizer` - - the type of tokenizer to use CLASSIC or DEFAULT
      `stopWords` - - Provide a set of user defined stop words
      `addToDefault` - - If set to true, the stopSet words will be added to the Lucene default stop set. If false, then only the user provided words will be used as the stop set
      `stemFilterType` - a preferred `LuceneAnalyzerUtil.StemFilterType` to use. Can be one of `LuceneAnalyzerUtil.StemFilterType.PORTERSTEM_FILTER`, `LuceneAnalyzerUtil.StemFilterType.ENGLISHMINIMALSTEM_FILTER`, or `LuceneAnalyzerUtil.StemFilterType.NONE`

  - ### LuceneTokenizer

    public LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, LuceneAnalyzerUtil.StemFilterType stemFilterType, int mingram, int maxgram)
    Creates a tokenizer for the ngram model based on param values
    Parameters:
      `content` - - The text to tokenize
      `tokenizer` - - the type of tokenizer to use CLASSIC or DEFAULT
      `stemFilterType` - - Type of stemming to perform
      `mingram` - - Value of mingram for tokenizing
      `maxgram` - - Value of maxgram for tokenizing

- ## Method Details

  - ### getTokenStream

    public org.apache.lucene.analysis.TokenStream getTokenStream()
    get the tokenStream created by `Tokenizer`
    Returns:
      The `TokenStream`