Package org.apache.nutch.segment

Interface SegmentMergeFilter

public interface SegmentMergeFilter

Interface used to filter segments during segment merge. It allows filtering on more sophisticated criteria than just URLs. In particular it allows filtering based on metadata collected while parsing page.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	X_POINT_ID	The name of the extension point.

Method Summary

Modifier and Type	Method	Description
boolean	filter(Text key, CrawlDatum generateData, CrawlDatum fetchData, CrawlDatum sigData, Content content, ParseData parseData, ParseText parseText, Collection<CrawlDatum> linked)	The filtering method which gets all information being merged for a given key (URL).

Field Details

X_POINT_ID

static final String X_POINT_ID

The name of the extension point.

Method Details

filter

boolean filter(Text key,
 CrawlDatum generateData,
 CrawlDatum fetchData,
 CrawlDatum sigData,
 Content content,
 ParseData parseData,
 ParseText parseText,
 Collection<CrawlDatum> linked)

The filtering method which gets all information being merged for a given key (URL).

Parameters:

key - the segment record key

generateData - directory and data produced by the generation phase

fetchData - directory and data produced by the fetch phase

sigData - directory and data produced by the parse phase

content - directory and data produced by the parse phase

parseData - directory and data produced by the parse phase

parseText - directory and data produced by the parse phase

linked - all LINKED values from the latest segment

Returns:

true values for this key (URL) should be merged into the new segment.