Package org.apache.nutch.indexer.geoip

Class GeoIPIndexingFilter

java.lang.Object

org.apache.nutch.indexer.geoip.GeoIPIndexingFilter

All Implemented Interfaces:

Closeable, AutoCloseable, Configurable, IndexingFilter, Pluggable

public class GeoIPIndexingFilter
extends Object
implements IndexingFilter, Closeable

This plugin implements an indexing filter which takes advantage of the GeoIP2-java API.

The third party library distribution provides an API for the GeoIP2 Precision web services, GeoLite2 (free) and databases.

Multiple databases can be configured simultaneously. Configure each database type you want to use by setting its corresponding property:

• index.geoip.db.anonymous - Anonymous IP database
• index.geoip.db.asn - ASN database
• index.geoip.db.city - City database
• index.geoip.db.connection - Connection Type database
• index.geoip.db.country - Country database
• index.geoip.db.domain - Domain database
• index.geoip.db.isp - ISP database

Alternatively, use the MaxMind Insights web service by setting index.geoip.insights.userid and index.geoip.insights.licensekey.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	GeoIPIndexingFilter.DatabaseType	Supported GeoIP database types.

Field Summary

Fields inherited from interface org.apache.nutch.indexer.IndexingFilter

X_POINT_ID

Constructor Summary

Constructors

Constructor	Description
GeoIPIndexingFilter()	Default constructor for this plugin

Method Summary

Modifier and Type	Method	Description
void	close()
NutchDocument	filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)	Adds fields or otherwise modifies the document that will be indexed for a parse.
Configuration	getConf()
void	setConf(Configuration config)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

GeoIPIndexingFilter

public GeoIPIndexingFilter()

Default constructor for this plugin

Method Details

getConf

public Configuration getConf()

Specified by:

getConf in interface Configurable

setConf

public void setConf(Configuration config)

Specified by:

setConf in interface Configurable

filter

public NutchDocument filter(NutchDocument doc,
 Parse parse,
 Text url,
 CrawlDatum datum,
 Inlinks inlinks)
                     throws IndexingException

Description copied from interface: IndexingFilter

Adds fields or otherwise modifies the document that will be indexed for a parse. Unwanted documents can be removed from indexing by returning a null value.

Specified by:

filter in interface IndexingFilter

Parameters:

doc - document instance for collecting fields

parse - parse data instance

url - page url

datum - crawl datum for the page (fetch datum from segment containing fetch status and fetch time)

inlinks - page inlinks

Returns:

modified (or a new) document instance, or null (meaning the document should be discarded)

Throws:

IndexingException - if an error occurs during during filtering

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException