Uses of Class
org.apache.nutch.indexer.NutchDocument
Packages that use NutchDocument
Package
Description
Text document language identifier.
Control code for exchange component, which acts in indexing job and decides to
which index writer a document should be routed, based on plugins behavior.
Plugin of Exchange component based on JEXL expressions.
Index content, configure and run indexing and cleaning jobs to
add, update, and delete documents from an index.
An indexing plugin for inbound anchor text.
Indexing filter to add document arbitrary data to the index
from the output of a user-specified class.
A basic indexing plugin, adds basic fields: url, host, title, content, etc.
Indexing filter to index meta data from RSS feeds.
This plugin implements an indexing filter which takes
advantage of the
GeoIP2-java API.
This plugin implements a dynamic indexing filter which uses JEXL
expressions to allow filtering based on the page's metadata
Indexing filter to add document metadata to the index.
A more indexing plugin, adds "more" index fields:last modified
date, MIME type, content length.
Indexing filter to allow pattern replacements on metadata.
A simple plugin called at indexing that adds fields with static data.
Indexing filter to assign documents to subcollections.
Top Level Domain Indexing plugin.
URL Meta Tag Indexing Plugin
Index writer plugin to write a plain CSV file.
Index writer plugin for debugging, writes pairs of <action, url> to a
text file, action is one of "add", "update", or "delete".
Index writer plugin for Elasticsearch.
Index writer plugin to produce JSON messages to Kafka.
Index writer plugin for OpenSearch.
Index writer plugin for Apache Solr.
A microformats Rel-Tag
Parser/Indexer/Querier plugin.
The
ScoringFilter interface.Scoring filter to stop crawling at a configurable depth
(number of "hops" from seed URLs).
Scoring filter used in conjunction with
WebGraph.Scoring filter implementing a variant of the Online Page Importance Computation
(OPIC) algorithm.
Miscellaneous tools.
Sample plugins that parse and index Creative Commons metadata.
-
Uses of NutchDocument in org.apache.nutch.analysis.lang
Methods in org.apache.nutch.analysis.lang that return NutchDocumentModifier and TypeMethodDescriptionLanguageIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Methods in org.apache.nutch.analysis.lang with parameters of type NutchDocumentModifier and TypeMethodDescriptionLanguageIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of NutchDocument in org.apache.nutch.exchange
Methods in org.apache.nutch.exchange with parameters of type NutchDocumentModifier and TypeMethodDescriptionString[]Exchanges.indexWriters(NutchDocument nutchDocument) Returns all the indexers where the document must be sent to.booleanExchange.match(NutchDocument doc) Determines if the document must go to the related index writers. -
Uses of NutchDocument in org.apache.nutch.exchange.jexl
Methods in org.apache.nutch.exchange.jexl with parameters of type NutchDocumentModifier and TypeMethodDescriptionbooleanJexlExchange.match(NutchDocument doc) Determines if the document must go to the related index writers. -
Uses of NutchDocument in org.apache.nutch.indexer
Fields in org.apache.nutch.indexer declared as NutchDocumentMethods in org.apache.nutch.indexer that return NutchDocumentModifier and TypeMethodDescriptionNutchDocument.clone()IndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Adds fields or otherwise modifies the document that will be indexed for a parse.IndexingFilters.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Run all defined filters.Methods in org.apache.nutch.indexer with parameters of type NutchDocumentModifier and TypeMethodDescriptionIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Adds fields or otherwise modifies the document that will be indexed for a parse.IndexingFilters.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Run all defined filters.voidIndexWriter.update(NutchDocument doc) voidIndexWriters.update(NutchDocument doc) voidIndexWriter.write(NutchDocument doc) voidIndexWriters.write(NutchDocument doc) Constructors in org.apache.nutch.indexer with parameters of type NutchDocument -
Uses of NutchDocument in org.apache.nutch.indexer.anchor
Methods in org.apache.nutch.indexer.anchor that return NutchDocumentModifier and TypeMethodDescriptionAnchorIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) TheAnchorIndexingFilterfilter object which supports boolean configuration settings for the deduplication of anchors.Methods in org.apache.nutch.indexer.anchor with parameters of type NutchDocumentModifier and TypeMethodDescriptionAnchorIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) TheAnchorIndexingFilterfilter object which supports boolean configuration settings for the deduplication of anchors. -
Uses of NutchDocument in org.apache.nutch.indexer.arbitrary
Methods in org.apache.nutch.indexer.arbitrary that return NutchDocumentModifier and TypeMethodDescriptionArbitraryIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) TheArbitraryIndexingFilterfilter object uses reflection to instantiate the configured class and invoke the configured method.Methods in org.apache.nutch.indexer.arbitrary with parameters of type NutchDocumentModifier and TypeMethodDescriptionArbitraryIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) TheArbitraryIndexingFilterfilter object uses reflection to instantiate the configured class and invoke the configured method. -
Uses of NutchDocument in org.apache.nutch.indexer.basic
Methods in org.apache.nutch.indexer.basic that return NutchDocumentModifier and TypeMethodDescriptionBasicIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) TheBasicIndexingFilterfilter object which supports few configuration settings for adding basic searchable fields.Methods in org.apache.nutch.indexer.basic with parameters of type NutchDocumentModifier and TypeMethodDescriptionBasicIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) TheBasicIndexingFilterfilter object which supports few configuration settings for adding basic searchable fields. -
Uses of NutchDocument in org.apache.nutch.indexer.feed
Methods in org.apache.nutch.indexer.feed that return NutchDocumentModifier and TypeMethodDescriptionFeedIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Extracts out the relevant fields: FEED_AUTHOR FEED_TAGS FEED_PUBLISHED FEED_UPDATED FEED And sends them to theIndexerfor indexing within the Nutch index.Methods in org.apache.nutch.indexer.feed with parameters of type NutchDocumentModifier and TypeMethodDescriptionFeedIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Extracts out the relevant fields: FEED_AUTHOR FEED_TAGS FEED_PUBLISHED FEED_UPDATED FEED And sends them to theIndexerfor indexing within the Nutch index. -
Uses of NutchDocument in org.apache.nutch.indexer.filter
Methods in org.apache.nutch.indexer.filter that return NutchDocumentModifier and TypeMethodDescriptionMimeTypeIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Methods in org.apache.nutch.indexer.filter with parameters of type NutchDocumentModifier and TypeMethodDescriptionMimeTypeIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of NutchDocument in org.apache.nutch.indexer.geoip
Methods in org.apache.nutch.indexer.geoip that return NutchDocumentModifier and TypeMethodDescriptionstatic NutchDocumentGeoIPDocumentCreator.createDocFromAnonymousIpDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in Anonymous IP database.static NutchDocumentGeoIPDocumentCreator.createDocFromAsnDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in ASN database.static NutchDocumentGeoIPDocumentCreator.createDocFromCityDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in City database.static NutchDocumentGeoIPDocumentCreator.createDocFromConnectionDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in ConnectionDb.static NutchDocumentGeoIPDocumentCreator.createDocFromCountryDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in Country database.static NutchDocumentGeoIPDocumentCreator.createDocFromDomainDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in Domain database.static NutchDocumentGeoIPDocumentCreator.createDocFromInsightsService(String serverIp, NutchDocument doc, com.maxmind.geoip2.WebServiceClient client) Populate aNutchDocumentbased on lookup of IP using the MaxMind Insights web service.static NutchDocumentGeoIPDocumentCreator.createDocFromIspDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in ISP database.GeoIPIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Methods in org.apache.nutch.indexer.geoip with parameters of type NutchDocumentModifier and TypeMethodDescriptionstatic NutchDocumentGeoIPDocumentCreator.createDocFromAnonymousIpDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in Anonymous IP database.static NutchDocumentGeoIPDocumentCreator.createDocFromAsnDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in ASN database.static NutchDocumentGeoIPDocumentCreator.createDocFromCityDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in City database.static NutchDocumentGeoIPDocumentCreator.createDocFromConnectionDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in ConnectionDb.static NutchDocumentGeoIPDocumentCreator.createDocFromCountryDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in Country database.static NutchDocumentGeoIPDocumentCreator.createDocFromDomainDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in Domain database.static NutchDocumentGeoIPDocumentCreator.createDocFromInsightsService(String serverIp, NutchDocument doc, com.maxmind.geoip2.WebServiceClient client) Populate aNutchDocumentbased on lookup of IP using the MaxMind Insights web service.static NutchDocumentGeoIPDocumentCreator.createDocFromIspDb(String serverIp, NutchDocument doc, com.maxmind.geoip2.DatabaseReader reader) Populate aNutchDocumentbased on lookup of IP in ISP database.GeoIPIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of NutchDocument in org.apache.nutch.indexer.jexl
Methods in org.apache.nutch.indexer.jexl that return NutchDocumentModifier and TypeMethodDescriptionJexlIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Methods in org.apache.nutch.indexer.jexl with parameters of type NutchDocumentModifier and TypeMethodDescriptionJexlIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of NutchDocument in org.apache.nutch.indexer.links
Methods in org.apache.nutch.indexer.links that return NutchDocumentModifier and TypeMethodDescriptionLinksIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Methods in org.apache.nutch.indexer.links with parameters of type NutchDocumentModifier and TypeMethodDescriptionLinksIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of NutchDocument in org.apache.nutch.indexer.metadata
Methods in org.apache.nutch.indexer.metadata that return NutchDocumentModifier and TypeMethodDescriptionMetadataIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Methods in org.apache.nutch.indexer.metadata with parameters of type NutchDocumentModifier and TypeMethodDescriptionprotected voidMetadataIndexer.add(NutchDocument doc, String key, String value) MetadataIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of NutchDocument in org.apache.nutch.indexer.more
Methods in org.apache.nutch.indexer.more that return NutchDocumentModifier and TypeMethodDescriptionMoreIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Methods in org.apache.nutch.indexer.more with parameters of type NutchDocumentModifier and TypeMethodDescriptionMoreIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of NutchDocument in org.apache.nutch.indexer.replace
Methods in org.apache.nutch.indexer.replace that return NutchDocumentModifier and TypeMethodDescriptionReplaceIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Methods in org.apache.nutch.indexer.replace with parameters of type NutchDocumentModifier and TypeMethodDescriptionReplaceIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of NutchDocument in org.apache.nutch.indexer.staticfield
Methods in org.apache.nutch.indexer.staticfield that return NutchDocumentModifier and TypeMethodDescriptionStaticFieldIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) TheStaticFieldIndexerfilter object which adds fields as per configuration setting.Methods in org.apache.nutch.indexer.staticfield with parameters of type NutchDocumentModifier and TypeMethodDescriptionStaticFieldIndexer.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) TheStaticFieldIndexerfilter object which adds fields as per configuration setting. -
Uses of NutchDocument in org.apache.nutch.indexer.subcollection
Methods in org.apache.nutch.indexer.subcollection that return NutchDocumentModifier and TypeMethodDescriptionSubcollectionIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Methods in org.apache.nutch.indexer.subcollection with parameters of type NutchDocumentModifier and TypeMethodDescriptionSubcollectionIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of NutchDocument in org.apache.nutch.indexer.tld
Methods in org.apache.nutch.indexer.tld that return NutchDocumentModifier and TypeMethodDescriptionTLDIndexingFilter.filter(NutchDocument doc, Parse parse, Text urlText, CrawlDatum datum, Inlinks inlinks) Methods in org.apache.nutch.indexer.tld with parameters of type NutchDocumentModifier and TypeMethodDescriptionTLDIndexingFilter.filter(NutchDocument doc, Parse parse, Text urlText, CrawlDatum datum, Inlinks inlinks) -
Uses of NutchDocument in org.apache.nutch.indexer.urlmeta
Methods in org.apache.nutch.indexer.urlmeta that return NutchDocumentModifier and TypeMethodDescriptionURLMetaIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) This will take the metatags that you have listed in your "urlmeta.tags" property, and looks for them inside the CrawlDatum object.Methods in org.apache.nutch.indexer.urlmeta with parameters of type NutchDocumentModifier and TypeMethodDescriptionURLMetaIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) This will take the metatags that you have listed in your "urlmeta.tags" property, and looks for them inside the CrawlDatum object. -
Uses of NutchDocument in org.apache.nutch.indexwriter.cloudsearch
Methods in org.apache.nutch.indexwriter.cloudsearch with parameters of type NutchDocumentModifier and TypeMethodDescriptionvoidCloudSearchIndexWriter.update(NutchDocument doc) voidCloudSearchIndexWriter.write(NutchDocument doc) -
Uses of NutchDocument in org.apache.nutch.indexwriter.csv
Methods in org.apache.nutch.indexwriter.csv with parameters of type NutchDocumentModifier and TypeMethodDescriptionvoidCSVIndexWriter.update(NutchDocument doc) voidCSVIndexWriter.write(NutchDocument doc) -
Uses of NutchDocument in org.apache.nutch.indexwriter.dummy
Methods in org.apache.nutch.indexwriter.dummy with parameters of type NutchDocumentModifier and TypeMethodDescriptionvoidDummyIndexWriter.update(NutchDocument doc) voidDummyIndexWriter.write(NutchDocument doc) -
Uses of NutchDocument in org.apache.nutch.indexwriter.elastic
Methods in org.apache.nutch.indexwriter.elastic with parameters of type NutchDocumentModifier and TypeMethodDescriptionvoidElasticIndexWriter.update(NutchDocument doc) voidElasticIndexWriter.write(NutchDocument doc) -
Uses of NutchDocument in org.apache.nutch.indexwriter.kafka
Methods in org.apache.nutch.indexwriter.kafka with parameters of type NutchDocumentModifier and TypeMethodDescriptionvoidKafkaIndexWriter.update(NutchDocument doc) voidKafkaIndexWriter.write(NutchDocument doc) -
Uses of NutchDocument in org.apache.nutch.indexwriter.opensearch1x
Methods in org.apache.nutch.indexwriter.opensearch1x with parameters of type NutchDocumentModifier and TypeMethodDescriptionvoidOpenSearch1xIndexWriter.update(NutchDocument doc) voidOpenSearch1xIndexWriter.write(NutchDocument doc) -
Uses of NutchDocument in org.apache.nutch.indexwriter.rabbit
Methods in org.apache.nutch.indexwriter.rabbit with parameters of type NutchDocumentModifier and TypeMethodDescriptionvoidRabbitIndexWriter.update(NutchDocument doc) voidRabbitIndexWriter.write(NutchDocument doc) -
Uses of NutchDocument in org.apache.nutch.indexwriter.solr
Methods in org.apache.nutch.indexwriter.solr with parameters of type NutchDocumentModifier and TypeMethodDescriptionvoidSolrIndexWriter.update(NutchDocument doc) voidSolrIndexWriter.write(NutchDocument doc) -
Uses of NutchDocument in org.apache.nutch.microformats.reltag
Methods in org.apache.nutch.microformats.reltag that return NutchDocumentModifier and TypeMethodDescriptionRelTagIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Methods in org.apache.nutch.microformats.reltag with parameters of type NutchDocumentModifier and TypeMethodDescriptionRelTagIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) -
Uses of NutchDocument in org.apache.nutch.scoring
Methods in org.apache.nutch.scoring with parameters of type NutchDocumentModifier and TypeMethodDescriptionfloatAbstractScoringFilter.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore) floatScoringFilter.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore) This method calculates a indexed document score/boost.floatScoringFilters.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore) -
Uses of NutchDocument in org.apache.nutch.scoring.depth
Methods in org.apache.nutch.scoring.depth with parameters of type NutchDocumentModifier and TypeMethodDescriptionfloatDepthScoringFilter.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore) -
Uses of NutchDocument in org.apache.nutch.scoring.link
Methods in org.apache.nutch.scoring.link with parameters of type NutchDocumentModifier and TypeMethodDescriptionfloatLinkAnalysisScoringFilter.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore) -
Uses of NutchDocument in org.apache.nutch.scoring.opic
Methods in org.apache.nutch.scoring.opic with parameters of type NutchDocumentModifier and TypeMethodDescriptionfloatOPICScoringFilter.indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse, Inlinks inlinks, float initScore) Dampen the boost value by scorePower. -
Uses of NutchDocument in org.apache.nutch.tools
Methods in org.apache.nutch.tools with parameters of type NutchDocumentModifier and TypeMethodDescriptionstatic final org.archive.io.warc.WARCRecordInfoWARCUtils.docToMetadata(NutchDocument doc) -
Uses of NutchDocument in org.creativecommons.nutch
Methods in org.creativecommons.nutch that return NutchDocumentModifier and TypeMethodDescriptionCCIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) Methods in org.creativecommons.nutch with parameters of type NutchDocumentModifier and TypeMethodDescriptionvoidCCIndexingFilter.addUrlFeatures(NutchDocument doc, String urlString) Add the features represented by a license URL.CCIndexingFilter.filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks)