Nutch has added pluggable URL normalization ('canonicalization' in heritrix-speak) and there is also now the ability to filter by URL at each of the indexing steps. Nutchwax picks up this feature in this release (See Flexible URL normalization NUTCH-365). Normally its operation will be of no concern, particularly as the default behavior is mild equating 'http://www.archive.org:80/' and 'http://www.archive.org/' for instance but it may become an issue if the engine you are using to rendering found pages uses an index other than the one made by nutchwax: e.g., a wayback that made its own CDX or bdbje index, Here there may be times when the nutch normalization disagrees with the wayback normalization and lookups into the alternate wayback index will fail. The configuration that manages urlnormalization is in nutch-default.xml. Its the property: urlnormalizer.order combined with mention in plugin.includes.

Complaints from outlinks parser are now much tidier occuping a single line rather than dumping a MalformedURLException stacktrace. Here are samples:

2006-12-15 11:19:16,749 WARN  parse.OutlinkExtractor - Invalid url: 'NOTE:DON', skipping.
2006-12-15 11:19:16,750 WARN  parse.OutlinkExtractor - Invalid url: 'DH0:LEVELS/', skipping.

With this release, the default pdf parser has been switched to the nutch parse-pdf PDFBox-based parser. Previous, a nutchwax plugin called parse-waxext was the default. The parse-waxext plugin ran an external dependency named xpdf -- via the wrapper script parse-pdf.sh -- parsing application/pdf document types. In primitive testing, nutch's parse-pdf comes close enough to the nutchwax parse-waxext plugin in the number of PDFs successfully parsed (80% vs. 90% of all PDFs in an ARC that contained 158). We make the move in the name of minimizing the number of NutchWAX external dependencies and in the hope that parse-pdf will continue to improve with time. Should you run into problems with parse-pdf -- it used to hang on PDFs from time-to-time in the past -- or you require that ingest parse the maximum number of PDFs, switching back to parse-waxext is just a matter of configuration. After ensuring xpdf is installed on all nodes, edit hadoop-sites.xml. Add in the plugin.includes from wax-default.xml and edit it so that rather than parse-pdf, instead it references parse-waxext. You'll then need to copy to your hadoop conf directory the wax-parse-plugins.xml and change references to parse-pdf to parse-waxext.

Its now possible to configure the open source wayback to use NutchWAX indices finding pages (and a page's embeds). One useful setup, has the wayback and NutchWAX WARs deployed in the same container with NutchWAX using the colocated wayback as the search result page renderer. See HOWTO: Configure Wayback to use NutchWAX index

Patches made to the Nutch 0.8.1 included in Nutch are listed in the NutchWAX README. NutchWAX 0.8.0 will only run on a hadoop 0.5.0. It will fail to run on later versions.

You'll see lots of output like the below during the import step. Its harmless even though its reported at the ERROR log level (Subsequent to the 0.8.1 release of nutch, these messages are no longer reported at ERROR log level). The HTML parser is reporting something that looked like a link is not of a supported protocol. In the below example margin-bottom is not a supported protocol:

06/12/04 15:36:35 ERROR parse.OutlinkExtractor: getOutlinks
java.net.MalformedURLException: unknown protocol: margin-bottom
	at java.net.URL.<init>(URL.java:574)
	at java.net.URL.<init>(URL.java:464)
	at java.net.URL.<init>(URL.java:413)
	at org.apache.nutch.net.BasicUrlNormalizer.normalize(BasicUrlNormalizer.java:78)
	at org.apache.nutch.parse.Outlink.<init>(Outlink.java:35)
	at org.apache.nutch.parse.OutlinkExtractor.getOutlinks(OutlinkExtractor.java:111)
	at org.apache.nutch.parse.OutlinkExtractor.getOutlinks(OutlinkExtractor.java:70)
	at org.apache.nutch.parse.text.TextParser.getParse(TextParser.java:47)
	at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:82)
	at org.archive.access.nutch.ImportArcs.processRecord(ImportArcs.java:513)
	at org.archive.access.nutch.ImportArcs$IndexingThread.run(ImportArcs.java:324)

Incremental indexing does not work in 0.6.0 [See [1477183] [nutchwax] Incremental indexing broken].

Indexes and segments made with 0.4.x NutchWAX will not work with the 0.6.0 release (and vice versa).

PDFs whose size is greater than 10megs are skipped completely. Legitimate PDFs whose http content-length does not strictly gree with the ARC length are also skipped.