Apache Nutch

Uses your system voice; voice and rate in settings.

Snapshot

≈ 1 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

Apache Nutch is a highly extensible and scalable open source web crawler software project.

Apache Nutch is a highly extensible and scalable open source web crawler software project.

Release history

Scalability

IBM Research studied the performance of Nutch/Lucene as part of its Commercial Scale Out (CSO) project. Their findings were that a scale-out system, such as Nutch/Lucene, could achieve a performance level on a cluster of blades that was not achievable on any scale-up computer such as the POWER5. The ClueWeb09 dataset (used in e.g. TREC) was gathered using Nutch, with an average speed of 755.31 documents per second.

Related projects

Hadoop – Java framework that supports distributed applications running on large clusters.

Search engines built with Nutch

Common Crawl – publicly available internet-wide crawls, started using Nutch in 2014. Creative Commons Search – an implementation of Nutch, used in the period of 2004–2006. DiscoverEd – Open educational resources search prototype developed by Creative Commons Krugle uses Nutch to crawl web pages for code, archives and technically interesting content. mozDex (inactive) Wikia Search - launched 2008, closed down 2009

See also

References

Bibliography

External links

Text derived from https://en.wikipedia.org/w/index.php?oldid=1319004893 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

15 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.