Apache Nutch
≈ 1 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
Apache Nutch is a highly extensible and scalable open source web crawler software project.
Apache Nutch is a highly extensible and scalable open source web crawler software project.
Release history
Scalability
IBM Research studied the performance of Nutch/Lucene as part of its Commercial Scale Out (CSO) project. Their findings were that a scale-out system, such as Nutch/Lucene, could achieve a performance level on a cluster of blades that was not achievable on any scale-up computer such as the POWER5. The ClueWeb09 dataset (used in e.g. TREC) was gathered using Nutch, with an average speed of 755.31 documents per second.
Related projects
Hadoop – Java framework that supports distributed applications running on large clusters.
Search engines built with Nutch
Common Crawl – publicly available internet-wide crawls, started using Nutch in 2014. Creative Commons Search – an implementation of Nutch, used in the period of 2004–2006. DiscoverEd – Open educational resources search prototype developed by Creative Commons Krugle uses Nutch to crawl web pages for code, archives and technically interesting content. mozDex (inactive) Wikia Search - launched 2008, closed down 2009
See also
References
Bibliography
External links
Text derived from https://en.wikipedia.org/w/index.php?oldid=1319004893 under CC BY-SA 4.0; modifications have been made.