Apache Druid
Druid is a column-oriented, open-source, distributed data store written in Java.
≈ 7 min read
Apache Druid. encyclopegia.org, snapshot enwiki-001346335485. /a/apache-druid
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
Druid is a column-oriented, open-source, distributed data store written in Java. Druid is designed to quickly ingest massive quantities of event data, and provide low-latency queries on top of the data.[3] The name Druid comes from the shapeshifting Druid class in many role-playing games, to reflect that the architecture of the system can shift to solve different types of data problems.
Druid is commonly used in business intelligence-OLAP applications to analyze high volumes of real-time and historical data.[4] Druid is used in production by technology companies such as Alibaba,[4] Airbnb,[4] Nielsen,[4] Cisco,[5][4] eBay,[6] Lyft,[7] Netflix,[8] PayPal,[4] Pinterest,[9] Reddit,[10] Twitter,[11] Walmart,[12] Wikimedia Foundation[13] and Yahoo.[14]
History
edit edit sourceDruid was started in 2011 by Eric Tschetter, Fangjin Yang, Gian Merlino and Vadim Ogievetsky[15] to power the analytics product of Metamarkets. The project was open-sourced under the GPL license in October 2012,[16][17][18] and moved to an Apache License in February 2015.[19][20]
Architecture
edit edit sourceFully deployed, Druid runs as a cluster of specialized processes (called nodes in Druid) to support a fault-tolerant architecture[21] where data is stored redundantly, and there is no single point of failure.[22] The cluster includes external dependencies for coordination (Apache ZooKeeper), metadata storage (e.g. MySQL, PostgreSQL, or Derby), and a deep storage facility (e.g. HDFS, or Amazon S3) for permanent data backup.
Query management
edit edit sourceClient queries first hit broker nodes, which forward them to the appropriate data nodes (either historical or real-time). Since Druid segments may be partitioned, an incoming query can require data from multiple segments and partitions (or shards) stored on different nodes in the cluster. Brokers are able to learn which nodes have the required data, and also merge partial results before returning the aggregated result.
Cluster management
edit edit sourceOperations relating to data management in historical nodes are overseen by coordinator nodes. Apache ZooKeeper is used to register all nodes, manage certain aspects of internode communications, and provide for leader elections.
Features
edit edit source- Low latency (streaming) data ingestion.
- Arbitrary slice and dice data exploration.
- Sub-second analytic queries.
- Approximate and exact computations.
Performance
edit edit sourceIn 2019, researchers compared the performance of Hive, Presto, and Druid using a denormalized Star Schema Benchmark based on the TPC-H standard. Druid was tested using both a “Druid Best” configuration using tables with hashed partitions and a “Druid Suboptimal” configuration which does not use hashed partitions.[23]
Tests were conducted by running the 13 TPC-H queries using TPC-H Scale Factor 30 (a 30GB database), Scale Factor 100 (a 100GB database), and Scale Factor 300 (a 300GB database).
| Scale Factor | Hive | Presto | Druid Best | Druid Suboptimal |
|---|---|---|---|---|
| 30 | 256s | 33s | 2.09s | 3.21s |
| 100 | 424s | 90s | 6.12s | 8.08s |
| 300 | 982s | 452s | 7.60s | 20.02s |
Druid performance was measured as at least 98% faster than Hive and at least 90% faster than Presto in each scenario, even when using the Druid Suboptimized configuration.
See also
edit edit sourceReferences
edit edit source- ^ "Apache Druid at GitHub". github.com. Retrieved 4 May 2021. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ "Release 38.0.0". 1 October 2026. Retrieved 2 October 2026. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ Hemsoth, Nicole. ""Druid Summons Strength in Real-Time"". Archived from the original on 2013-02-27. Retrieved 2014-02-07., Datanami, 8 November 2012 Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ a b c d e f druid. "Druid | Powered by Druid". druid.apache.org. Retrieved 2016-06-29. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ Butler, Brandon (20 June 2016). "Under the hood of Cisco's Tetration Analytics platform". Archived from the original on 2024-04-26. Retrieved 2016-06-23. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ "Druid at Pulsar - ebay的专栏 - 博客频道 - CSDN.NET". blog.csdn.net. Retrieved 2016-06-23. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ Streaming SQL and Druid by Arup Malakar, 23 August 2018, retrieved 2020-01-29 Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ "The Netflix Tech Blog: Announcing Suro: Backbone of Netflix's Data Pipeline". techblog.netflix.com. Retrieved 2016-06-23. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ Pinterest: Powering Ad Analytics with Apache Druid, 23 October 2019, retrieved 2020-01-29 Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ "Scaling Reporting at Reddit - Upvoted". www.redditinc.com. 26 February 2021. Retrieved 2022-09-13. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ "Interactive Analytics at MoPub: Querying Terabytes of Data in Seconds". blog.twitter.com. Retrieved 2020-01-29. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ Nayak, Amaresh (2018-02-23). "Event Stream Analytics at Walmart with Druid". Medium. Retrieved 2020-01-29. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ "Conferences - O'Reilly Media". Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ "Complementing Hadoop at Yahoo: Interactive Analytics with Druid". Retrieved 2016-06-23. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ "Druid: A Real-time Analytical Data Store" (PDF). Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ Tschetter, Eric. ""Introducing Druid"". Archived from the original on 2022-02-08. Retrieved 2019-06-12., druid.apache.org, 24 October 2012 Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ Higginbotham, Stacey. ""Metamarkets open sources Druid, its in-memory database"". Archived from the original on 2021-09-18. Retrieved 2014-02-07., GigaOM, 24 October 2012 Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ "Metamarkets Open Sources Druid, Streaming Real-Time Data Store". Yahoo News. 2012-10-24. Retrieved 2023-07-24. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ Harris, Derrick (2015-02-20). "The Druid real-time database moves to an Apache license". Archived from the original on 2015-08-22. Retrieved 2015-08-04. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ "Druid Gets Open Source-ier Under the Apache License". Retrieved 2015-08-04. Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ "Druid Project Documentation". Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ Yang, Fangjin; Tschetter, Eric; Léauté, Xavier; Ray, Nelson; Merlino, Gian; Ganguli, Deep. ""Druid: A Real-time Analytical Data Store"" (PDF)., Metamarkets, retrieved 6 February 2014 Wikipedia revision 1346335485, retrieved 2026-11-19
- ^ Correia, José; Costa, Carlos; Santos, Maribel Yasmina (2019). "Challenging SQL-on-Hadoop Performance with Apache Druid". In Abramowicz, Witold; Corchuelo, Rafael (eds.). Business Information Systems. Lecture Notes in Business Information Processing. Vol. 353. Cham: Springer International Publishing. pp. 149–161. doi:10.1007/978-3-030-20485-3_12. hdl:1822/66785. ISBN 978-3-030-20485-3. S2CID 190005302. Wikipedia revision 1346335485, retrieved 2026-11-19
External links
edit edit source The Apache Software Foundationprojects
- Accumulo
- ActiveMQ
- Airavata
- Airflow
- Allura
- Ambari
- Ant
- Aries
- Arrow
- Apache HTTP Server
- APR
- Avro
- Axis
- Axis2
- Beam
- Bloodhound
- Brooklyn
- Calcite
- Camel
- CarbonData
- Cassandra
- Cayenne
- CloudStack
- Cordova
- CouchDB
- cTAKES
- CXF
- Derby
- Directory
- Drill
- Druid
- Empire-db
- Felix
- Flex
- Flink
- Flume
- FreeMarker
- Geronimo
- Groovy
- Guacamole
- Gump
- Hadoop
- HBase
- Helix
- Hive
- Iceberg
- Ignite
- Impala
- Jackrabbit
- James
- Jena
- JMeter
- Kafka
- Kudu
- Kylin
- Lucene
- Mahout
- Maven
- MINA
- mod_perl
- MyFaces
- Mynewt
- NiFi
- NetBeans
- Nutch
- NuttX
- OFBiz
- Oozie
- OpenEJB
- OpenJPA
- OpenNLP
- OрenOffice
- ORC
- PDFBox
- Parquet
- Phoenix
- POI
- Pig
- Pinot
- Pivot
- Qpid
- Roller
- RocketMQ
- Samza
- Shiro
- SINGA
- Sling
- Solr
- Spark
- Storm
- SpamAssassin
- Struts
- Subversion
- Superset
- SystemDS
- Tapestry
- Thrift
- Tika
- TinkerPop
- Tomcat
- Trafodion
- Traffic Server
- UIMA
- Velocity
- Wicket
- Xalan
- Xerces
- XMLBeans
- Yetus
- ZooKeeper
Text derived from https://en.wikipedia.org/w/index.php?oldid=1346335485 under CC BY-SA 4.0; modifications have been made.