Apache Hive

Apache Hive is a data warehouse software project. It is built on top of Apache Hadoop for providing data query and analysis. Hive gives an SQL-like interface to query data stored in various databases and file systems that integrate with Hadoop. Traditional SQL queries must be implemented in the MapReduce Java API to execute SQL applications and queries over distributed data. Hive provides the necessary SQL abstraction to integrate SQL-like queries (HiveQL) into the underlying Java without the need to implement queries in the low-level Java API. Hive facilitates the integration of SQL-based querying languages with Hadoop, which is commonly used in data warehousing applications. While initially developed by Facebook, Apache Hive is used and developed by other companies such as Netflix and the Financial Industry Regulatory Authority (FINRA). Amazon maintains a software fork of Apache Hive included in Amazon Elastic MapReduce on Amazon Web Services. ==Features== Apache Hive supports the analysis of large datasets stored in Hadoop's HDFS and compatible file systems such as Amazon S3 filesystem and Alluxio. It provides a SQL-like query language called HiveQL with schema on read and transparently converts queries to MapReduce, Apache Tez and Spark jobs. All three execution engines can run in Hadoop's resource negotiator, YARN (Yet Another Resource Negotiator). To accelerate queries, it provided indexes, but this feature was removed in version 3.0 Other features of Hive include: Different storage types such as plain text, RCFile, HBase, ORC, and others. Metadata storage in a relational database management system, significantly reduces the time to perform semantic checks during query execution. Operating on compressed data stored in the Hadoop ecosystem using algorithms including DEFLATE, BWT, Snappy, etc. Built-in user-defined functions (UDFs) to manipulate dates, strings, and other data-mining tools. Hive supports extending the UDF set to handle use cases not supported by built-in functions. SQL-like queries (HiveQL), which are implicitly converted into MapReduce or Tez, or Spark jobs. By default, Hive stores metadata in an embedded Apache Derby database, and other client/server databases like MySQL can optionally be used. The first four file formats supported in Hive were plain text, sequence file, optimized row columnar (ORC) format and RCFile. Apache Parquet can be read via plugin in versions later than 0.10 and natively starting at 0.13. ==HiveQL== While based on SQL, HiveQL does not strictly follow the full SQL-92 standard. HiveQL offers extensions not in SQL, including multi-table inserts, and creates tables as select. HiveQL lacked support for transactions and materialized views, and offered only limited subquery support. Support for insert, update, and delete with full ACID functionality was made available with release 0.14. Internally, a compiler translates HiveQL statements into a directed acyclic graph of MapReduce, Tez, or Spark jobs, which are submitted to Hadoop for execution. DROP TABLE IF EXISTS docs; CREATE TABLE docs (line STRING); LOAD DATA INPATH 'input_file' OVERWRITE INTO TABLE docs; CREATE TABLE word_counts AS SELECT word, count(1) AS count FROM (SELECT explode(split(line, '\s')) AS word FROM docs) temp GROUP BY word ORDER BY word; A brief explanation of each of the statements is as follows: DROP TABLE IF EXISTS docs; CREATE TABLE docs (line STRING); Checks if table exists and drops it if it does. Creates a new table called with a single column of type called . LOAD DATA INPATH 'input_file' OVERWRITE INTO TABLE docs; Loads the specified file or directory (In this case “input_file”) into the table. specifies that the target table to which the data is being loaded into is to be re-written; Otherwise, the data would be appended. CREATE TABLE word_counts AS SELECT word, count(1) AS count FROM (SELECT explode(split(line, '\s')) AS word FROM docs) temp GROUP BY word ORDER BY word; The query creates a table called with two columns: and . This query draws its input from the inner query , which splits the input words into different rows of a temporary table aliased as . The clause groups the results based on their keys, so that the column holds the number of occurrences for each word of the column. The clause sorts the words alphabetically. ==Security== Hive v0.7.0 added integration with Hadoop security. Hadoop began using Kerberos authorization support to provide security. Kerberos allows for mutual authentication between client and server, where the client's request for a ticket is passed along with the request. In earlier versions of Hadoop, users could modify usernames via the hadoop.job.ugi property, and MapReduce operations ran under a shared default user identity (such as Hadoop or mapred). Following Hive v0.7.0's integration with Hadoop security, TaskTracker jobs are executed by the user who initiated them, and the username can no longer be spoofed by setting the hadoop.job.ugi property. Permissions for newly created files in Hive are dictated by the HDFS. The Hadoop distributed file system authorization model uses three entities: user, group and others with three permissions: read, write and execute. The default permissions for newly created files can be set by changing the unmask value for the Hive configuration variable hive.files.umask.value . . Permissions for newly created files in Hive are dictated by the HDFS. The Hadoop distributed file system authorization model uses three entities: user, group and others with three permissions: read, write and execute. The default permissions for newly created files can be set by changing the unmask value for the Hive configuration variable hive.files.umask.value . ==See also== Apache Pig Sqoop Apache Impala Apache Drill Apache Flume Apache HBase Trino (SQL query engine) ==Notes== ==References== ==External links==

Text derived from https://en.wikipedia.org/w/index.php?oldid=1360128781 under CC BY-SA 4.0; modifications have been made.

Names in other languages