Snappy (compression)

Uses your system voice; voice and rate in settings.

Snapshot

≈ 4 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

Snappy (previously known as Zippy) is a fast data compression and decompression library written in C++ by Google based on ideas from LZ77 and open-sourced in 2011. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. Compression speed is 250 MB/s and decompression speed is 500 MB/s using a single core of a circa 2011 "Westmere" 2.26 GHz Core i7 processor running in 64-bit mode. The compression ratio is 20–100% lower than gzip. Snappy is widely used in Google projects like Bigtable, MapReduce and in compressing data for Google's internal RPC systems. It can be used in open-source projects like MariaDB ColumnStore, Cassandra, Couchbase, Hadoop, LevelDB, MongoDB, RocksDB, Lucene, Spark, Parquet, InfluxDB, and Ceph. Firefox uses Snappy to compress data in localStorage. Decompression is tested to detect any errors in the compressed stream. Snappy does not use inline assembler (except some optimizations ) and is portable.

Snappy (previously known as Zippy) is a fast data compression and decompression library written in C++ by Google based on ideas from LZ77 and open-sourced in 2011. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. Compression speed is 250 MB/s and decompression speed is 500 MB/s using a single core of a circa 2011 "Westmere" 2.26 GHz Core i7 processor running in 64-bit mode. The compression ratio is 20–100% lower than gzip. Snappy is widely used in Google projects like Bigtable, MapReduce and in compressing data for Google's internal RPC systems. It can be used in open-source projects like MariaDB ColumnStore, Cassandra, Couchbase, Hadoop, LevelDB, MongoDB, RocksDB, Lucene, Spark, Parquet, InfluxDB, and Ceph. Firefox uses Snappy to compress data in localStorage. Decompression is tested to detect any errors in the compressed stream. Snappy does not use inline assembler (except some optimizations ) and is portable. The stream starts with the length of the uncompressed data as a varint, so the first byte, with the high bit clear, corresponds to a length of 51 16 =81 bytes. The first block must be a literal, and f042 corresponds thereto: the first byte is broken down as f0 16 ⇒ len−1=111100 2 ;type=00 2 ; type 0 signifies a literal, and a length−1 of 111100 2 =60 means the length is read from the following byte, in this case 42 16 =66. The first 66 bytes of the text ("Wikipedia is a free, web-based, collaborative, multilingual encyclo") follow. 000040 6e 63 79 63 6c 6f 09 3f 1c 70 72 6f 6a 65 63 74 >ncyclo.?.project . 16 ⇒ off h =000 2 ,len−4=010 2 ;type=01 2 : type 1 indicates a "copy with 1-byte offset": the length to copy works out to 010 2 +4=6 bytes, and the offset is an 11-bit integer whose top bits are off h and whose low bits are the next byte: 3f, so {off h }{3f 16 }=00000111111 2 =63. This means to copy 6 bytes, starting 63 bytes ago – since 67 bytes have already been copied this evaluates to copying 6 bytes starting at position 4 (from the fifth byte), which produces "pedia ". This block has no other content, and thus the following block starts immediately after – 1c 16 ⇒ len−1=000111 2 ;type=00 2 , i.e. a literal of length 000111 2 +1=8. The final part of the text ("project.") follows. In this example, all common substrings with four or more characters were eliminated by the compression process. More common compressors can compress this better. Unlike compression methods such as gzip and bzip2, there is no entropy encoding used to pack alphabet into the bit stream.

External links

Snappy mailing list

Text derived from https://en.wikipedia.org/w/index.php?oldid=1330311696 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

18 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.