Zstd
Zstandard is a lossless data compression algorithm developed by Yann Collet at Facebook.
≈ 14 min read
Zstd. encyclopegia.org, snapshot enwiki-001367433854. /a/zstd
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
Zstandard is a lossless data compression algorithm developed by Yann Collet at Facebook. Zstd is the corresponding reference implementation in C, released as open-source software on 31 August 2016.[3][4]
The algorithm was published in 2018 as RFC 8478, which also defines an associated media type "application/zstd", filename extension "zst", and HTTP content encoding "zstd".[5]
In February 2021, RFC 8878 replaced and obsoleted RFC 8478. RFC 8878 is an Informational publication rather than an Internet Standards Track specification.[6]
Features
edit edit sourceZstandard was designed to give a compression ratio comparable to that of the DEFLATE algorithm (developed in 1991 and used in the original ZIP and gzip programs), but faster, especially for decompression. It is tunable with compression levels ranging from negative 7 (fastest)[7] to 22 (slowest in compression speed, but best compression ratio).
Starting from version 1.3.2 (October 2017), zstd optionally implements very-long-range search and deduplication (--long, 128 MiB window) similar to rzip or lrzip.[8]
Compression speed can vary by a factor of 20 or more between the fastest and slowest levels, while decompression is uniformly fast, varying by less than 20% between the fastest and slowest levels.[9] The Zstandard command-line has an "adaptive" (--adapt) mode that varies compression level depending on I/O conditions, mainly how fast it can write the output.
Zstandard at its maximum compression level gives a compression ratio close to lzma, lzham, and ppmx.[10][11] As of 2019, Zstandard reaches the Pareto frontier for decompression, as it produces output faster than any other open source algorithm with a similar or better compression ratio.[12]
Dictionaries can have a large impact on the compression ratio of small files, so Zstandard can use a user-provided compression dictionary. It also offers a training mode, able to generate a dictionary from a set of samples.[13][14] In particular, one dictionary can be loaded to process large sets of files with redundancy between files, but not necessarily within each file, such as for log files.
Design
edit edit sourceZstandard combines a dictionary-matching stage (LZ77) with a large search window and a fast entropy-coding stage. It uses Huffman coding alongside finite-state entropy (FSE), a variant of tANS.[15]
Usage
edit edit sourceThe Linux kernel has included Zstandard since November 2017 (version 4.14) as a compression method for the btrfs and squashfs filesystems,[17][18][19] as well as for loadable kernel modules.
In 2017, Allan Jude integrated Zstandard into the FreeBSD kernel,[20] and it was subsequently integrated as a compressor option for core dumps (both user programs and kernel panics). It was also used to create a proof-of-concept OpenZFS compression method[9] which was integrated in 2020.[21]
The AWS Redshift and RocksDB databases include support for field compression using Zstandard.[22]
In March 2018, Canonical tested[23] the use of zstd as a deb package compression method by default for the Ubuntu Linux distribution. Compared with xz compression of deb packages, zstd at level 19 decompresses significantly faster, but at the cost of 6% larger package files. Support was added to Debian (and subsequently, Ubuntu) in April 2018 (in version 1.6~rc1).[24][23][25]
Fedora added ZStandard support to RPM in May 2018 (Fedora release 28) and used it for packaging the release in October 2019 (Fedora 31).[26] In Fedora 33, the filesystem is compressed by default with zstd.[27][28]
Arch Linux added support for zstd as a package compression method in October 2019 with the release of the pacman 5.2 package manager[29] and in January 2020 switched from xz to zstd for the packages in the official repository. Arch uses zstd -c -T0 --ultra -20 -; while the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 14 times faster, decompression memory increased by 50 MiB when using multiple threads, and compression memory increased but scales with the number of threads used.[30][31][32] Arch Linux later also switched to zstd as the default compression algorithm for mkinitcpio initial ramdisk generator.[33]
On 15 June 2020, Zstandard was implemented in version 6.3.8 of the zip file format with codec number 93, deprecating the previous codec number of 20 as it was implemented in version 6.3.7, released on 1 June.[34][35]
On 31 October 2023, official zstd support for compression/decompression was added to Windows Explorer in Windows 11 (via update package KB5031455)[36]
In March 2024, Google Chrome version 123 (and Chromium-based browsers such as Brave or Microsoft Edge) added zstd support in the HTTP header Content-Encoding.[37] In May 2024, Firefox release 126.0 added zstd support in the HTTP header Content-Encoding.[38]
Python offers native support for zstd since Python 3.14 which has been released in October 2025.[39][40]
License
edit edit sourceThe reference implementation is licensed under the BSD license, published at GitHub.[41] Since version 1.0, published 31 August 2016,[42] it had an additional Grant of Patent Rights.[43]
From version 1.3.1, released 20 August 2017,[44] this patent grant was dropped and the license was changed to a BSD + GPLv2 dual license.[45]
See also
edit edit source- LZ4 (compression algorithm) – a fast member of the LZ77 family
- LZFSE – a similar algorithm by Apple used since iOS 9 and OS X 10.11 and made open source on 1 June 2016
- Zlib
- Brotli – also integrated into browsers
- Gzip – one of the most widely used compression tools
References
edit edit source- ^ "Contributors to facebook/zstd". github.com. Retrieved 26 January 2021. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Release Zstandard v1.5.7". Retrieved 21 February 2025. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ De Simone, Sergio (2 September 2016). "Facebook Open-Sources New Compression Algorithm Outperforming Zlib". InfoQ. Archived from the original on 7 October 2021. Retrieved 20 April 2019. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Williams, Chris (31 August 2016). "Life imitates satire: Facebook touts zlib killer just like Silicon Valley's Pied Piper". The Register. Archived from the original on 3 September 2016. Retrieved 6 September 2016. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Collet, Yann (October 2018). Kucherawy, Murray S. (ed.). Zstandard Compression and the application/zstd Media Type. Internet Engineering Task Force Request for Comments. IETF. doi:10.17487/RFC8478. RFC 8478. Retrieved 7 October 2020. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "RFC 8878: Zstandard Compression and the application/zstd Media Type". RFC Editor. February 2021. Retrieved 3 August 2026. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Release Zstandard v1.3.4 - faster everything · facebook/zstd". GitHub. Archived from the original on 11 September 2021. Retrieved 27 March 2024. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Command Line Interface for Zstandard library". GitHub. Facebook. 28 October 2021. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ a b Jude, Allan (2017). "ZStandard in ZFS" (PDF). open-zfs.org. Archived (PDF) from the original on 18 December 2019. Retrieved 20 April 2019. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Mahoney, Matt. "Silesia Open Source Compression Benchmark". Archived from the original on 21 January 2022. Retrieved 10 May 2019. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Mahoney, Matt (29 August 2016). "Large Text Compression Benchmark, .2157 zstd". Archived from the original on 31 March 2022. Retrieved 10 May 2019. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ TurboBench: Static/Dynamic web content compression benchmark, PowTurbo, archived from the original on 17 March 2022, retrieved 6 January 2026 Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Bockelman, Brian; Pivraski, Jim; Shadura, Oksana; Zhang, Zhe (11 October 2017). "DIANA Contributions Update" (PDF). Fermilab. Archived (PDF) from the original on 25 January 2018. Retrieved 27 March 2024. Facebook developers report massive speedups and compression ratio improvements when using dictionaries Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Collet, Yann; Turner, Chip (31 August 2016). "Smaller and faster data compression with Zstandard". Facebook. Archived from the original on 27 September 2016. Retrieved 3 September 2016. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Zstandard Compression Format". GitHub. Facebook. 28 October 2021. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ a b c d Collet, Yann (February 2021). Kucherawy, Murray S. (ed.). Zstandard Compression and the application/zstd Media Type. Internet Engineering Task Force Request for Comments. IETF. doi:10.17487/RFC8878. RFC 8878. Retrieved 26 February 2023. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Corbet, Jonathan (17 September 2017). "The rest of the 4.14 merge window [LWN.net]". lwn.net. Archived from the original on 22 November 2021. Retrieved 27 March 2024. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Linux_4.14". Kernelnewbies.org. 30 December 2017. Archived from the original on 10 January 2018. Retrieved 16 August 2018. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Larabel, Michael (8 September 2017). "Zstd Compression For Btrfs & Squashfs Set For Linux 4.14, Already Used Within Facebook". www.phoronix.com. Archived from the original on 25 July 2019. Retrieved 13 November 2017. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Integrate ZSTD into the kernel · freebsd/Freebsd-SRC@28ef165". GitHub. The FreeBSD Project. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Add ZSTD support to ZFS · openzfs/ZFS@10b3c7f". GitHub. OpenZFS. Archived from the original on 10 September 2020. Retrieved 12 October 2020. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Zstandard Encoding - Amazon Redshift". Amazon Web Services. 20 April 2019. Archived from the original on 14 August 2021. Retrieved 24 January 2018. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ a b Larabel, Michael (12 March 2018). "Canonical Working On Zstd-Compressed Debian Packages For Ubuntu". phoronix.com. Phoronix Media. Archived from the original on 16 August 2021. Retrieved 29 October 2019. The developers at Canonical are considering a feature freeze exception to get this newly-developed Zstd Apt/Dpkg support in Ubuntu 18.04 LTS. In doing so, they mention they would be looking at enabling Zstd compression for packages by default in Ubuntu 18.10. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Nestor, Marius (12 March 2018). "New Ubuntu Installs Could Be Speed Up by 10% with the Zstd Compression Algorithm". Softpedia. Archived from the original on 6 October 2021. Retrieved 13 August 2018. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Debian Changelog for apt". Debian. 19 April 2021. Retrieved 7 November 2022. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Changes/Switch RPMS to ZSTD compression". Fedora Project Wiki. Archived from the original on 2 June 2019. Retrieved 8 July 2020. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Thurman, Andrew (14 April 2021). "Fedora Workstation 34 feature focus: Btrfs transparent compression". Fedora Magazine. Retrieved 12 May 2022. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Changes/BtrfsTransparentCompression". Fedora Project Wiki. Retrieved 12 May 2022. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Larabel, Michael (16 October 2019). "Arch Linux Nears Roll-Out of ZSTD Compressed Packages for Faster Pacman Installs". Phoronix. Archived from the original on 18 March 2022. Retrieved 21 October 2019. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Broda, Mara (4 January 2020). "Now using Zstandard instead of xz for package compression". Arch Linux. Archived from the original on 18 March 2022. Retrieved 5 January 2020. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Broda, Mara (25 March 2019). "RFC: (devtools) Changing default compression method to zstd". arch-dev-public (Mailing list). Archived from the original on 17 August 2021. Retrieved 5 January 2020. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Broda, Mara; Polyak, Levente (27 December 2019). "makepkg.conf: change default compression method to zstd". GitHub. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Razzolini, Giancarlo (19 February 2021). "News: Moving to Zstandard images by default on mkinitcpio". Arch Linux. Retrieved 28 December 2021. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.8, PKWARE, 15 June 2020, retrieved 7 July 2020 Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.7, PKWARE, 1 June 2020, retrieved 6 June 2020 Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "October 31, 2023—KB5031455 (OS Builds 22621.2506 and 22631.2506) Preview". support.microsoft.com. Retrieved 23 July 2026. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Jara, Adriana (19 March 2024). "New in Chrome 123 | Chrome Blog". Chrome for Developers. Google. Retrieved 16 April 2024. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Firefox 126.0, See All New Features, Updates and Fixes". May 2024. Archived from the original on 13 May 2024. Retrieved 15 May 2024. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "What's new in Python 3.14". Python documentation. Retrieved 23 July 2026. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "PEP 784 – Adding Zstandard to the standard library | peps.python.org". Python Enhancement Proposals (PEPs). Retrieved 23 July 2026. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ Dignan, Larry (31 August 2016). "Facebook open sources Zstandard data compression algorithm, aims to replace technology behind Zip". ZDnet. Retrieved 1 September 2016. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Zstandard v1.0". GitHub. Facebook. 31 August 2016. Archived from the original on 7 April 2023. Retrieved 23 January 2025. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "v1.3.0/PATENTS · facebook/zstd". GitHub. Facebook. 30 August 2016. Archived from the original on 15 May 2021. Retrieved 27 March 2024. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "Release Zstandard v1.3.1 · facebook/zstd". GitHub. Facebook. 20 August 2017. Archived from the original on 12 September 2020. Retrieved 27 March 2024. Wikipedia revision 1367433854, retrieved 2026-11-22
- ^ "New license by Cyan4973 · Pull Request #801 · facebook/zstd". GitHub. Facebook. 19 August 2017. Archived from the original on 12 September 2020. Retrieved 27 March 2024. Wikipedia revision 1367433854, retrieved 2026-11-22
External links
edit edit source- Official website
- zstd on GitHub
- 7zip with Zstandard on GitHub
- "Smaller and faster data compression with Zstandard", Yann Collet and Chip Turner, 31 August 2016, Facebook Announcement
- The Guardian is using ZStandard instead of zlib
type
type
and compressing
and distributing
and distributing
Text derived from https://en.wikipedia.org/w/index.php?oldid=1367433854 under CC BY-SA 4.0; modifications have been made.
References
3 claims could not be tied to a specific sentence and are listed in the table below instead.
- ↩Wikipedia “Zstd” publication date (×2) · confidence 0.95 revision URL