Bfloat16 floating-point format
≈ 20 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
The bfloat16 (brain floating point) floating-point format is a computer number format occupying 16 bits in computer memory; it represents a wide dynamic range of numeric values by using a floating radix point. This format is a shortened (16-bit) version of the 32-bit IEEE 754 single-precision floating-point format (binary32) with the intent of accelerating machine learning and near-sensor computing. It preserves the approximate dynamic range of 32-bit floating-point numbers by retaining 8 exponent bits, but supports only an 8-bit precision rather than the 24-bit significand of the binary32 format. More so than single-precision 32-bit floating-point numbers, bfloat16 numbers are unsuitable for integer calculations, but this is not their intended use. Bfloat16 is used to reduce the storage requirements and increase the calculation speed of machine learning algorithms. The bfloat16 format was developed by Google Brain, an artificial intelligence research group at Google, for use in its TPU v2, released in 20171. It is utilized in many CPUs, GPUs, and AI processors, such as Intel Xeon processors (AVX-512 BF16 extensions), Intel Data Center GPU, Intel Nervana NNP-L1000, Intel FPGAs, AMD Zen, AMD Instinct, NVIDIA GPUs, Google Cloud TPUs, AWS Inferentia, AWS Trainium, ARMv8.6-A, and Apple's M2 and therefore A15 chips and later.…
The bfloat16 (brain floating point) floating-point format is a computer number format occupying 16 bits in computer memory; it represents a wide dynamic range of numeric values by using a floating radix point. This format is a shortened (16-bit) version of the 32-bit IEEE 754 single-precision floating-point format (binary32) with the intent of accelerating machine learning and near-sensor computing. It preserves the approximate dynamic range of 32-bit floating-point numbers by retaining 8 exponent bits, but supports only an 8-bit precision rather than the 24-bit significand of the binary32 format. More so than single-precision 32-bit floating-point numbers, bfloat16 numbers are unsuitable for integer calculations, but this is not their intended use. Bfloat16 is used to reduce the storage requirements and increase the calculation speed of machine learning algorithms. The bfloat16 format was developed by Google Brain, an artificial intelligence research group at Google, for use in its TPU v2, released in 2017. It is utilized in many CPUs, GPUs, and AI processors, such as Intel Xeon processors (AVX-512 BF16 extensions), Intel Data Center GPU, Intel Nervana NNP-L1000, Intel FPGAs, AMD Zen, AMD Instinct, NVIDIA GPUs, Google Cloud TPUs, AWS Inferentia, AWS Trainium, ARMv8.6-A, and Apple's M2 and therefore A15 chips and later. Many libraries support bfloat16, such as CUDA, Intel oneAPI Math Kernel Library, AMD ROCm, AMD Optimizing CPU Libraries, PyTorch, and TensorFlow. On these platforms, bfloat16 may also be used in mixed-precision arithmetic, where bfloat16 numbers may be operated on and expanded to wider data types. | style="border-width:0; background:#FFFFFF;" colspan="10"| |- | style="border-width:0; background:#FFFFFF;" | || style="background:#C4FCFF;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 1 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 |- style="font-size:9pt;" | style="border-width:0; background:#FFFFFF;" | || style="border-width:0; background:#FFFFFF;" | 15 || style="border-width:0; background:#FFFFFF;" | 14 || style="border-width:0; background:#FFFFFF;" colspan="3" | || style="border-width:0; background:#FFFFFF;" | 10 || style="border-width:0; background:#FFFFFF;" | 9 || style="border-width:0; background:#FFFFFF;" colspan="8" | || style="border-width:0; background:#FFFFFF;" | 0 |- | style="border-width:0; background:#FFFFFF;" | |- | style="border-width:0; background:#FFFFFF; text-align:left" colspan="17" | bfloat16 |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | sign | style="border-width:0; background:#FFFFFF;" colspan="8" | exponent (8 bit) | style="border-width:0; background:#FFFFFF;" colspan="7" | fraction (7 bit) |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | ┃ | style="border-width:0; background:#FFFFFF;" colspan="8" | | style="border-width:0; background:#FFFFFF;" colspan="7" | |- | style="border-width:0; background:#FFFFFF;" | || style="background:#C4FCFF;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 1 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 |- style="font-size:9pt;" | style="border-width:0; background:#FFFFFF;" | || style="border-width:0; background:#FFFFFF;" | 15 || style="border-width:0; background:#FFFFFF;" | 14 || style="border-width:0; background:#FFFFFF;" colspan="6" | || style="border-width:0; background:#FFFFFF;" | 7 || style="border-width:0; background:#FFFFFF;" | 6 || style="border-width:0; background:#FFFFFF;" colspan="5" | || style="border-width:0; background:#FFFFFF;" | 0 |- | style="border-width:0; background:#FFFFFF;" | |- | style="border-width:0; background:#FFFFFF; text-align:left" colspan="17" | IBM's DLFloat (16 bits) |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | sign | style="border-width:0; background:#FFFFFF;" colspan="6" | exponent (6 bit) | style="border-width:0; background:#FFFFFF;" colspan="9" | fraction (9 bit) |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | ┃ | style="border-width:0; background:#FFFFFF;" colspan="6" | | style="border-width:0; background:#FFFFFF;" colspan="9" | |- | style="border-width:0; background:#FFFFFF;" | || style="background:#C4FCFF;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 1 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 |- style="font-size:9pt;" | style="border-width:0; background:#FFFFFF;" | || style="border-width:0; background:#FFFFFF;" | 15 || style="border-width:0; background:#FFFFFF;" | 14 || style="border-width:0; background:#FFFFFF;" colspan="4" | || style="border-width:0; background:#FFFFFF;" | 9 || style="border-width:0; background:#FFFFFF;" | 8 || style="border-width:0; background:#FFFFFF;" colspan="7" | || style="border-width:0; background:#FFFFFF;" | 0 |- | style="border-width:0; background:#FFFFFF;" | |- | style="border-width:0; background:#FFFFFF; text-align:left" colspan="20" | Nvidia's TensorFloat-32 (19 bits) |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | sign | style="border-width:0; background:#FFFFFF;" colspan="8" | exponent (8 bit) | style="border-width:0; background:#FFFFFF;" colspan="10"| fraction (10 bit) |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | ┃ | style="border-width:0; background:#FFFFFF;" colspan="8" | | style="border-width:0; background:#FFFFFF;" colspan="10"| |- | style="border-width:0; background:#FFFFFF;" | || style="background:#C4FCFF;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 1 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 |- style="font-size:9pt;" | style="border-width:0; background:#FFFFFF;" | || style="border-width:0; background:#FFFFFF;" | 18 || style="border-width:0; background:#FFFFFF;" | 17 || style="border-width:0; background:#FFFFFF;" colspan="6" | || style="border-width:0; background:#FFFFFF;" | 10 || style="border-width:0; background:#FFFFFF;" | 9 || style="border-width:0; background:#FFFFFF;" colspan="8" | || style="border-width:0; background:#FFFFFF;" | 0 |- | style="border-width:0; background:#FFFFFF;" | |- | style="border-width:0; background:#FFFFFF; text-align:left" colspan="25" | ATI's fp24 format |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | sign | style="border-width:0; background:#FFFFFF;" colspan="7" | exponent (7 bit) | style="border-width:0; background:#FFFFFF;" colspan="16"| fraction (16 bit) |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | ┃ | style="border-width:0; background:#FFFFFF;" colspan="7" | | style="border-width:0; background:#FFFFFF;" colspan="16"| |- | style="border-width:0; background:#FFFFFF;" | || style="background:#C4FCFF;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 1 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 |- style="font-size:9pt;" | style="border-width:0; background:#FFFFFF;" | || style="border-width:0; background:#FFFFFF;" | 23 || style="border-width:0; background:#FFFFFF;" | 22 || style="border-width:0; background:#FFFFFF;" colspan="5" | || style="border-width:0; background:#FFFFFF;" | 16 || style="border-width:0; background:#FFFFFF;" | 15 || style="border-width:0; background:#FFFFFF;" colspan="14" | || style="border-width:0; background:#FFFFFF;" | 0 |- | style="border-width:0; background:#FFFFFF;" | |- | style="border-width:0; background:#FFFFFF; text-align:left" colspan="25" | Pixar's PXR24 format |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | sign | style="border-width:0; background:#FFFFFF;" colspan="8" | exponent (8 bit) | style="border-width:0; background:#FFFFFF;" colspan="15"| fraction (15 bit) |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | ┃ | style="border-width:0; background:#FFFFFF;" colspan="8" | | style="border-width:0; background:#FFFFFF;" colspan="15"| |- | style="border-width:0; background:#FFFFFF;" | || style="background:#C4FCFF;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#FFACAC;" | 1 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 |- style="font-size:9pt;" | style="border-width:0; background:#FFFFFF;" | || style="border-width:0; background:#FFFFFF;" | 23 || style="border-width:0; background:#FFFFFF;" | 22 || style="border-width:0; background:#FFFFFF;" colspan="6" | || style="border-width:0; background:#FFFFFF;" | 15 || style="border-width:0; background:#FFFFFF;" | 14 || style="border-width:0; background:#FFFFFF;" colspan="13" | || style="border-width:0; background:#FFFFFF;" | 0 |- | style="border-width:0; background:#FFFFFF;" | |- | style="border-width:0; background:#FFFFFF; text-align:left" colspan="33" | IEEE 754 single-precision 32-bit float |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | sign | style="border-width:0; background:#FFFFFF;" colspan="8" | exponent (8 bit) | style="border-width:0; background:#FFFFFF;" colspan="23"| fraction (23 bit) |- style="line-height:70%;" | style="border-width:0; background:#FFFFFF;" colspan="2" | ┃ | style="border-width:0; background:#FFFFFF;" colspan="8" | | style="border-width:0; background:#FFFFFF;" colspan="23"| |- | style="border-width:0; background:#FFFFFF;" | || style="background:#C4FCFF;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 1 || style="background:#9FFFAC;" | 0 || style="background:#9FFFAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 1 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 || style="background:#FFACAC;" | 0 |- style="font-size:9pt;" | style="border-width:0; background:#FFFFFF;" | || style="border-width:0; background:#FFFFFF;" | 31 || style="border-width:0; background:#FFFFFF;" | 30 || style="border-width:0; background:#FFFFFF;" colspan="6" | || style="border-width:0; background:#FFFFFF;" | 23 || style="border-width:0; background:#FFFFFF;" | 22 || style="border-width:0; background:#FFFFFF;" colspan="21" | || style="border-width:0; background:#FFFFFF;" | 0 |} E min = 01 H −7F H = −126 E max = FE H −7F H = 127 Exponent bias = 7F H = 127 Thus, in order to get the true exponent as defined by the offset-binary representation, the offset of 127 has to be subtracted from the value of the exponent field. The minimum and maximum values of the exponent field (00 H and FF H ) are interpreted specially, like in the IEEE 754 standard formats. The minimum positive normal value is 2 −126 ≈ 1.18 × 10 −38 and the minimum positive (subnormal) value is 2 −126−7 = 2 −133 ≈ 9.2 × 10 −41 . Just as in IEEE 754, positive and negative infinity are represented with their corresponding sign bits, all 8 exponent bits set (FF hex ) and all significand bits zero. Explicitly, val s_exponent_signcnd +inf = 0_11111111_0000000 -inf = 1_11111111_0000000 Just as in IEEE 754, NaN values are represented with either sign bit, all 8 exponent bits set (FF hex ) and not all significand bits zero. Explicitly, val s_exponent_signcnd +NaN = 0_11111111_klmnopq -NaN = 1_11111111_klmnopq where at least one of k, l, m, n, o, p, or q is 1. As with IEEE 754, NaN values can be quiet or signaling, although there are no known uses of signaling bfloat16 NaNs. 3f80 = 0 01111111 0000000 = 1 c000 = 1 10000000 0000000 = −2 7f7f = 0 11111110 1111111 = (2 8 − 1) × 2 −7 × 2 127 ≈ 3.38953139 × 10 38 (max finite positive value in bfloat16 precision) 0080 = 0 00000001 0000000 = 2 −126 ≈ 1.175494351 × 10 −38 (min normalized positive value in bfloat16 precision and single-precision floating point) The maximum positive finite value of a normal bfloat16 number is 3.38953139 × 10 38 , slightly below (2 24 − 1) × 2 −23 × 2 127 = 3.402823466 × 10 38 , the max finite positive value representable in single precision. 0000 = 0 00000000 0000000 = 0 8000 = 1 00000000 0000000 = −0 7f80 = 0 11111111 0000000 = infinity ff80 = 1 11111111 0000000 = −infinity 4049 = 0 10000000 1001001 = 3.140625 ≈ π ( pi ) 3eab = 0 01111101 0101011 = 0.333984375 ≈ 1/3 ffc1 = x 11111111 1000001 => qNaN ff81 = x 11111111 0000001 => sNaN
References
Text derived from https://en.wikipedia.org/w/index.php?oldid=1370000678 under CC BY-SA 4.0; modifications have been made.
References
- ↩Wikipedia “Bfloat16 floating-point format” release date (×1) · confidence 0.95 revision URL