Precision (computer science)

Uses your system voice; voice and rate in settings.

Snapshot

≈ 3 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

In computer science, the precision of a numerical quantity is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to precision in mathematics, which describes the number of digits that are used to express a value. Some of the standardized floating-point precision formats are: * Half-precision floating-point format * Single-precision floating-point format * Double-precision floating-point format * Quadruple-precision floating-point format * Octuple-precision floating-point format Of these, octuple-precision format is rarely used. The names refer to the size of a computer word, commonly 32-bits on many current computer architectures. A single-precision value occupies one word, a half-precision half of a word, double-precision two words, etc. The single- and double-precision formats are most widely used and supported on nearly all platforms. The use of half-precision format and minifloat formats has been increasing especially in the field of machine learning since many machine learning algorithms are inherently error-tolerant.

In computer science, the precision of a numerical quantity is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to precision in mathematics, which describes the number of digits that are used to express a value. Some of the standardized floating-point precision formats are: Half-precision floating-point format Single-precision floating-point format Double-precision floating-point format Quadruple-precision floating-point format Octuple-precision floating-point format Of these, octuple-precision format is rarely used. The names refer to the size of a computer word, commonly 32-bits on many current computer architectures. A single-precision value occupies one word, a half-precision half of a word, double-precision two words, etc. The single- and double-precision formats are most widely used and supported on nearly all platforms. The use of half-precision format and minifloat formats has been increasing especially in the field of machine learning since many machine learning algorithms are inherently error-tolerant.

Rounding errors

Precision is often the source of rounding errors in computation. The number of bits used to store a number will often cause some loss of accuracy. An example would be to store "sin(0.1)" in IEEE single-precision floating point standard. The error is then often magnified as subsequent computations are made using the data (although it can also be reduced).

See also

Approximate computing Arbitrary-precision arithmetic Extended precision Granularity IEEE754 (IEEE floating point standard) Integer (computer science) Minifloat (8-bit and less) Significant figures Truncation

References

Text derived from https://en.wikipedia.org/w/index.php?oldid=1320793526 under CC BY-SA 4.0; modifications have been made.

Provenance

snapshot enwiki-001320793526 · captured 1762456885.0 · version history

Cite this page as: encyclopegia.org, “Precision (computer science)”, snapshot enwiki-001320793526. https://encyclopegia.org/a/precision-computer-science

Sources cited in this revision

Extracted from the cited revision's reference blocks; absent citations are not invented.