Flaky test

Uses your system voice; voice and rate in settings.

Snapshot

≈ 3 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

Flaky test (or flaky test case) is a software test that exhibits non-deterministic behavior, i.e., it may pass or fail inconsistently without any changes in the underlying code. This flakiness can arise from a number of causes, including concurrency issues, timing dependencies, reliance on external systems, or lack of sufficient isolation between tests. Flaky tests are problematic in continuous integration environments because they decrease trust in automated test suites and can hide real defects, leading to wasted debugging effort and decreased productivity. Flaky tests can be mitigated by improving test isolation, controlling sources of nondeterminism (e.g., time and randomness), mocking of external dependencies, and rerunning of tests to confirm failures. Flakiness is a well-known issue in large-scale software development, and there are many techniques to tackle it, such as test retries, quarantine mechanisms, or better test design practices. Nevertheless, flaky tests remain an active research area in software engineering, especially for distributed systems and large code bases where nondeterminism is more prevalent. Research has also investigated automated detection of flaky tests using machine-learning techniques.…

Flaky test (or flaky test case) is a software test that exhibits non-deterministic behavior, i.e., it may pass or fail inconsistently without any changes in the underlying code. This flakiness can arise from a number of causes, including concurrency issues, timing dependencies, reliance on external systems, or lack of sufficient isolation between tests. Flaky tests are problematic in continuous integration environments because they decrease trust in automated test suites and can hide real defects, leading to wasted debugging effort and decreased productivity. Flaky tests can be mitigated by improving test isolation, controlling sources of nondeterminism (e.g., time and randomness), mocking of external dependencies, and rerunning of tests to confirm failures. Flakiness is a well-known issue in large-scale software development, and there are many techniques to tackle it, such as test retries, quarantine mechanisms, or better test design practices. Nevertheless, flaky tests remain an active research area in software engineering, especially for distributed systems and large code bases where nondeterminism is more prevalent. Research has also investigated automated detection of flaky tests using machine-learning techniques. One study evaluated machine-learning models for detecting both order-dependent and non-order-dependent flaky tests using test suites from 26 Python projects containing more than 67,000 tests.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1376046395 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

1 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.