Continuous testing is becoming part of mainstream software delivery, but adoption does not mean every team has fast, reliable feedback. In 2024, 92% of surveyed testing organizations used Agile or an Agile-like model, while only 2% said automation had replaced all manual testing. Sauce Labs benchmark data from January through June 2024 adds a practical view: the average virtual-device-cloud pass rate was 70%, and the average test runtime was 92 seconds.
Table of contents
- Adoption and automation depth
- CI/CD participation and test readiness
- Pass rates and quality benchmarks
- Runtime and feedback speed
- Platform coverage and testing tools
- Outcomes, metrics, and AI
Adoption and automation depth
The 2024 State of Testing survey shows that iterative delivery practices are common among testing organizations. Agile or an Agile-like model was used by 92% of surveyed organizations in 2024. DevOps was reported by 45%, while 26% used Behavior-Driven Development and 23% used Test-Driven Development. These figures describe reported practices among survey respondents; they are not a measure of every software organization worldwide. (STATE OF TESTING™ 2024)
Automation coverage was uneven. Only 2% of respondents said automation had replaced all of their previously manual testing efforts. A further 19% said automation had replaced about 75% or more of manual testing, while 30% reported replacement of about 50%. At the lower end, 22% said automation had replaced about 25% or less of their previous manual testing. Another 9% reported that their organization did not have test automation in 2024. (STATE OF TESTING™ 2024)
| Reported automation replacement | Share of respondents, 2024 |
|---|---|
| All manual testing | 2% |
| About 75% or more | 19% |
| About 50% | 30% |
| About 25% or less | 22% |
| No test automation | 9% |
The distribution suggests that continuous testing maturity is better understood as a spectrum than as a simple adopted/not-adopted label. A team may use Agile and automated checks while still relying substantially on manual testing. The survey does not establish why organizations fell into each category, so these percentages should be used to describe reported automation depth rather than to infer causes.
CI/CD participation and test readiness
Testing involvement in CI/CD varied considerably in the 2024 State of Testing results. Forty-eight percent of respondents said testers were active in defining and maintaining their CI/CD process. Fifteen percent said testers had access to CI/CD reports and used them as feedback for testing. In contrast, 20% knew their organization had CI/CD but said testers were not part of the process, and 10% said their organization did not deploy CI/CD or that testers were not part of it. (STATE OF TESTING™ 2024)
Test-case readiness was also mixed. Forty percent of respondents described their test cases as well-written and well-maintained. Another 27% said their test cases were well-written but not well-maintained. Eighteen percent tested ad hoc because they did not have organized test cases, and 15% reported many duplications in their test cases. These measures point to two different requirements for continuous testing: integration into the delivery pipeline and test assets that remain usable as the product changes. (STATE OF TESTING™ 2024)
The reported figures also caution against treating a CI/CD label as proof of a mature testing feedback loop. A pipeline can exist while testers remain outside its design, maintenance, or reporting flow. Similarly, automated execution can be constrained when cases are duplicated, unorganized, or not maintained.
Pass rates and quality benchmarks
Sauce Labs defined continuous-testing excellence in its January–June 2024 benchmark as an average test pass rate of at least 85% across desktop and mobile tests. Only 38% of organizations passed its virtual-device-cloud desktop test-quality benchmark, and 37% passed the virtual-device-cloud mobile benchmark. The average virtual-device-cloud test pass rate was 70%, below the 85% excellence threshold. (The Continuous Testing Benchmark Report)
The industry comparison in the same report found that 41% of software-industry companies passed the virtual-device-cloud test-quality benchmark, the highest rate among the listed industries. Real-device-cloud performance was lower: 26% of organizations passed the real-device-cloud test-quality benchmark, and the average real-device-cloud mobile test pass rate was 65%. The virtual-device and real-device results are separate benchmark views and should not be combined into a single pass-rate estimate. (The Continuous Testing Benchmark Report)
The State of Testing survey measured which quality indicators teams used, rather than reporting one universal pass-rate result. Defect metrics were measured by 50% of surveyed teams in 2024. Test-coverage metrics were measured by 48%, test-execution metrics by 43%, and escaped-defect rate by 27%. (STATE OF TESTING™ 2024)
Together, the sources separate three ideas that are often blended: whether tests pass, which quality signals teams measure, and whether results arrive quickly enough to influence development. A high test count or coverage measure alone does not establish the Sauce Labs pass-rate benchmark, and the benchmark does not describe every quality practice used by survey respondents.
Runtime and feedback speed
Sauce Labs defined runtime excellence as average test runs of two minutes or less in its January–June 2024 benchmark. Seventy-seven percent of organizations completed virtual-device-cloud desktop tests in an average of two minutes or less, compared with 71% for virtual-device-cloud mobile tests. Only 24% completed real-device-cloud mobile tests within the same two-minute threshold. The average test runtime across the benchmark data was 92 seconds. (The Continuous Testing Benchmark Report)
Sauce Labs reported that tests completing in two minutes or less were nearly two times as likely to pass as longer tests. This is a reported benchmark relationship, not proof that shortening any individual test will cause it to pass. The report also found that 78% of organizations in regulation-heavy industries passed the two-minute virtual-device-cloud runtime benchmark. By contrast, only 24% passed the real-device-cloud runtime benchmark, where average runtime was around six minutes. (The Continuous Testing Benchmark Report)
| Sauce Labs runtime benchmark | Result, January–June 2024 |
|---|---|
| Virtual-device-cloud desktop tests at two minutes or less | 77% |
| Virtual-device-cloud mobile tests at two minutes or less | 71% |
| Real-device-cloud mobile tests at two minutes or less | 24% |
| Average test runtime | 92 seconds |
| Real-device-cloud average runtime | Around 6 minutes |
For teams designing a continuous-testing feedback loop, these results make device type a material consideration. Virtual-device-cloud tests met the two-minute threshold more often than real-device-cloud mobile tests in this benchmark. The figures do not indicate that one approach is universally preferable; they show that the speed profile differed across the tested environments.
Platform coverage and testing tools
Sauce Labs defined platform-coverage excellence as testing against at least five platforms on average. Seventy-two percent of organizations passed the virtual-device-cloud benchmark of five or more platforms, and the average number of virtual-device-cloud platforms tested was 7.67. For real mobile devices, 79% of organizations reported testing on 10 or more devices, while 59.5% reported testing on 30 or more real mobile devices. (The Continuous Testing Benchmark Report)
The report’s 2024 case study on Walmart described testing its apps on more than 300 unique real devices using Sauce Labs. This is a case-study figure for Walmart, not an average across organizations. It illustrates the scale that device diversity can reach in a large application context without establishing a universal coverage target. (The Continuous Testing Benchmark Report)
The State of Testing survey provides a separate view of tools in testing-related technology stacks. Jira was reported by 42% of surveyed testing organizations in 2024, Selenium by 22%, and Postman by 18%. These are reported tool-use shares, not measures of tool quality, test volume, or successful integration into CI/CD. (STATE OF TESTING™ 2024)
Coverage therefore has at least two dimensions in these sources: the number of platforms or devices exercised and the tools used to organize or execute testing. Neither source supplies a single formula that combines device breadth, test pass rate, runtime, and tool adoption into one maturity score.
Outcomes, metrics, and AI
In the 2024 State of Testing survey, 81% of respondents said testing had a say in release-readiness decisions. Respondents also reported perceived effects of adopting iterative models. Forty-nine percent agreed that iterative models increased feature and functionality releases, including 25% who strongly agreed. Forty-nine percent agreed that iterative models improved overall testing levels, including 24% who strongly agreed. (STATE OF TESTING™ 2024)
Forty-five percent agreed that adopting iterative models reduced serious bugs escaping into production, and 23% strongly agreed. These are survey responses about perceived outcomes, not measured production-defect reductions supplied by the report. The distinction matters when comparing opinion-based outcome measures with Sauce Labs’ benchmark observations about pass rates and runtimes. (STATE OF TESTING™ 2024)
AI use remained limited relative to non-use. Sixty percent of respondents said they were not using AI tools for testing in 2024. Among the listed uses, 25% used AI for test-case creation, 23% for test-case optimization, and 20% for test planning. These categories may overlap because respondents could report more than one use; the supplied survey results do not state that the percentages form a mutually exclusive distribution. (STATE OF TESTING™ 2024)
The combined picture is specific: continuous-testing practices were widely connected to Agile delivery, but deep automation, fast feedback, high pass rates, broad platform coverage, and AI use were not uniform. The most useful operational baseline is therefore multidimensional—track involvement in CI/CD, test-case maintenance, pass rate, runtime, escaped defects, coverage, and release-readiness influence separately, with the measurement period and testing environment kept explicit.