Mobile app testing spans a large and changing technical surface. Applause’s 2023 digital-quality dataset covered more than 748,000 test runs across mobile apps and mobile web in real-world scenarios, while Kobiton’s October 2023 survey shows how frequently teams release and how long their test cycles take. Together, these sources describe testing shaped by device fragmentation, delivery pressure, uneven automation, and growing interest in AI-assisted quality work.
Contents
- Coverage and device fragmentation
- Release cadence and delivery constraints
- Manual and automated execution time
- Automation coverage and business impact
- How teams expand automation
- Barriers and maintenance effort
- AI, cloud, and geolocation priorities
Coverage and device fragmentation
The scale of a mobile test program is easier to understand when coverage is described in concrete dimensions. The State of Digital Quality 2024 reports on Applause’s 2023 test cycles, as reported in 2024. Its dataset included more than 748,000 test runs involving mobile apps and mobile web in real-world scenarios.
That dataset covered 2,102 mobile models, 17 mobile operating systems, and 420 mobile operating-system versions. It also included 37 mobile web browsers, 527 mobile carriers, and 126 mobile makes. These figures are coverage counts for the Applause dataset, not a complete count of every device or software configuration in the market.
Payment flows add another layer of variation. Applause’s payment-testing data included 87 mobile wallets. A team validating login, checkout, notifications, responsive layouts, and payment behavior therefore faces several overlapping matrices rather than a single device list.
| Coverage dimension | Applause dataset measure | Period or scope |
|---|---|---|
| Test runs | More than 748,000 | 2023 test cycles; reported 2024 |
| Mobile models | 2,102 | 2023 test cycles; reported 2024 |
| Mobile operating systems | 17 | 2023 test cycles; reported 2024 |
| Operating-system versions | 420 | 2023 test cycles; reported 2024 |
| Mobile web browsers | 37 | 2023 test cycles; reported 2024 |
| Mobile carriers | 527 | 2023 test cycles; reported 2024 |
| Mobile makes | 126 | 2023 test cycles; reported 2024 |
| Mobile wallets in payment testing | 87 | 2023 test cycles; reported 2024 |
Release cadence and delivery constraints
Kobiton surveyed mobile-app development and testing professionals from October 9–24, 2023. In that survey, 38% said their organizations released mobile-app updates weekly and 20% said they released updates daily. Another 27% reported monthly releases, while 14% released quarterly and 1% less than once per quarter. The results indicate that most respondents worked with a release rhythm at least as frequent as monthly, but the survey categories should not be treated as a universal industry distribution.
Respondents also identified several factors slowing mobile-app delivery. Limited financial resources in development and QA was selected by 50%, inefficient development and QA processes by 47%, and a lack of skilled development and QA labor by 40%. About one-third estimated that slow release times cost their companies more than $500,000 annually. That is a respondent estimate, not a measured total across all companies.
The same survey asked about the effect of test automation on time to market. Twenty-five percent said automation reduced mobile-app time to market by less than 25%, and 37% reported a reduction of 25%–50%. A separate 18% reported a reduction of more than 50%. These are survey responses about perceived or reported impact; they do not establish a causal benchmark for every team.
Manual and automated execution time
Manual regression work can consume a substantial part of a release window. Kobiton respondents reported the following time required to execute manual mobile test cases before each release:
- 8% said up to 1 day.
- 31% said 1–2 days.
- 40% said 3–5 days.
- 12% said 6–10 days.
- 9% said more than 10 days.
The survey also measured automated mobile tests for daily builds. Eleven percent said those tests took up to 1 hour, 28% said 1–3 hours, and 32% said 3–6 hours. A further 21% reported 6–9 hours, while 8% said 10 hours or more. These figures describe the respondents’ reported execution times and do not specify that every test ran serially or on the same device set.
The comparison is useful because automation does not mean that a test suite is instantaneous. Automated execution can still occupy a large build window, particularly when a suite spans many mobile configurations. At the same time, manual execution introduces a different constraint: the time before each release depends on people completing repeatable checks, recording results, and handling the devices and environments in scope.
Automation coverage and business impact
Kobiton found that 47% of respondents executed at least 100 manual mobile test cases with each app release. The size of that recurring manual workload helps explain why teams consider automation coverage alongside execution time and delivery speed.
Current and ideal coverage were not the same. Twenty-seven percent said they currently automated less than 10% of their mobile test cases. When asked about an ideal level, 28% selected 10%–24%, 20% selected 50%–75%, and 17% selected more than 75%. The survey does not provide a single ideal percentage for all organizations; the responses show a range of ambitions.
Reported time-to-market effects also varied. Twenty-five percent said automation reduced time to market by less than 25%, 37% said it reduced time to market by 25%–50%, and 18% said the reduction was more than 50%. The figures are best read as an indication of differing outcomes across surveyed organizations, not as a forecast for a proposed automation project.
For planning purposes, coverage should be tied to risk. A team may prioritize smoke tests, authentication, payments, deep links, offline behavior, or high-value device and operating-system combinations before attempting broad end-to-end automation. The statistics show why coverage and business impact are connected, but they do not identify which test cases any particular organization should automate first.
How teams expand automation
The Kobiton survey recorded several approaches teams used or planned to use to expand mobile test automation. Team training was selected by 45% of respondents, while 34% planned to hire more automation engineers. Low-code or no-code automation tools were selected by 30%.
Script-generation practices varied. Forty-three percent said they built scripts from scratch with native iOS and Android frameworks. Thirty percent built automation scripts from scratch in Appium, and 23% used low-code tools that generate Appium scripts. Fifty-three percent said they used generative AI to generate automation scripts.
These categories can overlap: a respondent could use more than one framework or strategy. The results therefore describe a mix of practices rather than mutually exclusive market shares. They also show that expanding automation is not only a tooling decision. Training, staffing, framework choice, and script-generation methods all appeared in the survey responses.
Barriers and maintenance effort
Instability was the most frequently selected top barrier in the Kobiton survey: 28% identified test instability, while 24% identified code complexity. Other top barriers included release repair at 20%, the gap between automation hype and reality at 13%, and device repair at 12%. Three percent selected not testing as their top mobile-automation barrier.
Maintenance remains part of the automation workload. Fifteen percent of respondents said updating mobile-automation scripts for a new release took 1 day or less. That figure does not describe the time reported by the remaining respondents, so it should not be expanded into an average or used to estimate a typical maintenance cycle.
The barrier results give context to the execution-time figures. A suite may run automatically for several hours and still require engineering work when tests are unstable, application behavior changes, or devices and releases need repair. The survey’s wording also matters: the percentages identify respondents’ top barriers, not the total amount of time each barrier consumed.
AI, cloud, and geolocation priorities
The World Quality Report 15th Edition 2023–24 surveyed quality professionals about automation priorities for the following 12 months. Testing AI was selected as a top-three automation focus by 55% of respondents, mobile by 40%, and security by 36%. These are forward-looking priorities reported in the 2023–24 survey period, not completed outcomes.
Among the report’s mobile-testing respondents, 76% said they used AI- and data-science-based approaches to make mobile application testing more effective. Forty-one percent said they included mobile application testing in DevOps toolchains. The report also recorded 35% using geolocation testing for enhanced global performance, 28% integrating cloud-based mobile device labs, and 20% pursuing higher mobile test automation.
The figures point to several distinct directions. AI and data-science approaches relate to test effectiveness; DevOps integration relates to where mobile testing fits in delivery workflows; geolocation testing addresses global performance; and cloud-based device labs address access to device environments. The supplied World Quality Report figures do not quantify defect reduction, return on investment, or the accuracy of AI-generated tests.
For readers comparing mobile testing programs, the most useful baseline is therefore multidimensional: record the device, operating-system, browser, carrier, and wallet coverage in scope; measure manual and automated execution windows; document current automation coverage; and track the maintenance burden created by instability and release changes. Those measures keep the reported survey percentages and dataset counts tied to the operational questions they can actually inform.