Software quality is measured through a mixture of test outcomes, defect leakage, automation, delivery performance, staffing, and customer-facing failures. In the 2024 TestRail survey, 70% of QA teams tracked test pass/fail rate, while 60% tracked production defects. Other 2024 surveys show strong interest in automated testing and AI, but also continuing challenges with coverage, environments, localization, and payment flows.
Contents
- Software quality metrics and KPIs
- Test automation and coverage
- Release speed, defects, and delivery reliability
- QA processes, staffing, and quality challenges
- AI adoption and software testing quality
- Digital quality failures and user experience
Software quality metrics and KPIs
The most common quality indicators in the Software Testing & Quality Report (4e) were close to the testing workflow itself. In its 2024 survey, 70% of QA teams tracked test pass/fail rate. This was followed by production defects, tracked by 60%, and testing progress—such as passed, failed, or blocked tests—tracked by 56%.
Half of the measured activity related to test volume and automation. A total of 51% of QA teams tracked the number of tests executed, 43% measured automated test creation, and 42% measured automated test execution. These figures describe what teams reported tracking; they do not establish that one metric is universally better than another or that the measures were defined identically across organizations.
The State of Software Quality Report 2024 provides a complementary view based on perceived effectiveness. In the Katalon survey, 69% of respondents rated automated integration or system testing as an effective QA practice, while 66% rated automated unit testing as effective in 2024. The two findings are survey ratings, not measurements of defect rates or proof that automation alone caused higher quality.
| Quality measure or practice | Reported share | Measurement period |
|---|---|---|
| Test pass/fail rate tracked | 70% | TestRail, 2024 survey |
| Production defects tracked | 60% | TestRail, 2024 survey |
| Testing progress tracked | 56% | TestRail, 2024 survey |
| Automated test creation measured | 43% | TestRail, 2024 survey |
| Automated test execution measured | 42% | TestRail, 2024 survey |
| Automated integration or system testing rated effective | 69% | Katalon, 2024 survey |
Together, the results suggest that software quality dashboards commonly combine immediate test status with post-release evidence. A team that reports only pass rates may miss defects found in production; a team that reports only production defects may lack visibility into the speed and completeness of testing before release.
Test automation and coverage
Automation was concentrated in several established testing layers in the TestRail 2024 survey. Regression testing was automated by 56% of QA teams, functional testing by 52%, end-to-end testing by 47%, unit testing by 42%, and smoke testing by 40%. Performance testing was automated by 30%.
These percentages should be read as separate shares of surveyed teams. They are not additive, because one team can automate multiple types of testing. The pattern also distinguishes the presence of automation from its scale: automating regression tests does not indicate how many tests run, how often they run, or how stable their results are.
The reported daily test volumes varied substantially:
- 41% of QA teams ran fewer than 100 automated tests per day.
- 33% ran 101 to 1,000 automated tests per day.
- 18% ran 1,001 to 10,000 automated tests per day.
- Just over 8% ran more than 10,000 automated tests per day.
- 3% ran more than 100,000 automated tests per day.
The volume categories overlap conceptually with different operating models, but they should not be treated as a direct quality ranking. A large suite can still have weak coverage or unstable tests, while a smaller suite may target critical paths effectively. The survey figures quantify execution volume, not the business importance or accuracy of each test.
The gap between adoption and effectiveness is important for interpreting automation statistics. Katalon respondents gave automated integration or system testing the highest effectiveness rating among the two practices reported here, at 69%, compared with 66% for automated unit testing. TestRail’s findings show that automation spans multiple layers, but the data does not provide a single overall automation coverage percentage.
Release speed, defects, and delivery reliability
Rapid delivery creates a quality-management tradeoff in the TestRail findings. In the 2024 survey, 58% of QA teams said rapid releases lead to defects slipping into production. At the same time, among teams with strong automation and CI/CD integration, 86% reported faster release cycles and 71% reported reduced defect leakage.
The comparison is descriptive rather than causal. The figures do not prove that strong automation and CI/CD integration alone produced faster releases or less leakage. They do show that the teams reporting those capabilities also reported these outcomes, while rapid release pressure remained associated with production defects for many QA teams.
Release cadence was distributed across several patterns. The TestRail report found that 15% of organizations deployed daily or more than once daily, 19% released on a slower bi-monthly or quarterly schedule, and 11% followed a flexible or ad hoc cadence. Daily deployments increased from 6% in the 2023 survey to 9% in the 2024 survey among TestRail respondents. The 15% figure uses a broader daily-or-more-than-daily category, so it should not be compared as though it were the same measure as the 9% daily-deployment figure.
Team structure was also connected with delivery practice: 45% of TestRail respondents said CI/CD adoption improves with QA team size in the 2024 survey. This is a respondent-reported relationship, not a universal staffing rule. Release frequency, automation depth, integration quality, and organizational scale can all describe different parts of delivery reliability.
QA processes, staffing, and quality challenges
Overall process sentiment was relatively positive in the TestRail 2024 survey: 78% of respondents were satisfied or very satisfied with their current testing process. Among those satisfied teams, 90% made test automation a top priority, 85% focused on increasing test coverage, and 82% involved QA earlier in the software lifecycle.
Efficiency and prevention were equally prominent priorities among satisfied teams. The report says 88% treated efficiency as a core testing objective, and another 88% emphasized preventing bugs before production. These percentages describe the satisfied subset, not all TestRail respondents, and they should not be combined with the overall satisfaction rate as if they represented a single population total.
Staffing remained a constraint. Only 51% of TestRail respondents said their QA teams were appropriately staffed for current testing needs in 2024. Conversely, 47% said their team was somewhat or severely understaffed. Within the understaffed respondents, 38% reported moderate resource gaps and 9% reported severe understaffing. Those subgroup percentages have a different denominator from the overall staffing figures.
The leading reported process challenges were practical and technically connected. End-to-end testing complexity was identified as the biggest QA challenge by 33% of TestRail teams, while 32% struggled with developing automated tests. Separately, 35% prioritized better test data and environment stability. The results indicate that quality work depends not only on writing tests, but also on dependable data, environments, and cross-system workflows.
AI adoption and software testing quality
AI was a major quality-engineering objective in the State of Software Quality Report 2024. In the Katalon survey, 47% of participants named integrating AI into QA processes as their main QA objective in 2024. Among managers and senior managers, 58% selected AI integration into QA as a key goal for the coming years. The most anticipated AI use reported by 52% of Katalon respondents was generating manual-test cases.
The TestRail 2024 survey shows a more mixed picture of current impact. Thirty-six percent of respondents said AI had minimal impact on their QA role, 26% said it required them to upskill or learn new tools, and 14% expressed concerns about job stability. At the same time, 20% said AI enabled advanced testing such as predictive analytics or anomaly detection.
Integration barriers were also quantified. Data privacy and security were cited as an obstacle by 34% of TestRail respondents, while 30% cited a lack of in-team expertise. These findings identify adoption concerns, not a measurement of AI-generated test accuracy or a forecast of workforce reduction.
The 2024 Global DevSecOps Report places QA-related AI adoption within broader software development. GitLab reported that 78% of respondents were using AI in software development or planned to within two years in 2024, up from 64% in 2023. It also reported that 67% said their software development lifecycle was mostly or completely automated in 2024. GitLab’s broader development population and measures should be kept distinct from the TestRail and Katalon QA-specific findings.
Digital quality failures and user experience
Quality failures are not limited to test dashboards. Applause’s State of Digital Quality 2024 combines survey responses with testing data collected in 2023. In its localization testing dataset, poor translation represented 47.1% of 5,226 localization issues. Missing translations represented 24.5%, truncation and overlap 5.8%, and currency and number-format issues 2.6%.
The consumer consequences were measured separately in a 2024 Applause survey of 2,402 consumers who had abandoned an app or site over localization problems. Incorrect words or syntax were cited by 47.8%, misspellings by 37.7%, unfamiliar units of measure or currency by 38.8%, and culturally insensitive language or imagery by 32.8%. These percentages use the subgroup of consumers who reported abandoning an app or site because of localization problems; they are not estimates for all digital users.
Payment testing also showed a difference between transaction types. In Applause’s 2024 payment survey, 76.7% of organizations tested purchase transactions, while 39.7% tested returns. The figures do not establish that untested returns caused customer losses, but they show that purchase and post-purchase flows were not tested by the same reported share of organizations.
For software quality programs, these findings widen the KPI set beyond unit and regression results. Localization defects, accessibility or presentation failures, and return-flow gaps can affect whether a user completes a task even when core application tests pass. Because Applause’s localization issue data was collected in 2023 and reported in 2024, its testing period should be distinguished from the 2024 consumer and payment survey periods.