Abstract

Despite the central role of test suites in the software development process, there is surprisingly limited information on how code and tests co-evolve to exercise different parts of the codebase.

Ten years ago, the Covrig project examined the code, test, and coverage evolution in six mature open-source projects, spanning a combined development time of twelve years. In this study, we significantly expand the analysis to nine mature projects and a combined period of 78 years of development time. Our focus is on understanding how development practices have changed and how these changes have impacted the way in which software is tested.

We report on the co-evolution of code and tests; the adoption of CI, coverage, and fuzzing services; the changes to the overall code coverage achieved by developer test suites; the distribution of patch coverage across revisions; how different kinds of code changes impact coverage; and the occurrence and evolution of flaky tests.

Our large-scale study paints a mixed picture in terms of how software development and testing have changed over the past ten years. While developers put more emphasis on software testing and the overall code coverage achieved by developer test suites has increased in most projects, coverage and fuzzing services are not widely adopted, many patches are still poorly tested, and the fraction of flaky tests has increased.