An empirical study and the underlying infrastructure for analyzing code, test and coverage evolution in real software using both static and dynamic metrics.
Overview
Software repositories provide rich information about the construction and evolution of software systems. While static data that can be mined directly from version control systems has been extensively studied, dynamic metrics concerning the execution of the software have received much less attention, due to the inherent difficulty of running and monitoring a large number of software versions.
We present Covrig, a flexible infrastructure that can be used to run each version of a system in isolation and collect static and dynamic software metrics, using a lightweight virtual machine environment that can be deployed on a cluster of local or cloud machines.
We use Covrig to conduct an empirical study examining how code and tests co-evolve in 6 popular open-source systems. We report the main characteristics of software patches, analyse the evolution of program and patch coverage, assess the impact of nondeterminism on the execution of the test suite, and investigate whether the coverage of code containing bugs and bug fixes is higher than average.
Download
The raw data for the six projects analysed is available. As it is large, please email Cristian Cadar for the link.
Research Support
This research project was generously sponsored by the UK EPSRC through a DTA studentship and the grant EP/J00636X/1, and by Google through a European Doctoral Fellowship.
Publications
-
Covrig: A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software
Paul Dan Marinescu, Petr Hosek, Cristian Cadar
International Symposium on Software Testing and Analysis (ISSTA 2014)
Talks
-
Covrig: A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software
Conference talk @ International Symposium on Software Testing and Analysis (ISSTA 2014)