An empirical study and the underlying infrastructure for analyzing code, test and coverage evolution in real software using both static and dynamic metrics.
Software repositories provide rich information about the construction and evolution of software systems. While static data that can be mined directly from version control systems has been extensively studied, dynamic metrics concerning the execution of the software have received much less attention, due to the inherent difficulty of running and monitoring a large number of software versions.
We present Covrig, a flexible infrastructure that can be used to run each version of a system in isolation and collect static and dynamic software metrics, using a lightweight virtual machine environment that can be deployed on a cluster of local or cloud machines.
We use Covrig to conduct an empirical study examining how code and tests co-evolve in 6 popular open-source systems. We report the main characteristics of software patches, analyse the evolution of program and patch coverage, assess the impact of nondeterminism on the execution of the test suite, and investigate whether the coverage of code containing bugs and bug fixes is higher than average.
The raw data for the six projects analysed is available for download.
A presentation of our work is available in ppsx format
This research project is generously sponsored by the UK EPSRC through a DTA studentship and the grant EP/J00636X/1, and by Google through a European Doctoral Fellowship.