An empirical study and the underlying infrastructure for analyzing code, test and coverage evolution in real software using both static and dynamic metrics.

Overview

Software repositories provide rich information about the construction and evolution of software systems. While static data that can be mined directly from version control systems has been extensively studied, dynamic metrics concerning the execution of the software have received much less attention, due to the inherent difficulty of running and monitoring a large number of software versions.

We present Covrig, a flexible infrastructure that can be used to run each version of a system in isolation and collect static and dynamic software metrics, using a lightweight virtual machine environment that can be deployed on a cluster of local or cloud machines.

We use Covrig to conduct an empirical study examining how code and tests co-evolve in 6 popular open-source systems. We report the main characteristics of software patches, analyse the evolution of program and patch coverage, assess the impact of nondeterminism on the execution of the test suite, and investigate whether the coverage of code containing bugs and bug fixes is higher than average.

Download

The raw data for the six projects analysed is available for download.

A presentation of our work is available in ppsx format

Research Support

This research project is generously sponsored by the UK EPSRC through a DTA studentship and the grant EP/J00636X/1, and by Google through a European Doctoral Fellowship.

Publications

  • Covrig: A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software

    Paul Dan Marinescu, Petr Hosek, Cristian Cadar

    International Symposium on Software Testing and Analysis (ISSTA 2014)