Output Diversity as a Test Selection Criterion

The uniqueness (or otherwise) of test outputs ought to have a bearing on test effectiveness, yet it has not previously been studied. We investigate the whitebox coverage and fault detection achieved by Output Uniqueness, a newly proposed blackbox test criterion, using 6 web applications. We find that output uniqueness exhibits average correlation coefficients of 0.85, 0.83 and 0.97 with statement, branch and path coverage respectively. More interestingly, output uniqueness finds 92% of the real faults found by branch coverage (and a further 47% that remained undetected by such whitebox techniques). These results suggest that output uniqueness may provide a useful surrogate when whitebox techniques are inapplicable and an effective complement where they are.

Nadia Alshahwan is a research associate at University College London (UCL) working on semantic malware detection and classification with Dr David Clark on the SeMaMatch project funded by EPSRC and GCHQ. Nadia received her PhD from UCL in 2012 under the supervision of Professor Mark Harman. The subject of her thesis is automated testing of server-side web application code. Nadia also has six years of industry experience as a developer, tester and system administrator.