Evidence-based defect assessment and prediction for software product lines

Abstract

<p>The systematic reuse provided by software product lines provides</p> <p>opportunities to achieve increased quality and reliability as a product line</p> <p>matures. This has led to a widely accepted assumption that as a product line evolves, its reliability improves. However, evidence in terms of empirical investigation of the relationship among change, reuse and reliability in evolving software product lines is lacking.</p> <p>To address the problem this work investigates: 1) whether reliability as measured by post-deployment failures improves as the products and components in a software product line change over time, and 2)</p> <p>whether the stabilizing effect of shared artifacts enables accurate prediction of failure-prone files in the product line.</p> <p>The first part of this work performs defect assessment and investigates defect trends in Eclipse, an open-source software product line. It analyzes the evolution of the product line over time in terms of the total number of defects, the percentage of severe defects and the relationship between defects and changes.</p> <p>The second part of this work explores prediction of failure-prone files in the Eclipse product line to determine whether prediction improves as the</p> <p>product line evolves over time. In addition, this part investigates the effect</p> <p>of defect and data collection periods on the prediction performance.</p> <p>The main contributions of this work include findings that the majority of files with severe defects are reused files rather than new files, but that</p> <p>common components experience less change than variation components. The work also found that there is a consistent set of metrics which serve as prominent predictors across multiple products and reuse categories over time. Classification of post-release, failure-prone files using change data for the Eclipse product line gives better recall and false positive rates as compared to classification using static code metrics. The work also found that on-going change in product lines hinders the ability to predict failure-prone files, and that predicting post-release defects using pre-release change data for the Eclipse case study is difficult. For example, using more data from the past to predict future failure-prone files does not necessarily give better results than</p> <p>using data only from the recent past. The empirical investigation of product line change and defect data leads to an improved understanding of the interplay among change, reuse and reliability as a product line evolves.</p>