Experience Report: Automated System Level Regression Test Prioritization Using Multiple Factors

EasyChair Preprint no. 72

Abstract

We propose a new method of determining an effective ordering of regression test cases, and describe its implementation as an automated tool called Suite Builder developed by Westermo. The tool generates an efficient order to run the cases in existing test suites by combining priorities of multiple factors associated with test cases, including previous fault detection success, interval since last executed, expected or observed test duration, and modifications to the code tested. The method and tool were developed to address problems in the traditional process of regression testing, such as lack of time to complete a regression suite, failure to detect bugs in time, and tests that are repeatedly omitted. The tool has been integrated into the existing nightly test framework for Westermo software that runs on large-scale data communication systems. In experimental evaluation of the tool, we found significant improvement in regression testing results. The re-ordered test suites finish within the available time, the majority of fault-detecting test cases are located in the first third of the suite, no important test case is omitted, and the necessity for manually reconfiguring the nightly testing is greatly reduced.

Keyphrases: nightly testing, regression testing, software testing, test case prioritization, test selection

@Booklet{EasyChair:72,
  author = {Per Erik Strandberg and Daniel Sundmark and Wasif Afzal and Thomas J. Ostrand and Elaine J. Weyuker},
  title = {Experience Report: Automated System Level Regression Test Prioritization Using Multiple Factors},
  howpublished = {EasyChair Preprint no. 72},
  doi = {10.29007/nr1s},
  year = {EasyChair, 2018}}