Test Case Prioritization: A Family of Empirical Studies
S. Elbaum, A. Malishevsky, and G. Rothermel
IEEE Transactions on Software Engineering,
Volume 28, no. 2, February, 2002, pages 159-182
To reduce the cost of regression testing, software testers may
prioritize their test cases so that those which are more important,
by some measure, are run earlier in the regression testing process.
One potential goal of such prioritization is to increase
a test suite's rate of fault detection.
Previous work reported the results of studies that
showed that prioritization techniques can
significantly improve rate of fault detection.
Those studies, however, raised several additional questions:
(1) can prioritization techniques be effective when targeted at
specific modified versions; (2) what tradeoffs exist between fine
granularity and coarse granularity prioritization techniques; (3)
can the incorporation of measures of fault proneness into
prioritization techniques improve their effectiveness? To address
these questions, we have performed several new studies, in which
we empirically compared prioritization techniques using both
controlled experiments and case studies. The results of these
studies show that each of the prioritization techniques considered
can improve the rate of fault detection
of test suites overall. Fine-granularity techniques
typically outperformed coarse-granularity techniques, but
only by a relatively small margin overall; in other words,
the relative imprecision in coarse-granularity analysis
did not dramatically reduce the techniques' ability
to improve rate of fault detection.
Incorporation of fault-proneness techniques
produced relatively small improvements
over other techniques in terms rate of fault detection,
a result which ran contrary to our expectations.
Our studies also show that the relative effectiveness
of various techniques can vary significantly across
target programs. Furthermore, our analysis shows that whether the
effectiveness differences observed will result in savings in
practice varies substantially with the cost factors associated
with particular testing processes. Further work to understand
the sources of this variance, and to incorporate such understanding into
prioritization techniques and the choice of techniques, would be beneficial.