Combinatorial Interaction Testing Portal

Covering Arrays by Simulated Annealing (CASA)

Evaluation Artifacts: SSBSE2009, Table 2

Average Size and Times Over 50 Runs for Constrained Two-Way Problems
			Size		Time (s)		Break-even (s/cfg.)
Name	Model	Constraints	mAETG Best	All	mAETG Best	All
SPIN-S	2¹³4⁵	2¹³	27.0	19.4	0.2	8.6	1.11
SPIN-V	2⁴²3²4¹¹	2⁴⁷3²	42.5	36.8	11.3	102.1	15.93
GCC	2¹⁸⁹3¹⁰	2³⁷3³	24.7	21.1	204.0	1902.0	471.67
Apache	2¹⁵⁸3⁸4⁴5¹6¹	2³3¹4²5¹	42.6	32.3	76.4	109.1	3.17
Bugzilla	2⁴⁹3¹4²	2⁴3¹	21.8	16.2	1.9	9.1	1.29

1.	2⁸⁶3³4¹5⁵6²	2²⁰3³4¹	53.3	38.6	24.4	179.5	10.55
2.	2⁸⁶3³4³5¹6¹	2¹⁹3³	40.5	31.0	14.7	25.4	1.13
3.	2²⁷4²	2⁹3¹	20.8	18.0	0.2	3.4	1.14
4.	2⁵¹3⁴4²5¹	2¹⁵3²	28.6	21.0	3.1	29.3	3.45
5.	2¹⁵⁵3⁷4³5⁵6⁴	2³²3⁶4¹	63.8	47.7	134.8	655.9	32.37
6.	2⁷³4³6¹	2²⁶3⁴	34.0	24.2	7.2	18.2	1.12
7.	2²⁹3¹	2¹³3²	12.5	9.0	0.3	2.1	0.51
8.	2¹⁰⁹3²4²5³6³	2³²3⁴4¹	55.6	40.5	45.1	249.9	13.56
9.	2⁵⁷3¹4¹5¹6¹	2³⁰3⁷	26.0	20.0	3.0	29.8	4.47
10.	2¹³⁰3⁶4⁵5²6⁴	2⁴⁰3⁷	60.4	44.0	74.3	357.3	17.26
11.	2⁸⁴3⁴4²5²6⁴	2²⁸3⁴	58.3	41.9	23.8	240.7	13.23
12.	2¹³⁶3⁴4³5¹6³	2²³3⁴	54.5	40.4	68.0	221.1	10.86
13.	2¹²⁴3⁴4¹5²6²	2²²3⁴	48.6	36.5	45.5	60.5	1.24
14.	2⁸¹3⁵4³6³	2¹³3²	51.8	36.9	20.2	58.1	2.54
15.	2⁵⁰3⁴4¹5²6¹	2²⁰3²	40.4	30.7	4.9	19.3	1.48
16.	2⁸¹3³4²6¹	2³⁰3⁴	33.4	24.1	9.7	19.7	1.08
17.	2¹²⁸3³4²5¹6³	2²⁵3⁴	53.4	39.2	46.9	335.5	20.32
18.	2¹²⁷3²4⁴5⁶6²	2²³3⁴4¹	57.3	42.3	60.1	303.5	16.23
19.	2¹⁷²3⁹4⁹5³6⁴	2³⁸3⁵	64.7	47.8	168.4	823.6	38.77
20.	2¹³⁸3⁴4⁵5⁴6⁷	2⁴²3⁶	71.5	53.2	121.1	1133.3	55.31
21.	2⁷⁶3³4²5¹6³	2⁴⁰3⁶	51.7	36.3	14.3	46.2	2.07
22.	2⁷²3⁴4¹6²	2³¹3⁴	26.2	36.0	6.7	12.3	N/A
23.	2²⁵3¹6¹	2¹³3²	15.7	12.6	0.6	10.1	3.06
24.	2¹¹⁰3²5³6⁴	2²⁵3⁴	58.3	42.8	40.6	304.5	17.03
25.	2¹¹⁸3⁶4²5²6⁶	2²³3³4¹	65.7	48.3	61.7	507.8	25.64
26.	2⁸⁷3¹4³5⁴	2²⁸3⁴	42.0	30.7	15.5	71.2	4.93
27.	2⁵⁵3²4²5¹6²	2¹⁷3³	45.8	36.0	4.6	10.8	0.63
28.	2¹⁶⁷3¹⁶4²5³6⁶	2³¹3⁶	68.5	50.7	170.4	1522.3	75.95
29.	2¹³⁴3⁷5³	2¹⁹3³	38.4	29.7	38.9	89.3	5.79
30.	2⁷³3³4³	2²⁰3²	45.8	19.3	10.3	30.5	0.76

Sum			1546.1	1155.4	1533.1	9501.8	20.40