Citportal
beta
Login
Sign Up
Academic
Industry
Covering Arrays by Simulated Annealing (CASA)
Evaluation Artifacts: EMSE2010, Table 2
Average sizes and times over 5 runs for constrained 2-way problems
Size
Run Time (s)
Name
Model
Constraints
Control
Grouped
TSR
OSN
All
Control
Grouped
TSR
OSN
All
SPIN-S
2
13
4
5
2
13
20.2
19.6
20.0
20.8
19.4
414.94
407.27
2.08
159.15
7.23
SPIN-V
2
42
3
2
4
11
2
47
3
2
33.2
32.6
43.2
49.4
36.8
4,010.73
4,710.78
27.58
1,080.60
60.03
GCC
2
189
3
10
2
37
3
3
19.8
19.6
26.8
19.2
21.8
36,801.39
41,181.63
717.54
366,713.76
1,708.92
Apache
2
158
3
8
4
4
5
1
6
1
2
3
3
1
4
2
5
1
32.6
31.2
34.8
38.8
33.0
23,589.09
18,326.97
59.90
27,274.91
59.35
Bugzilla
2
49
3
1
4
2
2
4
3
1
16.0
16.0
18.0
17.0
16.4
58.04
45.90
4.14
322.11
6.08
1.
2
86
3
3
4
1
5
5
6
2
2
20
3
3
4
1
39.6
39.4
43.0
55.6
39.0
10,596.51
13,190.71
26.37
12,581.48
80.82
2.
2
86
3
3
4
3
5
1
6
1
2
19
3
3
30.0
30.0
32.6
33.0
31.4
1,888.95
1,163.55
20.16
6,085.74
22.59
3.
2
27
4
2
2
9
3
1
18.0
18.0
18.0
18.0
18.0
506.12
629.50
2.80
60.58
5.43
4.
2
51
3
4
4
2
5
1
2
15
3
2
25.6
20.2
22.0
20.8
21.6
2,145.45
690.53
7.27
3,282.58
143.89
5.
2
155
3
7
4
3
5
5
6
4
2
32
3
6
4
1
48.4
49.0
52.0
80.6
47.0
61,775.54
48,312.01
122.17
405,339.65
816.86
6.
2
73
4
3
6
1
2
26
3
4
24.0
24.0
24.0
27.6
24.2
603.51
414.45
8.88
3,241.36
20.98
7.
2
29
3
1
2
13
3
2
9.0
9.0
9.0
9.2
9.0
378.11
655.98
1.51
25.66
2.12
8.
2
109
3
2
4
2
5
3
6
3
2
32
3
4
4
1
41.4
42.2
44.2
98.6
40.4
18,046.28
12,355.77
39.04
149,367.29
121.22
9.
2
57
3
1
4
1
5
1
6
1
2
30
3
7
30.0
30.0
30.0
20.0
20.0
11.59
3.23
3.33
418.29
26.37
10.
2
130
3
6
4
5
5
2
6
4
2
40
3
7
45.0
45.6
48.8
73.4
44.4
32,034.07
24,105.66
59.69
16,511.05
322.95
11.
2
84
3
4
4
2
5
2
6
4
2
28
3
4
43.2
43.8
45.8
57.6
41.4
15,605.26
11,467.99
28.10
13,408.84
380.83
12.
2
136
3
4
4
3
5
1
6
3
2
23
3
4
40.2
40.0
43.4
50.0
41.2
23,708.98
26,328.56
56.71
549,157.34
109.31
13.
2
124
3
4
4
1
5
2
6
2
2
22
3
4
36.0
36.2
37.4
38.6
36.6
4,721.45
5,513.18
40.88
24,168.64
56.44
14.
2
81
3
5
4
3
6
3
2
13
3
2
37.4
36.8
39.6
43.4
36.4
4,962.51
6,915.28
25.43
27,614.18
76.75
15.
2
50
3
4
4
1
5
2
6
1
2
20
3
2
67.0
30.0
31.8
39.8
31.0
6,032.95
1,231.50
9.75
1,594.78
14.26
16.
2
81
3
3
4
2
6
1
2
30
3
4
24.0
24.0
24.0
25.8
24.2
1,501.38
254.08
10.16
3,649.28
18.98
17.
2
128
3
3
4
2
5
1
6
3
2
25
3
4
38.8
39.4
43.0
53.8
40.0
22,742.49
22,096.80
46.15
8,306.83
99.34
18.
2
127
3
2
4
4
5
6
6
2
2
23
3
4
4
1
42.6
43.6
46.6
63.4
42.0
36,907.75
18,813.12
50.77
15,663.36
245.32
19.
2
172
3
9
4
9
5
3
6
4
2
38
3
5
49.6
49.6
51.8
77.0
47.6
89,240.98
62,418.58
124.46
169,470.59
747.64
20.
2
138
3
4
4
5
5
4
6
7
2
42
3
6
55.4
54.8
57.8
67.2
54.2
41,527.58
38,089.49
86.11
1,177,089.69
425.07
21.
2
76
3
3
4
2
5
1
6
3
2
40
3
6
36.6
36.6
38.6
52.2
36.0
4,064.05
3,261.35
18.40
517.35
72.37
22.
2
72
3
4
4
1
6
2
2
31
3
4
38.0
36.0
36.0
39.0
36.0
848.69
191.24
8.91
2,168.70
9.87
23.
2
25
3
1
6
1
2
13
3
2
18.0
18.0
18.0
12.0
13.0
0.70
0.25
0.50
109.24
10.45
24.
2
110
3
2
5
3
6
4
2
25
3
4
43.4
43.8
47.4
60.8
42.6
23,949.69
22,376.39
37.15
78,771.41
251.14
25.
2
118
3
6
4
2
5
2
6
6
2
23
3
3
4
1
49.8
49.8
53.4
76.8
48.4
33,822.78
25,175.35
49.28
24,897.32
363.73
26.
2
87
3
1
4
3
5
4
2
28
3
4
29.8
30.4
35.8
39.8
30.8
12,286.50
9,893.79
22.66
43,046.04
82.16
27.
2
55
3
2
4
2
5
1
6
2
2
17
3
3
36.0
36.0
37.0
36.0
36.2
368.33
300.34
8.46
3,142.71
13.41
28.
2
167
3
16
4
2
5
3
6
6
2
31
3
6
51.8
53.4
56.0
59.6
51.2
93,493.65
59,797.45
113.67
1,314,793.81
578.47
29.
2
134
3
7
5
3
2
19
3
3
29.0
28.8
31.4
34.0
29.2
20,088.01
23,313.04
41.77
182,819.54
68.52
30.
2
73
3
3
4
3
2
20
3
2
18.2
18.2
20.2
19.0
18.8
4,285.60
5,481.25
25.74
10,136.84
24.68
Sum
1,217.6
1,175.6
1,261.4
1,527.8
1,159.2
633,019.65
509,113.01
1,907.52
4,642,990.67
7,053.57
Coefficient of Variation
2.9%
0.4%
0.7%
5.8%
0.6%
4.9%
8.6%
9.5%
64.5%
25.8%