Combinatorial Interaction Testing Portal

Covering Arrays by Simulated Annealing (CASA)

Evaluation Artifacts: EMSE2010, Table 5

Average size and times over 50 runs for constrained 2-way problems
	Size			Run Time (s)			Break-even (s/cfg.)			Coefficient of Variation			Coefficient of Variation
							TSR vs.	All vs.		in Size (%)			in Run Time (%)
	mAETG			mAETG			mAETG	mAETG	All vs.	mAETG			mAETG
Name	Best	TSR	All	Best	TSR	All	Best	Best	TSR	Best	TSR	All	Best	TSR	All

SPIN-S	27.0	20.6	19.4	0.2	2.2	8.6	0.32	1.11	5.31	3.3	5.7	3.4	4.2	40.4	75.6
SPIN-V	42.5	42.9	36.8	11.3	32.5	102.1	N/A	15.93	11.44	3.0	6.4	6.3	3.0	59.5	121.6
GCC	24.7	25.8	21.1	204.0	661.7	1,902.0	N/A	471.67	266.15	4.4	8.5	11.7	4.4	39.4	88.1
Apache	42.6	34.6	32.3	76.4	61.8	109.1	0.00	3.17	20.55	3.6	4.0	5.0	3.6	11.5	137.9
Bugzilla	21.8	16.9	16.2	1.9	3.4	9.1	0.31	1.29	8.32	5.2	6.4	3.1	5.3	42.6	74.8

1.	53.3	42.5	38.6	24.4	27.9	179.5	0.32	10.55	38.68	3.1	2.9	2.8	3.9	21.6	138.3
2.	40.5	32.9	31.0	14.7	23.5	25.4	1.16	1.13	1.00	4.1	7.0	3.6	4.0	73.7	86.9
3.	20.8	18.3	18.0	0.2	2.5	3.4	0.90	1.14	3.33	6.9	3.3	1.1	6.8	32.5	61.5
4.	28.6	21.8	21.0	3.1	7.3	29.3	0.61	3.45	29.01	4.0	5.1	5.5	4.3	23.4	323.0
5.	63.8	54.8	47.7	134.8	165.6	655.9	3.43	32.37	69.05	1.9	8.8	3.2	4.2	95.7	133.3
6.	34.0	24.2	24.2	7.2	9.1	18.2	0.19	1.12	456.71	4.5	2.7	1.5	4.6	26.6	54.9
7.	12.5	9.0	9.0	0.3	1.5	2.1	0.33	0.51	N/A	4.9	0.0	0.0	8.7	2.4	4.0
8.	55.6	44.0	40.5	45.1	41.2	249.9	0.00	13.56	60.32	3.3	3.6	3.3	3.3	23.2	220.4
9.	26.0	30.0	20.0	3.0	3.4	29.8	N/A	4.47	2.64	5.3	0.0	1.0	5.3	2.7	54.0
10.	60.4	48.2	44.0	74.3	60.8	357.3	0.00	17.26	70.26	2.6	2.4	2.1	2.6	15.1	89.2
11.	58.3	46.1	41.9	23.8	27.0	240.7	0.26	13.23	50.65	2.5	2.2	1.9	2.8	20.0	94.5
12.	54.5	43.2	40.4	68.0	53.7	221.1	0.00	10.86	58.95	3.5	2.5	4.1	3.6	15.3	151.6
13.	48.6	37.7	36.5	45.5	40.1	60.5	0.00	1.24	17.28	5.4	3.1	2.2	5.4	12.6	94.0
14.	51.8	39.6	36.9	20.2	23.9	58.1	0.30	2.54	12.68	4.0	3.1	2.0	4.1	18.4	71.6
15.	40.4	31.8	30.7	4.9	9.8	19.3	0.57	1.48	8.65	3.4	3.9	2.7	3.3	18.5	76.4
16.	33.4	24.3	24.1	9.7	11.2	19.7	0.16	1.08	38.81	5.3	3.6	1.0	5.5	21.2	35.1
17.	53.4	42.6	39.2	46.9	45.8	335.5	0.00	20.32	84.21	3.3	2.9	3.9	3.4	14.7	171.8
18.	57.3	45.9	42.3	60.1	52.8	303.5	0.00	16.23	70.43	2.1	2.7	3.6	2.2	13.6	108.3
19.	64.7	52.3	47.8	168.4	124.2	823.6	0.00	38.77	154.72	2.4	4.5	2.4	2.5	47.7	86.1
20.	71.5	58.5	53.2	121.1	86.3	1,133.3	0.00	55.31	199.05	1.7	4.8	1.7	1.8	63.4	171.4
21.	51.7	38.1	36.3	14.3	18.7	46.2	0.32	2.07	15.29	3.9	3.4	1.3	3.9	17.1	89.1
22.	26.2	36.0	36.0	6.7	8.9	12.3	N/A	N/A	86.24	4.6	0.5	0.0	4.5	14.8	39.4
23.	15.7	18.0	12.6	0.6	0.5	10.1	0.04	3.06	1.78	5.4	0.0	7.5	6.0	2.8	52.2
24.	58.3	47.0	42.8	40.6	39.6	304.5	0.00	17.03	63.38	2.7	2.6	2.3	2.6	22.8	128.1
25.	65.7	53.1	48.3	61.7	53.1	507.8	0.00	25.64	93.94	2.0	2.1	1.7	7.7	21.0	104.0
26.	42.0	32.9	30.7	15.5	22.0	71.2	0.71	4.93	22.18	2.8	5.9	2.5	3.3	26.6	75.8
27.	45.8	36.5	36.0	4.6	8.2	10.8	0.39	0.63	5.15	4.6	2.0	0.4	4.6	24.5	60.5
28.	68.5	55.4	50.7	170.4	121.6	1,522.3	0.00	75.95	296.75	2.1	2.9	2.4	2.1	27.4	119.9
29.	38.4	31.6	29.7	38.9	41.0	89.3	0.31	5.79	25.41	4.1	3.1	3.3	4.1	17.7	117.0
30.	45.8	20.4	19.3	10.3	17.5	30.5	0.28	0.76	12.23	6.9	4.3	4.9	6.8	113.2	129.0

Sum	1,546.1	1,257.4	1,155.4	1,533.1	1,910.4	9,501.8	1.31	20.40	74.40	0.6	0.8	0.6	0.9	16.8	37.6