Beyond the Rainbow: Self-Adaptive Failure Avoidance in Configurable Systems

Jacob Swanson, Myra B. Cohen, Matthew B. Dwyer, Brady J. Garvin and Justin Firestone

Supplementary Data – FSE 2014

Abstract

Self-adaptive software systems monitor their state and then adapt when certain conditions are met, guided by a global utility function. In prior work we developed algorithms and conducted a post-hoc analysis demonstrating the possibility of adapting to software failures by judiciously changing configurations. In this paper we present the REFRACT framework that realizes this idea in practice by building on the self-adaptive Rainbow architecture. REFRACT extends Rainbow with new components and algorithms targeting failure avoidance. We use REFRACT in a case study running four independently executing Firefox clients with 36 passing test cases and 7 seeded faults. The study show that workarounds for all but one of the seeded faults are found and the one that is not found, never fails -- it is guarded from failing by a related workaround. Moreover, REFRACT finds workarounds for 8 configuration related failures from tests that were expected to pass (and did under the default configuration). Finally, the data show that when a failure and its workaround are found, configuration guards prevent the failure from appearing again. In a simulation lasting 24 hours we see over 150 guard activations and no failures with workarounds beyond 16 hours.

Supplemental Data

Firefox Feature Model (in XML) used in study.
Normal Test Cases. These are the Mozmill test cases that we expected to pass.
Seeded Faulty Test Cases. These seven tests contains faults that we recreated based on real Firefox faults.
MS Thesis: Tech Report.

Results

Random-10 3-hour, Run One. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	FW	2P	2P	P	P	2P	3P	3P	G	3P	5P	3P	1
306208	F	FWAPG	3P	P	PG	4P	4P	4P	2P	PG	2P	6P	3
797945	FWPG	G	5P	3P	4P	2P	2P	2P	4P	3P	3P	PG	3
808290	-	P	P	2P	P	4P	P	PFW	2P	3P	P	-	0
840411	3P	2P	5P	FPW2P	3P	-	6P	P	PG	-	3P	PG	2
442970	3P	-	2P	P	2P	2P	2P	2P	P	3P	2P	4P	0
505548	-	3P	4P	3P	4P	2P	-	P	2P	2P	P	2P	0
Not Expected to Fail
testOpenSearchAutodiscovery	P	P	6P	5P	3P	2P	3P	2P	5P	PF	W3P	2P	0
testSSLDisabledErrorPage	FP	PA	-	2P	3P	2P	2P	5P	4P	P	3P	3P	0
testUntrustedConnectionErrorPage	FN	P	3P	4P	3P	P	2P	P	2P	3P	2P	2P	0
Total Guards													9

Random-10 3-hour, Run Two. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	FW	2P	4P	5P	-	PG	4P	2P	8P	G7PG3P	3P	4PG2P	4
306208	P	2P	P	5P	5P	3P	3P	4P	2P	P	5P	5P	0
797945	-	FWP	3P2GP	2P	3P	5P	3P	5P	2P	6P	2PGP	PG5PG	5
808290	P	3P	P	4P	5P	P	P	3P	4P	4P	6P	P	0
840411	2P	4P	2P	3P	-	4P	6P	3P	8P	4P	2P	6P	0
442970	P	2P	4P	P	4P	P	3P	2P	5P	2P	2P	5P	0
505548	7P	7P	-	4P	5P	4P	4P	3P	2P	6P	6P	3P	0
Not Expected to Fail
ClearFormHistory	4P	3P	4P	3P	FPW5P	3P	P	5P	P	3PG	4P	GP	2
PopupsBlocked	5P	3P	P	4P	P	PF2PWP	6P	3P	5P	6P	2P	5P	0
AddMozSearchProvider	3P	3PF	PW2P	P	4P	4P	7P	6P	5P	4P	4P	3P	0
Total Guards													11

Random-10 3-hour, Run Three. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	FW	2P	G2P2G	3P	7P	6P	4P	8P	7P	P	P	3P	3
306208	P	6P	P	7P	6P	2P	P	4P	4P	P	3P	2P	0
797945	2FWA	5P	4P	3P	5P	5P	6PG	4P	4P	2PGP	5P	PG3P	3
808290	2P	4P	4P	5P	-	4P	2P	6P	3P	6P	5P	2PFPWP	0
840411	5P	FPW7P	5P	4P	2PGPG4P	8P	6PGP	2PGP	3P	3P	4P	2PGP	5
442970	3P	6P	P	4P	6P	6P	6P	4P	2P	PF2P	NFPW	7P	0
505548	6P	2P	3P	9P	5P	5P	P	4P	5P	2P	4P	2P	0
Not Expected to Fail
testPopupsAllowed	4P	6P	4P	5P	5P	P	7P	5PF	PW3P	8P	P	2P	0
Total Guards													11

Random-10 3-hour, Run Four. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	FWP	G5P	3P	5P	7P	4P	G4P	3P	6P	4P	4P	6P	0
306208	P	3P	5P	6P	5P	8P	PF	PW3P	P	4P	4P	2P	0
797945	2FWA	PG4P	4PG	6PG2PG	6P	2G2P	5P	2PG	2PG2PG	6P	4P	2P	9
808290	P	2P	6P	P	4P	5P	2P	4P	P	3P	5P	2P	0
840411	2P	4P	3P	5P	2P	9P	2P	2P	3P	3P	3P	4P	0
442970	4P	3P	2P	3P	6P	5P	3P	5P	P	3P	-	3P	0
505548	P	3P	2P	5P	3P	4P	-	4P	P	6P	-	7P	0
Not Expected to Fail
testOpenSearchAutodiscovery	P	3P	3P	7P	-	4P	3P	3P	3P	2PF2P	W5P	3P	0
testClearFormHistory	P	5P	3P	4P	5P	5P	5P	2PF	WP	3P	3P	6P	0
testAutoCompleteOff	5P	3PF	PN2P	2P	4P	P	4P	5P	5P	4P	5P	4P	0
Total Guards													9

Random-10 3-hour, Run Five. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	FWGP	G2P	7P	4P	G4P	G2P	2P	2PG2P	2P	2P	2PG	2P	6
306208	3P	3P	4P	3P	P	5P	P	2P	4P	2P	4P	P	0
797945	F	W2P	3P	4P	2PG2P	2P	6P	3P	P	4P	5P	3P	1
808290	3P	$-$	P	P	4P	P	P	2P	2P	6P	7P	3PF	0
840411	5P	4P	2P	$-$	3P	4P	5P	5P	PFWP	2P	2PGP	G	2
442970	3P	5P	6P	P	2P	3P	4P	$-$	3P	5P	4P	2P	0
505548	4P	2P	2P	3P	2P	2P	3P	2P	3P	5P	3P	3P	0
Not Expected to Fail
testPopupsAllowed	P	P	5P	FP	PW2P	PGP	G5P	4PG3P	2P	3P	5P	3P	3
Total Guards													12

2-CA 3-hour, Run One. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	F	WP	2PG	5P	-	2P	3P	P	2PF3PNP	5P	5P	PG	2
306208	FW	P	P	4P	2P	4P	2P	3PFP	PN3P	3PG	3P	5P	1
797945	F	2F	W2A5P	2PGP	4P	2P	7P	2P	5P	2P	6P	2P	0
808290	P	P	3P	2P	FWP	P	2P	3P	4P	2P	5P	PGP	1
840411	FW	2P	-	G4P	GP	5P	3P	3PG	PG2P	G4P	GP	2P	6
442970	-	P	FWP	2P	5P	2P	P	P	3P	2P	6P	2P	0
505548	2P	-	2P	3P	2P	5P	4P	P	3P	3P	2P	4P	0
Not Expected to Fail
testClearFormHistory	4P	P	2P	2P	3PF	3P	PN2PFP	PW3P	4P	PG	2P	4P	1
testSubmitUnencryptedInfoWarning	-	-	P	6P	3P	P	6P	3P	3PF	PW	6P	3P	0
Total Guards													11

2-CA 3-hour, Run Two. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	-	F	W	-	-	P	3P	2P	2P	2P	3P	P	0
306208	-	-	-	-	-	-	P	2F	WPA	GP	2PG	2P	2
797945	-	2FWA	-	-	-	-	3P	2P	3P	2P	P	3P	0
808290	2P	-	-	-	-	-	2P	P	-	2PF2P	WP	2PGP	1
840411	2P	2P	P	P	P	5P	P	-	-	-	-	-	0
442970	-	P	-	P	P	-	P	-	-	-	-	-	0
505548	-	2P	4P	3P	2P	2P	-	-	-	-	-	-	0
Not Expected to Fail
testAutoCompleteOff	P	-	2P	-	-	-	FP	W3P	P	-	P	P	0
testDisableFormManager	3P	-	FN	-	-	-	2P	-	P	-	4P	2P	0
Total Guards													3

2-CA 3-hour, Run Three. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	F	FWP	A2P	P	FN2P	2P	2P	P	3P	5P	3P	P	0
306208	2FWP	AP	2P	5P	2P	2P	2P	P	3P	P	3P	P	0
797945	FWFP	2P	N4P	7P	P	2P	3P	3P	-	4P	3P	4P	0
808290	-	P	P	-	-	2P	2PF	P	W3P	2P	G2P	3P	1
840411	2P	-	2P	P	P	2P	P	P	-	2P	PF	PW4P	0
442970	P	3PF	PW2P	2P2GP	3P	4P	4P	P	PF3P	PN	5P	2P	2
505548	-	P	2P	3P	3P	2P	PF	NP	2P	7P	-	3P	0
Not Expected to Fail
testPasteLocationBar	2P	2P	-	P	-	2P	2P	2P	P	5P	FPW	3P	0
testClearFormHistory	2P	P	FPW	4P	3P	3PG	2P	P	P	-	P	2P	1
testDisableFormManager	3P	3P	2P	-	3P	P	2P	3P	4P	P	-	FPN	0
testOpenSearchAutodiscovery	-	4P	3P	3P	F	WPG	3P	-	5P	3P	3P	P	1
Total Guards													5

2-CA 3-hour, Run Four. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	F	FWA	P	PGP	P	3P	P	P	-	-	-	-	1
306208	2FWA	-	P	P	2P	P	P	G	-	-	-	-	1
797945	P2F	WA3P	5P	2P	P	4P	2P	P	-	-	-	-	0
808290	2P	P	7P	-	2P	2P	3P	-	-	-	-	-	0
840411	2P	2P	P	P	P	5P	P	-	-	-	-	-	0
442970	-	P	-	P	P	-	P	-	-	-	-	-	0
505548	-	2P	4P	3P	2P	2P	-	-	-	-	-	-	0
Not Expected to Fail
testSetToCurrentPage	2P	3P	PF	WP	2P	2P	3P	-	-	-	-	-	0
testOpenSearchAutodiscovery	2P	P	PF	F	W2PW	2PG	P	P	-	-	-	-	1
Total Guards													3

2-CA 3-hour, Run Five. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	F	-	-	FWA	3P	2PG	4P	P	-	2P	4PF	4PNP	1
306208	-	-	F	W2P	2P	2P	3P	7P	PGP	3P	4PGP	3P	2
797945	FN	-	-	F	FWA	2P	P	3P	3P	5P	4PG3P	P	1
808290	2P	-	-	2P	P	P	P	2PFW	5P	G2PGP	6P	P	2
840411	2P	-	-	-	2PF3P	W3P	-	-	2P	4P	4P	G2P	1
442970	-	-	-	3P	2P	FW3P	4P	P	2PG2P	4P	3P	P	1
505548	-	-	-	-	-	4P	2P	2P	P	5P	5P	3P	0
Not Expected to Fail
testSetToCurrentPage	PFN	-	-	-	-	2P	5P	5P	3P	4P	3P	-	0
testEnablePrivilege	P	-	-	-	P	P	5P	F	PW4PG3P	2P	2P	3P	1
testSSLDisabledErrorPage	2P	-	-	FA	-	3P	3P	3P	P	3P	5P	2P	0
Total Guards													9

2-CA 3-hour, Alternative Covering Array One. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	-	-	-	-	-	-	2FWA	-	-	-	-	P	0
306208	F	F	-	WPA	-	-	-	P	-	-	-	2P	0
797945	-	-	-	FW	-	-	P	3P	P	-	P	-	0
808290	-	-	-	P	-	-	-	-	-	-	-	P	0
840411	-	-	-	-	-	-	P	-	-	-	-	3P	0
442970	-	-	-	3P	-	-	P	3P	P	-	P	-	0
505548	-	-	-	-	-	-	2P	-	-	-	-	P	0
Not Expected to Fail
testGoButton	FPN	-	-	P	-	-	-	2P	P	-	-	P	0
testSetToCurrentPage	-	-	-	FP	N	-	3P	-	FN	-	-	-	0
Total Guards													0

2-CA 3-hour, Alternative Covering Array Two. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	F	2FW	AGA3P	P	P	-	-	-	P	2P	P	-	1
306208	FW	-	3P	2P	PG	2GP	-	-	P	4P	3P	-	3
797945	2F	FWP2AP	2P	P	P	G2P	P	P	-	2P	2P	P	1
808290	3P	P	P	-	P	2P	P	-	-	5P	3P	-	0
840411	-	P	-	2P	P	4P	-	-	-	3P	2P	P	0
442970	-	P	2P	P	2P	3P	-	-	-	4P	P	P	0
505548	2P	P	P	3P	P	P	-	-	3P	-	3P	P	0
Not Expected to Fail
testClearFormHistory	P	P	3P	P	FWP	P	-	-	P	3P	2P	-	0
testPopupsBlocked	-	-	2P	P	2P	2P	-	-	PF	PW5P	-	P	0
testPaneRetention	-	-	-	2P	P	-	F3PN	-	-	5P	2P	P	0
testSearchViaShortcut	P	-	P	2P	PF	P	N	-	-	P	P	2P	0
Total Guards													5

2-CA 3-hour, Alternative Covering Array Three. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	-	-	-	-	-	-	-	F	-	F	WAP	-	0
306208	F	-	2F	WAGA	-	P	-	P	-	-	P	P	1
797945	F	-	WG	2P	5P	-	-	P	P	-	3P	-	1
808290	P	P	2P	2P	2P	P	-	2P	-	-	-	P	0
840411	-	-	-	-	P	-	-	P	-	-	-	-	0
442970	-	-	-	-	P	P	-	2P	-	-	-	4P	0
505548	-	-	-	P	2P	-	-	P	-	-	-	P	0
Not Expected to Fail
testClearFormHistory	FN	-	-	P	-	P	-	P	-	-	P	P	0
testSearchViaShortcut	P	-	-	2P	-	-	-	FN	-	-	-	2P	0
testDVCertificate	-	-	3P	F	2P	N	-	-	-	P	P	-	0
Total Guards													2

2-CA 3-hour, Alternative Covering Array Four. 15 minute intervals shown.
P—pass, F—failure, W—workaround, A—workaround already found, N—no workaround found, G—Guard.
Test	0:15	0:30	0:45	1:00	1:15	1:30	1:45	2:00	2:15	2:30	2:45	3:00	# Guards
Expected to Fail
344189	-	FWFA	5P	G	P	P	2PF	N	2P	P	2P	-	1
306208	F	WF	NP	GP	3P	P	P	2P	2P	4P	-	P	1
797945	FW	P	2P	PFN	P	G2P	2P	2P	PF	GN	2P	-	2
808290	-	P	3P	P	-	3P	2P	P	2P	2P	P	4P	0
840411	-	P	2P	-	-	P	PFW	2P	P	3P	2PG2P	3P	1
442970	P	-	3P	3P	FP	W5P	3P	-	-	P	-	-	0
505548	-	2P	-	3P	-	P	4P	-	4P	-	3P	5P	0
Not Expected to Fail
testLocationBarSearches	P	-	3P	3P	-	4P	4P	P	PF	WP	-	-	0
testOpenDownloadManager	N	P	2P	-	-	P	-	-	2P	2P	-	-	0
testClearFormHistory	P	P	-	5P	3P	P	PF	PW	P	P	P	P	0
Total Guards													5