AutoInSpec: Using Missing Test Coverage to Infer Specifications in GUIs
Myra B. Cohen, Si Huang, Atif M. Memon
Supplementary Data -- ISSRE 2012
Developers of a software's graphical user interface (GUI) often fail to document the interface specifications. Without these, models used for automated test generation and execution remain imperfect and incomplete. This leads to unexpected behavior that creates unrecoverable situations for test harnesses, and missed coverage. In this paper, we present AutoInSpec, a technique to infer an important class of specifications, temporal and state-based invariants between GUI events that have been incorrectly modeled. Unlike existing specification mining approaches that require full execution traces, or source code, and that mine all invariants, we simplify the problem. We guide AutoInSpec with coverage criteria and use a previously developed repair framework that builds coverage-adequate test suites, removing unexecutable sub-sequences from consideration. These failing sub-sequences are input to a logic-based inference engine, armed with known invariant templates, to obtain the missing specifications. We validate AutoInSpec on a set of well studied GUI applications.

We thank B. Nguyen for help with the replayer infrastructure and X. Yuan for sharing her experimental artifacts. This work was partially supported by the US National Science Foundation under grants CCF-0747009, CNS-1205472, CNS-0855139, CNS-1205501 and CNS-0855055, the Air Force Office of Scientific Research through award FA9550-10-1-0406, and the Office of Naval Research under grant N00014-05-1-0421.

Experimental Subjects for RQ1. The Java source code and a description of each of these can be found at:
No. Program Name (Descriptive Name) Number of Events Missing t-sets: Prolog Input
1 2Cons (Consecutive) 3 2Cons-Missing
2 Excl (Excludes) 3 Excl-Missing
3 Disb (Disabled) 3 Disb-Missing
4 Reqs (Requires) 3 Reqs-Missing
5 Cmpd (Compound) 5 Cmpd-Missing
Experimental Subjects for RQ2 and RQ3.
For each application, we provide a link to the application, an XML mapping of events, the initial covering array of test sequences (before the GA). Events are represented as integers (see the XML mapping), and the missing pairs of events after the genetic algorithm completes.
No. Program Name Lines of Code Group Number of Events Task Description CA Missing t-sets (Prolog Input)
1 TerpPaint 3.0 13315 4 11 Clipboard Operations TPGa-CA TPa-G4
2 TerpPresent 3.0 44591 5 14 Content TPrG5-CA TPr-G5
3 TerpWord 3.0 22806 1 14 Table Operations TPwG1-CA TW-G1
4 TerpSpreadSheet 3.0 6337 1 14 Format Cell TPsG1-CA TS-G1
5 5 8 Table Format TPsG5-CA TS-G5
6 CrosswordSage 0.35 3220 4 14 Preference Settings CS-G4-CA CS-G4
7 FreeMind 0.80 24665 1 11 Map Operations FM-G1-CA FM-G1
8 2 18 Format FM-G2-CA FM-G4
9 4 10 Clipboard Operations FM-G4-CA FM-G4
Prolog Output
This data is the output from running our prolog programs. We break out the results by the type of invariant. The first column represents the invariants for RQ1 (known invariants) while the second column represents the ivnariants found on the real applications (RQ2 and RQ3).
RQ1: Invariant RQ2/3: Invariant
DisabledAtPosition DisabledAtPosition
Disabled Disabled
Consecutive Consecutive
Excludes Excludes
Requires Requires