PAClab - a Program Analysis ColLABoratory

Benchmark programs play an important role in evaluating the advances in program analysis research. Thus, the confidence of the evaluation depends on the quantity and the quality of the benchmark programs. The larger their number and the closer they resemble real-world programs, the higher confidence in the evaluation. Currently, research in the field of program analysis often relies on a few outdated or deficient benchmark programs for evaluation purposes. The difficulty in obtaining sufficient benchmark programs comes from the requirements imposed on the benchmark programs by program analyzers. Usually researchers manually obtain, inspect, and transform potential benchmark programs to create benchmark programs that meet those requirements. Such an unscalable approach often fails to produce adequate benchmark programs. The goal of this research is to automate this process and make it available through an online, public infrastructure. The infrastructure facilitates obtaining, selecting, and transforming open-source software projects for use as benchmark programs. In addition, the infrastructure gives researchers the ability to easily curate benchmark programs for program analysis and share the results of running their analyzers on those benchmarks.

The objectives of this project are to solicit requirements and feedback from the program analysis community on automating benchmark program generation, and to prototype a Program Analysis Collaboratory (PAClab) research infrastructure that implements that process. Using researchers' specifications for adequate benchmark program generation, PAClab locates and obtains potential benchmark programs in open-source repositories; performs necessary program transformations as specified by researchers; and outputs the adequate benchmark programs. In addition, PAClab enables researchers to share their program analyzers using container technology, the transformed benchmark programs, and the results of running their analyzers on those benchmark programs. The intellectual merit of the project lies in investigating techniques for specifying and automating program transformations for targeted program analyzers. The broader impacts of the project stem from its potential to broaden adaptation of program analysis advances through the increased confidence of their evaluations; to lower the entry barrier for new program analysis researchers; and to accelerate the pace in program analysis research. PAClab's educational benefits include the ability for students to easily locate real-world programs and evaluate their implementations of program analyzers.