home · mobile · calendar · defenses · 2005-2006 · 

Thesis Defense - Rutherford

ADDSALT: Adequate System-Level Testing of Distributed Systems
Computer Science PhD Candidate
5/16/2006
2:00pm-4:00pm

Test adequacy criteria are rules that provide an objective stopping condition on test input creation by defining a finite set of test requirements that must be satisfied. Adequacy criteria have been a focus of research activity for many years but existing testing criteria do not address the unique features of distributed applications. The contributions of this dissertation are: (1) a study of reported failure scenarios of seven distributed applications; (2) a novel testing technique based on discrete-event simulations that serves as a basis for adequacy criteria for distributed systems; (3) a fault-based analysis technique that addresses the fundamental risks associated with using adequacy criteria; and (4) a case-study evaluation of the simulation-based and fault-based techniques.

Instead of inventing a new specification formalism, we adapt the common practice of using discrete-event simulations for the design and understanding of distributed systems to the testing activity. Our key observation is that these simulations can be viewed as specifications of the expected behavior of the system. Using simulations to test implementations of a system is therefore a matter of selecting inputs to cover the simulation according to some criterion, and then mapping them into the implementation domain. As simulations are sequential programs themselves, virtually all black- and white-box criteria can be used with our simulation-based technique.

We evaluate the simulation-based technique and the companion fault-based analysis method on three distributed systems. The results of these experiments are striking. First, we confirm that discrete-event simulations can indeed be used in testing, and that white-box techniques based on the simulation compare favorably to randomly selected test suites of the same size. Second, we demonstrate the power of our fault-based analyses by predicting exactly the relationships between different criteria, and by significantly improving the effectiveness of adequate suites within each criterion.

Committee: Antonio Carzaniga, Research Assistant Professor (Co-Chair)
Alexander Wolf, Professor (Co-Chair)
Amer Diwan, Assistant Professor
Shivakant Mishra, Associate Professor
Alessandro Orso, Georgia Institute of Technology
Department of Computer Science
University of Colorado Boulder
Boulder, CO 80309-0430 USA
webmaster@cs.colorado.edu
www.cs.colorado.edu
May 5, 2012 (14:20)
XHTML 1.0/CSS2
©2012