Model-Based Testing for Software Product Lines

Abstract

A Software Product Line (SPL), or family of systems, is a collection of applications that have so many features in common that it is worthwhile to study and analyze the common features as well as analyzing the features that differentiate these applications. Model-based design and development for SPLs extends modeling concepts for single applications to model the commonality and variability among the members of the SPL. Previous research on model-based functional testing methods for SPLs use existing requirement models, such as feature and use case models, to create reusable test specifications that can be configured for applications derived from a SPL. Feature-based test coverage criteria can be applied to determine what applications to test, when it is not feasible to test all possible applications of a SPL. However, previous research on functional testing methods for SPLs does not apply feature-based test coverage criteria together with a use case-based approach of creating reusable test specifications for a SPL. This research describes a functional test design method for SPLs (Customizable Activity diagrams, Decision tables and Test specifications, or CADeT) that applies feature-based test coverage criteria together with a use case-based approach of creating reusable test specifications for a SPL. Features from a feature model are associated with test models created from the use cases of a SPL using feature condition variables. The values of a feature condition represent possible feature selections, so that selecting a value for the feature condition selects and customizes the test models associated with that feature. With CADeT, activity diagrams are created from the use case descriptions of a SPL. Reusable test specifications are traced from the use case activity diagrams and described in decision tables. The relationships of features to activity diagrams are also portrayed in decision tables, and then analyzed to apply a feature-based test coverage criterion to the SPL. Representative applications configurations are generated to cover all features, all use case scenarios, and all relevant feature combinations of a SPL. Reusable test specifications are selected and customized for each application configuration, and then used to test the corresponding application implementation. Furthermore, CADeT is extended to use separation of concerns to customize the reusable test specifications during feature-based test derivation (CADeT-SoC). Instead of using feature conditions to customize these test specifications, CADeT-SoC separates the variable test steps from the test specifications, and then weaves selected test steps with these test specifications during feature-based test derivation. CADeT-SoC is more suitable than CADeT for customizing the test specifications of a SPL with many variation points repeated across several use cases. Using CADeT-SoC reduced the effort needed to define variable test steps for the variation points in test specifications in each SPL. The feasibility of the CADeT and CADeT-SoC methods was evaluated in three studies on two SPLs: an Automated Highway Toll System (AHTS) SPL, and a Banking System SPL. The results of these studies show that CADeT and CADeT-SoC can be used to create reusable test specifications to cover all use case scenarios, all features, and all relevant feature combinations on each of these two SPLs. The feature model of each SPL, and the relationships of features to test specifications were analyzed to determine the relevant feature combinations, and a feature-based coverage criterion was applied to reduce the number of application configurations to test. Using CADeT also reduced the number of test specifications needed to satisfy these criteria, as compared with using two alternative approaches. The contribution of this research is CADeT, a model-based test specification design method, and CADeT-SoC, an extension of CADeT that uses separation of concerns to customize the test specifications for an application derived from the SPL. CADeT and CADeT-SoC can help a test engineer create reusable test specifications to cover all use case scenarios, features and relevant feature combinations of a SPL. These test specifications can be customized during feature-based test derivation for a set of applications derived from a SPL. Using CADeT and CADeT-SoC reduces the number of application configurations and test specifications that need to be created to cover all use case scenarios, features and relevant feature combinations in a SPL.