Bachelor and Master Theses

Title: On the Use of Base Choice Strategy for Testing Industrial Control Software
Subject: Computer Science
Level: Basic
Description: Background
Software testing is an important practical engineering activity, used to improve software quality and it is sometimes based on satisfying different testing purposes from code coverage criteria to combinatorial coverage [1]. In software testing literature, a software fault is a static defect in the software, while a software error is an incorrect internal state that is the manifestation of some fault. A software failure refers to the incorrect behavior with respect to the requirements or other description of the expected behavior. Combinatorial test suite design is a test generation approach, popular in part due to its ability to efficiently create test cases that can help engineers find faults in their software.

Problem formulation
Some case studies have appeared presenting practical applications of different combinatorial test design techniques (e.g., base choice (BC) strategy). Recent empirical studies [2, 3] showed the value of base choice coverage and its fault detection capability. However, it is not very well studied how BC could contribute to the development of reliable software and how it compares to manual testing performed in practice. Three questions arose from this:

How does BC and manual tests compare in terms of achieved code coverage?
Are BC tests less costly to use than manual tests?
Are BC tests more effective in fault detection than manual tests?

These questions could lead to promising results and may be beneficial for companies (such as Bombardier Transportation AB) when testing the software in their control systems.

A research methodology is a structured and systematic process that fulfills a certain research objective. Our research started with finding a problem or opportunity of using BC testing in industrial software development, and ends with performing an empirical evaluation [4]. The case study design started with the formulation of the research objective that was broken down into the three research questions which are investigated in this thesis.

An evaluation with regard to its success in meeting code coverage will be performed using a case study. In addition, in the same case study, we compare BC with random and manual testing in terms of cost and fault detection using mutation analysis. The resulted data will be analyzed using descriptive and inferential statistics. Descriptive statistics will be used to visualize and provide an overview of the data, e.g., in plots and graphs. In contrast, inferential statistics will be used in this paper to compare and evaluate data to find patterns, e.g., to compare if two data sets are significantly different from each other.

The set of programs used and tested in this thesis are written in IEC 61131-3 language. A combinatorial testing tool will be used to generate test cases. The combinatorial testing tool used in this thesis was developed by seven students at MDH, which has base choice and random algorithm implemented. The test cases will be used to test the programs and fault detection, code coverage in terms of branch coverage and cost will be measured using the execution software used by Bombardier. The tests created manually by engineers at Bombardier will also be measured using the same execution framework.

Expected Outcome(s)
In this thesis we present the results from a comparative evaluation of three combination strategies: base choice, random and manual tests produced by real industrial engineers. We perform this evaluation using real-world programs developed in industry and comparing these three test design techniques in terms of achieved code coverage, fault detection and cost. The results from this evaluation would be used to show evidence on the usage of base choice criteria in a real industrial scenario. The results will show if BC is better, the same or worse than manual testing in terms of achieved branch coverage and fault detection.

Initial Timeplan
Initial time plan consists of:
Background : March 28, 2016 - April 11, 2016
Related work
Research/Implementation: April 12, 2016 - May 12, 2016
Experimentation/Analysis: May 13, 2016 - May 31, 2016
Report writing: March 28, 2016 - June 5, 2016

We will use one combinatorial testing tool for BC developed at MDH. There are many tools for generating test and these may give different results. Another limitation is that the results are mainly based on programs provided by one company (i.e., Bombardier Transportation AB in Sweden). Even if this can be seen as a weak point, using real industrial programs from the safety-critical domain can be representative to other systems. More studies are needed to generalize the results of this thesis to other systems and domains.

To test the fault detection effectiveness of BC, random and manual test strategy, mutation analysis will be used. Since mutation analysis is a proxy for fault detection that is used for injecting faults into the software, this can be a limitation to our results. There will be no real faults used to evaluate these strategies and this fact may affect the outcome of the results. However, mutants have been shown to be a valid substitute for real faults in fault detection according to recent studies [5], which would support the results from a mutation analysis to be scientifically valid.


[1] Ammann, Paul, and Jeff Offutt. Introduction to software testing. Cambridge University Press, 2008.

[2] Grindal, Mats, Jeff Offutt, and Sten F. Andler. "Combination testing strategies: a survey." Software Testing, Verification and Reliability 15.3 (2005): 167-199.

[3] Grindal, Mats, et al. "An evaluation of combination strategies for test case selection." Empirical Software Engineering 11.4 (2006): 583-611.

[4] Runeson, Per, and Martin Höst. "Guidelines for conducting and reporting case study research in software engineering." Empirical software engineering 14.2 (2009): 131-164.

[5] Just, René, et al. "Are mutants a valid substitute for real faults in software testing?." Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 2014.
Prel. end date: 2016-06-5
Presentation date: 2016-06-3
Student: Simon Eklund
IDT supervisor: Eduard Paul Enoiu, +46-21-101624
Examinator: Daniel Sundmark
Daniel Sundmark, +46-21-103145

Rapport och bilagor


Senaste uppdatering



2016-08-03, 11:25

  • Mälardalen University |
  • Box 883 |
  • 721 23 Västerås/Eskilstuna |
  • 021-101300, 016-153600 |
  • webmaster |
  • Latest update: 2017.08.23