Lysaker, Norway
Lysaker, Norway

Time filter

Source Type

Wu J.,Beihang University | Ali S.,Certus Software Vandnter | Yue T.,Certus Software Vandnter | Tian J.,Beihang University
2013 IEEE 24th International Symposium on Software Reliability Engineering, ISSRE 2013 | Year: 2013

Real Time Operating System for Avionics (RTOS4A) is responsible for providing an operating environment for avionics application software. Avionics software being safety-critical in nature poses several safety and reliability requirements on RTOS4A in addition to the requirements imposed by standards, for instance, DO-178B. Due to this reason, reliability assessment of RTOS4A is very critical to demonstrate confidence about its reliability to its relevant stakeholders. One common way of assessing reliability is by systematic analyses of testing data such as number of tests, number of failures, and coverage using appropriate statistical tests. In this paper, we report our experience of assessing the reliability of an industrial RTOS4A based on testing data collected for 17 months on eight continuous releases. We studied correlation among various measures including: Testing Effort Measures (e.g., complexity of test cases), Testing Effectiveness Measures (e.g., number of failures), and Complexity Measures (e.g., number of functions in a release) and provide in this paper a set of recommendations to assess the reliability of RTOS4A, which serve as guidelines to practitioners in the domain of RTOS4A. © 2013 IEEE.

Ali S.,Certus Software Vandnter | Zohaib Iqbal M.,National University of Computer and Emerging Sciences | Arcuri A.,Certus Software Vandnter | Briand L.C.,University of Luxembourg
IEEE Transactions on Software Engineering | Year: 2013

Model-based testing (MBT) aims at automated, scalable, and systematic testing solutions for complex industrial software systems. To increase chances of adoption in industrial contexts, software systems can be modeled using well-established standards such as the Unified Modeling Language (UML) and the Object Constraint Language (OCL). Given that test data generation is one of the major challenges to automate MBT, we focus on test data generation from OCL constraints in this paper. This endeavor is all the more challenging given the numerous OCL constructs and operations that are designed to facilitate the definition of constraints. Though search-based software testing has been applied to test data generation for white-box testing (e.g., branch coverage), its application to the MBT of industrial software systems has been limited. In this paper, we propose a set of search heuristics targeted to OCL constraints to guide test data generation and automate MBT in industrial applications. We evaluate these heuristics for three search algorithms: Genetic Algorithm, (1+1) Evolutionary Algorithm, and Alternating Variable Method. We empirically evaluate our heuristics using complex artificial problems, followed by empirical analyses of the feasibility of our approach on one industrial system in the context of robustness testing. Our approach is also compared with the most widely referenced OCL solver (UMLtoCSP) in the literature and shows to be significantly more efficient. © 2013 IEEE.

Ali S.,Certus Software Vandnter | Yue T.,Certus Software Vandnter | Rubab I.,University of Luxembourg
Proceedings - International Conference on Quality Software | Year: 2014

Aspect state machines (ASMs) are extended UML state machines that use stereotypes from a UML profile called AspectSM. In our previous experiments, we empirically evaluated ASMs from the perspectives of readability, comprehensibility, understand ability, modeling errors, and modeling effort and the results showed that ASMs are significantly better than the standard UML state machines for modeling robustness behavior for testing. However, a fundamental question still remained to be answered about how modelers/testers modeling ASMs feel about their use. With this in mind, we report results from a series of controlled experiments that were conducted to evaluate subjective opinions of modelers/testers from various perspectives using several questionnaires. The results of the experiment showed that the participants found it difficult to apply AspectSM and weren't confident about their solutions. We further observed that the participants' understand ability and experience of applying AspectSM improved after performing various modeling activities. Although, our results seem very generic, but notice that these results provide preliminary evidence about these observations, which is missing in the Aspect-Oriented Modeling literature. © 2014 IEEE.

Behjati R.,Certus Software Vandnter | Nejati S.,Certus Software Vandnter | Briand L.C.,University of Luxembourg
ACM Transactions on Software Engineering and Methodology | Year: 2014

Configuration in the domain of Integrated Control Systems (ICS) is largely manual, laborious, and error prone. In this article, we propose a model-based configuration approach that provides automation support for reducing configuration effort and the likelihood of configuration errors in the ICS domain. We ground our approach on component-based specifications of ICS families. We then develop a configuration algorithm using constraint satisfaction techniques over finite domains to generate products that are consistent with respect to their ICS family specifications. We reason about the termination and consistency of our configuration algorithm analytically. We evaluate the effectiveness of our configuration approach by applying it to a real subsea oil production system. Specifically, we have rebuilt a number of existing verified product configurations of our industry partner. Our experience shows that our approach can automatically infer up to 50% of the configuration decisions, and reduces the complexity of making configuration decisions. © 2014 ACM.

Ali S.,Certus Software Vandnter | Yue T.,Certus Software Vandnter | Briand L.C.,Certus Software Vandnter | Briand L.C.,University of Luxembourg
Software and Systems Modeling | Year: 2014

Aspect-oriented modeling (AOM) is a relatively recent and very active field of research, whose application has, however, been limited in practice. AOM is assumed to yield several potential benefits such as enhanced modularization, easier evolution, increased reusability, and improved readability of models, as well as reduced modeling effort. However, credible, solid empirical evidence of such benefits is lacking. We evaluate the "readability" of state machines when modeling crosscutting behavior using AOM and more specifically AspectSM, a recently published UML profile. This profile extends the UML state machine notation with mechanisms to define aspects using state machines. Readability is indirectly measured through defect identification and fixing rates in state machines, and the scores obtained when answering a comprehension questionnaire about the system behavior. With AspectSM, crosscutting behavior is modeled using so-called "aspect state machines". Their readability is compared with that of system state machines directly modeling crosscutting and standard behavior together. An initial controlled experiment and a much larger replication were conducted with trained graduate students, in two different institutions and countries, to achieve the above objective. We use two baselines of comparisons-standard UML state machines without hierarchical features (flat state machines) and standard state machines with hierarchical/concurrent features (hierarchical state machines). The results showed that defect identification and fixing rates are significantly better with AspectSM than with both flat and hierarchical state machines. However, in terms of comprehension scores and inspection effort, no significant difference was observed between any of the approaches. Results of the experiments suggest that one should use, when possible, aspect state machines along with hierarchical and/or concurrent features of UML state machines to model crosscutting behaviors. © 2012 Springer-Verlag Berlin Heidelberg.

Gotlieb A.,French Institute for Research in Computer Science and Automation | Gotlieb A.,Certus Software Vandnter
Knowledge Engineering Review | Year: 2012

Safety-critical software must be thoroughly verified before being exploited in commercial applications. In particular, any TCAS (Traffic Alert and Collision Avoidance System) implementation must be verified against safety properties extracted from the anti-collision theory that regulates the controlled airspace. This verification step is currently realized with manual code reviews and testing. In our work, we explore the capabilities of Constraint Programming for automated software verification and testing. We built a dedicated constraint solving procedure that combines constraint propagation with Linear Programming to solve conditional disjunctive constraint systems over bounded integers extracted from computer programs and safety properties. An experience we made on verifying a publicly available TCAS component implementation against a set of safety-critical properties showed that this approach is viable and efficient. © 2012 Cambridge University Press.

Ali S.,Certus Software Vandnter | Yue T.,Certus Software Vandnter
Proceedings - Asia-Pacific Software Engineering Conference, APSEC | Year: 2012

Aspect-oriented Modeling (AOM) is a relatively recent, very active field of research and is assumed to yield several potential benefits such as enhanced modularization, easier evolution, increased reusability, and improved readability and understandability of models, as well as reduced modeling effort. However, credible, solid empirical evidence of such benefits is very rare. In this paper, we evaluate the understandability of state machines, when modeling crosscutting behavior using AOM and more specifically AspectSM, a UML profile extending the UML state machine notation to provide mechanisms to define aspects using state machines. With AspectSM, crosscutting behavior is modeled using so-called aspect state machines, which are woven into a base state machine (modeling core functionality) to produce a woven state machine. Modeling aspect state machines separately from the base state machine no doubt offers several abovementioned benefits but on the other hand poses extra challenges for a modeler to understand them because of implicit interactions between both aspect and base state machines. This paper reports a study, which was specifically designed to evaluate the understandability of aspect state machines. The understandability of aspect state machines in conjunction with a base state machine is evaluated by comparing woven state machines produced by subjects (subject woven state machines) with the woven state machines automatically generated by our weaver (reference woven state machines). Understandability is measured from the aspects of Completeness and Redundancy of a subject's woven state machine when compared with the corresponding reference woven state machine. Results of the study show that on average, we observed completeness of 71%, whereas we observed approximately 6% of redundancy. We also observed that subjects took significantly more time to weave more complex aspect state machines (with more states, transitions, and pointcuts). © 2012 IEEE.

Fraser G.,University of Sheffield | Arcuri A.,Certus Software Vandnter
Proceedings - IEEE 6th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2013 | Year: 2013

EvoSuite is a mature research prototype that automatically generates unit tests for Java code. This paper summarizes the results and experiences in participating at the unit testing competition held at SBST 2013, where EvoSuite ranked first with a score of 156.95. © 2013 IEEE.

Arcuri A.,Certus Software Vandnter | Iqbal M.Z.,Certus Software Vandnter | Briand L.,Certus Software Vandnter
IEEE Transactions on Software Engineering | Year: 2012

A substantial amount of work has shed light on whether random testing is actually a useful testing technique. Despite its simplicity, several successful real-world applications have been reported in the literature. Although it is not going to solve all possible testing problems, random testing appears to be an essential tool in the hands of software testers. In this paper, we review and analyze the debate about random testing. Its benefits and drawbacks are discussed. Novel results addressing general questions about random testing are also presented, such as how long does random testing need, on average, to achieve testing targets (e.g., coverage), how does it scale, and how likely is it to yield similar results if we rerun it on the same testing problem (predictability). Due to its simplicity that makes the mathematical analysis of random testing tractable, we provide precise and rigorous answers to these questions. Results show that there are practical situations in which random testing is a viable option. Our theorems are backed up by simulations and we show how they can be applied to most types of software and testing criteria. In light of these results, we then assess the validity of empirical analyzes reported in the literature and derive guidelines for both practitioners and scientists. © 2012 IEEE.

Wang S.,Certus Software Vandnter | Ali S.,Certus Software Vandnter | Gotlieb A.,Certus Software Vandnter
CEUR Workshop Proceedings | Year: 2013

Testing products in a cost-efficient way remains an attractive topic for Model-Based Testing (MBT) of product lines in both academia and industry, which can be addressed by employing systematic and automated approaches based on models (such as feature models and UML models). Cost-effective testing products can be divided into three main problems, i.e., test selection, test generation, and test minimization. Driven by the needs of our industrial problems for testing Video Conferencing Systems (VCSs) product line developed by Cisco, Norway, this paper presents Product Line Model-based Testing Methodologies (PL-MTM) to tackle the above-mentioned three problems for cost-effective testing a product in product line, which includes: 1) an systematic and automated test selection methodology; 2) an automated test minimization approach; and 3) an automated and systematic test generation methodology.

Loading Certus Software Vandnter collaborators
Loading Certus Software Vandnter collaborators