Uma abordagem sistemática para implementação, gerenciamento e customização de testes de linhas de produto de software

Through the adoption of the software product line (SPL) approach, several benefits are achieved when compared to the conventional development processes that are based on creating a single software system at a time. The process of developing a SPL differs from traditional software construction, since it has two essential phases: the domain engineering - when common and variables elements of the SPL are defined and implemented; and the application engineering - when one or more applications (specific products) are derived from the reuse of artifacts created in the domain engineering. The test activity is also fundamental and aims to detect defects in the artifacts produced in SPL development. However, the characteristics of an SPL bring new challenges to this activity that must be considered. Several approaches have been recently proposed for the testing process of product lines, but they have been shown limited and have only provided general guidelines. In addition, there is also a lack of tools to support the variability management and customization of automated case tests for SPLs. In this context, this dissertation has the goal of proposing a systematic approach to software product line testing. The approach offers: (i) automated SPL test strategies to be applied in the domain and application engineering, (ii) explicit guidelines to support the implementation and reuse of automated test cases at the unit, integration and system levels in domain and application engineering; and (iii) tooling support for automating the variability management and customization of test cases. The approach is evaluated through its application in a software product line for web systems. The results of this work have shown that the proposed approach can help the developers to deal with the challenges imposed by the characteristics of SPLs during the testing process

COLLABORATIVE TESTING ACROSS SHARED SOFTWARE COMPONENTS

Large component-based systems are often built from many of the same components. As individual component-based software systems are developed, tested and maintained, these shared components are repeatedly manipulated. As a result there are often significant overlaps and synergies across and among the different test efforts of different component-based systems. However, in practice, testers of different systems rarely collaborate, taking a test-all-by-yourself approach. As a result, redundant effort is spent testing common components, and important information that could be used to improve testing quality is lost. The goal of this research is to demonstrate that, if done properly, testers of shared software components can save effort by avoiding redundant work, and can improve the test effectiveness for each component as well as for each component-based software system by using information obtained when testing across multiple components. To achieve this goal I have developed collaborative testing techniques and tools for developers and testers of component-based systems with shared components, applied the techniques to subject systems, and evaluated the cost and effectiveness of applying the techniques. The dissertation research is organized in three parts. First, I investigated current testing practices for component-based software systems to find the testing overlap and synergy we conjectured exists. Second, I designed and implemented infrastructure and related tools to facilitate communication and data sharing between testers. Third, I designed two testing processes to implement different collaborative testing algorithms and applied them to large actively developed software systems. This dissertation has shown the benefits of collaborative testing across component developers who share their components. With collaborative testing, researchers can design algorithms and tools to support collaboration processes, achieve better efficiency in testing configurations, and discover inter-component compatibility faults within a minimal time window after they are introduced.

A Binary Classifier for Test Case Feasibility Applied to Automatically Generated Tests of Event-Driven Software

Modern software application testing, such as the testing of software driven by graphical user interfaces (GUIs) or leveraging event-driven architectures in general, requires paying careful attention to context. Model-based testing (MBT) approaches first acquire a model of an application, then use the model to construct test cases covering relevant contexts. A major shortcoming of state-of-the-art automated model-based testing is that many test cases proposed by the model are not actually executable. These \textit{infeasible} test cases threaten the integrity of the entire model-based suite, and any coverage of contexts the suite aims to provide. In this research, I develop and evaluate a novel approach for classifying the feasibility of test cases. I identify a set of pertinent features for the classifier, and develop novel methods for extracting these features from the outputs of MBT tools. I use a supervised logistic regression approach to obtain a model of test case feasibility from a randomly selected training suite of test cases. I evaluate this approach with a set of experiments. The outcomes of this investigation are as follows: I confirm that infeasibility is prevalent in MBT, even for test suites designed to cover a relatively small number of unique contexts. I confirm that the frequency of infeasibility varies widely across applications. I develop and train a binary classifier for feasibility with average overall error, false positive, and false negative rates under 5\%. I find that unique event IDs are key features of the feasibility classifier, while model-specific event types are not. I construct three types of features from the event IDs associated with test cases, and evaluate the relative effectiveness of each within the classifier. To support this study, I also develop a number of tools and infrastructure components for scalable execution of automated jobs, which use state-of-the-art container and continuous integration technologies to enable parallel test execution and the persistence of all experimental artifacts.

Estudo e implementação de testes de software em desenvolvimento ágil

Dissertação de mestrado em Engenharia de Sistemas

Computer integrated testing for the transport refrigeration systems industry.

This is a study of a state of the art implementation of a new computer integrated testing (CIT) facility within a company that designs and manufactures transport refrigeration systems. The aim was to use state of the art hardware, software and planning procedures in the design and implementation of three CIT systems. Typical CIT system components include data acquisition (DAQ) equipment, application and analysis software, communication devices, computer-based instrumentation and computer technology. It is shown that the introduction of computer technology into the area of testing can have a major effect on such issues as efficiency, flexibility, data accuracy, test quality, data integrity and much more. Findings reaffirm how the overall area of computer integration continues to benefit any organisation, but with more recent advances in computer technology, communication methods and software capabilities, less expensive more sophisticated test solutions are now possible. This allows more organisations to benefit from the many advantages associated with CIT. Examples of computer integration test set-ups and the benefits associated with computer integration have been discussed.

3G SGSN Toiminnallisuustestauksen Automatisointi

3G SGSN (3rd Generation Serving GPRS Support Node) onUMTS (Universal Mobile Telecommunications System) pakettiverkon verkkoelementti, joka toimii linkkinä ulkoisen verkon(kuten Internet) ja radioverkon välillä. 3G SGSN ylläpitää tilaajan tietoja ja paikannustietoja päätelaitteen liikkuessa verkon sisällä ja välittää myös dataa ulkoisen verkon ja mobiilipäätelaitteen välillä. Tämän diplomityön aiheena on 3G SGSN toiminnallisuus-testauksen automatisointi. Työssä kehitetään 3G SGSN toiminnallisuustestaukseen soveltuva testauksen automatisointijärjestelmä Linux-ympäristössä AuTS (Automated Test Sequences)-automatisointityökalulla. Kehitystyöhön kuuluu järjestelmän suunnittelu ja toteutus. Toteutettu automatisointijärjestelmä suorittaa 3G SGSN toiminnallisuustestauksen istunnon- ja liikkuvuudenhallinnan testitapauksia.Suoritettujen testitapausten tuloksista luodaan HTML (Hypertext Markup Language)-raportti, joka sisältää tiedot ja analyysit suoritetuista testitapauksista sekä linkit testitapausten lokitietoihin. Hyvä automatisointi vaatii suunnittelua, jossa huomioidaan käytettävyys ja ylläpidettävyys. Näihin ominaisuuksiin on toteutuksessa kiinnitetty erityistä huomiota. Lopputuloksena saatiin toimiva automatisointijärjestelmä, joka osoittaa automatisoinnin tuovan huomattavia etuja toiminnallisuustestauksen tehokkuuteen. Työssä käsitelläänmyös ohjelmistotestausta sekä testauksen automatisointia ja niiden periaatteita.

Unlicensed Mobile Access Concept from Testing Point of View

Viime vuosikymmenien aikana kommunikaatioteknologiat ovat kehittyneet erittäin paljon. Uusia verkkoja, liityntätekniikoita, protokollia ja päätelaitteita on luotu alati kehittyvällä vauhdilla, eikä hidastumisen merkkejä ole näkyvissä. Varsinkin mobiilisovellukset ovat kasvattaneet markkinaosuuksiaan viime aikoina. Unlicensed MobileAccess (UMA) on uusi liityntätekniikka mobiilipäätelaitteille, joka mahdollistaa liitynnän GSM- runkoverkkoon WLAN- tai Bluetooth - tekniikoiden avulla. Tämä diplomityö keskittyy UMAan liittyviin teknologioihin, joita tarkastellaan lähemmin ensimmäisissä kappaleissa. Tavoitteena on esitellä, mitä UMA merkitsee, ja kuinka eri tekniikoita voidaan soveltaa sen toteutuksissa. Ennenkuin uusia teknologioita voidaan soveltaa kaupallisesti, täytyy niiden olla kokonaisvaltaisesti testattuja. Erilaisia testausmenetelmiä sovelletaan laitteistonja ohjelmiston testaukseen, mutta tavoite on kuitenkin sama, eli vähentää testattavan tuotteen epäluotettavuutta ja lisätä sen laatua. Vaikka UMA käsittääkin pääasiassa jo olemassa olevia tekniikoita, tuo se silti mukanaan uuden verkkoelementin ja kaksi uutta kommunikaatioprotokollaa. Ennen kuin mitään UMAa tukevia ratkaisuja voidaan tuoda markkinoille, monia erilaisia testausmenetelmiä on suoritettava, jotta varmistutaan uuden tuotteen oikeasta toiminnallisuudesta. Koska tämä diplomityö käsittelee uutta tekniikkaa, on myös testausmenetelmien yleisen testausteorian käsittelemiselle varattu oma kappale. Kappale esittelee erilaisia testauksen näkökulmia ja niihin perustuen rakennetaan myös testausohjelmisto. Tavoitteena on luoda ohjelmisto, jota voidaan käyttää UMA-RR protokollan toiminnan varmentamiseen kohdeympäristössä.

Using time and stochastic models for software reliability forecasting and analysis

Tämä työ luo katsauksen ajallisiin ja stokastisiin ohjelmien luotettavuus malleihin sekä tutkii muutamia malleja käytännössä. Työn teoriaosuus sisältää ohjelmien luotettavuuden kuvauksessa ja arvioinnissa käytetyt keskeiset määritelmät ja metriikan sekä varsinaiset mallien kuvaukset. Työssä esitellään kaksi ohjelmien luotettavuusryhmää. Ensimmäinen ryhmä ovat riskiin perustuvat mallit. Toinen ryhmä käsittää virheiden ”kylvöön” ja merkitsevyyteen perustuvat mallit. Työn empiirinen osa sisältää kokeiden kuvaukset ja tulokset. Kokeet suoritettiin käyttämällä kolmea ensimmäiseen ryhmään kuuluvaa mallia: Jelinski-Moranda mallia, ensimmäistä geometrista mallia sekä yksinkertaista eksponenttimallia. Kokeiden tarkoituksena oli tutkia, kuinka syötetyn datan distribuutio vaikuttaa mallien toimivuuteen sekä kuinka herkkiä mallit ovat syötetyn datan määrän muutoksille. Jelinski-Moranda malli osoittautui herkimmäksi distribuutiolle konvergaatio-ongelmien vuoksi, ensimmäinen geometrinen malli herkimmäksi datan määrän muutoksille.

Implementation of counter testing in Nokia Automation System

In this study the performance measurement, a part of the research and development of the RNC, was improved by implementing counter testing to the Nokia Automation System. The automation of counter testing is a feature the customer ordered, because performing counter testing manually is rather complex. The objective was to implement an automated counter testing system, which once configured correctly, would manage to run the testing and perform the analysis. The requirements for the counter testing were first studied. It was investigated if the auto-mation of the feature was feasible in the meetings with the customer. The basic functionality required for the automation was also drawn. The technologies used in the architecture of the Nokia Automation System were studied. Based on the results of the study, a new technology, wxWidgets, was introduced. The new technology was necessary to facilitate the implementing of the required feature. Finally the implementation of the counter testing was defined and implemented. The result of this study was the automation of the counter testing method developed as a new feature for the Nokia Automation System. The feature meets the specifications and requirements set by the customer. The performing of the counter testing feature is totally automated. Only configuration of the test cases is done by the user. The customer has presented new requests to further develop the feature and there are plans by the Nokia Automation System developers to implement those in the near future. The study describes the implementation of the counter testing feature introduced. The results of the study give guidelines for further developing the feature.

Evaluation of Software Development Practices through Error Analysis

This thesis studies evaluation of software development practices through an error analysis. The work presents software development process, software testing, software errors, error classification and software process improvement methods. The practical part of the work presents results from the error analysis of one software process. It also gives improvement ideas for the project. It was noticed that the classification of the error data was inadequate in the project. Because of this it was impossible to use the error data effectively. With the error analysis we were able to show that there were deficiencies in design and analyzing phases, implementation phase and in testing phase. The work gives ideas for improving error classification and for software development practices.

WWW-pohjaisen sovelluksen automatisoitu testaus ja sitä edistävät ohjelmointikäytännöt

Ohjelmistotestauksen avulla voidaan tarkastella sovelluksen vastaavuutta vaatimuksiin. Tavoitteena on löytää sovelluksesta virheitä, ja siten parantaa sovelluksen laatua. Sovelluksen laatu voidaan määritellä useilla mittareilla, kuten esimerkiksi testattavuudella. Tässä työssä tarkastellaan WWW-sovelluksen automatisoidun testauksen toteutusta, jossa käytetään apuna testauskehystä. Automatisoituun testaukseen kuuluu testitapausten suunnittelu sekä toteutus, joiden lopputuloksena on uudelleenajettavia testitapauksia. Testaus keskittyy sovelluksen toiminnallisuuteen ja jättää tietokantaan päivitettävien tietojen tarkastamisen tekemättä. Testaus suoritetaan ilman tarkempaa tietoa sovelluksen sisäisestä toiminnasta. Testattava sovellus on Mobilding-hankkeessa toteutettu WWW-sovellus, jonka avulla hallinnoidaan rakennuksen elementtejä. Työssä vertaillaan WWW-sovelluksen käyttöliittymän testaukseen soveltuvia testauskehyksiä, ja pyritään tuomaan esille niiden ominaispiirteitä. Työn tuloksena on uudelleensuoritettavia testitapahtumia. Lisäksi pohditaan ohjelmointikäytäntöjä, joilla voidaan edistää automatisoitua testausta. Ohjelmointikäytännöt perustuvat työn toteutuksen aikana havaittuihin ongelmiin.

Ketterien sovelluskehityskäytäntöjen soveltaminen J2EE-kehitykseen

Yritysohjelmistojen kehittäminen on monesti aikataulukriittistä ja tuottavuusvaatimukset ovat korkeita. Työssä kehitettiin tyypillisen web-käyttöliittymän omaavan J2EE-sovelluksen koontiin soveltuva koontimenetelmä, joka hyödyntää XP:n (Extreme Programming) sovelluskehityskäytäntöjä. Tärkeimpiä näistä on automatisoitu testaus, jonka päätyypit ovat automatisoitu yksikkö- ja integrointitestaus. Työn aikana asetettiin tavoitteet koontimenetelmälle sekä evaluointiin ja valittiin markkinoilla olevista työkaluista toimeksiantajan ympäristöön sopiva joukko. Koontimenetelmän toteutuksen tarkastelussa havaittiin, että XP:n käytännöt soveltuvat J2EE-sovelluskehitykseen ja tarpeelliset työkalut tähän ovat olemassa.

Planning and ramping up a new SW Testing site

During recent years, mobile phone markets have changed significantly. Asian markets have become vital for the manufacturers with their millions of end users and multiple major mobile network operators. This has resulted in software development as global companies have research and development sites running in multiple locations, including Asia. The reasons behind this are not only in reducing labor costs but also in capitalizing on the local knowledge and knowhow. A ramp-up site has multiple effects in the software development and software release activities. This thesis focuses on representing the importance of software testing as part of software development process and highlighting issues that need to be considered during ramp-up activities. In addition this work tries to emphasize the importance of communication between parties and information gathering prior to setting up the ramp-up site. The output of this thesis was successful software testing site ramp-up within the set time limits. The quality of software testing work was assured and the ramp-up -project requirements were achieved.

Test automation scheme for LTE core network element

Modern sophisticated telecommunication devices require even more and more comprehensive testing to ensure quality. The test case amount to ensure well enough coverage of testing has increased rapidly and this increased demand cannot be fulfilled anymore only by using manual testing. Also new agile development models require execution of all test cases with every iteration. This has lead manufactures to use test automation more than ever to achieve adequate testing coverage and quality. This thesis is separated into three parts. Evolution of cellular networks is presented at the beginning of the first part. Also software testing, test automation and the influence of development model for testing are examined in the first part. The second part describes a process which was used to implement test automation scheme for functional testing of LTE core network MME element. In implementation of the test automation scheme agile development models and Robot Framework test automation tool were used. In the third part two alternative models are presented for integrating this test automation scheme as part of a continuous integration process. As a result, the test automation scheme for functional testing was implemented. Almost all new functional level testing test cases can now be automated with this scheme. In addition, two models for integrating this scheme to be part of a wider continuous integration pipe were introduced. Also shift from usage of a traditional waterfall model to a new agile development based model in testing stated to be successful.