A Survey and Classification of Software Testing Tools

Software testing is one of the essential parts in software engineering process. The objective of the study was to describe software testing tools and the corresponding use. The thesis contains examples of software testing tools usage. The study was conducted as a literature study, with focus on current software testing practices and quality assurance standards. In the paper a tool classifier was employed, and testing tools presented in study were classified according to it. We found that it is difficult to distinguish current available tools by certain testing activities as many of them contain functionality that exceeds scopes of a single testing type.

Testing of Android testing tools: development of a benchmark for the evaluation

With the ever growing trend of smart phones and tablets, Android is becoming more and more popular everyday. With more than one billion active users i to date, Android is the leading technology in smart phone arena. In addition to that, Android also runs on Android TV, Android smart watches and cars. Therefore, in recent years, Android applications have become one of the major development sectors in software industry. As of mid 2013, the number of published applications on Google Play had exceeded one million and the cumulative number of downloads was more than 50 billionii. A 2013 survey also revealed that 71% of the mobile application developers work on developing Android applicationsiii. Considering this size of Android applications, it is quite evident that people rely on these applications on a daily basis for the completion of simple tasks like keeping track of weather to rather complex tasks like managing one’s bank accounts. Hence, like every other kind of code, Android code also needs to be verified in order to work properly and achieve a certain confidence level. Because of the gigantic size of the number of applications, it becomes really hard to manually test Android applications specially when it has to be verified for various versions of the OS and also, various device configurations such as different screen sizes and different hardware availability. Hence, recently there has been a lot of work on developing different testing methods for Android applications in Computer Science fraternity. The model of Android attracts researchers because of its open source nature. It makes the whole research model more streamlined when the code for both, application and the platform are readily available to analyze. And hence, there has been a great deal of research in testing and static analysis of Android applications. A great deal of this research has been focused on the input test generation for Android applications. Hence, there are a several testing tools available now, which focus on automatic generation of test cases for Android applications. These tools differ with one another on the basis of their strategies and heuristics used for this generation of test cases. But there is still very little work done on the comparison of these testing tools and the strategies they use. Recently, some research work has been carried outiv in this regard that compared the performance of various available tools with respect to their respective code coverage, fault detection, ability to work on multiple platforms and their ease of use. It was done, by running these tools on a total of 60 real world Android applications. The results of this research showed that although effective, these strategies being used by the tools, also face limitations and hence, have room for improvement. The purpose of this thesis is to extend this research into a more specific and attribute-­‐ oriented way. Attributes refer to the tasks that can be completed using the Android platform. It can be anything ranging from a basic system call for receiving an SMS to more complex tasks like sending the user to another application from the current one. The idea is to develop a benchmark for Android testing tools, which is based on the performance related to these attributes. This will allow the comparison of these tools with respect to these attributes. For example, if there is an application that plays some audio file, will the testing tool be able to generate a test input that will warrant the execution of this audio file? Using multiple applications using different attributes, it can be visualized that which testing tool is more useful for which kinds of attributes. In this thesis, it was decided that 9 attributes covering the basic nature of tasks, will be targeted for the assessment of three testing tools. Later this can be done for much more attributes to compare even more testing tools. The aim of this work is to show that this approach is effective and can be used on a much larger scale. One of the flagship features of this work, which also differentiates it with the previous work, is that the applications used, are all specially made for this research. The reason for doing that is to analyze just that specific attribute in isolation, which the application is focused on, and not allow the tool to get bottlenecked by something trivial, which is not the main attribute under testing. This means 9 applications, each focused on one specific attribute. The main contributions of this thesis are: A summary of the three existing testing tools and their respective techniques for automatic test input generation of Android Applications. • A detailed study of the usage of these testing tools using the 9 applications specially designed and developed for this study. • The analysis of the obtained results of the study carried out. And a comparison of the performance of the selected tools.

Dicto: A Unified DSL for Testing Architectural Rules

Software architecture consists of a set of design choices that can be partially expressed in form of rules that the implementation must conform to. Architectural rules are intended to ensure properties that fulfill fundamental non-functional requirements. Verifying architectural rules is often a non- trivial activity: available tools are often not very usable and support only a narrow subset of the rules that are commonly specified by practitioners. In this paper we present a new highly-readable declarative language for specifying architectural rules. With our approach, users can specify a wide variety of rules using a single uniform notation. Rules can get tested by third-party tools by conforming to pre-defined specification templates. Practitioners can take advantage of the capabilities of a growing number of testing tools without dealing with them directly.

A Framework for Design-Time Testing of Service-Based Applications at BPEL Level

Software applications created on top of the service-oriented architecture (SOA) are increasingly popular but testing them remains a challenge. In this paper a framework named TASSA for testing the functional and non-functional behaviour of service-based applications is presented. The paper focuses on the concept of design time testing, the corresponding testing approach and architectural integration of the consisting TASSA tools. The individual TASSA tools with sample validation scenarios were already presented with a general view of their relation. This paper’s contribution is the structured testing approach, based on the integral use of the tools and their architectural integration. The framework is based on SOA principles and is composable depending on user requirements.

A self -testing approach for autonomic software

As researchers and practitioners move towards a vision of software systems that configure, optimize, protect, and heal themselves, they must also consider the implications of such self-management activities on software reliability. Autonomic computing (AC) describes a new generation of software systems that are characterized by dynamically adaptive self-management features. During dynamic adaptation, autonomic systems modify their own structure and/or behavior in response to environmental changes. Adaptation can result in new system configurations and capabilities, which need to be validated at runtime to prevent costly system failures. However, although the pioneers of AC recognize that validating autonomic systems is critical to the success of the paradigm, the architectural blueprint for AC does not provide a workflow or supporting design models for runtime testing. ^ This dissertation presents a novel approach for seamlessly integrating runtime testing into autonomic software. The approach introduces an implicit self-test feature into autonomic software by tailoring the existing self-management infrastructure to runtime testing. Autonomic self-testing facilitates activities such as test execution, code coverage analysis, timed test performance, and post-test evaluation. In addition, the approach is supported by automated testing tools, and a detailed design methodology. A case study that incorporates self-testing into three autonomic applications is also presented. The findings of the study reveal that autonomic self-testing provides a flexible approach for building safe, reliable autonomic software, while limiting the development and performance overhead through software reuse. ^

EMMON - EMbedded MONitoring

Despite the steady increase in experimental deployments, most of research work on WSNs has focused only on communication protocols and algorithms, with a clear lack of effective, feasible and usable system architectures, integrated in a modular platform able to address both functional and non–functional requirements. In this paper, we outline EMMON [1], a full WSN-based system architecture for large–scale, dense and real–time embedded monitoring [3] applications. EMMON provides a hierarchical communication architecture together with integrated middleware and command and control software. Then, EM-Set, the EMMON engineering toolset will be presented. EM-Set includes a network deployment planning, worst–case analysis and dimensioning, protocol simulation and automatic remote programming and hardware testing tools. This toolset was crucial for the development of EMMON which was designed to use standard commercially available technologies, while maintaining as much flexibility as possible to meet specific applications requirements. Finally, the EMMON architecture has been validated through extensive simulation and experimental evaluation, including a 300+ nodes testbed.

The dark side of DEMMON: what is behind the scene in engineering large-scale wireless sensor networks

Most research work on WSNs has focused on protocols or on specific applications. There is a clear lack of easy/ready-to-use WSN technologies and tools for planning, implementing, testing and commissioning WSN systems in an integrated fashion. While there exists a plethora of papers about network planning and deployment methodologies, to the best of our knowledge none of them helps the designer to match coverage requirements with network performance evaluation. In this paper we aim at filling this gap by presenting an unified toolset, i.e., a framework able to provide a global picture of the system, from the network deployment planning to system test and validation. This toolset has been designed to back up the EMMON WSN system architecture for large-scale, dense, real-time embedded monitoring. It includes network deployment planning, worst-case analysis and dimensioning, protocol simulation and automatic remote programming and hardware testing tools. This toolset has been paramount to validate the system architecture through DEMMON1, the first EMMON demonstrator, i.e., a 300+ node test-bed, which is, to the best of our knowledge, the largest single-site WSN test-bed in Europe to date.

Caso de demonstração de uma framework para automatização do teste de APIs de aplicações SaaS

Dissertação de mestrado integrado em Engenharia e Gestão de Sistemas de Informação

ATS3-testausjärjestelmän evaluointi

Työ tehtiin Nokia Technology Platforms S60 Mobile Runtime-yksikölle. Työn tavoitteena oli evaluoida ATS3-testausjärjestelmä. ATS3-järjestelmällä voidaan automatisoida ohjelmistotestausta. Evaluoinnilla pyrittiin selvittämään voidaanko tuotekehitysvaiheessa oleva järjestelmä ottaa jo käyttöön vai vaatiiko se vielä lisäkehitystä. Työssä kerrotaan aluksi yleisesti ohjelmistotestauksesta. Siinä käydään läpi myös testausprosessin vaiheet sekä kerrotaan lyhyesti avuksi kehitetyistä työkaluista. Työssä kerrotaan myös ATS3-järjestelmän edeltäjien ominaisuuksista, sekä miten ne eroavat tarkastelun kohteena olevasta järjestelmästä. Varsinainen evaluointi alkoi tutustumalla ja asentamalla järjestelmä. Tämän jälkeen suoritettiin samat testit sekä ATS2.x-järjestelmässä että ATS3-järjestelmäss. Testauksen jälkeen analysoitiin testitulosten yhdenmukaisuuden perusteella, voidaanko siirtyä käyttömään ATS3-järjestelmää. Testitulokset olivat käytännässä yhdenmukaiset suoritettaessa testejä matkapuhelimessa, mutta emulaattoritestaus täytyi jättää väliin. Lukuisista yrityksistä sekä järjestelmän kehittäjän avusta huolimatta testejä ei onnistuttu suorittamaan ATS3-järjestelmän emulaattoriympäristössä. Näin ollen päädyttiin tulokseen, ettei järjestelmää voida ottaa vielä käytöön emulaattoritestauksessa, vaan se vaatii lisätutkimusta sekä perehtymistä järjestelmään.

Design Guide for Improved Quality of Roadway Subgrades and Subbases, TR-525, 2008

The performance of a pavement depends on the quality of its subgrade and subbase layers; these foundational layers play a key role in mitigating the effects of climate and the stresses generated by traffic. Therefore, building a stable subgrade and a properly drained subbase is vital for constructing an effective and long lasting pavement system. This manual has been developed to help Iowa highway engineers improve the design, construction, and testing of a pavement system’s subgrade and subbase layers, thereby extending pavement life. The manual synthesizes current and previous research conducted in Iowa and other states into a practical geotechnical design guide [proposed as Chapter 6 of the Statewide Urban Design and Specifications (SUDAS) Design Manual] and construction specifications (proposed as Section 2010 of the SUDAS Standard Specifications) for subgrades and subbases. Topics covered include the important characteristics of Iowa soils, the key parameters and field properties of optimum foundations, embankment construction, geotechnical treatments, drainage systems, and field testing tools, among others.

Ohjelmistotestauksen markkinatutkimus

Tutkimuksen tavoitteena oli selvittää ohjelmistotestauksen tarve ohjelmistoliiketoiminnan yrityksissä. Ensin tutkittiin ohjelmistotestauksen sekä markkinatutkimuksen teoriaa, kirjallisuuden ja lehtiartikkelien avulla. Markkinatutkimus toteutettiin kirjekyselynä. Erityistä huomiota kiinnitettiin testauksen ulkoistamiseen, yritysten haasteisiin sekä tarpeisiin liittyen ohjelmistokehitykseen ja –testaukseen. Ohjelmistotestauksen avulla saavutettava laatu on olennainen osa suomalaista ohjelmistoliiketoiminnan kilpailukykyä ja se on keino erottua edukseen muista ohjelmistotuotteista kilpailtaessa samoista asiakkaista kansainvälisillä markkinoilla. Markkinatutkimuksessa saatiin merkittäviä tuloksia ohjelmistotestausmarkkinoista, yritysten sisäisestä sekä ulkoisesta ohjelmistotestauksesta ja testaustyökalujen käytöstä. Tutkimuksissa selvisi, että yritysten suurimpana haasteena oli ohjelmistokehitys- ja testausprosessien kehittäminen. Tätä aihetta on Suomessa tutkinut Teknillinen korkeakoulusta SoberIT-yksikkö ja he ovat kehittäneet SEMS-ohjelmistotuotannon ohjausjärjestelmän pk-ohjelmistotuoteyrityksille. SoftaTest aikoo hyödyntää näitä tutkimustuloksia mm. perustamalla liiketoimintaan tähtäävän projektin vahvistamaan suomalaisten ohjelmistotuoteyritysten menestymistä markkinoilla.

Kuormitustestausvälineen valinta Microsoft Office SharePoint Server 2007 – toteutuksessa

Kuormitustestaus on osa web-sovellusten kehitystä ja käyttöönottoa. Sillä varmistetaan sovellusten toimivuus ennalta määrätyn kuorman alla. Microsoft Office SharePoint Server 2007 (MOSS) on palvelintuote nykyaikaisten web-sovellusten luontiin ja ylläpitoon. Työssä vertaillaan kahta eri uormitustestaustyökalua: SilkPerformer 2008 ja Visual Studio Team System 2008 Test Edition ja valitaan MOSS – web-sovelluksille paremmin sopiva työkalu. Työssä vertaillaan työkaluja niiden ominaisuuksien perusteella sekä suorittamalla kuormitustestausta testausta varten luodulle MOSS – web-sovellukselle. Vaikuttavien tekijöiden perusteella työkaluja arvioidaan ja tämän perusteella saadaan tulos vertailulle. Työn tuloksena Visual Studio Team System 2008 Test Edition sopii paremmin MOSS – web-sovelluksen kuormitustestausvälineeksi. Vertailussa kuitenkin havaittiin, että työkalut ovat melko tasavertaisia, ja käytännön tilanteesta riippuu, kumpi sopii paremmin. Tämä työ auttaa valinnan teossa.

Testverktyg för test av mobila applikationer : En jämförande fallstudie

Användandet av mobila applikationer har växt radikalt de senaste åren och de samverkar med många system. Därför ställs det högre krav på kvaliteten och att applikationen ska anpassas till många olika enheter, operativsystem samt plattformar. Detta gör att test av mobila applikationer blivit viktigare och större. Detta arbete har bedrivits som en jämförande fallstudie inom området test av mobila applikationer samt testverktyg. Syftet har varit att beskriva hur testning av mobila applikationer sker idag vilket gjorts genom litteraturstudier och intervjuer med IT-företag. Ett annat syfte har varit att utvärdera fyra testverktyg, deras för- och nackdelar samt hur de kan användas vid testning av mobila applikationer och jämföras mot manuell testning utan testverktyg. Detta har gjorts genom att skapa förstahandserfarenheter baserat på användandet av testverktygen. Under arbetet har vi utgått från mobila applikationer som vi fått tillgång till av Triona, som varit vår samarbetspartner.Idag finns många olika testverktyg som kan användas som stöd för testningen men få företag har implementerat något eftersom det kräver både tid och kompetens samt valet av testverktyg kan vara svårt. Testverktygen har olika för- och nackdelar vilket gör att de passar olika bra beroende på typ av projekt och applikation. Fördelar med att använda testverktyg är möjligheten att kunna automatisera, testa på flera enheter samtidigt samt få tillgång till enheter via molnet. Utmaningarna är att det kan vara svårt att installera och lära sig testverktyget samt att licenserna kan vara dyra. Det är därför viktigt att redan innan implementationen veta vilka tester och applikationer testverktygen ska användas till samt vem som ska använda det. Utifrån vår studie kan slutsatsen dras att inget testverktyg är helt komplett men de kan bidra med olika funktioner vilket effektiviserar delar av testningen av mobila applikationer.

The resistivity piezocone for high resolution geo-environmental site investigation: A Brazilian example

A suit able decision-making on managing a contaminated site characterization program is strongly dependent of the diagnosis process. A detailed diagnosis can be done based on a Conceptual Site Model (CSM) elaboration using high resolution site characterization tools. The piezocone (CPTu) test is a high resolution tool which allows attaching several specific sensors, like the resistivity probe. This hybrid device is called the resistivity piezocone (RCPTu). A simulated geo-environmental site characterization program was performed on an erosion site using different tools (direct push tools soil samplers, hollow stem auger (HSA) drilling and RCPTu tests) to develop the CSM for a site similar to the Brazilian conditions. It was observed a good agreement between the site profiles interpreted by the different methods. The resistivity sensor attached to the piezocone improved the interpretation and the decision-making process on site was significantly better for the CSM elaboration. The RCPTu test data also allowed identifying the hydrogeological heterogeneities. The present study shows that the RCPTu test is also a useful and powerful tool to development an accurate CSM in a Brazilian condition, especially in an approach that prioritizes high resolution geo-environmental investigation. © 2013 Taylor & Francis Group.