The Risk of Using the Q Heterogeneity Estimator for Software Engineering Experiments

All meta-analyses should include a heterogeneity analysis. Even so, it is not easy to decide whether a set of studies are homogeneous or heterogeneous because of the low statistical power of the statistics used (usually the Q test). Objective: Determine a set of rules enabling SE researchers to find out, based on the characteristics of the experiments to be aggregated, whether or not it is feasible to accurately detect heterogeneity. Method: Evaluate the statistical power of heterogeneity detection methods using a Monte Carlo simulation process. Results: The Q test is not powerful when the meta-analysis contains up to a total of about 200 experimental subjects and the effect size difference is less than 1. Conclusions: The Q test cannot be used as a decision-making criterion for meta-analysis in small sample settings like SE. Random effects models should be used instead of fixed effects models. Caution should be exercised when applying Q test-mediated decomposition into subgroups.

Quantitative Determination of the Relationship between Internal Validity and Bias in Software Engineering Experiments: Consequences for Systematic Literature Reviews

Quality assessment is one of the activities performed as part of systematic literature reviews. It is commonly accepted that a good quality experiment is bias free. Bias is considered to be related to internal validity (e.g., how adequately the experiment is planned, executed and analysed). Quality assessment is usually conducted using checklists and quality scales. It has not yet been proven;however, that quality is related to experimental bias. Aim: Identify whether there is a relationship between internal validity and bias in software engineering experiments. Method: We built a quality scale to determine the quality of the studies, which we applied to 28 experiments included in two systematic literature reviews. We proposed an objective indicator of experimental bias, which we applied to the same 28 experiments. Finally, we analysed the correlations between the quality scores and the proposed measure of bias. Results: We failed to find a relationship between the global quality score (resulting from the quality scale) and bias; however, we did identify interesting correlations between bias and some particular aspects of internal validity measured by the instrument. Conclusions: There is an empirically provable relationship between internal validity and bias. It is feasible to apply quality assessment in systematic literature reviews, subject to limits on the internal validity aspects for consideration.

Configuration management and product lines to enhance the replication process in software engineering

This research is concerned with the experimental software engineering area, specifically experiment replication. Replication has traditionally been viewed as a complex task in software engineering. This is possibly due to the present immaturity of the experimental paradigm applied to software development. Researchers usually use replication packages to replicate an experiment. However, replication packages are not the solution to all the information management problems that crop up when successive replications of an experiment accumulate. This research borrows ideas from the software configuration management and software product line paradigms to support the replication process. We believe that configuration management can help to manage and administer information from one replication to another: hypotheses, designs, data analysis, etc. The software product line paradigm can help to organize and manage any changes introduced into the experiment by each replication. We expect the union of the two paradigms in replication to improve the planning, design and execution of further replications and their alignment with existing replications. Additionally, this research work will contribute a web support environment for archiving information related to different experiment replications. Additionally, it will provide flexible enough information management support for running replications with different numbers and types of changes. Finally, it will afford massive storage of data from different replications. Experimenters working collaboratively on the same experiment must all have access to the different experiments.

Replication Data Management,needs and solutions: an initial evaluation of conceptual approaches for integrating heterogeneous replication study data

Replication Data Management (RDM) aims at enabling the use of data collections from several iterations of an experiment. However, there are several major challenges to RDM from integrating data models and data from empirical study infrastructures that were not designed to cooperate, e.g., data model variation of local data sources. [Objective] In this paper we analyze RDM needs and evaluate conceptual RDM approaches to support replication researchers. [Method] We adapted the ATAM evaluation process to (a) analyze RDM use cases and needs of empirical replication study research groups and (b) compare three conceptual approaches to address these RDM needs: central data repositories with a fixed data model, heterogeneous local repositories, and an empirical ecosystem. [Results] While the central and local approaches have major issues that are hard to resolve in practice, the empirical ecosystem allows bridging current gaps in RDM from heterogeneous data sources. [Conclusions] The empirical ecosystem approach should be explored in diverse empirical environments.

Definition of a support infrastructure for replicating and aggregating families of software engineering experiments

Experimental software engineering includes several processes, the most representative being run experiments, run replications and synthesize the results of multiple replications. Of these processes, only the first is relatively well established in software engineering. Problems of information management and communication among researchers are one of the obstacles to progress in the replication and synthesis processes. Software engineering experimentation has expanded considerably over the last few years. This has brought with it the invention of experimental process support proposals. However, few of these proposals provide integral support, including replication and synthesis processes. Most of the proposals focus on experiment execution. This paper proposes an infrastructure providing integral support for the experimental research process, specializing in the replication and synthesis of a family of experiments. The research has been divided into stages or phases, whose transition milestones are marked by the attainment of their goals. Each goal exactly matches an artifact or product. Within each stage, we will adopt cycles of successive approximations (generateand- test cycles), where each approximation includes a diferent viewpoint or input. Each cycle will end with the product approval.

Nuevas arquitecturas de transpondedores de microondas = New transponder architectures at microwave frequencies = Nouvelles architectures des transpondeurs d’hyperfréquences

En la última década, los sistemas de telecomunicación de alta frecuencia han evolucionado tremendamente. Las bandas de frecuencias, los anchos de banda del usuario, las técnicas de modulación y otras características eléctricas están en constante cambio de acuerdo a la evolución de la tecnología y la aparición de nuevas aplicaciones. Las arquitecturas de los transceptores modernos son diferentes de las tradicionales. Muchas de las funciones convencionalmente realizadas por circuitos analógicos han sido asignadas gradualmente a procesadores digitales de señal, de esta manera, las fronteras entre la banda base y las funcionalidades de RF se difuminan. Además, los transceptores inalámbricos digitales modernos son capaces de soportar protocolos de datos de alta velocidad, por lo que emplean una elevada escala de integración para muchos de los subsistemas que componen las diferentes etapas. Uno de los objetivos de este trabajo de investigación es realizar un estudio de las nuevas configuraciones en el desarrollo de demostradores de radiofrecuencia (un receptor y un transmisor) y transpondedores para fines de comunicaciones y militares, respectivamente. Algunos trabajos se han llevado a cabo en el marco del proyecto TECRAIL, donde se ha implementado un demostrador de la capa física LTE para evaluar la viabilidad del estándar LTE en el entorno ferroviario. En el ámbito militar y asociado al proyecto de calibración de radares (CALRADAR), se ha efectuado una actividad importante en el campo de la calibración de radares balísticos Doppler donde se ha analizado cuidadosamente su precisión y se ha desarrollado la unidad generadora de Doppler de un patrón electrónico para la calibración de estos radares. Dicha unidad Doppler es la responsable de la elevada resolución en frecuencia del generador de “blancos” radar construido. Por otro lado, se ha elaborado un análisis completo de las incertidumbres del sistema para optimizar el proceso de calibración. En una segunda fase se han propuesto soluciones en el desarrollo de dispositivos electro-ópticos para aplicaciones de comunicaciones. Estos dispositivos son considerados, debido a sus ventajas, tecnologías de soporte para futuros dispositivos y subsistemas de RF/microondas. Algunas demandas de radio definida por software podrían cubrirse aplicando nuevos conceptos de circuitos sintonizables mediante parámetros programables de un modo dinámico. También se ha realizado una contribución relacionada con el diseño de filtros paso banda con topología “Hairpin”, los cuales son compactos y se pueden integrar fácilmente en circuitos de microondas en una amplia gama de aplicaciones destinadas a las comunicaciones y a los sistemas militares. Como importante aportación final, se ha presentado una propuesta para ecualizar y mejorar las transmisiones de señales discretas de temporización entre los TRMs y otras unidades de procesamiento, en el satélite de última generación SEOSAR/PAZ. Tras un análisis exhaustivo, se ha obtenido la configuración óptima de los buses de transmisión de datos de alta velocidad basadas en una red de transceptores. ABSTRACT In the last decade, high-frequency telecommunications systems have extremely evolved. Frequency bands, user bandwidths, modulation techniques and other electrical characteristics of these systems are constantly changing following to the evolution of technology and the emergence of new applications. The architectures of modern transceivers are different from the traditional ones. Many of the functions conventionally performed by analog circuitry have gradually been assigned to digital signal processors. In this way, boundaries between baseband and RF functionalities are diffused. The design of modern digital wireless transceivers are capable of supporting high-speed data protocols. Therefore, a high integration scale is required for many of the components in the block chain. One of the goals of this research work is to investigate new configurations in the development of RF demonstrators (a receiver and a transmitter) and transponders for communications and military purposes, respectively. A LTE physical layer demonstrator has been implemented to assess the viability of LTE in railway scenario under the framework of the TECRAIL project. An important activity, related to the CALRADAR project, for the calibration of Doppler radars with extremely high precision has been performed. The contribution is the Doppler unit of the radar target generator developed that reveals a high frequency resolution. In order to assure the accuracy of radar calibration process, a complete analysis of the uncertainty in the above mentioned procedure has been carried out. Another important research topic has been the development of photonic devices that are considered enabling technologies for future RF and microwave devices and subsystems. Some Software Defined Radio demands are addressed by the proposed novel circuit concepts based on photonically tunable elements with dynamically programmable parameters. A small contribution has been made in the field of Hairpin-line bandpass filters. These filters are compact and can also be easily integrated into microwave circuits finding a wide range of applications in communication and military systems. In this research field, the contributions made have been the improvements in the design and the simulations of wideband filters. Finally, an important proposal to balance and enhance transmissions of discrete timing signals between TRMs and other processing units into the state of the art SEOSAR/PAZ Satellite has been carried out obtaining the optimal configuration of the high-speed data transmission buses based on a transceiver network. RÉSUMÉ Les systèmes d'hyperfréquence dédiés aux télécommunications ont beaucoup évolué dans la dernière décennie. Les bandes de fréquences, les bandes passantes par utilisateur, les techniques de modulation et d'autres caractéristiques électriques sont en constant changement en fonction de l'évolution des technologies et l'émergence de nouvelles applications. Les architectures modernes des transcepteurs sont différentes des traditionnelles. Un grand nombre d’opérations normalement effectuées par les circuits analogiques a été progressivement alloué à des processeurs de signaux numériques. Ainsi, les frontières entre la bande de base et la fonctionnalité RF sont floues. Les transcepteurs sans fils numériques modernes sont capables de transférer des données à haute vitesse selon les différents protocoles de communication utilisés. C'est pour cette raison qu’un niveau élevé d'intégration est nécessaire pour un grand nombre de composants qui constitue les différentes étapes des systèmes. L'un des objectifs de cette recherche est d'étudier les nouvelles configurations dans le développement des démonstrateurs RF (récepteur et émetteur) et des transpondeurs à des fins militaire et de communication. Certains travaux ont été réalisés dans le cadre du projet TECRAIL, où un démonstrateur de la couche physique LTE a été mis en place pour évaluer la faisabilité de la norme LTE dans l'environnement ferroviaire. Une contribution importante, liée au projet CALRADAR, est proposée dans le domaine des systèmes d’étalonnage de radar Doppler de haute précision. Cette contribution est le module Doppler de génération d’hyperfréquence intégré dans le système électronique de génération de cibles radar virtuelles que présente une résolution de fréquence très élevée. Une analyse complète de l'incertitude dans l'étalonnage des radars Doppler a été effectuée, afin d'assurer la précision du calibrage. La conception et la mise en oeuvre de quelques dispositifs photoniques sont un autre sujet important du travail de recherche présenté dans cette thèse. De tels dispositifs sont considérés comme étant des technologies habilitantes clés pour les futurs dispositifs et sous-systèmes RF et micro-ondes grâce à leurs avantages. Certaines demandes de radio définies par logiciel pourraient être supportées par nouveaux concepts de circuits basés sur des éléments dynamiquement programmables en utilisant des paramètres ajustables. Une petite contribution a été apportée pour améliorer la conception et les simulations des filtres passe-bande Hairpin à large bande. Ces filtres sont compacts et peuvent également être intégrés dans des circuits à micro-ondes compatibles avec un large éventail d'applications dans les systèmes militaires et de communication. Finalement, une proposition a été effectuée visant à équilibrer et améliorer la transmission des signaux discrets de synchronisation entre les TRMs et d'autres unités de traitement dans le satellite SEOSAR/PAZ de dernière génération et permettant l’obtention de la configuration optimale des bus de transmission de données à grande vitesse basés sur un réseau de transcepteurs.

User experience design

Inmersos en un mundo digital en el que participan cada vez más personas con diferentes niveles de experiencia en tecnología, ha aparecido la necesidad de diseñar las aplicaciones pensando en el usuario. El principal problema que un desarrollador se encuentra hoy en día es la falta de tiempo o agilidad para conceder al usuario una experiencia a medida. Muchas empresas han conseguido destacar en un mundo donde la competencia es muy grande gracias al aporte de valor añadido de una buena experiencia de usuario. Empezando por el análisis psicológico y las reacciones que tenemos cuando estamos interactuando con un sistema digital se intenta conseguir un método para diseñar la experiencia de usuario de un modo satisfactorio para ambas partes. Con el objetivo de probar el método diseñado, en el cual se incorporan muchas técnicas de otros métodos, se diseña parte de la experiencia de usuario de “ESEM 2014”, una de las conferencias informáticas más importantes del norte de Italia. Los resultados obtenidos en este trabajo son muy favorables, ya que mejoran substancialmente el resultado final y permiten al desarrollador seguir mejorando la aplicación hasta el momento y objetivo deseado. Resumen Español---ABSTRACT---In a world where more people with different level of expertise is engaging with technology, designing for the user has become one of the main concerns for the developers. Lack of time or agility are the main problems from the developers’ point of view in order to provide a good user experience. One of most important differentiators for companies is the provided experience as the technology and knowledge is highly available for everyone. The main goal of this project is to compose a method to include experience design in current agile development where short iterations is the main characteristic. In order to test the designed method, were many parts of other techniques have been included, part of the “ESEM 2014” conference has been designed. The results achieved with this guidelines has been very positive from both sides of the development, users and developers. Having short iterations allows improvement until the desired experience is achieved.

Reviewing technical approaches for sharing and preservation of experimental data

Empirical Software Engineering (ESE) replication researchers need to store and manipulate experimental data for several purposes, in particular analysis and reporting. Current research needs call for sharing and preservation of experimental data as well. In a previous work, we analyzed Replication Data Management (RDM) needs. A novel concept, called Experimental Ecosystem, was proposed to solve current deficiencies in RDMapproaches. The empirical ecosystem provides replication researchers with a common framework that integrates transparently local heterogeneous data sources. A typical situation where the Empirical Ecosystem is applicable, is when several members of a research group, or several research groups collaborating together, need to share and access each other experimental results. However, to be able to apply the Empirical Ecosystem concept and deliver all promised benefits, it is necessary to analyze the software architectures and tools that can properly support it.

A systematic mapping study on testing technique experiments: has the situation changed since 2000?

Context: Empirical Software Engineering (ESE) replication researchers need to store and manipulate experimental data for several purposes, in particular analysis and reporting. Current research needs call for sharing and preservation of experimental data as well. In a previous work, we analyzed Replication Data Management (RDM) needs. A novel concept, called Experimental Ecosystem, was proposed to solve current deficiencies in RDM approaches. The empirical ecosystem provides replication researchers with a common framework that integrates transparently local heterogeneous data sources. A typical situation where the Empirical Ecosystem is applicable, is when several members of a research group, or several research groups collaborating together, need to share and access each other experimental results. However, to be able to apply the Empirical Ecosystem concept and deliver all promised benefits, it is necessary to analyze the software architectures and tools that can properly support it.