Quantitative Determination of the Relationship between Internal Validity and Bias in Software Engineering Experiments: Consequences for Systematic Literature Reviews

Quality assessment is one of the activities performed as part of systematic literature reviews. It is commonly accepted that a good quality experiment is bias free. Bias is considered to be related to internal validity (e.g., how adequately the experiment is planned, executed and analysed). Quality assessment is usually conducted using checklists and quality scales. It has not yet been proven;however, that quality is related to experimental bias. Aim: Identify whether there is a relationship between internal validity and bias in software engineering experiments. Method: We built a quality scale to determine the quality of the studies, which we applied to 28 experiments included in two systematic literature reviews. We proposed an objective indicator of experimental bias, which we applied to the same 28 experiments. Finally, we analysed the correlations between the quality scores and the proposed measure of bias. Results: We failed to find a relationship between the global quality score (resulting from the quality scale) and bias; however, we did identify interesting correlations between bias and some particular aspects of internal validity measured by the instrument. Conclusions: There is an empirically provable relationship between internal validity and bias. It is feasible to apply quality assessment in systematic literature reviews, subject to limits on the internal validity aspects for consideration.

Sample Size vs. Bias in Defect Prediction

Most empirical disciplines promote the reuse and sharing of datasets, as it leads to greater possibility of replication. While this is increasingly the case in Empirical Software Engineering, some of the most popular bug-fix datasets are now known to be biased. This raises two significants concerns: first, that sample bias may lead to underperforming prediction models, and second, that the external validity of the studies based on biased datasets may be suspect. This issue has raised considerable consternation in the ESE literature in recent years. However, there is a confounding factor of these datasets that has not been examined carefully: size. Biased datasets are sampling only some of the data that could be sampled, and doing so in a biased fashion; but biased samples could be smaller, or larger. Smaller data sets in general provide less reliable bases for estimating models, and thus could lead to inferior model performance. In this setting, we ask the question, what affects performance more? bias, or size? We conduct a detailed, large-scale meta-analysis, using simulated datasets sampled with bias from a high-quality dataset which is relatively free of bias. Our results suggest that size always matters just as much bias direction, and in fact much more than bias direction when considering information-retrieval measures such as AUC and F-score. This indicates that at least for prediction models, even when dealing with sampling bias, simply finding larger samples can sometimes be sufficient. Our analysis also exposes the complexity of the bias issue, and raises further issues to be explored in the future.

Generic multisensor multitarget bias estimation architecture

Current bias estimation algorithms for air traffic control (ATC) surveillance are focused on radar sensors, but the integration of new sensors (especially automatic dependent surveillance-broadcast and wide area multilateration) demands the extension of traditional procedures. This study describes a generic architecture for bias estimation applicable to multisensor multitarget surveillance systems. It consists on first performing bias estimations using measurements from each target, of a subset of sensors, assumed to be reliable, forming track bias estimations. All track bias estimations are combined to obtain, for each of those sensors, the corresponding sensor bias. Then, sensor bias terms are corrected, to subsequently calculate the target or sensor-target pair specific biases. Once these target-specific biases are corrected, the process is repeated recursively for other sets of less reliable sensors, assuming bias corrected measures from previous iterations are unbiased. This study describes the architecture and outlines the methodology for the estimation and the bias estimation design processes. Then the approach is validated through simulation, and compared with previous methods in the literature. Finally, the study describes the application of the methodology to the design of the bias estimation procedures for a modern ATC surveillance application, specifically for off-line assessment of ATC surveillance performance.

Assessing user bias in affect detection within context-based spoken dialog systems

This paper presents an empirical evidence of user bias within a laboratory-oriented evaluation of a Spoken Dialog System. Specifically, we addressed user bias in their satisfaction judgements. We question the reliability of this data for modeling user emotion, focusing on contentment and frustration in a spoken dialog system. This bias is detected through machine learning experiments that were conducted on two datasets, users and annotators, which were then compared in order to assess the reliability of these datasets. The target used was the satisfaction rating and the predictors were conversational/dialog features. Our results indicated that standard classifiers were significantly more successful in discriminating frustration and contentment and the intensities of these emotions (reflected by user satisfaction ratings) from annotator data than from user data. Indirectly, the results showed that conversational features are reliable predictors of the two abovementioned emotions.

Aspects of a phased array radar feed and bias network design

Radar technologies have been developed to improve the efficiency when detecting targets. Radar is a system composed by several devices connected and working together. Depending on the type of radar, the improvements are focused on different functionalities of the radar. One of the most important devices composing a radar is the antenna, that sends the radio-frequency signal to the space in order to detect targets. This project is focused on a specific type of radar called phased array radar. This type of radar is characterized by its antenna, which consist on a linear array of radiating elements, in this particular case, eight dipoles working at the frequency band S. The main advantage introduced by the phased array antenna is that using the fundamentals of arrays, the directivity of the antenna can change by shifting the phase of the signal at the input of each radiating element. This can be done using phase shifters. Phase shifter consists on a device which produces a phase shift in the radio-frequency input signal depending on a control DC voltage. Using a phased array antenna allows changing the directivity of the antenna without a mechanical rotating system. The objective of this project is to design the feed network and the bias network of the phased antenna. The feed network consists on a parallel-fed network composed by power dividers that sends the radio-frequency signal from the source to each radiating element of the antenna. The bias network consists on a system that generates the control DC voltages supplied to the phase shifters in order to change the directivity. The architecture of the bias network is composed by a software, implemented in Matlab and run in a laptop which is connected to a micro-controller by a serial communication port. The software calculates the control DC voltages needed to obtain a determined directivity or scan angle. These values are sent by the serial communication port to the micro-controller as data. Then the micro-controller generates the desired control DC voltages and supplies them to the phase shifters. In this project two solutions for bias network are designed. Each one is tested and final conclusions are obtained to determine the advantages and disadvantages. Finally a graphic user interface is developed in order to make the system easy to use. RESUMEN. Las tecnologías empleadas por lo dispositivos radar se han ido desarrollando para mejorar su eficiencia y usabilidad. Un radar es un sistema formado por varios subsistemas conectados entre sí. Por lo que dependiendo del tipo de radar las mejoras se centran en los subsistemas correspondientes. Uno de los elementos más importantes de un radar es la antena. Esta se emplea para enviar la señal de radiofrecuencia al espacio y así poder detectar los posibles obstáculos del entorno. Este proyecto se centra en un tipo específico de radar llamado phased array radar. Este tipo de radar se caracteriza por la antena que es un array de antenas, en concreto para este proyecto se trata de un array lineal de ocho dipolos en la banda de frequencia S. El uso de una antena de tipo phased array supone una ventaja importante. Empleando los fundamentos de radiación aplicado a array de antenas se obtiene que la directividad de la antena puede ser modificada. Esto se consigue aplicando distintos desfasajes a la señal de radiofrecuencia que alimenta a cada elemento del array. Para aplicar los desfasajes se emplea un desplazador de fase, este dispositivo aplica una diferencia de fase a su salida con respecto a la señal de entrada dependiendo de una tensión continua de control. Por tanto el empleo de una antena de tipo phased array supone una gran ventaja puesto que no se necesita un sistema de rotación para cambiar la directividad de la antena. El objetivo principal del proyecto consiste en el diseño de la red de alimentación y la red de polarización de la antena de tipo phased array. La red de alimentación consiste en un circuito pasivo que permite alimentar a cada elemento del array con la misma cantidad de señal. Dicha red estará formada por divisores de potencia pasivos y su configuración será en paralelo. Por otro lado la red de polarización consiste en el diseño de un sistema automático que permite cambiar la directividad de la antena. Este sistema consiste en un programa en Matlab que es ejecutado en un ordenador conectado a un micro-controlador mediante una comunicación serie. El funcionamiento se basa en calcular las tensiones continuas de control, que necesitan los desplazadores de fase, mediante un programa en Matlab y enviarlos como datos al micro-controlador. Dicho micro-controlador genera las tensiones de control deseadas y las proporciona a cada desplazador de fase, obteniendo así la directividad deseada. Debido al amplio abanico de posibilidades, se obtienen dos soluciones que son sometidas a pruebas. Se obtienen las ventajas y desventajas de cada una. Finalmente se implementa una interfaz gráfica de usuario con el objetivo de hacer dicho sistema manejable y entendible para cualquier usuario.

Lidar uncertainty in complex terrain development of a bias correction methodology

El autor ha trabajado como parte del equipo de investigación en mediciones de viento en el Centro Nacional de Energías Renovables (CENER), España, en cooperación con la Universidad Politécnica de Madrid y la Universidad Técnica de Dinamarca. El presente reporte recapitula el trabajo de investigación realizado durante los últimos 4.5 años en el estudio de las fuentes de error de los sistemas de medición remota de viento, basados en la tecnología lidar, enfocado al error causado por los efectos del terreno complejo. Este trabajo corresponde a una tarea del paquete de trabajo dedicado al estudio de sistemas remotos de medición de viento, perteneciente al proyecto de intestigación europeo del 7mo programa marco WAUDIT. Adicionalmente, los datos de viento reales han sido obtenidos durante las campañas de medición en terreno llano y terreno complejo, pertenecientes al también proyecto de intestigación europeo del 7mo programa marco SAFEWIND. El principal objetivo de este trabajo de investigación es determinar los efectos del terreno complejo en el error de medición de la velocidad del viento obtenida con los sistemas de medición remota lidar. Con este conocimiento, es posible proponer una metodología de corrección del error de las mediciones del lidar. Esta metodología está basada en la estimación de las variaciones del campo de viento no uniforme dentro del volumen de medición del lidar. Las variaciones promedio del campo de viento son predichas a partir de los resultados de las simulaciones computacionales de viento RANS, realizadas para el parque experimental de Alaiz. La metodología de corrección es verificada con los resultados de las simulaciones RANS y validadas con las mediciones reales adquiridas en la campaña de medición en terreno complejo. Al inicio de este reporte, el marco teórico describiendo el principio de medición de la tecnología lidar utilizada, es presentado con el fin de familiarizar al lector con los principales conceptos a utilizar a lo largo de este trabajo. Posteriormente, el estado del arte es presentado en donde se describe los avances realizados en el desarrollo de la la tecnología lidar aplicados al sector de la energía eólica. En la parte experimental de este trabajo de investigación se ha estudiado los datos adquiridos durante las dos campañas de medición realizadas. Estas campañas has sido realizadas en terreno llano y complejo, con el fin de complementar los conocimiento adquiridos en casa una de ellas y poder comparar los efectos del terreno en las mediciones de viento realizadas con sistemas remotos lidar. La primer campaña experimental se desarrollo en terreno llano, en el parque de ensayos de aerogeneradores H0vs0re, propiedad de DTU Wind Energy (anteriormente Ris0). La segunda campaña experimental se llevó a cabo en el parque de ensayos de aerogeneradores Alaiz, propiedad de CENER. Exactamente los mismos dos equipos lidar fueron utilizados en estas campañas, haciendo de estos experimentos altamente relevantes en el contexto de evaluación del recurso eólico. Un equipo lidar está basado en tecnología de onda continua, mientras que el otro está basado en tecnología de onda pulsada. La velocidad del viento fue medida, además de con los equipos lidar, con anemómetros de cazoletas, veletas y anemómetros verticales, instalados en mástiles meteorológicos. Los sensores del mástil meteorológico son considerados como las mediciones de referencia en el presente estudio. En primera instancia, se han analizado los promedios diez minútales de las medidas de viento. El objetivo es identificar las principales fuentes de error en las mediciones de los equipos lidar causadas por diferentes condiciones atmosféricas y por el flujo no uniforme de viento causado por el terreno complejo. El error del lidar ha sido estudiado como función de varias propiedades estadísticas del viento, como lo son el ángulo vertical de inclinación, la intensidad de turbulencia, la velocidad vertical, la estabilidad atmosférica y las características del terreno. El propósito es usar este conocimiento con el fin de definir criterios de filtrado de datos. Seguidamente, se propone una metodología para corregir el error del lidar causado por el campo de viento no uniforme, producido por la presencia de terreno complejo. Esta metodología está basada en el análisis matemático inicial sobre el proceso de cálculo de la velocidad de viento por los equipos lidar de onda continua. La metodología de corrección propuesta hace uso de las variaciones de viento calculadas a partir de las simulaciones RANS realizadas para el parque experimental de Alaiz. Una ventaja importante que presenta esta metodología es que las propiedades el campo de viento real, presentes en las mediciones instantáneas del lidar de onda continua, puede dar paso a análisis adicionales como parte del trabajo a futuro. Dentro del marco del proyecto, el trabajo diario se realizó en las instalaciones de CENER, con supervisión cercana de la UPM, incluyendo una estancia de 1.5 meses en la universidad. Durante esta estancia, se definió el análisis matemático de las mediciones de viento realizadas por el equipo lidar de onda continua. Adicionalmente, los efectos del campo de viento no uniforme sobre el error de medición del lidar fueron analíticamente definidos, después de asumir algunas simplificaciones. Adicionalmente, durante la etapa inicial de este proyecto se desarrollo una importante trabajo de cooperación con DTU Wind Energy. Gracias a esto, el autor realizó una estancia de 1.5 meses en Dinamarca. Durante esta estancia, el autor realizó una visita a la campaña de medición en terreno llano con el fin de aprender los aspectos básicos del diseño de campañas de medidas experimentales, el estudio del terreno y los alrededores y familiarizarse con la instrumentación del mástil meteorológico, el sistema de adquisición y almacenamiento de datos, así como de el estudio y reporte del análisis de mediciones. ABSTRACT The present report summarizes the research work performed during last 4.5 years of investigation on the sources of lidar bias due to complex terrain. This work corresponds to one task of the remote sensing work package, belonging to the FP7 WAUDIT project. Furthermore, the field data from the wind velocity measurement campaigns of the FP7 SafeWind project have been used in this report. The main objective of this research work is to determine the terrain effects on the lidar bias in the measured wind velocity. With this knowledge, it is possible to propose a lidar bias correction methodology. This methodology is based on an estimation of the wind field variations within the lidar scan volume. The wind field variations are calculated from RANS simulations performed from the Alaiz test site. The methodology is validated against real scale measurements recorded during an eight month measurement campaign at the Alaiz test site. Firstly, the mathematical framework of the lidar sensing principle is introduced and an overview of the state of the art is presented. The experimental part includes the study of two different, but complementary experiments. The first experiment was a measurement campaign performed in flat terrain, at DTU Wind Energy H0vs0re test site, while the second experiment was performed in complex terrain at CENER Alaiz test site. Exactly the same two lidar devices, based on continuous wave and pulsed wave systems, have been used in the two consecutive measurement campaigns, making this a relevant experiment in the context of wind resource assessment. The wind velocity was sensed by the lidars and standard cup anemometry and wind vanes (installed on a met mast). The met mast sensors are considered as the reference wind velocity measurements. The first analysis of the experimental data is dedicated to identify the main sources of lidar bias present in the 10 minute average values. The purpose is to identify the bias magnitude introduced by different atmospheric conditions and by the non-uniform wind flow resultant of the terrain irregularities. The lidar bias as function of several statistical properties of the wind flow like the tilt angle, turbulence intensity, vertical velocity, atmospheric stability and the terrain characteristics have been studied. The aim of this exercise is to use this knowledge in order to define useful lidar bias data filters. Then, a methodology to correct the lidar bias caused by non-uniform wind flow is proposed, based on the initial mathematical analysis of the lidar measurements. The proposed lidar bias correction methodology has been developed focusing on the the continuous wave lidar system. In a last step, the proposed lidar bias correction methodology is validated with the data of the complex terrain measurement campaign. The methodology makes use of the wind field variations obtained from the RANS analysis. The results are presented and discussed. The advantage of this methodology is that the wind field properties at the Alaiz test site can be studied with more detail, based on the instantaneous measurements of the CW lidar. Within the project framework, the daily basis work has been done at CENER, with close guidance and support from the UPM, including an exchange period of 1.5 months. During this exchange period, the mathematical analysis of the lidar sensing of the wind velocity was defined. Furthermore, the effects of non-uniform wind fields on the lidar bias were analytically defined, after making some assumptions for the sake of simplification. Moreover, there has been an important cooperation with DTU Wind Energy, where a secondment period of 1.5 months has been done as well. During the secondment period at DTU Wind Energy, an important introductory learning has taken place. The learned aspects include the design of an experimental measurement campaign in flat terrain, the site assessment study of obstacles and terrain conditions, the data acquisition and processing, as well as the study and reporting of the measurement analysis.

Evidence of the presence of bias in subjective metrics: analysis within a family of experiments

Context: Measurement is crucial and important to empirical software engineering. Although reliability and validity are two important properties warranting consideration in measurement processes, they may be influenced by random or systematic error (bias) depending on which metric is used. Aim: Check whether, the simple subjective metrics used in empirical software engineering studies are prone to bias. Method: Comparison of the reliability of a family of empirical studies on requirements elicitation that explore the same phenomenon using different design types and objective and subjective metrics. Results: The objectively measured variables (experience and knowledge) tend to achieve more reliable results, whereas subjective metrics using Likert scales (expertise and familiarity) tend to be influenced by systematic error or bias. Conclusions: Studies that predominantly use variables measured subjectively, like opinion polls or expert opinion acquisition.