Evidence of the presence of bias in subjective metrics: analysis within a family of experiments

Context: Measurement is crucial and important to empirical software engineering. Although reliability and validity are two important properties warranting consideration in measurement processes, they may be influenced by random or systematic error (bias) depending on which metric is used. Aim: Check whether, the simple subjective metrics used in empirical software engineering studies are prone to bias. Method: Comparison of the reliability of a family of empirical studies on requirements elicitation that explore the same phenomenon using different design types and objective and subjective metrics. Results: The objectively measured variables (experience and knowledge) tend to achieve more reliable results, whereas subjective metrics using Likert scales (expertise and familiarity) tend to be influenced by systematic error or bias. Conclusions: Studies that predominantly use variables measured subjectively, like opinion polls or expert opinion acquisition.

A systematic mapping study on testing technique experiments: has the situation changed since 2000?

Context: Empirical Software Engineering (ESE) replication researchers need to store and manipulate experimental data for several purposes, in particular analysis and reporting. Current research needs call for sharing and preservation of experimental data as well. In a previous work, we analyzed Replication Data Management (RDM) needs. A novel concept, called Experimental Ecosystem, was proposed to solve current deficiencies in RDM approaches. The empirical ecosystem provides replication researchers with a common framework that integrates transparently local heterogeneous data sources. A typical situation where the Empirical Ecosystem is applicable, is when several members of a research group, or several research groups collaborating together, need to share and access each other experimental results. However, to be able to apply the Empirical Ecosystem concept and deliver all promised benefits, it is necessary to analyze the software architectures and tools that can properly support it.

Topic selection in industry experiments

This paper shares our experience with initial negotiation and topic elicitation process for conducting industry experiments in six software development organizations in Finland. The process involved interaction with company representatives in the form of both multiple group discussions and separate face-to-face meetings. Fitness criteria developed by researchers were applied to the list of generated topics to decide on a common topic. The challenges we faced include diversity of proposed topics, communication gaps, skepticism about research methods, initial disconnect between research and industry needs, and lack of prior work relationship. Lessons learned include having enough time to establish trust with partners, importance of leveraging the benefits of training and skill development that are inherent in the experimental approach, uniquely positioning the experimental approach within the landscape of other validation approaches more familiar to industrial partners, and introducing the fitness criteria early in the process.

Effects of a bifenthrin-treated net on the natural enemies Aphidius colemani (Haliday) and Adalia bipunctata L. in a cucumber crop in Central Spain. Semi-field experiments

Pest management practices that rely on pesticides are growing increasingly less effective and environmentally inappropriate in many cases and the search of alternatives is under focus nowadays. Exclusion of pests from the crop by means of pesticide-treated screens can be an eco-friendly method to protect crops, especially if pests are vectors of important diseases. The mesh size of nets is crucial to determine if insects can eventually cross the barrier or exclude them because there is a great variation in insect size depending on the species. Long-lasting insecticide-treated (LLITN) nets, factory pre-treated, have been used since years to fight against mosquitoes vector of malaria and are able to retain their biological efficacy under field for 3 years. In agriculture, treated nets with different insecticides have shown efficacy in controlling some insects and mites, so they seem to be a good tool in helping to solve some pest problems. However, treated nets must be carefully evaluated because can diminish air flow, increase temperature and humidity and decrease light transmission, which may affect plant growth, pests and natural enemies. As biological control is considered a key factor in IPM nowadays, the potential negative effects of treated nets on natural enemies need to be studied carefully. In this work, the effects of a bifentrhin-treated net (3 g/Kg) (supplied by the company Intelligent Insect Control, IIC) on natural enemies of aphids were tested on a cucumber crop in Central Spain in autumn 2011. The crop was sown in 8x6.5 m tunnels divided in 2 sealed compartments with control or treated nets, which were simple yellow netting with 25 mesh (10 x 10 threads/cm2; 1 x 1 mm hole size). Pieces of 2 m high of the treated-net were placed along the lateral sides of one of the two tunnel compartments in each of the 3 available tunnels (replicates); the rest was covered by a commercial untreated net of a similar mesh. The pest, Aphis gossypii Glover (Aphidae), the parasitoid Aphidius colemani (Haliday) (Braconidae) and the predator Adalia bipunctata L. (Coccinellidae) were artificially introduced in the crop. Weekly sampling was done determining the presence or absence of the pest and the natural enemies (NE) in the 42 plants/compartment as well as the number of insects in 11 marked plants. Environmental conditions (temperature, relative humidity, UV and PAR radiation) were recorded. Results show that when aphids were artificially released inside the tunnels, neither its number/plant nor their distribution was affected by the treated net. A lack of negative effect of the insecticide-treated net on natural enemies was also observed. Adalia bipunctata did not establish in the crop and only a short term control of aphids was observed one week after release. On the other hand, A. colemani did establish in the crop and a more long-term effect on the numbers of aphids/plant was detected irrespective of the type of net. KEY WORDS: bifenthrin-treated net, Adalia bipunctata, Aphidius colemani, Aphis gossypii, semi-field

Conservation of Computational Scientific Execution Environments for Workflow-based Experiments Using Ontologies

La reproducibilidad de estudios y resultados científicos es una meta a tener en cuenta por cualquier científico a la hora de publicar el producto de una investigación. El auge de la ciencia computacional, como una forma de llevar a cabo estudios empíricos haciendo uso de modelos matemáticos y simulaciones, ha derivado en una serie de nuevos retos con respecto a la reproducibilidad de dichos experimentos. La adopción de los flujos de trabajo como método para especificar el procedimiento científico de estos experimentos, así como las iniciativas orientadas a la conservación de los datos experimentales desarrolladas en las últimas décadas, han solucionado parcialmente este problema. Sin embargo, para afrontarlo de forma completa, la conservación y reproducibilidad del equipamiento computacional asociado a los flujos de trabajo científicos deben ser tenidas en cuenta. La amplia gama de recursos hardware y software necesarios para ejecutar un flujo de trabajo científico hace que sea necesario aportar una descripción completa detallando que recursos son necesarios y como estos deben de ser configurados. En esta tesis abordamos la reproducibilidad de los entornos de ejecución para flujos de trabajo científicos, mediante su documentación usando un modelo formal que puede ser usado para obtener un entorno equivalente. Para ello, se ha propuesto un conjunto de modelos para representar y relacionar los conceptos relevantes de dichos entornos, así como un conjunto de herramientas que hacen uso de dichos módulos para generar una descripción de la infraestructura, y un algoritmo capaz de generar una nueva especificación de entorno de ejecución a partir de dicha descripción, la cual puede ser usada para recrearlo usando técnicas de virtualización. Estas contribuciones han sido aplicadas a un conjunto representativo de experimentos científicos pertenecientes a diferentes dominios de la ciencia, exponiendo cada uno de ellos diferentes requisitos hardware y software. Los resultados obtenidos muestran la viabilidad de propuesta desarrollada, reproduciendo de forma satisfactoria los experimentos estudiados en diferentes entornos de virtualización. ABSTRACT Reproducibility of scientific studies and results is a goal that every scientist must pursuit when announcing research outcomes. The rise of computational science, as a way of conducting empirical studies by using mathematical models and simulations, have opened a new range of challenges in this context. The adoption of workflows as a way of detailing the scientific procedure of these experiments, along with the experimental data conservation initiatives that have been undertaken during last decades, have partially eased this problem. However, in order to fully address it, the conservation and reproducibility of the computational equipment related to them must be also considered. The wide range of software and hardware resources required to execute a scientific workflow implies that a comprehensive description detailing what those resources are and how they are arranged is necessary. In this thesis we address the issue of reproducibility of execution environments for scientific workflows, by documenting them in a formalized way, which can be later used to obtain and equivalent one. In order to do so, we propose a set of semantic models for representing and relating the relevant information of those environments, as well as a set of tools that uses these models for generating a description of the infrastructure, and an algorithmic process that consumes these descriptions for deriving a new execution environment specification, which can be enacted into a new equivalent one using virtualization solutions. We apply these three contributions to a set of representative scientific experiments, belonging to different scientific domains, and exposing different software and hardware requirements. The obtained results prove the feasibility of the proposed approach, by successfully reproducing the target experiments under different virtualization environments.