Basics on design and analysis of SE experiments: widespread shortcomings

Keynote speech about experimental design

Identification and correction of besouro's faults: impact on test‐driven development experiments

Context. This thesis is framed in experimental software engineering. More concretely, it addresses the problems arisen when assessing process conformance in test-driven development experiments conducted by UPM's Experimental Software Engineering group. Process conformance was studied using the Eclipse's plug-in tool Besouro. It has been observed that Besouro does not work correctly in some circumstances. It creates doubts about the correction of the existing experimental data which render it useless. Aim. The main objective of this work is the identification and correction of Besouro's faults. A secondary goal is fixing the datasets already obtained in past experiments to the maximum possible extent. This way, existing experimental results could be used with confidence. Method. (1) Testing Besouro using different sequences of events (creation methods, assertions etc..) to identify the underlying faults. (2) Fix the code and (3) fix the datasets using code specially created for this purpose. Results. (1) We confirmed the existence of several fault in Besouro's code that affected to Test-First and Test-Last episode identification. These faults caused the incorrect identification of 20% of episodes. (2) We were able to fix Besouro's code. (3) The correction of existing datasets was possible, subjected to some restrictions (such us the impossibility of tracing code size increase to programming time. Conclusion. The results of past experiments dependent upon Besouro's data could no be trustable. We have the suspicion that more faults remain in Besouro's code, whose identification requires further analysis.

New experiments on speaker diarization for unsupervised speaking style voice building for speech synthesis

El uso universal de síntesis de voz en diferentes aplicaciones requeriría un desarrollo sencillo de las nuevas voces con poca intervención manual. Teniendo en cuenta la cantidad de datos multimedia disponibles en Internet y los medios de comunicación, un objetivo interesante es el desarrollo de herramientas y métodos para construir automáticamente las voces de estilo de varios de ellos. En un trabajo anterior se esbozó una metodología para la construcción de este tipo de herramientas, y se presentaron experimentos preliminares con una base de datos multiestilo. En este artículo investigamos más a fondo esta tarea y proponemos varias mejoras basadas en la selección del número apropiado de hablantes iniciales, el uso o no de filtros de reducción de ruido, el uso de la F0 y el uso de un algoritmo de detección de música. Hemos demostrado que el mejor sistema usando un algoritmo de detección de música disminuye el error de precisión 22,36% relativo para el conjunto de desarrollo y 39,64% relativo para el montaje de ensayo en comparación con el sistema base, sin degradar el factor de mérito. La precisión media para el conjunto de prueba es 90.62% desde 76.18% para los reportajes de 99,93% para los informes meteorológicos.

Understanding replication of experiments in software engineering: a classification

Context: Replication plays an important role in experimental disciplines. There are still many uncertain- ties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experiment- ers, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment? Objective: To improve our understanding of SE experiment replication, in this work we propose a classi- fication which is intend to provide experimenters with guidance about what types of replication they can perform. Method: The research approach followed is structured according to the following activities: (1) a litera- ture review of experiment replication in SE and in other disciplines, (2) identification of typical elements that compose an experimental configuration, (3) identification of different replications purposes and (4) development of a classification of experiment replications for SE. Results: We propose a classification of replications which provides experimenters in SE with guidance about what changes can they make in a replication and, based on these, what verification purposes such a replication can serve. The proposed classification helped to accommodate opposing views within a broader framework, it is capable of accounting for less similar replications to more similar ones regarding the baseline experiment. Conclusion: The aim of replication is to verify results, but different types of replication serve special ver- ification purposes and afford different degrees of change. Each replication type helps to discover partic- ular experimental conditions that might influence the results. The proposed classification can be used to identify changes in a replication and, based on these, understand the level of verification.

Evidence of the presence of bias in subjective metrics: analysis within a family of experiments

Context: Measurement is crucial and important to empirical software engineering. Although reliability and validity are two important properties warranting consideration in measurement processes, they may be influenced by random or systematic error (bias) depending on which metric is used. Aim: Check whether, the simple subjective metrics used in empirical software engineering studies are prone to bias. Method: Comparison of the reliability of a family of empirical studies on requirements elicitation that explore the same phenomenon using different design types and objective and subjective metrics. Results: The objectively measured variables (experience and knowledge) tend to achieve more reliable results, whereas subjective metrics using Likert scales (expertise and familiarity) tend to be influenced by systematic error or bias. Conclusions: Studies that predominantly use variables measured subjectively, like opinion polls or expert opinion acquisition.

A systematic mapping study on testing technique experiments: has the situation changed since 2000?

Context: Empirical Software Engineering (ESE) replication researchers need to store and manipulate experimental data for several purposes, in particular analysis and reporting. Current research needs call for sharing and preservation of experimental data as well. In a previous work, we analyzed Replication Data Management (RDM) needs. A novel concept, called Experimental Ecosystem, was proposed to solve current deficiencies in RDM approaches. The empirical ecosystem provides replication researchers with a common framework that integrates transparently local heterogeneous data sources. A typical situation where the Empirical Ecosystem is applicable, is when several members of a research group, or several research groups collaborating together, need to share and access each other experimental results. However, to be able to apply the Empirical Ecosystem concept and deliver all promised benefits, it is necessary to analyze the software architectures and tools that can properly support it.

Topic selection in industry experiments

This paper shares our experience with initial negotiation and topic elicitation process for conducting industry experiments in six software development organizations in Finland. The process involved interaction with company representatives in the form of both multiple group discussions and separate face-to-face meetings. Fitness criteria developed by researchers were applied to the list of generated topics to decide on a common topic. The challenges we faced include diversity of proposed topics, communication gaps, skepticism about research methods, initial disconnect between research and industry needs, and lack of prior work relationship. Lessons learned include having enough time to establish trust with partners, importance of leveraging the benefits of training and skill development that are inherent in the experimental approach, uniquely positioning the experimental approach within the landscape of other validation approaches more familiar to industrial partners, and introducing the fitness criteria early in the process.

Effects of a bifenthrin-treated net on the natural enemies Aphidius colemani (Haliday) and Adalia bipunctata L. in a cucumber crop in Central Spain. Semi-field experiments

Pest management practices that rely on pesticides are growing increasingly less effective and environmentally inappropriate in many cases and the search of alternatives is under focus nowadays. Exclusion of pests from the crop by means of pesticide-treated screens can be an eco-friendly method to protect crops, especially if pests are vectors of important diseases. The mesh size of nets is crucial to determine if insects can eventually cross the barrier or exclude them because there is a great variation in insect size depending on the species. Long-lasting insecticide-treated (LLITN) nets, factory pre-treated, have been used since years to fight against mosquitoes vector of malaria and are able to retain their biological efficacy under field for 3 years. In agriculture, treated nets with different insecticides have shown efficacy in controlling some insects and mites, so they seem to be a good tool in helping to solve some pest problems. However, treated nets must be carefully evaluated because can diminish air flow, increase temperature and humidity and decrease light transmission, which may affect plant growth, pests and natural enemies. As biological control is considered a key factor in IPM nowadays, the potential negative effects of treated nets on natural enemies need to be studied carefully. In this work, the effects of a bifentrhin-treated net (3 g/Kg) (supplied by the company Intelligent Insect Control, IIC) on natural enemies of aphids were tested on a cucumber crop in Central Spain in autumn 2011. The crop was sown in 8x6.5 m tunnels divided in 2 sealed compartments with control or treated nets, which were simple yellow netting with 25 mesh (10 x 10 threads/cm2; 1 x 1 mm hole size). Pieces of 2 m high of the treated-net were placed along the lateral sides of one of the two tunnel compartments in each of the 3 available tunnels (replicates); the rest was covered by a commercial untreated net of a similar mesh. The pest, Aphis gossypii Glover (Aphidae), the parasitoid Aphidius colemani (Haliday) (Braconidae) and the predator Adalia bipunctata L. (Coccinellidae) were artificially introduced in the crop. Weekly sampling was done determining the presence or absence of the pest and the natural enemies (NE) in the 42 plants/compartment as well as the number of insects in 11 marked plants. Environmental conditions (temperature, relative humidity, UV and PAR radiation) were recorded. Results show that when aphids were artificially released inside the tunnels, neither its number/plant nor their distribution was affected by the treated net. A lack of negative effect of the insecticide-treated net on natural enemies was also observed. Adalia bipunctata did not establish in the crop and only a short term control of aphids was observed one week after release. On the other hand, A. colemani did establish in the crop and a more long-term effect on the numbers of aphids/plant was detected irrespective of the type of net. KEY WORDS: bifenthrin-treated net, Adalia bipunctata, Aphidius colemani, Aphis gossypii, semi-field

Conservation of Computational Scientific Execution Environments for Workflow-based Experiments Using Ontologies

La reproducibilidad de estudios y resultados científicos es una meta a tener en cuenta por cualquier científico a la hora de publicar el producto de una investigación. El auge de la ciencia computacional, como una forma de llevar a cabo estudios empíricos haciendo uso de modelos matemáticos y simulaciones, ha derivado en una serie de nuevos retos con respecto a la reproducibilidad de dichos experimentos. La adopción de los flujos de trabajo como método para especificar el procedimiento científico de estos experimentos, así como las iniciativas orientadas a la conservación de los datos experimentales desarrolladas en las últimas décadas, han solucionado parcialmente este problema. Sin embargo, para afrontarlo de forma completa, la conservación y reproducibilidad del equipamiento computacional asociado a los flujos de trabajo científicos deben ser tenidas en cuenta. La amplia gama de recursos hardware y software necesarios para ejecutar un flujo de trabajo científico hace que sea necesario aportar una descripción completa detallando que recursos son necesarios y como estos deben de ser configurados. En esta tesis abordamos la reproducibilidad de los entornos de ejecución para flujos de trabajo científicos, mediante su documentación usando un modelo formal que puede ser usado para obtener un entorno equivalente. Para ello, se ha propuesto un conjunto de modelos para representar y relacionar los conceptos relevantes de dichos entornos, así como un conjunto de herramientas que hacen uso de dichos módulos para generar una descripción de la infraestructura, y un algoritmo capaz de generar una nueva especificación de entorno de ejecución a partir de dicha descripción, la cual puede ser usada para recrearlo usando técnicas de virtualización. Estas contribuciones han sido aplicadas a un conjunto representativo de experimentos científicos pertenecientes a diferentes dominios de la ciencia, exponiendo cada uno de ellos diferentes requisitos hardware y software. Los resultados obtenidos muestran la viabilidad de propuesta desarrollada, reproduciendo de forma satisfactoria los experimentos estudiados en diferentes entornos de virtualización. ABSTRACT Reproducibility of scientific studies and results is a goal that every scientist must pursuit when announcing research outcomes. The rise of computational science, as a way of conducting empirical studies by using mathematical models and simulations, have opened a new range of challenges in this context. The adoption of workflows as a way of detailing the scientific procedure of these experiments, along with the experimental data conservation initiatives that have been undertaken during last decades, have partially eased this problem. However, in order to fully address it, the conservation and reproducibility of the computational equipment related to them must be also considered. The wide range of software and hardware resources required to execute a scientific workflow implies that a comprehensive description detailing what those resources are and how they are arranged is necessary. In this thesis we address the issue of reproducibility of execution environments for scientific workflows, by documenting them in a formalized way, which can be later used to obtain and equivalent one. In order to do so, we propose a set of semantic models for representing and relating the relevant information of those environments, as well as a set of tools that uses these models for generating a description of the infrastructure, and an algorithmic process that consumes these descriptions for deriving a new execution environment specification, which can be enacted into a new equivalent one using virtualization solutions. We apply these three contributions to a set of representative scientific experiments, belonging to different scientific domains, and exposing different software and hardware requirements. The obtained results prove the feasibility of the proposed approach, by successfully reproducing the target experiments under different virtualization environments.

The R1 component of mammalian ribonucleotide reductase has malignancy-suppressing activity as demonstrated by gene transfer experiments

Our recent studies have shown that deregulated expression of R2, the rate-limiting component of ribonucleotide reductase, enhances transformation and malignant potential by cooperating with activated oncogenes. We now demonstrate that the R1 component of ribonucleotide reductase has tumor-suppressing activity. Stable expression of a biologically active ectopic R1 in ras-transformed mouse fibroblast 10T½ cell lines, with or without R2 overexpression, led to significantly reduced colony-forming efficiency in soft agar. The decreased anchorage independence was accompanied by markedly suppressed malignant potential in vivo. In three ras-transformed cell lines, R1 overexpression resulted in abrogation or marked suppression of tumorigenicity. In addition, the ability to form lung metastases by cells overexpressing R1 was reduced by >85%. Metastasis suppressing activity also was observed in the highly malignant mouse 10T½ derived RMP-6 cell line, which was transformed by a combination of oncogenic ras, myc, and mutant p53. Furthermore, in support of the above observations with the R1 overexpressing cells, NIH 3T3 cells cotransfected with an R1 antisense sequence and oncogenic ras showed significantly increased anchorage independence as compared with control ras-transfected cells. Finally, characteristics of reduced malignant potential also were demonstrated with R1 overexpressing human colon carcinoma cells. Taken together, these results indicate that the two components of ribonucleotide reductase both are unique malignancy determinants playing opposing roles in its regulation, that there is a novel control point important in mechanisms of malignancy, which involves a balance in the levels of R1 and R2 expression, and that alterations in this balance can significantly modify transformation, tumorigenicity, and metastatic potential.

Vibrational population relaxation of carbon monoxide in the heme pocket of photolyzed carbonmonoxy myoglobin: Comparison of time-resolved mid-IR absorbance experiments and molecular dynamics simulations

The vibrational energy relaxation of carbon monoxide in the heme pocket of sperm whale myoglobin was studied by using molecular dynamics simulation and normal mode analysis methods. Molecular dynamics trajectories of solvated myoglobin were run at 300 K for both the δ- and ɛ-tautomers of the distal His-64. Vibrational population relaxation times of 335 ± 115 ps for the δ-tautomer and 640 ± 185 ps for the ɛ-tautomer were estimated by using the Landau–Teller model. Normal mode analysis was used to identify those protein residues that act as the primary “doorway” modes in the vibrational relaxation of the oscillator. Although the CO relaxation rates in both the ɛ- and δ-tautomers are similar in magnitude, the simulations predict that the vibrational relaxation of the CO is faster in the δ-tautomer with the distal His playing an important role in the energy relaxation mechanism. Time-resolved mid-IR absorbance measurements were performed on photolyzed carbonmonoxy hemoglobin (Hb13CO). From these measurements, a T1 time of 600 ± 150 ps was determined. The simulation and experimental estimates are compared and discussed.

TROSY in triple-resonance experiments: New perspectives for sequential NMR assignment of large proteins

The NMR assignment of 13C, 15N-labeled proteins with the use of triple resonance experiments is limited to molecular weights below ∼25,000 Daltons, mainly because of low sensitivity due to rapid transverse nuclear spin relaxation during the evolution and recording periods. For experiments that exclusively correlate the amide proton (1HN), the amide nitrogen (15N), and 13C atoms, this size limit has been previously extended by additional labeling with deuterium (2H). The present paper shows that the implementation of transverse relaxation-optimized spectroscopy ([15N,1H]-TROSY) into triple resonance experiments results in several-fold improved sensitivity for 2H/13C/15N-labeled proteins and approximately twofold sensitivity gain for 13C/15N-labeled proteins. Pulse schemes and spectra recorded with deuterated and protonated proteins are presented for the [15N, 1H]-TROSY-HNCA and [15N, 1H]-TROSY-HNCO experiments. A theoretical analysis of the HNCA experiment shows that the primary TROSY effect is on the transverse relaxation of 15N, which is only little affected by deuteration, and predicts sensitivity enhancements that are in close agreement with the experimental data.

Detection of alkali metal ions in DNA crystals using state-of-the-art X-ray diffraction experiments

The observation of light metal ions in nucleic acids crystals is generally a fortuitous event. Sodium ions in particular are notoriously difficult to detect because their X-ray scattering contributions are virtually identical to those of water and Na+…O distances are only slightly shorter than strong hydrogen bonds between well-ordered water molecules. We demonstrate here that replacement of Na+ by K+, Rb+ or Cs+ and precise measurements of anomalous differences in intensities provide a particularly sensitive method for detecting alkali metal ion-binding sites in nucleic acid crystals. Not only can alkali metal ions be readily located in such structures, but the presence of Rb+ or Cs+ also allows structure determination by the single wavelength anomalous diffraction technique. Besides allowing identification of high occupancy binding sites, the combination of high resolution and anomalous diffraction data established here can also pinpoint binding sites that feature only partial occupancy. Conversely, high resolution of the data alone does not necessarily allow differentiation between water and partially ordered metal ions, as demonstrated with the crystal structure of a DNA duplex determined to a resolution of 0.6 Å.

Free energy reconstruction from nonequilibrium single-molecule pulling experiments

Laser tweezers and atomic force microscopes are increasingly used to probe the interactions and mechanical properties of individual molecules. Unfortunately, using such time-dependent perturbations to force rare molecular events also drives the system away from equilibrium. Nevertheless, we show how equilibrium free energy profiles can be extracted rigorously from repeated nonequilibrium force measurements on the basis of an extension of Jarzynski's remarkable identity between free energies and the irreversible work.