The Risk of Using the Q Heterogeneity Estimator for Software Engineering Experiments

All meta-analyses should include a heterogeneity analysis. Even so, it is not easy to decide whether a set of studies are homogeneous or heterogeneous because of the low statistical power of the statistics used (usually the Q test). Objective: Determine a set of rules enabling SE researchers to find out, based on the characteristics of the experiments to be aggregated, whether or not it is feasible to accurately detect heterogeneity. Method: Evaluate the statistical power of heterogeneity detection methods using a Monte Carlo simulation process. Results: The Q test is not powerful when the meta-analysis contains up to a total of about 200 experimental subjects and the effect size difference is less than 1. Conclusions: The Q test cannot be used as a decision-making criterion for meta-analysis in small sample settings like SE. Random effects models should be used instead of fixed effects models. Caution should be exercised when applying Q test-mediated decomposition into subgroups.

Quantitative Determination of the Relationship between Internal Validity and Bias in Software Engineering Experiments: Consequences for Systematic Literature Reviews

Quality assessment is one of the activities performed as part of systematic literature reviews. It is commonly accepted that a good quality experiment is bias free. Bias is considered to be related to internal validity (e.g., how adequately the experiment is planned, executed and analysed). Quality assessment is usually conducted using checklists and quality scales. It has not yet been proven;however, that quality is related to experimental bias. Aim: Identify whether there is a relationship between internal validity and bias in software engineering experiments. Method: We built a quality scale to determine the quality of the studies, which we applied to 28 experiments included in two systematic literature reviews. We proposed an objective indicator of experimental bias, which we applied to the same 28 experiments. Finally, we analysed the correlations between the quality scores and the proposed measure of bias. Results: We failed to find a relationship between the global quality score (resulting from the quality scale) and bias; however, we did identify interesting correlations between bias and some particular aspects of internal validity measured by the instrument. Conclusions: There is an empirically provable relationship between internal validity and bias. It is feasible to apply quality assessment in systematic literature reviews, subject to limits on the internal validity aspects for consideration.

Identifying Moderator Variables Through Requirements Elicitation Experiments Limitations

Interviews are the most widely used elicitation technique in Requirements Engineering (RE). Despite its importance, research in interviews is quite limited, in particular from an experimental perspective. We have performed a series of experiments exploring the relative effectiveness of structured and unstructured interviews. This line of research has been active in Information Systems in the past years, so that our experiments can be aggregated together with existing ones to obtain guidelines for practice. Experimental aggregation is a demanding task. It requires not only a large number of experiments, but also considering the influence of the existing moderators. However, in the current state of the practice in RE, those moderators are unknown. We believe that analyzing the threats to validity in interviewing experiments may give insight about how to improve further replications and the corresponding aggregations. It is likely that this strategy may be applied in other Software Engineering areas as well.

MIRACLE at GeoCLEF 2005: First Experiments in Geographical IR

This paper presents the 2005 MIRACLE team’s approach to Cross-Language Geographical Retrieval (GeoCLEF). The main goal of the GeoCLEF participation of the MIRACLE team was to test the effect that geographical information retrieval techniques have on information retrieval. The baseline approach is based on the development of named entity recognition and geospatial information retrieval tools and on its combination with linguistic techniques to carry out indexing and retrieval tasks.

Automatic Understanding of ATC Speech: Study of Prospectives and Field Experiments for Several Controller Positions

Although there has been a lot of interest in recognizing and understanding air traffic control (ATC) speech, none of the published works have obtained detailed field data results. We have developed a system able to identify the language spoken and recognize and understand sentences in both Spanish and English. We also present field results for several in-tower controller positions. To the best of our knowledge, this is the first time that field ATC speech (not simulated) is captured, processed, and analyzed. The use of stochastic grammars allows variations in the standard phraseology that appear in field data. The robust understanding algorithm developed has 95% concept accuracy from ATC text input. It also allows changes in the presentation order of the concepts and the correction of errors created by the speech recognition engine improving it by 17% and 25%, respectively, absolute in the percentage of fully correctly understood sentences for English and Spanish in relation to the percentages of fully correctly recognized sentences. The analysis of errors due to the spontaneity of the speech and its comparison to read speech is also carried out. A 96% word accuracy for read speech is reduced to 86% word accuracy for field ATC data for Spanish for the "clearances" task confirming that field data is needed to estimate the performance of a system. A literature review and a critical discussion on the possibilities of speech recognition and understanding technology applied to ATC speech are also given.

DAEDALUS at ImageCLEF Medical Retrieval 2011: Textual, Visual and Multimodal Experiments

This paper describes the participation of DAEDALUS at ImageCLEF 2011 Medical Retrieval task. We have focused on multimodal (or mixed) experiments that combine textual and visual retrieval. The main objective of our research has been to evaluate the effect on the medical retrieval process of the existence of an extended corpus that is annotated with the image type, associated to both the image itself and also to its textual description. For this purpose, an image classifier has been developed to tag each document with its class (1st level of the hierarchy: Radiology, Microscopy, Photograph, Graphic, Other) and subclass (2nd level: AN, CT, MR, etc.). For the textual-based experiments, several runs using different semantic expansion techniques have been performed. For the visual-based retrieval, different runs are defined by the corpus used in the retrieval process and the strategy for obtaining the class and/or subclass. The best results are achieved in runs that make use of the image subclass based on the classification of the sample images. Although different multimodal strategies have been submitted, none of them has shown to be able to provide results that are at least comparable to the ones achieved by the textual retrieval alone. We believe that we have been unable to find a metric for the assessment of the relevance of the results provided by the visual and textual processes

Fluid-physics-module experiments

Some floating-liquid-zone experiments performed under reduced-gravity conditions are reviewed. Several types of instabilities are discussed, together with the relevant parameters controlling them. It is shown that the bounding values of these parameters could be increased, by orders of magnitude in several instances, by selecting appropriate liquids. Two of the many problems that a Fluid-Physics Module, devised to perform experiments on floating zones in a space laboratory, would involve are discussed: namely (i) procedures for disturbing the zoneunder controlled conditions, and (ii) visualisation of the inner flow pattern. Several topics connected with the nonisothermal nature and the phase-changes of floating zones are presented. In particular, a mode of propagation through the liquid zone for disturbances which could appear in the melting solid/liquid interface is suggested. Although most research on floating liquid zones is aimed at improving the crystal-growth process, some additional applications are suggested.

ILP Experiments in Detecting Traffic Problems

The paper describes experiments in automated acquisition of knowledge in traffic problem detection. Preliminary results show that ILP can be used to successfully learn to detect traffic problems.

Experiments in context-sensitive analysis of modular programs

Several models for context-sensitive analysis of modular programs have been proposed, each with different characteristics and representing different trade-offs. The advantage of these context-sensitive analyses is that they provide information which is potentially more accurate than that provided by context-free analyses. Such information can then be applied to validating/debugging the program and/or to specializing the program in order to obtain important performance improvements. Some very preliminary experimental results have also been reported for some of these models which provided initial evidence on their potential. However, further experimentation, which is needed in order to understand the many issues left open and to show that the proposed modes scale and are usable in the context of large, real-life modular programs, was left as future work. The aim of this paper is two-fold. On one hand we provide an empirical comparison of the different models proposed in previous work, as well as experimental data on the different choices left open in those designs. On the other hand we explore the scalability of these models by using larger modular programs as benchmarks. The results have been obtained from a realistic implementation of the models, integrated in a production-quality compiler (CiaoPP/Ciao). Our experimental results shed light on the practical implications of the different design choices and of the models themselves. We also show that contextsensitive analysis of modular programs is indeed feasible in practice, and that in certain critical cases it provides better performance results than those achievable by analyzing the whole program at once, specially in terms of memory consumption and when reanalyzing after making changes to a program, as is often the case during program development.

Experiments in abstract interpretation-based code certification for pervasive systems

Proof carrying code (PCC) is a general is originally a roof in ñrst-order logic of certain vermethodology for certifying that the execution of an un- ification onditions and the checking process involves trusted mobile code is safe. The baste idea is that the ensuring that the certifícate is indeed a valid ñrst-order code supplier attaches a certifícate to the mobile code proof. which the consumer checks in order to ensure that the The main practical difñculty of PCC techniques is in code is indeed safe. The potential benefit is that the generating safety certiñeates which at the same time: i) consumer's task is reduced from the level of proving to allow expressing interesting safety properties, ii) can be the level of checking. Recently, the abstract interpre- generated automatically and, iii) are easy and efficient tation techniques developed, in logic programming have to check. In [1], the abstract interpretation techniques been proposed as a basis for PCC. This extended ab- [5] developed in logic programming1 are proposed as stract reports on experiments which illustrate several is- a basis for PCC. They offer a number of advantages sues involved in abstract interpretation-based certifica- for dealing with the aforementioned issues. In particution. First, we describe the implementation of our sys- lar, the xpressiveness of existing abstract domains will tem in the context of CiaoPP: the preprocessor of the be implicitly available in abstract interpretation-based Ciao multi-paradigm programming system. Then, by code certification to deñne a wide range of safety propermeans of some experiments, we show how code certifi- ties. Furthermore, the approach inherits the automation catión is aided in the implementation of the framework. and inference power of the abstract interpretation en- Finally, we discuss the application of our method within gines used in (Constraint) Logic Programming, (C)LP. the área, of pervasive systems

Abstract interpretation-based code certification for pervasive systems: Preliminary experiments

Proof carrying code is a general methodology for certifying that the execution of an untrusted mobile code is safe, according to a predefined safety policy. The basic idea is that the code supplier attaches a certifícate (or proof) to the mobile code which, then, the consumer checks in order to ensure that the code is indeed safe. The potential benefit is that the consumer's task is reduced from the level of proving to the level of checking, a much simpler task. Recently, the abstract interpretation techniques developed in logic programming have been proposed as a basis for proof carrying code [1]. To this end, the certifícate is generated from an abstract interpretation-based proof of safety. Intuitively, the verification condition is extracted from a set of assertions guaranteeing safety and the answer table generated during the analysis. Given this information, it is relatively simple and fast to verify that the code does meet this proof and so its execution is safe. This extended abstract reports on experiments which illustrate several issues involved in abstract interpretation-based code certification. First, we describe the implementation of our system in the context of CiaoPP: the preprocessor of the Ciao multi-paradigm (constraint) logic programming system. Then, by means of some experiments, we show how code certification is aided in the implementation of the framework. Finally, we discuss the application of our method within the área of pervasive systems which may lack the necessary computing resources to verify safety on their own. We herein illustrate the relevance of the information inferred by existing cost analysis to control resource usage in this context. Moreover, since the (rather complex) analysis phase is replaced by a simpler, efficient checking process at the code consumer side, we believe that our abstract interpretation-based approach to proof-carrying code becomes practically applicable to this kind of systems.

Selectivity of diacylhydrazine insecticides to the predatory bug Orius laevigatus: In vivo and modeling/docking experiments

BACKGROUND: Knowledge of pesticide selectivity to natural enemies is necessary for a successful implementation of biological and chemical control methods in integrated pest management (IPM) programs. Diacylhydrazine (DAH)-based ecdysone agonists also known as molting-accelerating compounds (MACs) are considered a selective group of insecticides, and their compatibility with predatory Heteroptera, which are used as biological control agents, is known. However, their molecular mode of action has not been explored in beneficial insects such as Orius laevigatus (Fieber) (Hemiptera: Anthocoridae). RESULTS: In this project in vivo toxicity assays demonstrated that the DAH-based RH-5849, tebufenozide and methoxyfenozide have no toxic effect against O. laevigatus. The ligand-binding domain (LBD) of the ecdysone receptor (EcR) of O. laevigatus was sequenced and a homology protein model was constructed which confirmed a cavity structure with 12 ?-helixes, harboring the natural insect molting hormone 20-hydroxyecdysone. However, docking studies showed that a steric clash occurred for the DAH-based insecticides due to a restricted extent of the ligand-binding cavity of the EcR of O. laevigatus. CONCLUSIONS: The insect toxicity assays demonstrated that MACs are selective for O. laevigatus. The modeling/docking experiments are indications that these pesticides do not bind with the LBD-EcR of O. laevigatus and support that they show no biological effects in the predatory bug. These data help in explaining the compatible use of MACs together with predatory bugs in IPM programs. Keywords: Orius laevigatus, selectivity, diacylhydrazine insecticides, ecdysone receptor, homology modelling, docking studies.

Electron temperature versus laser intensity times wavelength squared: a comparison of theory and experiments

The peak temperature in the corona of plasma ejected by a laser-irradiated slab is discussed in terms of a one-electron-temperature model. Both heat-flux saturation and pulse rise-time effects are considered;the intensity in the rising half of the pulse is approximated by a linear function of time, I(t) = Iot/r. The temperature is found to be proportional to (IQX2)273 and a function of I0X4/r. Above a certain value of I0X4/T, the plasma presents two characteristic temperatures (at saturation and at the critical surface) which can be identified with experimentally observed cold- and hot-electron temperatures. The results are compared with extensive experimental data available for both nd and CO2 lasers, I0(W'cnf2) X2 (/um) starting around 1012. The agreement is good if substantial flux inhibition is assumed (flux-limit factor f = 0.03), and fails for I0X2 above 1O1S. Results for both ablation pressure and mass ablation rate are also given.

Dynamics in the governance of collective irrigation systems: Evidence from field experiments in Nicaragua.

In this paper, we investigate the real demand for climate protection when the purely individual perspective of existing revealed preference studies is relaxed. This is achieved in two treatments; first, we determine the information subjects receive about the demand revealed by other subjects in a similar decision making situation, second, collective action is implemented whereby all subjects are required to purchase the group?s median quantity at a given price. Participants in the experiment were offered the opportunity to contribute to climate protection by purchasing European Union Allowances. Allowances purchased were withdrawn from the European Emissions Trading Scheme. In our experiment, information about other subjects? behaviour has no treatment effect on the demand for climate protection. Under collective action however, the probability of purchasing allowances is higher compared to the reference treatment situation, an individual contribution mechanism. Furthermore, we observe a strong correlation between subjects? demand and their expectations about other participants? behaviour. When collective action is not available, subjects? e xpectations are consistent with free rider behaviour.

Dynamics in the governance of collective irrigation systems: Evidence from field experiments in Nicaragua.

The objective of this study is to analyze the common pool resource appropriation and public good provisiondecisions in a dynamic setting, testing the differences in behavior and performance between lab and field subjects. We performeda total of 45 games in Nicaragua, including 88 villagers in rural communities and 92 undergraduate students. In order to analyze sequential decision making, we introduce a dynamic and asymmetric irrigation game that combines the typical social dilemmas associated to irrigation systems management.In addition, in 9 out of 22 villagers’ groups, we implemented a treatment that included the disclosure of subjects’ appropriation of the common pool resource. The results reveal that the provision of individuals’ appropriation level results in higher appropriation in subsequent rounds. In addition, the results show that non-treated villagers provide more public good than treated villagers but if compared with students the differences are not significant. The results also suggest that appropriation levels are below the Nash prediction of full appropriation, but above the social efficient level. This results in an efficiency loss in the game that can be explained to a large extent by individual decisions on appropriation and public good contribution and by group appropriation behavior.