Stepwise multiple testing as formalized data snooping

It is common in econometric applications that several hypothesis tests arecarried out at the same time. The problem then becomes how to decide whichhypotheses to reject, accounting for the multitude of tests. In this paper,we suggest a stepwise multiple testing procedure which asymptoticallycontrols the familywise error rate at a desired level. Compared to relatedsingle-step methods, our procedure is more powerful in the sense that itoften will reject more false hypotheses. In addition, we advocate the useof studentization when it is feasible. Unlike some stepwise methods, ourmethod implicitly captures the joint dependence structure of the teststatistics, which results in increased ability to detect alternativehypotheses. We prove our method asymptotically controls the familywise errorrate under minimal assumptions. We present our methodology in the context ofcomparing several strategies to a common benchmark and deciding whichstrategies actually beat the benchmark. However, our ideas can easily beextended and/or modied to other contexts, such as making inference for theindividual regression coecients in a multiple regression framework. Somesimulation studies show the improvements of our methods over previous proposals. We also provide an application to a set of real data.

Exact and approximate stepdown methods for multiple hypothesis testing

Consider the problem of testing k hypotheses simultaneously. In this paper,we discuss finite and large sample theory of stepdown methods that providecontrol of the familywise error rate (FWE). In order to improve upon theBonferroni method or Holm's (1979) stepdown method, Westfall and Young(1993) make eective use of resampling to construct stepdown methods thatimplicitly estimate the dependence structure of the test statistics. However,their methods depend on an assumption called subset pivotality. The goalof this paper is to construct general stepdown methods that do not requiresuch an assumption. In order to accomplish this, we take a close look atwhat makes stepdown procedures work, and a key component is a monotonicityrequirement of critical values. By imposing such monotonicity on estimatedcritical values (which is not an assumption on the model but an assumptionon the method), it is demonstrated that the problem of constructing a validmultiple test procedure which controls the FWE can be reduced to the problemof contructing a single test which controls the usual probability of a Type 1error. This reduction allows us to draw upon an enormous resamplingliterature as a general means of test contruction.

Automated quantitative pupillometry for the prognostication of coma after cardiac arrest.

BACKGROUND: Sedation and therapeutic hypothermia (TH) delay neurological responses and might reduce the accuracy of clinical examination to predict outcome after cardiac arrest (CA). We examined the accuracy of quantitative pupillary light reactivity (PLR), using an automated infrared pupillometry, to predict outcome of post-CA coma in comparison to standard PLR, EEG, and somato-sensory evoked potentials (SSEP). METHODS: We prospectively studied over a 1-year period (June 2012-June 2013) 50 consecutive comatose CA patients treated with TH (33 °C, 24 h). Quantitative PLR (expressed as the % of pupillary response to a calibrated light stimulus) and standard PLR were measured at day 1 (TH and sedation; on average 16 h after CA) and day 2 (normothermia, off sedation: on average 46 h after CA). Neurological outcome was assessed at 90 days with Cerebral Performance Categories (CPC), dichotomized as good (CPC 1-2) versus poor (CPC 3-5). Predictive performance was analyzed using area under the ROC curves (AUC). RESULTS: Patients with good outcome [n = 23 (46 %)] had higher quantitative PLR than those with poor outcome [n = 27; 16 (range 9-23) vs. 10 (1-30) % at day 1, and 20 (13-39) vs. 11 (1-55) % at day 2, both p < 0.001]. Best cut-off for outcome prediction of quantitative PLR was <13 %. The AUC to predict poor outcome was higher for quantitative than for standard PLR at both time points (day 1, 0.79 vs. 0.56, p = 0.005; day 2, 0.81 vs. 0.64, p = 0.006). Prognostic accuracy of quantitative PLR was comparable to that of EEG and SSEP (0.81 vs. 0.80 and 0.73, respectively, both p > 0.20). CONCLUSIONS: Quantitative PLR is more accurate than standard PLR in predicting outcome of post-anoxic coma, irrespective of temperature and sedation, and has comparable prognostic accuracy than EEG and SSEP.

P value and the theory of hypothesis testing: an explanation for new researchers.

In the 1920s, Ronald Fisher developed the theory behind the p value and Jerzy Neyman and Egon Pearson developed the theory of hypothesis testing. These distinct theories have provided researchers important quantitative tools to confirm or refute their hypotheses. The p value is the probability to obtain an effect equal to or more extreme than the one observed presuming the null hypothesis of no effect is true; it gives researchers a measure of the strength of evidence against the null hypothesis. As commonly used, investigators will select a threshold p value below which they will reject the null hypothesis. The theory of hypothesis testing allows researchers to reject a null hypothesis in favor of an alternative hypothesis of some effect. As commonly used, investigators choose Type I error (rejecting the null hypothesis when it is true) and Type II error (accepting the null hypothesis when it is false) levels and determine some critical region. If the test statistic falls into that critical region, the null hypothesis is rejected in favor of the alternative hypothesis. Despite similarities between the two, the p value and the theory of hypothesis testing are different theories that often are misunderstood and confused, leading researchers to improper conclusions. Perhaps the most common misconception is to consider the p value as the probability that the null hypothesis is true rather than the probability of obtaining the difference observed, or one that is more extreme, considering the null is true. Another concern is the risk that an important proportion of statistically significant results are falsely significant. Researchers should have a minimum understanding of these two theories so that they are better able to plan, conduct, interpret, and report scientific experiments.

FastEpistasis: a high performance computing solution for quantitative trait epistasis.

Motivation: Genome-wide association studies have become widely used tools to study effects of genetic variants on complex diseases. While it is of great interest to extend existing analysis methods by considering interaction effects between pairs of loci, the large number of possible tests presents a significant computational challenge. The number of computations is further multiplied in the study of gene expression quantitative trait mapping, in which tests are performed for thousands of gene phenotypes simultaneously. Results: We present FastEpistasis, an efficient parallel solution extending the PLINK epistasis module, designed to test for epistasis effects when analyzing continuous phenotypes. Our results show that the algorithm scales with the number of processors and offers a reduction in computation time when several phenotypes are analyzed simultaneously. FastEpistasis is capable of testing the association of a continuous trait with all single nucleotide polymorphism ( SNP) pairs from 500 000 SNPs, totaling 125 billion tests, in a population of 5000 individuals in 29, 4 or 0.5 days using 8, 64 or 512 processors.

Assessment of Nondestructive Testing Technologies for Quality Control/Quality Assurance of Asphalt Mixtures, TR-653, 2015

Asphalt pavements suffer various failures due to insufficient quality within their design lives. The American Association of State Highway and Transportation Officials (AASHTO) Mechanistic-Empirical Pavement Design Guide (MEPDG) has been proposed to improve pavement quality through quantitative performance prediction. Evaluation of the actual performance (quality) of pavements requires in situ nondestructive testing (NDT) techniques that can accurately measure the most critical, objective, and sensitive properties of pavement systems. The purpose of this study is to assess existing as well as promising new NDT technologies for quality control/quality assurance (QC/QA) of asphalt mixtures. Specifically, this study examined field measurements of density via the PaveTracker electromagnetic gage, shear-wave velocity via surface-wave testing methods, and dynamic stiffness via the Humboldt GeoGauge for five representative paving projects covering a range of mixes and traffic loads. The in situ tests were compared against laboratory measurements of core density and dynamic modulus. The in situ PaveTracker density had a low correlation with laboratory density and was not sensitive to variations in temperature or asphalt mix type. The in situ shear-wave velocity measured by surface-wave methods was most sensitive to variations in temperature and asphalt mix type. The in situ density and in situ shear-wave velocity were combined to calculate an in situ dynamic modulus, which is a performance-based quality measurement. The in situ GeoGauge stiffness measured on hot asphalt mixtures several hours after paving had a high correlation with the in situ dynamic modulus and the laboratory density, whereas the stiffness measurement of asphalt mixtures cooled with dry ice or at ambient temperature one or more days after paving had a very low correlation with the other measurements. To transform the in situ moduli from surface-wave testing into quantitative quality measurements, a QC/QA procedure was developed to first correct the in situ moduli measured at different field temperatures to the moduli at a common reference temperature based on master curves from laboratory dynamic modulus tests. The corrected in situ moduli can then be compared against the design moduli for an assessment of the actual pavement performance. A preliminary study of microelectromechanical systems- (MEMS)-based sensors for QC/QA and health monitoring of asphalt pavements was also performed.

A general theory of rank testing

This paper develops an approach to rank testing that nests all existing rank tests andsimplifies their asymptotics. The approach is based on the fact that implicit in every ranktest there are estimators of the null spaces of the matrix in question. The approach yieldsmany new insights about the behavior of rank testing statistics under the null as well as localand global alternatives in both the standard and the cointegration setting. The approach alsosuggests many new rank tests based on alternative estimates of the null spaces as well as thenew fixed-b theory. A brief Monte Carlo study illustrates the results.

Carbonate Rock Pore Size Distribution Determination through Iowa Pore Index Testing, MLR-15-01, 2015

The Iowa Pore Index (IPI) measures the pore system of carbonate (limestone and dolomite) rocks using pressurized water to infiltrate the pore system. This technique provides quantitative results for the primary and capillary (secondary) pores in carbonate rocks. These results are used in conjunction with chemical and mineralogical test results to calculate a quality number, which is used as a predictor of aggregate performance in Portland cement concrete (PCC) leading to the durability classification of the aggregate. This study had two main objectives: to determine the effect different aggregate size has on IPI test results and to establish the precision of IPI test and test apparatus. It was found that smaller aggregate size fractions could be correlated to the standard 1/2”-3/4” size sample. Generally, a particle size decrease was accompanied by a slight decrease in IPI values. The IPI testing also showed fairly good agreement of the secondary pore index number between the 1/2”-3/4”and the 3/8”-1/2” fraction. The #4-3/8” showed a greater difference of the secondary number from the 1/2”-3/4” fraction. The precision of the IPI test was established as a standard deviation (Sr) of 2.85 (Primary) and 0.87 (Secondary) with a repeatability limit (%r) of 8.5% and 14.9% for the primary and secondary values, respectively.

Results of An International Study of Quantitative BCR/ABL Assays Using Defined RNA Samples

Molecular monitoring of BCR/ABL transcripts by real time quantitative reverse transcription PCR (qRT-PCR) is an essential technique for clinical management of patients with BCR/ABL-positive CML and ALL. Though quantitative BCR/ABL assays are performed in hundreds of laboratories worldwide, results among these laboratories cannot be reliably compared due to heterogeneity in test methods, data analysis, reporting, and lack of quantitative standards. Recent efforts towards standardization have been limited in scope. Aliquots of RNA were sent to clinical test centers worldwide in order to evaluate methods and reporting for e1a2, b2a2, and b3a2 transcript levels using their own qRT-PCR assays. Total RNA was isolated from tissue culture cells that expressed each of the different BCR/ABL transcripts. Serial log dilutions were prepared, ranging from 100 to 10-5, in RNA isolated from HL60 cells. Laboratories performed 5 independent qRT-PCR reactions for each sample type at each dilution. In addition, 15 qRT-PCR reactions of the 10-3 b3a2 RNA dilution were run to assess reproducibility within and between laboratories. Participants were asked to run the samples following their standard protocols and to report cycle threshold (Ct), quantitative values for BCR/ABL and housekeeping genes, and ratios of BCR/ABL to housekeeping genes for each sample RNA. Thirty-seven (n=37) participants have submitted qRT-PCR results for analysis (36, 37, and 34 labs generated data for b2a2, b3a2, and e1a2, respectively). The limit of detection for this study was defined as the lowest dilution that a Ct value could be detected for all 5 replicates. For b2a2, 15, 16, 4, and 1 lab(s) showed a limit of detection at the 10-5, 10-4, 10-3, and 10-2 dilutions, respectively. For b3a2, 20, 13, and 4 labs showed a limit of detection at the 10-5, 10-4, and 10-3 dilutions, respectively. For e1a2, 10, 21, 2, and 1 lab(s) showed a limit of detection at the 10-5, 10-4, 10-3, and 10-2 dilutions, respectively. Log %BCR/ABL ratio values provided a method for comparing results between the different laboratories for each BCR/ABL dilution series. Linear regression analysis revealed concordance among the majority of participant data over the 10-1 to 10-4 dilutions. The overall slope values showed comparable results among the majority of b2a2 (mean=0.939; median=0.9627; range (0.399 - 1.1872)), b3a2 (mean=0.925; median=0.922; range (0.625 - 1.140)), and e1a2 (mean=0.897; median=0.909; range (0.5174 - 1.138)) laboratory results (Fig. 1-3)). Thirty-four (n=34) out of the 37 laboratories reported Ct values for all 15 replicates and only those with a complete data set were included in the inter-lab calculations. Eleven laboratories either did not report their copy number data or used other reporting units such as nanograms or cell numbers; therefore, only 26 laboratories were included in the overall analysis of copy numbers. The median copy number was 348.4, with a range from 15.6 to 547,000 copies (approximately a 4.5 log difference); the median intra-lab %CV was 19.2% with a range from 4.2% to 82.6%. While our international performance evaluation using serially diluted RNA samples has reinforced the fact that heterogeneity exists among clinical laboratories, it has also demonstrated that performance within a laboratory is overall very consistent. Accordingly, the availability of defined BCR/ABL RNAs may facilitate the validation of all phases of quantitative BCR/ABL analysis and may be extremely useful as a tool for monitoring assay performance. Ongoing analyses of these materials, along with the development of additional control materials, may solidify consensus around their application in routine laboratory testing and possible integration in worldwide efforts to standardize quantitative BCR/ABL testing.

Observations on Software Testing Practice

This thesis investigates factors that affect software testing practice. The thesis consists of empirical studies, in which the affecting factors were analyzed and interpreted using quantitative and qualitative methods. First, the Delphi method was used to specify the scope of the thesis. Secondly, for the quantitative analysis 40industry experts from 30 organizational units (OUs) were interviewed. The survey method was used to explore factors that affect software testing practice. Conclusions were derived using correlation and regression analysis. Thirdly, from these 30 OUs, five were further selected for an in-depth case study. The data was collected through 41 semi-structured interviews. The affecting factors and their relationships were interpreted with qualitative analysis using grounded theory as the research method. The practice of software testing was analyzed from the process improvement and knowledge management viewpoints. The qualitative and quantitativeresults were triangulated to increase the validity of the thesis. Results suggested that testing ought to be adjusted according to the business orientation of the OU; the business orientation affects the testing organization and knowledge management strategy, and the business orientation andthe knowledge management strategy affect outsourcing. As a special case, the complex relationship between testing schedules and knowledge transfer is discussed. The results of this thesis can be used in improvingtesting processes and knowledge management in software testing.

Sensory characteristics of 'Douradão' peaches submitted to modified atmosphere packaging

The sensory, physical and chemical characteristics of 'Douradão' peaches cold stored in different modified atmosphere packaging (LDPE bags of 30, 50, 60, 75µm thickness) were studied. After 14, 21 and 28 days of cold storage (1 ± 1 ºC and 90 ± 5% RH), samples were withdrawn from MAP and kept during 4 days in ambient air for ripening. Descriptive terminology and sensory profile of the peaches were developed by methodology based on the Quantitative Descriptive Analysis (QDA). The assessors consensually defined the sensory descriptors, their respective reference materials and the descriptive evaluation ballot. Fourteen individuals were selected as judges based on their discrimination capacity and reproducibility. Seven descriptors were generated showing similarities and differences among the samples. The data were analysed by ANOVA, Tukey test and Principal Component Analysis (PCA). The atmospheres that developed inside the different packaging materials during cold storage differed significantly. The PCA showed that MA50 and MA60 treatments were more characterized by the fresh peach flavour, fresh appearance, juiciness and flesh firmness, and were effective for keeping good quality of 'Douradão' peaches during 28 d of cold storage. The Control and MA30 treatments were characterized by the mealiness, the MA75 treatment showed lower intensity for all attributes evaluated and they were ineffective to maintain good quality of the fruits during cold storage. Higher correlation coefficients (positive) were found between fresh appearance and flesh firmness (0.95), fresh appearance and juiciness (0.97), ratio and intensity of fresh peach smell (0.81), as well as higher correlation coefficients (negative) between Hue angle and intensity of yellow colour (-0.91), fresh appearance and mealiness (-0.92), juiciness and mealiness (-0.95), firmness and mealiness (-0.94).

Testing a new multigroup inference approach to reconstructing past environmental conditions

A new, quantitative, inference model for environmental reconstruction (transfer function), based for the first time on the simultaneous analysis of multigroup species, has been developed. Quantitative reconstructions based on palaeoecological transfer functions provide a powerful tool for addressing questions of environmental change in a wide range of environments, from oceans to mountain lakes, and over a range of timescales, from decades to millions of years. Much progress has been made in the development of inferences based on multiple proxies but usually these have been considered separately, and the different numeric reconstructions compared and reconciled post-hoc. This paper presents a new method to combine information from multiple biological groups at the reconstruction stage. The aim of the multigroup work was to test the potential of the new approach to making improved inferences of past environmental change by improving upon current reconstruction methodologies. The taxonomic groups analysed include diatoms, chironomids and chrysophyte cysts. We test the new methodology using two cold-environment training-sets, namely mountain lakes from the Pyrenees and the Alps. The use of multiple groups, as opposed to single groupings, was only found to increase the reconstruction skill slightly, as measured by the root mean square error of prediction (leave-one-out cross-validation), in the case of alkalinity, dissolved inorganic carbon and altitude (a surrogate for air-temperature), but not for pH or dissolved CO2. Reasons why the improvement was less than might have been anticipated are discussed. These can include the different life-forms, environmental responses and reaction times of the groups under study.

Sensory profiles of chocolates produced from cocoa cultivars resistant to Moniliophtora Perniciosa

The present study evaluated the sensory quality of chocolates obtained from two cocoa cultivars (PH16 and SR162) resistant to Moniliophtora perniciosa mould comparing to a conventional cocoa that is not resistant to the disease. The acceptability of the chocolates was assessed and the promising cultivars with relevant sensory and commercial attributes could be indicated to cocoa producers and chocolate manufacturers. The descriptive terminology and the sensory profile of chocolates were developed by Quantitative Descriptive Analysis (QDA). Ten panelists, selected on the basis of their discriminatory capacity and reproducibility, defined eleven sensory descriptors, their respective reference materials and the descriptive evaluation ballot. The data were analyzed using ANOVA, Principal Component Analysis (PCA) and Tukey's test to compare the means. The results revealed significant differences among the sensory profiles of the chocolates. Chocolates from the PH16 cultivar were characterized by a darker brown color, more intense flavor and odor of chocolate, bitterness and a firmer texture, which are important sensory and commercial attributes. Chocolates from the SR162 cultivar were characterized by a greater sweetness and melting quality and chocolates from the conventional treatment presented intermediate sensory characteristics between those of the other two chocolates. All samples indicated high acceptance, but chocolates from the PH16 and conventional cultivars obtained higher purchase intention scores.

International STakeholder NETwork (ISTNET): creating a developmental neurotoxicity (DNT) testing road map for regulatory purposes.

A major problem in developmental neurotoxicity (DNT) risk assessment is the lack of toxicological hazard information for most compounds. Therefore, new approaches are being considered to provide adequate experimental data that allow regulatory decisions. This process requires a matching of regulatory needs on the one hand and the opportunities provided by new test systems and methods on the other hand. Alignment of academically and industrially driven assay development with regulatory needs in the field of DNT is a core mission of the International STakeholder NETwork (ISTNET) in DNT testing. The first meeting of ISTNET was held in Zurich on 23-24 January 2014 in order to explore the concept of adverse outcome pathway (AOP) to practical DNT testing. AOPs were considered promising tools to promote test systems development according to regulatory needs. Moreover, the AOP concept was identified as an important guiding principle to assemble predictive integrated testing strategies (ITSs) for DNT. The recommendations on a road map towards AOP-based DNT testing is considered a stepwise approach, operating initially with incomplete AOPs for compound grouping, and focussing on key events of neurodevelopment. Next steps to be considered in follow-up activities are the use of case studies to further apply the AOP concept in regulatory DNT testing, making use of AOP intersections (common key events) for economic development of screening assays, and addressing the transition from qualitative descriptions to quantitative network modelling.

Food Safety Testing: Rapid Molecular Methods for Chemical and Biological Hazards

The central goal of food safety policy in the European Union (EU) is to protect consumer health by guaranteeing a high level of food safety throughout the food chain. This goal can in part be achieved by testing foodstuffs for the presence of various chemical and biological hazards. The aim of this study was to facilitate food safety testing by providing rapid and user-friendly methods for the detection of particular food-related hazards. Heterogeneous competitive time-resolved fluoroimmunoassays were developed for the detection of selected veterinary residues, that is coccidiostat residues, in eggs and chicken liver. After a simplified sample preparation procedure, the immunoassays were performed either in manual format with dissociation-enhanced measurement or in automated format with pre-dried assay reagents and surface measurement. Although the assays were primarily designed for screening purposes providing only qualitative results, they could also be used in a quantitative mode. All the developed assays had good performance characteristics enabling reliable screening of samples at concentration levels required by the authorities. A novel polymerase chain reaction (PCR)-based assay system was developed for the detection of Salmonella spp. in food. The sample preparation included a short non-selective pre-enrichment step, after which the target cells were collected with immunomagnetic beads and applied to PCR reaction vessels containing all the reagents required for the assay in dry form. The homogeneous PCR assay was performed with a novel instrument platform, GenomEra^™, and the qualitative assay results were automatically interpreted based on end-point time-resolved fluorescence measurements and cut-off values. The assay was validated using various food matrices spiked with sub-lethally injured Salmonella cells at levels of 1-10 colony forming units (CFU)/25 g of food. The main advantage of the system was the exceptionally short time to result; the entire process starting from the pre-enrichment and ending with the PCR result could be completed in eight hours. In conclusion, molecular methods using state-of-the-art assay techniques were developed for food safety testing. The combination of time-resolved fluorescence detection and ready-to-use reagents enabled sensitive assays easily amenable to automation. Consequently, together with the simplified sample preparation, these methods could prove to be applicable in routine testing.