More on Testing for Validity Instead of Looking for It

Using Monte Carlo simulations and reanalyzing the data of a validation study of the AEIM emotional intelligence test, we demonstrated that an atheoretical approach and the use of weak statistical procedures can result in biased validity estimates. These procedures included stepwise regression-and the general case of failing to include important theoretical controls-extreme scores analysis, and ignoring heteroscedasticity as well as measurement error. The authors of the AEIM test responded by offering more complete information about their analyses, allowing us to further examine the perils of ignoring theory and correct statistical procedures. In this paper we show with extended analyses that the AEIM test is invalid.

Validity of an arterial pressure waveform analysis device: does the puncture site play a role in the agreement with intermittent pulmonary artery catheter thermodilution measurements?

OBJECTIVE: The measurement of cardiac output is a key element in the assessment of cardiac function. Recently, a pulse contour analysis-based device without need for calibration became available (FloTrac/Vigileo, Edwards Lifescience, Irvine, CA). This study was conducted to determine if there is an impact of the arterial catheter site and to investigate the accuracy of this system when compared with the pulmonary artery catheter using the bolus thermodilution technique (PAC). DESIGN: Prospective study. SETTING: The operating room of 1 university hospital. PARTICIPANTS: Twenty patients undergoing cardiac surgery. INTERVENTIONS: CO was determined in parallel by the use of the Flotrac/Vigileo systems in the radial and femoral position (CO_rad and CO_fem) and by PAC as the reference method. Data triplets were recorded at defined time points. The primary endpoint was the comparison of CO_rad and CO_fem, and the secondary endpoint was the comparison with the PAC. MEASUREMENTS AND MAIN RESULTS: Seventy-eight simultaneous data recordings were obtained. The Bland-Altman analysis for CO_fem and CO_rad showed a bias of 0.46 L/min, precision was 0.85 L/min, and the percentage error was 34%. The Bland-Altman analysis for CO_rad and PAC showed a bias of -0.35 L/min, the precision was 1.88 L/min, and the percentage error was 76%. The Bland-Altman analysis for CO_fem and PAC showed a bias of 0.11 L/min, the precision was 1.8 L/min, and the percentage error was 69%. CONCLUSION: The FloTrac/Vigileo system was shown to not produce exactly the same CO data when used in radial and femoral arteries, even though the percentage error was close to the clinically acceptable range. Thus, the impact of the introduction site of the arterial catheter is not negligible. The agreement with thermodilution was low.

Validity of Charlson Comorbidity Index in patients hospitalised with acute coronary syndrome. Insights from the nationwide AMIS Plus registry 2002-2012.

OBJECTIVE: This study aimed to assess the impact of individual comorbid conditions as well as the weight assignment, predictive properties and discriminating power of the Charlson Comorbidity Index (CCI) on outcome in patients with acute coronary syndrome (ACS). METHODS: A prospective multicentre observational study (AMIS Plus Registry) from 69 Swiss hospitals with 29 620 ACS patients enrolled from 2002 to 2012. The main outcome measures were in-hospital and 1-year follow-up mortality. RESULTS: Of the patients, 27% were female (age 72.1 ± 12.6 years) and 73% were male (64.2 ± 12.9 years). 46.8% had comorbidities and they were less likely to receive guideline-recommended drug therapy and reperfusion. Heart failure (adjusted OR 1.88; 95% CI 1.57 to 2.25), metastatic tumours (OR 2.25; 95% CI 1.60 to 3.19), renal diseases (OR 1.84; 95% CI 1.60 to 2.11) and diabetes (OR 1.35; 95% CI 1.19 to 1.54) were strong predictors of in-hospital mortality. In this population, CCI weighted the history of prior myocardial infarction higher (1 instead of -0.4, 95% CI -1.2 to 0.3 points) but heart failure (1 instead of 3.7, 95% CI 2.6 to 4.7) and renal disease (2 instead of 3.5, 95% CI 2.7 to 4.4) lower than the benchmark, where all comorbidities, age and gender were used as predictors. However, the model with CCI and age has an identical discrimination to this benchmark (areas under the receiver operating characteristic curves were both 0.76). CONCLUSIONS: Comorbidities greatly influenced clinical presentation, therapies received and the outcome of patients admitted with ACS. Heart failure, diabetes, renal disease or metastatic tumours had a major impact on mortality. CCI seems to be an appropriate prognostic indicator for in-hospital and 1-year outcomes in ACS patients. ClinicalTrials.gov Identifier: NCT01305785.

Induced quantum metric fluctuations and the validity of semiclassical gravity

We propose a criterion for the validity of semiclassical gravity (SCG) which is based on the stability of the solutions of SCG with respect to quantum metric fluctuations. We pay special attention to the two-point quantum correlation functions for the metric perturbations, which contain both intrinsic and induced fluctuations. These fluctuations can be described by the Einstein-Langevin equation obtained in the framework of stochastic gravity. Specifically, the Einstein-Langevin equation yields stochastic correlation functions for the metric perturbations which agree, to leading order in the large N limit, with the quantum correlation functions of the theory of gravity interacting with N matter fields. The homogeneous solutions of the Einstein-Langevin equation are equivalent to the solutions of the perturbed semiclassical equation, which describe the evolution of the expectation value of the quantum metric perturbations. The information on the intrinsic fluctuations, which are connected to the initial fluctuations of the metric perturbations, can also be retrieved entirely from the homogeneous solutions. However, the induced metric fluctuations proportional to the noise kernel can only be obtained from the Einstein-Langevin equation (the inhomogeneous term). These equations exhibit runaway solutions with exponential instabilities. A detailed discussion about different methods to deal with these instabilities is given. We illustrate our criterion by showing explicitly that flat space is stable and a description based on SCG is a valid approximation in that case.

Bilateral symmetry of radial pulse in high-level tennis players : implications for the validity of central aortic pulse wave analysis

Rapport de synthèse Ce travail de thèse s'articule autour de l'importance de l'évaluation de la fonction vasculaire et des répercussions au niveau central, cardiaque, des perturbations du réseau vasculaire. Les maladies cardiovasculaires sont prédominantes dans notre société et causes de morbidité et mortalité importante. La mesure de la pression artérielle classique reste le moyen le plus utilisé pour suivre la santé des vaisseaux, mais ne reflète pas directement ce qui se passe au niveau du coeur. La tonométrie d'aplanation permet depuis quelques années de mesurer l'onde de pouls radial, et par le biais d'une fonction mathématique de transfert validée, il est possible d'en déduire la forme et Γ amplitude de l'onde de pouls central, donc de la pression aortique centrale. Cette dernière est un reflet bien plus direct de la post-charge cardiaque, et de nombreuses études cliniques actuelles s'intéressent à cette mesure pour stratifier le risque ou évaluer l'effet d'un traitement vasculaire. Toutefois, bien que cet outil soit de plus en plus utilisé, il est rarement précisé si la latéralité de la mesure joue un rôle, sachant que certaines propriétés des membres supérieurs peuvent être affectées par un usage préférentiel (masse musculaire, densité osseuse, diamètre des artères, capillarisation musculaire, et même fonction endothéliale). On a en effet observé que ces divers paramètre étaient tous augmentés sur un bras entraîné. Dès lors on peut se poser la question de l'influence de ces adaptations physiologiques sur la mesure indirecte effectuée par le biais du pouls radial. Nous avons investigué les deux membres supérieurs de sujets jeunes et sédentaires (SED), ainsi que ceux de sujets sportifs avec un développement fortement asymétrique des bras, soit des joueurs de tennis de haut niveau (TEN). Des mesures anthropométriques incluant la composition corporelle et la circonférence des bras et avant-bras ont montré que TEN présente une asymétrie hautement significative aux deux mesures entre le bras dominant (entraîné) et l'autre, ce qui est aussi présent pour la force de serrage (mesurée au dynamomètre de Jamar). L'analyse des courbes centrales de pouls ne montre aucune différence entre les deux membres dans chaque groupe, par contre on peut observer une différence entre SED et TEN, avec un index d'augmentation diastolique qui est 50 % plus élevé chez TEN. Les index d'augmentation systolique sont identiques dans les deux groupes. On peut retenir de cette étude la validité de la méthode de tonométrie d'aplanation quel que soit le bras utilisé (dominant ou non-dominant) et ce même si une asymétrie conséquente est présente. Ces données sont clairement nouvelles et permettent de s'affranchir de cette variable dans la mesure d'un paramètre cardiovasculaire dont l'importance est actuellement grandissante. Les différences d'index diastolique sont expliquées par la fréquence cardiaque et la vitesse de conduction de l'onde de pouls plus basses chez TEN, causant un retard diastolique du retour de l'onde au niveau central, phénomène précédemment bien décrit dans la littérature.

The ability model of emotional intelligence: Searching for valid measures

Current measures of ability emotional intelligence (EI)--including the well-known Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT)--suffer from several limitations, including low discriminant validity and questionable construct and incremental validity. We show that the MSCEIT is largely predicted by personality dimensions, general intelligence, and demographics having multiple R's with the MSCEIT branches up to .66; for the general EI factor this relation was even stronger (Multiple R = .76). As concerns the factor structure of the MSCEIT, we found support for four first-order factors, which had differential relations with personality, but no support for a higher-order global EI factor. We discuss implications for employing the MSCEIT, including (a) using the single branches scores rather than the total score, (b) always controlling for personality and general intelligence to ensure unbiased parameter estimates in the EI factors, and (c) correcting for measurement error. Failure to account for these methodological aspects may severely compromise predictive validity testing. We also discuss avenues for the improvement of ability-based tests.

Reproducibility and relative validity of a food-frequency questionnaire for French-speaking Swiss adults.

BACKGROUND: Due to the distinct cultural and language differences that exist in Switzerland, there is little information on the dietary intake among the general Swiss population. Adequately assessing dietary intake is thus paramount if nutritional epidemiological studies are to be conducted. OBJECTIVE: To assess the reproducibility and validity of a food-frequency questionnaire (FFQ) developed for French-speaking Swiss adults. DESIGN: A total of 23 men and 17 women (43.1+/-2.0 years) filled out one FFQ and completed one 24-hour dietary recall at baseline and 1 month afterward. RESULTS: Crude Pearson's correlation coefficients between the first and the second FFQ ranged from 0.58 to 0.90, intraclass correlation coefficient (ICC) ranged between 0.53 and 0.92. Lin's concordance coefficients ranged between 0.55 and 0.87. Over 80% of participants were classified in the same or adjacent tertile using each FFQ. Macronutrient intakes estimated by both FFQs were significantly higher than those estimated from the 24-hour recall for protein and water, while no significant differences were found for energy, carbohydrate, fats (five groups), and alcohol. De-attenuated Pearson's correlation coefficients between the 24-hour recall and the first FFQ ranged between 0.31 and 0.49, while for the second FFQ the values ranged between 0.38 and 0.59. Over 40 and 95% of participants fell into the same or the adjacent energy and nutrient tertiles, respectively, using the FFQs and the 24-hour recall. CONCLUSIONS: This FFQ shows good reproducibility and can be used determining macronutrient intake in a French-speaking Swiss population in an epidemiological setting.

Looking for Validity or Testing It? The Perils of Stepwise Regression, Extreme-Scores Analysis, Heteroscedasticity, and Measurement Error

When researchers introduce a new test they have to demonstrate that it is valid, using unbiased designs and suitable statistical procedures. In this article we use Monte Carlo analyses to highlight how incorrect statistical procedures (i.e., stepwise regression, extreme scores analyses) or ignoring regression assumptions (e.g., heteroscedasticity) contribute to wrong validity estimates. Beyond these demonstrations, and as an example, we re-examined the results reported by Warwick, Nettelbeck, and Ward (2010) concerning the validity of the Ability Emotional Intelligence Measure (AEIM). Warwick et al. used the wrong statistical procedures to conclude that the AEIM was incrementally valid beyond intelligence and personality traits in predicting various outcomes. In our re-analysis, we found that the reliability-corrected multiple correlation of their measures with personality and intelligence was up to .69. Using robust statistical procedures and appropriate controls, we also found that the AEIM did not predict incremental variance in GPA, stress, loneliness, or well-being, demonstrating the importance for testing validity instead of looking for it.

The family alliance assessment scales : steps toward validity and reliability of an observational assessment tool for early family interactions

We present the first steps in the validation of an observational tool for father-mother-infant interactions: the FAAS (Family Alliance Assessment Scales). Family-level variables are acknowledged as unique contributors to the understanding of the socio-affective development of the child, yet producing reliable assessments of family-level interactions poses a methodological challenge. There is, therefore, a clear need for a validated and clinically relevant tool. This validation study has been carried out on three samples: one non-referred sample, of families taking part in a study on the transition to parenthood (normative sample; n = 30), one referred for medically assisted procreation (infertility sample; n = 30) and one referred for a psychiatric condition in one parent (clinical sample; n = 15). Results show that the FAAS scales have (1) good inter-rater reliability and (2) good validity, as assessed through known-group validity by comparing the three samples and through concurrent validity by checking family interactions against parents' self-reported marital satisfaction.

The External validity in economic experiments

One of the most important questions regarding experimental economics is the external validity of laboratory experiments. This paper goes through a study that tests the generalizability of a Dictator Game as a laboratory analogue for a naturally occurringdecision-making context of teacher absenteeism. Because lab and naturally-occurring environments systematically differ we then discuss other factors that might strongly affect the choices that individuals make. We conclude that the dichotomy drawn between labexperiments and data collected from natural settings is a false one. A combination of the two would provide deeper and better insights than either separately.

Use of the Cognitive Performance Scale (CPS) to detect cognitive impairment in the acute care setting: concurrent and predictive validity.

The Cognitive Performance Scale (CPS) was initially designed to assess cognition in long term care residents. Subsequently, the CPS has also been used among in-home, post-acute, and acute care populations even though CPS' clinimetric performance has not been studied in these settings. This study aimed to determine CPS agreement with the Mini Mental Status Exam (MMSE) and its predictive validity for institutionalization and death in a cohort (N=401) of elderly medical inpatients aged 75 years and over. Medical, physical and mental status were assessed upon admission. The same day, the patient's nurse completed the CPS by interview. Follow-up data were gathered from the central billing system (nursing home stay) and proxies (death). Cognitive impairment was present in 92 (23%) patients according to CPS (score >or= 2). Agreement with MMSE was moderate (kappa 0.52, P<.001). Analysis of discordant results suggested that cognitive impairment was overestimated by the CPS in dependent patients with comorbidities and depressive symptoms, and underestimated in older ones. During follow-up, subjects with abnormal CPS had increased risks of death (adjusted hazard ratio (adjHR) 1.7, 95% CI 1.0-2.8, P=.035) and institutionalization (adjHR 2.7, 95% CI 1.3-5.3, P=.006), independent of demographic, health and functional status. Interestingly, subjects with abnormal CPS were at increased risk of death only if they also had abnormal MMSE. The CPS predicted death and institutionalization during follow-up, but correlated moderately well with the MMSE. Combining CPS and MMSE provided additional predictive information, suggesting that domains other than cognition are assessed by professionals when using the CPS in elderly medical inpatients.

Validity of the acoustic approximation in full-waveform seismic crosshole tomography

Acoustic waveform inversions are an increasingly popular tool for extracting subsurface information from seismic data. They are computationally much more efficient than elastic inversions. Naturally, an inherent disadvantage is that any elastic effects present in the recorded data are ignored in acoustic inversions. We investigate the extent to which elastic effects influence seismic crosshole data. Our numerical modeling studies reveal that in the presence of high contrast interfaces, at which P-to-S conversions occur, elastic effects can dominate the seismic sections, even for experiments involving pressure sources and pressure receivers. Comparisons of waveform inversion results using a purely acoustic algorithm on synthetic data that is either acoustic or elastic, show that subsurface models comprising small low-to-medium contrast (?30%) structures can be successfully resolved in the acoustic approximation. However, in the presence of extended high-contrast anomalous bodies, P-to-S-conversions may substantially degrade the quality of the tomographic images. In particular, extended low-velocity zones are difficult to image. Likewise, relatively small low-velocity features are unresolved, even when advanced a priori information is included. One option for mitigating elastic effects is data windowing, which suppresses later arriving seismic arrivals, such as shear waves. Our tests of this approach found it to be inappropriate because elastic effects are also included in earlier arriving wavetrains. Furthermore, data windowing removes later arriving P-wave phases that may provide critical constraints on the tomograms. Finally, we investigated the extent to which acoustic inversions of elastic data are useful for time-lapse analyses of high contrast engineered structures, for which accurate reconstruction of the subsurface structure is not as critical as imaging differential changes between sequential experiments. Based on a realistic scenario for monitoring a radioactive waste repository, we demonstrated that acoustic inversions of elastic data yield substantial distortions of the tomograms and also unreliable information on trends in the velocity changes.