Towards criterion validity in classroom language analysis: methodological constraints of metadiscourse and inter-rater agreement

This paper reports on a process to validate a revised version of a system for coding classroom discourse in foreign language lessons, a context in which the dual role of language (as content and means of communication) and the speakers' specific pedagogical aims lead to a certain degree of ambiguity in language analysis. The language used by teachers and students has been extensively studied, and a framework of concepts concerning classroom discourse well-established. Models for coding classroom language need, however, to be revised when they are applied to specific research contexts. The application and revision of an initial framework can lead to the development of earlier models, and to the re-definition of previously established categories of analysis that have to be validated. The procedures followed to validate a coding system are related here as guidelines for conducting research under similar circumstances. The advantages of using instruments that incorporate two types of data, that is, quantitative measures and qualitative information from raters' metadiscourse, are discussed, and it is suggested that such procedure can contribute to the process of validation itself, towards attaining reliability of research results, as well as indicate some constraints of the adopted research methodology.

Reliability, construct and criterion validity of the KIDSCREEN-10 score: a short measure for children and adolescents' well-being and health-related quality of life

Background To assess the criterion and construct validity of the KIDSCREEN-10 well-being and health-related quality of life (HRQoL) score, a short version of the KIDSCREEN-52 and KIDSCREEN-27 instruments. Methods The child self-report and parent report versions of the KIDSCREEN-10 were tested in a sample of 22,830 European children and adolescents aged 8–18 and their parents (n = 16,237). Correlation with the KIDSCREEN-52 and associations with other generic HRQoL measures, physical and mental health, and socioeconomic status were examined. Score differences by age, gender, and country were investigated. Results Correlations between the 10-item KIDSCREEN score and KIDSCREEN-52 scales ranged from r = 0.24 to 0.72 (r = 0.27–0.72) for the self-report version (proxy-report version). Coefficients below r = 0.5 were observed for the KIDSCREEN-52 dimensions Financial Resources and Being Bullied only. Cronbach alpha was 0.82 (0.78), test–retest reliability was ICC = 0.70 (0.67) for the self- (proxy-)report version. Correlations between other children self-completed HRQoL questionnaires and KIDSCREEN-10 ranged from r = 0.43 to r = 0.63 for the KIDSCREEN children self-report and r = 0.22–0.40 for the KIDSCREEN parent proxy report. Known group differences in HRQoL between physically/mentally healthy and ill children were observed in the KIDSCREEN-10 self and proxy scores. Associations with self-reported psychosomatic complaints were r = −0.52 (−0.36) for the KIDSCREEN-10 self-report (proxy-report). Statistically significant differences in KIDSCREEN-10 self and proxy scores were found by socioeconomic status, age, and gender. Conclusions Our results indicate that the KIDSCREEN-10 provides a valid measure of a general HRQoL factor in children and adolescents, but the instrument does not represent well most of the single dimensions of the original KIDSCREEN-52. Test–retest reliability was slightly below a priori defined thresholds.

Development and Criterion Validity of Differentiated and Elevated Vocational Interests in Adolescence

Interest differentiation and elevation are supposed to provide important information about a person’s state of interest development, yet little is known about their development and criterion validity. The present study explored these constructs among a group of Swiss adolescents. Study 1 applied a cross-sectional design with 210 students in 11th grade. Study 2 applied a 1-year longitudinal design with 289 students in 7th to 8th grade. gender, personality traits, and career exploration were significant predictors of state and development of differentiation and elevation. Increase in differentiation predicted increase in career decidedness above traits. elevation could not predict increase in exploration behavior over traits. The results provide support for differentiation and elevation as important aspects of adolescents’ vocational interests.

Criterion-related validity of the short form of the international physical activity questionnaire in adults who are Deaf

Background: To implement appropriate programs for promoting physical activity (PA) in people who are Deaf, it is important to have valid instruments for assessing PA in this population. Objective: The main purpose of this study was to examine the criterion validity of the short form of the International Physical Activity Questionnaire (IPAQ-S) in Deaf adults. Method: This study included 44 adults (18e65 years) of both genders (63.6% were females) who met the inclusion criteria. Objective measures of PAwere collected using accelerometers, which were worn by each participant during one week. After using the accelerometer, the IPAQ-S was applied to assess participants’ physical activity during the last 7 days. Results: There was no significant correlation between the average time spent in moderate to vigorous physical activity (MVPA) as measured by the accelerometer (40.1 6 24.5 min/day) and by the IPAQ-S (41.3 6 57.5 min/day). The IPAQ-S significantly underestimated the time spent in sedentary behavior (7.6 6 2.7 h/day vs. 10.1 6 1.6 h/day). Sedentary behavior and MVPA as measured by the accelerometer and the IPAQ-S showed limited agreement. Conclusions: Our results show some limitations on the use of IPAQ-S for quantifying PA among adults who are Deaf. The IPAQ-S tends to overestimate the MVPA and to underestimate sedentary behavior in adults who are Deaf.

Detecting and measuring deprivation in primary care: development, validity and reliability of a self-reported questionnaire - the DiPCare-Q

Background: General practitioners play a central role in taking deprivation into consideration when caring for patients in primary care. Validated questions to identify deprivation in primary-care practices are still lacking. For both clinical and research purposes, this study therefore aims to develop and validate a standardized instrument measuring both material and social deprivation at an individual level. Methods: The Deprivation in Primary Care Questionnaire (DiPCare-Q) was developed using qualitative and quantitative approaches between 2008 and 2011. A systematic review identified 199 questions related to deprivation. Using judgmental item quality, these were reduced to 38 questions. Two focus groups (primary-care physicians, and primary-care researchers), structured interviews (10 laymen), and think aloud interviews (eight cleaning staff) assured face validity. Item response theory analysis was then used to derive the DiPCare-Q index using data obtained from a random sample of 200 patients who were to complete the questionnaire a second time over the phone. For construct and criterion validity, the final 16 questions were administered to a random sample of 1,898 patients attending one of 47 different private primary-care practices in western Switzerland (validation set) along with questions on subjective social status (subjective SES ladder), education, source of income, welfare status, and subjective poverty. Results: Deprivation was defined in three distinct dimensions (table); material deprivation (eight items), social deprivation (five items) and health deprivation (three items). Item consistency was high in both the derivation (KR20 = 0.827) and the validation set (KR20 = 0.778). The DiPCare-Q index was reliable (ICC = 0.847). For construct validity, we showed the DiPCare-Q index to be correlated to patients' estimation of their position on the subjective SES ladder (rs = 0.539). This position was correlated to both material and social deprivation independently suggesting two separate mechanisms enhancing the feeling of deprivation. Conclusion: The DiPCare-Q is a rapid, reliable and validated instrument useful for measuring both material and social deprivation in primary care. Questions from the DiPCare-Q are easy to use when investigating patients' social history and could improve clinicians' ability to detect underlying social distress related to deprivation.

Validity of the 'protocol of oro-facial myofunctional evaluation with scores' for young and adult subjects

The aims of this study were to analyse the validity, sensitivity and specificity of the protocol of oro-facial myofunctional evaluation with scores (OMES) for oro-facial myofunctional disorder (OMD) diagnosis in young and adult subjects. Eighty subjects were examined. The OMES was validated against the Nordic Orofacial Test-Screening (NOT-S) protocol (criterion validity) (Spearman correlation test). The construct validity was tested by analysis of the ability of the OMES (i) to differentiate healthy subjects (n = 22) from temporomandibular disorder (TMD) patients (n = 22), which frequently have OMD (MannWhitney test) and (ii) to measure the changes that occurred in a subgroup with TMD between the period before and after oro-facial myofunctional therapy (T group, n = 15) (Wilcoxon test). Two speech therapists trained with the OMES participated as examiners (E). There was a statistically significant correlation between the OMES and NOT-S protocols, which was negative because the two scales are inverse (r = -0.86, P < 0.01). There was a significant difference between the healthy and TMD subjects regarding the oro-facial myofunctional status (OMES total score, P = 0.003). After therapy, the T group showed improvement in the oro-facial myofunctional status (OMES total score, P = 0.001). Inter- and intra-examiner agreement was moderate, and the reliability coefficients ranged from good to excellent. The OMES protocol presented mean sensitivity and specificity = 0.80, positive predictive value = 0.76 and negative predictive value = 0.84. Conclusion: The OMES protocol is valid and reliable for clinical evaluation of young and adult subjects, among them patients with TMD.

The Development, Reliability, and Validity of the OARS Multidimensional Functional Assessment Questionnaire

This report outlines the development, validity, and reliability of Part A of the OARS Multidimensional Functional Assessment Questionnaire. Part A permits assessment of individuals' functioning on each of five dimensions (social, economic, mental health, physical health and self-care capacity), the detailed information in each area being summarized on a 6-point rating scale by a rater. Content and consensual validity were ensured by the manner of construction. Information on criterion validity was obtained for all dimensions except social. The criterion used and their associated Kendall's Tau values were: an objective economic scale (.62); ratings based on personal interviews by geropsychiatrists (.60); physician's associates (.82); and physical therapists (.89). For 11 geographically dispersed raters from research and clinic settings, intraclass correlational coefficients, based on 30 subjects, ranged from .66 on physical health to .87 in self-care capacity; 74% of the ratings were in complete agreement, 24% differed by one point.

Validity and reliability of using MVP 4 Function Walk4Life digital pedometers to assess physical activity levels among preschool-aged Head Start children

The Surgeon General recommends preschoolers 3-5 years old accumulate 60 minutes of moderate-to-vigorous physical activity (MVPA) per day. However, there is limited data measuring physical activity (PA) and MVPA amongst this population. The purpose of this cross-sectional study is to determine the validity, reliability, and feasibility of using MVP 4 Function Walk4Life digital pedometers (MVP-4) in measuring MVPA among preschoolers using the newly modified direct observational technique, System for Observing Fitness Instruction Time-Preschool Version (SOFIT-P) as the gold standard. An ethnically diverse population of 3-5 year old underserved children were recruited from two Harris County Department of Education (HCDE) Head Start centers. For 2 days at baseline and 2 days at post-test, 75 children enrolled wore MVP-4 pedometers for approximately 6-hours per observation day and were observed using SOFIT-P during predominantly active times. Statistical analyses used Pearson "r" correlation coefficients to determine mean minutes of PA and MVPA, convergent and criterion validity, and reliability. Significance was set at p = <0.05. Feasibility was determined through process evaluation information collected during this study via observations from data collectors and teacher input. Results show mean minutes of PA and MVPA ranged between 30-42 and 11-14 minutes, respectively. Convergent validity comparing BMI percentiles with MVP-4 PA outcomes show no significance at pre-test; however, each measurement at post-test showed significance for MVPA (p = 0.0247, p = 0.0056), respectively. Criterion validity comparing percent MVPA time between SOFIT-P and MVP-4 pedometers was determined; however, results deemed insufficient due to inconsistency in observation times while using the newly developed SOFIT-P. Reliability measures show no significance at pre-test, yet show significant results for all PA outcomes at post-test (p = 0.001, p = 0.001, p = 0.0010, p = 0.003), respectively. Finally, MVP-4 pedometers lacked feasibility due to logistical barriers in design. Researchers feel the significant results at post-test are secondary to increased familiarity and more accurate placement of pedometers across time. Researchers suggest manufacturers of MVP-4 pedometers further modify the instrument for ease of use with this population, following which future studies ought to determine validity using objective measures or all-day direct observation techniques.^

The Reliability and Validity of a Child and Adolescent Participation in Decision Making Questionnaire

Background
There is a growing impetus across the research, policy and practice communities for children and young people to participate in decisions that affect their lives. Furthermore, there is a dearth of general instruments that measure children and young people’s views on their participation in decision making. This paper presents the reliability and validity of the Child and Adolescent Participation in Decision Making Questionnaire (CAP-DMQ) and specifically looks at a population of looked-after children where a lack of participation in decision making is an acute issue.
Methods
The participants were 151 looked after children and adolescents between 10-23 years of age who completed the 10 item CAP-DMQ. Of the participants 113 were in receipt of an advocacy service that had an aim of increasing participation in decision-making with the remaining participants not having received this service.
Results
The results showed that the CAP-DMQ had good reliability (Cronbach’s alpha = .94) and showed promising uni-dimensional construct validity through an exploratory factor analysis. The items in the CAP-DMQ also demonstrated good content validity by overlapping with prominent models of child and adolescent participation (Lundy 2007) and decision making (Halpern 2014). A regression analysis showed that age and gender were not significant predictors of CAP-DMQ scores but receipt of advocacy was a significant predictor of scores (effect size d=.88), thus showing appropriate discriminant criterion validity.
Conclusion
Overall, the CAP-DMQ showed good reliability and validity. Therefore, the measure has excellent promise for theoretical investigation in the area of child and adolescent participation in decision making and equally shows empirical promise for use as a measure in evaluating services which have increasing the participation of children and adolescents in decision making as an intended outcome.

Validity and reliability of the Cyber-aggression Questionnaire for Adolescents (CYBA)

Cybercrime is a growing and worrisome problem, particularly when it involves minors. Cyber-aggression among adolescents in particular can result in negative legal and psychological consequences for people involved. Therefore, it is important to have instruments to detect these incidents early and understand the problem to propose effective measures for prevention and treatment. This paper aims to design a new self-report, the Cyber-Aggression Questionnaire for Adolescents (CYBA), to evaluate the extent to which the respondent conducts aggressions through a mobile phone or the internet and analyse the factorial and criterion validity and reliability of their scores in a sample of adolescents from Asturias, Spain. The CYBA was administered to 3,148 youth aged between 12 and 18 years old along with three self-reports to measure aggression at school, impulsivity, and empathy. Regarding factorial validity, the model that best represents the structure of the CYBA consists of three factors (Impersonation, Visual-sexual Cyber-aggression, and Verbal Cyber-aggression and Exclusion) and four additional indicators of Visual Cyber-aggression-Teasing/Happy Slapping. Regarding criterion validity, the score on the CYBA correlates positively with aggression at school and impulsivity and negatively with empathy. That is the way cyber-aggression correlates with these three variables, according to previous empirical evidence. The reliability of the scores on each item and factor of the CYBA are adequate. Therefore, the CYBA offers a valid and reliable measure of cyber-aggression in adolescents.

Validade e confiabilidade da versão brasileira do Índice de Capacidade para o Trabalho

OBJETIVO: Avaliar a validade e a confiabilidade da versão brasileira de índice de capacidade para o trabalho. MÉTODOS:Estudo transversal com amostra de 475 trabalhadores de empresa do setor elétrico no estado de São Paulo (dez municípios em Campinas e região), realizado em 2005. Foram avaliados os seguintes aspectos da versão brasileira do Índice de Capacidade para o Trabalho: validade de construto, por meio de análise fatorial confirmatória e da capacidade discriminante; validade de critério, correlacionado o escore do índice com medidas de saúde auto-referidas; e confiabilidade, por meio da análise da consistência interna utilizando o coeficiente alfa de Cronbach. RESULTADOS: A análise fatorial indicou três fatores do construto capacidade para o trabalho: questões relativas aos "recursos mentais" (20,6% da variância), à autopercepção da capacidade para o trabalho (18,9% da variância) e à presença de doenças e limitações decorrentes do estado de saúde (18,4% da variância). O índice discriminou os trabalhadores segundo nível de absenteísmo, identificando média estatisticamente significativa (p<0,001) entre aqueles com absenteísmo elevado (37,2 pontos) e baixo (42,3 pontos). A análise de critério mostrou correlação do índice com todas as dimensões do estado de saúde analisadas (p<0,0001). O índice apresentou boa confiabilidade com coeficiente alfa de Cronbach (0,72). CONCLUSÕES: A versão brasileira do Índice de Capacidade para o Trabalho mostrou propriedades psicométricas satisfatórias quanto à validade de construto, de critério e de confiabilidade, representando uma opção adequada para avaliação da capacidade para o trabalho em abordagens individuais e inquéritos populacionais.

Psychometric properties of the Child Health Assessment Questionnaire (CHAQ) applied to children and adolescents with cerebral palsy

Background: Cerebral palsy (CP) patients have motor limitations that can affect functionality and abilities for activities of daily living (ADL). Health related quality of life and health status instruments validated to be applied to these patients do not directly approach the concepts of functionality or ADL. The Child Health Assessment Questionnaire (CHAQ) seems to be a good instrument to approach this dimension, but it was never used for CP patients. The purpose of the study was to verify the psychometric properties of CHAQ applied to children and adolescents with CP. Methods: Parents or guardians of children and adolescents with CP, aged 5 to 18 years, answered the CHAQ. A healthy group of 314 children and adolescents was recruited during the validation of the CHAQ Brazilian-version. Data quality, reliability and validity were studied. The motor function was evaluated by the Gross Motor Function Measure (GMFM). Results: Ninety-six parents/guardians answered the questionnaire. The age of the patients ranged from 5 to 17.9 years (average: 9.3). The rate of missing data was low(< 9.3%). The floor effect was observed in two domains, being higher only in the visual analogue scales (<= 35.5%). The ceiling effect was significant in all domains and particularly high in patients with quadriplegia (81.8 to 90.9%) and extrapyramidal (45.4 to 91.0%). The Cronbach alpha coefficient ranged from 0.85 to 0.95. The validity was appropriate: for the discriminant validity the correlation of the disability index with the visual analogue scales was not significant; for the convergent validity CHAQ disability index had a strong correlation with the GMFM (0.77); for the divergent validity there was no correlation between GMFM and the pain and overall evaluation scales; for the criterion validity GMFM as well as CHAQ detected differences in the scores among the clinical type of CP (p < 0.01); for the construct validity, the patients' disability index score (mean: 2.16; SD: 0.72) was higher than the healthy group ( mean: 0.12; SD: 0.23)(p < 0.01). Conclusion: CHAQ reliability and validity were adequate to this population. However, further studies are necessary to verify the influence of the ceiling effect on the responsiveness of the instrument.

New methodology to assess activity status of occlusal caries in primary teeth using laser fluorescence device

An in vivo study was conducted to verify the ability of laser fluorescence (LF) to assess the activity status of occlusal caries in primary teeth, using different air-drying times. Occlusal sites (707) were examined using LF (DIAGNOdent) after air-drying for 3 s and 15 s, and the difference between readings (DIF15 s-3 s) was calculated. For concurrent validation of LF, visual criteria-Nyvad (NY) and Lesion Activity Assessment associated with the International Caries Detection and Assessment System (LAA-ICDAS)-were the reference standards for lesion activity. Histological exam using a pH-indicator dye (0.1% methyl red) was performed in 46 exfoliated/extracted teeth for criterion validation. LF readings and DIF15 s-3 s were compared using Kruskall-Wallis and Mann-Whitney tests. Receiver operating characteristic analyses were performed and validity parameters calculated, considering the caries activity assessment. Using NY, active lesions (3 s: 30.0 +/- 29.3; 15 s: 34.2 +/- 30.6) presented higher LF readings than inactive lesions (3 s: 17.0 +/- 16.3; 15 s: 19.2 +/- 17.3; p <0.05), different from LAA-ICDAS. Active cavitated caries resulted in higher LF readings (3 s: 50.3 +/- 3.5; 15 s: 54.7 +/- 30.2) than inactive cavitated caries (3 s: 19.9 +/- 16.3; 15 s: 22.8 +/- 16.8). Therefore, LF can distinguish cavitated active and inactive lesions classified by NY, but not by LAA-ICDAS; however, this difference might be related to the visual system rather than to LF. The air-drying time could be an alternative to improve the caries activity assessment; however, longer air-drying time is suggested to be tested subsequently. (C) 2010 Society of Photo-Optical Instrumentation Engineers. [DOI: 10.1117/1.3463007]

Validation of ASHA FACS-Functional Assessment of Communication Skills for Alzheimer Disease Population

This study vas aimed to validate the American Speech-Language-Hearing Association Functional Assessment Of Communication Skills (ASHA FACS) for a Brazilian population. The scale was translated and adapted into Portuguese. Thirty-two patients with mild Alzheimer disease (AD). 25 patients with moderate AD. and 51 elderly without dementia were examined with Mini Mental State Examination, Geriatric Depression Scale. and Alzheimer Disease Assessment Scale-Cognitive subscale (ADAS-cog). The ASHA FACS was answered by their relative/caregiver. The scale`s internal consistency. its inter-examiner and intra-examiner`s reproducibility. and scale`s criterion validity were researched by correlation with ADAS-cog,. The sensitivity and specificity Were also researched. Statistical analyses indicated that the ASHA FACS has excellent internal consistency (Cronbach alpha = 0.955), test-retest reliability (interclass correlation coefficient = 0.995; P < 0.001). and inter-examiners (interclass correlation coefficient = 0.998: P < 0.001). It showed excellent criterion validity when correlated with ADAS-cog,. The ASHA FACS scale showed good sensitivity (75.0%) and specificity (82.4%) values once it is an ecologic and broad evaluation. The ASHA FACS Portuguese version is a valid and reliable instrument to verify communication alterations in AD patients and fills an important gap of efficiency indicators for speech language therapy in our country.