Towards criterion validity in classroom language analysis: methodological constraints of metadiscourse and inter-rater agreement

This paper reports on a process to validate a revised version of a system for coding classroom discourse in foreign language lessons, a context in which the dual role of language (as content and means of communication) and the speakers' specific pedagogical aims lead to a certain degree of ambiguity in language analysis. The language used by teachers and students has been extensively studied, and a framework of concepts concerning classroom discourse well-established. Models for coding classroom language need, however, to be revised when they are applied to specific research contexts. The application and revision of an initial framework can lead to the development of earlier models, and to the re-definition of previously established categories of analysis that have to be validated. The procedures followed to validate a coding system are related here as guidelines for conducting research under similar circumstances. The advantages of using instruments that incorporate two types of data, that is, quantitative measures and qualitative information from raters' metadiscourse, are discussed, and it is suggested that such procedure can contribute to the process of validation itself, towards attaining reliability of research results, as well as indicate some constraints of the adopted research methodology.

Reliability-based design optimization strategies based on FORM: a review

In deterministic optimization, the uncertainties of the structural system (i.e. dimension, model, material, loads, etc) are not explicitly taken into account. Hence, resulting optimal solutions may lead to reduced reliability levels. The objective of reliability based design optimization (RBDO) is to optimize structures guaranteeing that a minimum level of reliability, chosen a priori by the designer, is maintained. Since reliability analysis using the First Order Reliability Method (FORM) is an optimization procedure itself, RBDO (in its classical version) is a double-loop strategy: the reliability analysis (inner loop) and the structural optimization (outer loop). The coupling of these two loops leads to very high computational costs. To reduce the computational burden of RBDO based on FORM, several authors propose decoupling the structural optimization and the reliability analysis. These procedures may be divided in two groups: (i) serial single loop methods and (ii) unilevel methods. The basic idea of serial single loop methods is to decouple the two loops and solve them sequentially, until some convergence criterion is achieved. On the other hand, uni-level methods employ different strategies to obtain a single loop of optimization to solve the RBDO problem. This paper presents a review of such RBDO strategies. A comparison of the performance (computational cost) of the main strategies is presented for several variants of two benchmark problems from the literature and for a structure modeled using the finite element method.

Reliability algorithms applied to reinforced concrete structures durability assessment

This paper addresses the analysis of probabilistic corrosion time initiation in reinforced concrete structures exposed to ions chloride penetration. Structural durability is an important criterion which must be evaluated in every type of structure, especially when these structures are constructed in aggressive atmospheres. Considering reinforced concrete members, chloride diffusion process is widely used to evaluate the durability. Therefore, at modelling this phenomenon, corrosion of reinforcements can be better estimated and prevented. These processes begin when a threshold level of chlorides concentration is reached at the steel bars of reinforcements. Despite the robustness of several models proposed in the literature, deterministic approaches fail to predict accurately the corrosion time initiation due to the inherently randomness observed in this process. In this regard, the durability can be more realistically represented using probabilistic approaches. A probabilistic analysis of ions chloride penetration is presented in this paper. The ions chloride penetration is simulated using the Fick's second law of diffusion. This law represents the chloride diffusion process, considering time dependent effects. The probability of failure is calculated using Monte Carlo simulation and the First Order Reliability Method (FORM) with a direct coupling approach. Some examples are considered in order to study these phenomena and a simplified method is proposed to determine optimal values for concrete cover.

The Development, Reliability, and Validity of the OARS Multidimensional Functional Assessment Questionnaire

This report outlines the development, validity, and reliability of Part A of the OARS Multidimensional Functional Assessment Questionnaire. Part A permits assessment of individuals' functioning on each of five dimensions (social, economic, mental health, physical health and self-care capacity), the detailed information in each area being summarized on a 6-point rating scale by a rater. Content and consensual validity were ensured by the manner of construction. Information on criterion validity was obtained for all dimensions except social. The criterion used and their associated Kendall's Tau values were: an objective economic scale (.62); ratings based on personal interviews by geropsychiatrists (.60); physician's associates (.82); and physical therapists (.89). For 11 geographically dispersed raters from research and clinic settings, intraclass correlational coefficients, based on 30 subjects, ranged from .66 on physical health to .87 in self-care capacity; 74% of the ratings were in complete agreement, 24% differed by one point.

Multi-node TOA-DOA cooperative LOS-NLOS localization : enabling high accuracy and reliability

This dissertation investigates high performance cooperative localization in wireless environments based on multi-node time-of-arrival (TOA) and direction-of-arrival (DOA) estimations in line-of-sight (LOS) and non-LOS (NLOS) scenarios. Here, two categories of nodes are assumed: base nodes (BNs) and target nodes (TNs). BNs are equipped with antenna arrays and capable of estimating TOA (range) and DOA (angle). TNs are equipped with Omni-directional antennas and communicate with BNs to allow BNs to localize TNs; thus, the proposed localization is maintained by BNs and TNs cooperation. First, a LOS localization method is proposed, which is based on semi-distributed multi-node TOA-DOA fusion. The proposed technique is applicable to mobile ad-hoc networks (MANETs). We assume LOS is available between BNs and TNs. One BN is selected as the reference BN, and other nodes are localized in the coordinates of the reference BN. Each BN can localize TNs located in its coverage area independently. In addition, a TN might be localized by multiple BNs. High performance localization is attainable via multi-node TOA-DOA fusion. The complexity of the semi-distributed multi-node TOA-DOA fusion is low because the total computational load is distributed across all BNs. To evaluate the localization accuracy of the proposed method, we compare the proposed method with global positioning system (GPS) aided TOA (DOA) fusion, which are applicable to MANETs. The comparison criterion is the localization circular error probability (CEP). The results confirm that the proposed method is suitable for moderate scale MANETs, while GPS-aided TOA fusion is suitable for large scale MANETs. Usually, TOA and DOA of TNs are periodically estimated by BNs. Thus, Kalman filter (KF) is integrated with multi-node TOA-DOA fusion to further improve its performance. The integration of KF and multi-node TOA-DOA fusion is compared with extended-KF (EKF) when it is applied to multiple TOA-DOA estimations made by multiple BNs. The comparison depicts that it is stable (no divergence takes place) and its accuracy is slightly lower than that of the EKF, if the EKF converges. However, the EKF may diverge while the integration of KF and multi-node TOA-DOA fusion does not; thus, the reliability of the proposed method is higher. In addition, the computational complexity of the integration of KF and multi-node TOA-DOA fusion is much lower than that of EKF. In wireless environments, LOS might be obstructed. This degrades the localization reliability. Antenna arrays installed at each BN is incorporated to allow each BN to identify NLOS scenarios independently. Here, a single BN measures the phase difference across two antenna elements using a synchronized bi-receiver system, and maps it into wireless channel’s K-factor. The larger K is, the more likely the channel would be a LOS one. Next, the K-factor is incorporated to identify NLOS scenarios. The performance of this system is characterized in terms of probability of LOS and NLOS identification. The latency of the method is small. Finally, a multi-node NLOS identification and localization method is proposed to improve localization reliability. In this case, multiple BNs engage in the process of NLOS identification, shared reflectors determination and localization, and NLOS TN localization. In NLOS scenarios, when there are three or more shared reflectors, those reflectors are localized via DOA fusion, and then a TN is localized via TOA fusion based on the localization of shared reflectors.

Validity and reliability of using MVP 4 Function Walk4Life digital pedometers to assess physical activity levels among preschool-aged Head Start children

The Surgeon General recommends preschoolers 3-5 years old accumulate 60 minutes of moderate-to-vigorous physical activity (MVPA) per day. However, there is limited data measuring physical activity (PA) and MVPA amongst this population. The purpose of this cross-sectional study is to determine the validity, reliability, and feasibility of using MVP 4 Function Walk4Life digital pedometers (MVP-4) in measuring MVPA among preschoolers using the newly modified direct observational technique, System for Observing Fitness Instruction Time-Preschool Version (SOFIT-P) as the gold standard. An ethnically diverse population of 3-5 year old underserved children were recruited from two Harris County Department of Education (HCDE) Head Start centers. For 2 days at baseline and 2 days at post-test, 75 children enrolled wore MVP-4 pedometers for approximately 6-hours per observation day and were observed using SOFIT-P during predominantly active times. Statistical analyses used Pearson "r" correlation coefficients to determine mean minutes of PA and MVPA, convergent and criterion validity, and reliability. Significance was set at p = <0.05. Feasibility was determined through process evaluation information collected during this study via observations from data collectors and teacher input. Results show mean minutes of PA and MVPA ranged between 30-42 and 11-14 minutes, respectively. Convergent validity comparing BMI percentiles with MVP-4 PA outcomes show no significance at pre-test; however, each measurement at post-test showed significance for MVPA (p = 0.0247, p = 0.0056), respectively. Criterion validity comparing percent MVPA time between SOFIT-P and MVP-4 pedometers was determined; however, results deemed insufficient due to inconsistency in observation times while using the newly developed SOFIT-P. Reliability measures show no significance at pre-test, yet show significant results for all PA outcomes at post-test (p = 0.001, p = 0.001, p = 0.0010, p = 0.003), respectively. Finally, MVP-4 pedometers lacked feasibility due to logistical barriers in design. Researchers feel the significant results at post-test are secondary to increased familiarity and more accurate placement of pedometers across time. Researchers suggest manufacturers of MVP-4 pedometers further modify the instrument for ease of use with this population, following which future studies ought to determine validity using objective measures or all-day direct observation techniques.^

The Reliability and Validity of a Child and Adolescent Participation in Decision Making Questionnaire

Background
There is a growing impetus across the research, policy and practice communities for children and young people to participate in decisions that affect their lives. Furthermore, there is a dearth of general instruments that measure children and young people’s views on their participation in decision making. This paper presents the reliability and validity of the Child and Adolescent Participation in Decision Making Questionnaire (CAP-DMQ) and specifically looks at a population of looked-after children where a lack of participation in decision making is an acute issue.
Methods
The participants were 151 looked after children and adolescents between 10-23 years of age who completed the 10 item CAP-DMQ. Of the participants 113 were in receipt of an advocacy service that had an aim of increasing participation in decision-making with the remaining participants not having received this service.
Results
The results showed that the CAP-DMQ had good reliability (Cronbach’s alpha = .94) and showed promising uni-dimensional construct validity through an exploratory factor analysis. The items in the CAP-DMQ also demonstrated good content validity by overlapping with prominent models of child and adolescent participation (Lundy 2007) and decision making (Halpern 2014). A regression analysis showed that age and gender were not significant predictors of CAP-DMQ scores but receipt of advocacy was a significant predictor of scores (effect size d=.88), thus showing appropriate discriminant criterion validity.
Conclusion
Overall, the CAP-DMQ showed good reliability and validity. Therefore, the measure has excellent promise for theoretical investigation in the area of child and adolescent participation in decision making and equally shows empirical promise for use as a measure in evaluating services which have increasing the participation of children and adolescents in decision making as an intended outcome.

Validity and reliability of the Cyber-aggression Questionnaire for Adolescents (CYBA)

Cybercrime is a growing and worrisome problem, particularly when it involves minors. Cyber-aggression among adolescents in particular can result in negative legal and psychological consequences for people involved. Therefore, it is important to have instruments to detect these incidents early and understand the problem to propose effective measures for prevention and treatment. This paper aims to design a new self-report, the Cyber-Aggression Questionnaire for Adolescents (CYBA), to evaluate the extent to which the respondent conducts aggressions through a mobile phone or the internet and analyse the factorial and criterion validity and reliability of their scores in a sample of adolescents from Asturias, Spain. The CYBA was administered to 3,148 youth aged between 12 and 18 years old along with three self-reports to measure aggression at school, impulsivity, and empathy. Regarding factorial validity, the model that best represents the structure of the CYBA consists of three factors (Impersonation, Visual-sexual Cyber-aggression, and Verbal Cyber-aggression and Exclusion) and four additional indicators of Visual Cyber-aggression-Teasing/Happy Slapping. Regarding criterion validity, the score on the CYBA correlates positively with aggression at school and impulsivity and negatively with empathy. That is the way cyber-aggression correlates with these three variables, according to previous empirical evidence. The reliability of the scores on each item and factor of the CYBA are adequate. Therefore, the CYBA offers a valid and reliable measure of cyber-aggression in adolescents.

Validity And Reliability Of A Pediatric Patient Classification Instrument.

to assess the construct validity and reliability of the Pediatric Patient Classification Instrument. correlation study developed at a teaching hospital. The classification involved 227 patients, using the pediatric patient classification instrument. The construct validity was assessed through the factor analysis approach and reliability through internal consistency. the Exploratory Factor Analysis identified three constructs with 67.5% of variance explanation and, in the reliability assessment, the following Cronbach's alpha coefficients were found: 0.92 for the instrument as a whole; 0.88 for the Patient domain; 0.81 for the Family domain; 0.44 for the Therapeutic procedures domain. the instrument evidenced its construct validity and reliability, and these analyses indicate the feasibility of the instrument. The validation of the Pediatric Patient Classification Instrument still represents a challenge, due to its relevance for a closer look at pediatric nursing care and management. Further research should be considered to explore its dimensionality and content validity.

Avaliação in vitro do aparelho DIAGNOdent para diagnóstico oclusal

The aim of this study was to determine the effectiveness and reliability of laser fluorescence measurements in relation to occlusal caries diagnosis. DIAGNOdent 2095 (Kavo, Biberach, Germany), which has been developed especially for caries diagnosis, was utilized. Five (5) teeth were examined in the pilot test; after that, ten (10) teeth were examined in order to calibrate both examiners. Data were obtained from 66 teeth (36 molars and 30 premolars), totalizing 144 sites identified through photographs of the occlusal surfaces. Reproducibility was evaluated in 10 teeth. The interexaminer Spearman correlation (r) was 0.89 and the intra-examiner, 0.93 and 0.97 (examiner A and B, respectively). Validation was carried out by histological examination (stereomicroscope). For the two examiners the sensitivity of the device was relatively high, varying from 0.81 to 1.00, while specificity varied according to which validation criterion was used (0.77 - 0.86: enamel lesion / 0.52 - 0.59: dentin lesion). It was concluded that DIAGNodent presented good capacity of identifying any alteration of the dental surface, nevertheless it presents the disadvantage of accomplishing many false-positive diagnosis when the validation criterion is dentin lesion.

Validação de uma bateria de testes de habilidades motoras para atletas de handebol em cadeira de rodas

Universidade Estadual de Campinas . Faculdade de Educação Física

Validity and reliability of methods for the detection of secondary caries around amalgam restorations in primary teeth

Secondary caries has been reported as the main reason for restoration replacement. The aim of this in vitro study was to evaluate the performance of different methods - visual inspection, laser fluorescence (DIAGNOdent), radiography and tactile examination - for secondary caries detection in primary molars restored with amalgam. Fifty-four primary molars were photographed and 73 suspect sites adjacent to amalgam restorations were selected. Two examiners evaluated independently these sites using all methods. Agreement between examiners was assessed by the Kappa test. To validate the methods, a caries-detector dye was used after restoration removal. The best cut-off points for the sample were found by a Receiver Operator Characteristic (ROC) analysis, and the area under the ROC curve (Az), and the sensitivity, specificity and accuracy of the methods were calculated for enamel (D2) and dentine (D3) thresholds. These parameters were found for each method and then compared by the McNemar test. The tactile examination and visual inspection presented the highest inter-examiner agreement for the D2 and D3 thresholds, respectively. The visual inspection also showed better performance than the other methods for both thresholds (Az = 0.861 and Az = 0.841, respectively). In conclusion, the visual inspection presented the best performance for detecting enamel and dentin secondary caries in primary teeth restored with amalgam.

Reliability and discriminatory power of methods for dental plaque quantification

OBJECTIVE: This in situ study evaluated the discriminatory power and reliability of methods of dental plaque quantification and the relationship between visual indices (VI) and fluorescence camera (FC) to detect plaque. MATERIAL AND METHODS: Six volunteers used palatal appliances with six bovine enamel blocks presenting different stages of plaque accumulation. The presence of plaque with and without disclosing was assessed using VI. Images were obtained with FC and digital camera in both conditions. The area covered by plaque was assessed. Examinations were done by two independent examiners. Data were analyzed by Kruskal-Wallis and Kappa tests to compare different conditions of samples and to assess the inter-examiner reproducibility. RESULTS: Some methods presented adequate reproducibility. The Turesky index and the assessment of area covered by disclosed plaque in the FC images presented the highest discriminatory powers. CONCLUSION: The Turesky index and images with FC with disclosing present good reliability and discriminatory power in quantifying dental plaque.

Validity and reliability of the Structured Clinical Interview for Mood Spectrum: Brazilian version (SCIMOODS-VB)

OBJECTIVE: The aim of this study was to translate the Structured Clinical Interview for Mood Spectrum into Brazilian Portuguese, measuring its reliability, validity, and defining scores for bipolar disorders. METHOD: Questionnaire was translated (into Brazilian Portuguese) and back-translated into English. Sample consisted of 47 subjects with bipolar disorder, 47 with major depressive disorder, 18 with schizophrenia and 22 controls. Inter-rater reliability was tested in 20 subjects with bipolar disorder and MDD. Internal consistency was measured using the Kuder Richardson formula. Forward stepwise discriminant analysis was performed. Scores were compared between groups; manic (M), depressive (D) and total (T) threshold scores were calculated through receiver operating characteristic (ROC) curves. RESULTS: Kuder Richardson coefficients were between 0.86 and 0.94. Intraclass correlation coefficient was 0.96 (CI 95 % 0.93-0.97). Subjects with bipolar disorder had higher M and T, and similar D scores, when compared to major depressive disorder (ANOVA, p < 0.001). The sub-domains that best discriminated unipolar and bipolar subjects were manic energy and manic mood. M had the best area under the curve (0.909), and values of M equal to or greater than 30 yielded 91.5% sensitivity and 74.5% specificity. CONCLUSION: Structured Clinical Interview for Mood Spectrum has good reliability and validity. Cut-off of 30 best differentiates subjects with bipolar disorder vs. unipolar depression. A cutoff score of 30 or higher in the mania sub-domain is appropriate to help make a distinction between subjects with bipolar disorder and those with unipolar depression.

Reproducibility, reliability and validity of measurements obtained from Cecile3 digital models

The aim of this study was to determine the reproducibility, reliability and validity of measurements in digital models compared to plaster models. Fifteen pairs of plaster models were obtained from orthodontic patients with permanent dentition before treatment. These were digitized to be evaluated with the program Cécile3 v2.554.2 beta. Two examiners measured three times the mesiodistal width of all the teeth present, intercanine, interpremolar and intermolar distances, overjet and overbite. The plaster models were measured using a digital vernier. The t-Student test for paired samples and interclass correlation coefficient (ICC) were used for statistical analysis. The ICC of the digital models were 0.84 ± 0.15 (intra-examiner) and 0.80 ± 0.19 (inter-examiner). The average mean difference of the digital models was 0.23 ± 0.14 and 0.24 ± 0.11 for each examiner, respectively. When the two types of measurements were compared, the values obtained from the digital models were lower than those obtained from the plaster models (p < 0.05), although the differences were considered clinically insignificant (differences < 0.1 mm). The Cécile digital models are a clinically acceptable alternative for use in Orthodontics.