Inter-rater Reliability of Criteria-Based Content Analysis of Children's Statements of Abuse.

The evaluation of children's statements of sexual abuse cases in forensic cases is critically important and must and reliable. Criteria-based content analysis (CBCA) is the main component of the statement validity assessment (SVA), which is the most frequently used approach in this setting. This study investigated the inter-rater reliability (IRR) of CBCA in a forensic context. Three independent raters evaluated the transcripts of 95 statements of sexual abuse. IRR was calculated for each criterion, total score, and overall evaluation. The IRR was variable for the criteria, with several being unsatisfactory. But high IRR was found for the total CBCA scores (Kendall's W = 0.84) and for overall evaluation (Kendall's W = 0.65). Despite some shortcomings, SVA remains a robust method to be used in the comprehensive evaluation of children's statements of sexual abuse in the forensic setting. However, the low IRR of some CBCA criteria could justify some technical improvements.

Assessing the psychometric properties of dream content questionnaires

The present study was the first of its kind to systematically explore the psychometric properties of dream content questionnaires as measures of dream experience. One hundred and six University students filled out the Dream Content Questionnaire (DCQ) and kept a 14-day dream diary on two separate occasions, in addition to filling out the NEO-PI-R and Multidimensional Personality Questionnaire and measures of spatial ability and imaginativeness. The DCQ's reliability was acceptable, as was its discriminant and construct validity. Six of eight predicted relationships between trait personality and DCQ reported dream content were significant. In contrast, dream diaries showed instability over time and were unrelated to personality traits. The DCQ's concurrent validity could not be adequately appraised due to the inconsistency in dream diary content over time. The results suggest that questionnaires may be used to measure dream experience; however, the precise utility of dream questionnaires remains unclear. The findings raise important questions concerning measures of dream experience.

Towards criterion validity in classroom language analysis: methodological constraints of metadiscourse and inter-rater agreement

This paper reports on a process to validate a revised version of a system for coding classroom discourse in foreign language lessons, a context in which the dual role of language (as content and means of communication) and the speakers' specific pedagogical aims lead to a certain degree of ambiguity in language analysis. The language used by teachers and students has been extensively studied, and a framework of concepts concerning classroom discourse well-established. Models for coding classroom language need, however, to be revised when they are applied to specific research contexts. The application and revision of an initial framework can lead to the development of earlier models, and to the re-definition of previously established categories of analysis that have to be validated. The procedures followed to validate a coding system are related here as guidelines for conducting research under similar circumstances. The advantages of using instruments that incorporate two types of data, that is, quantitative measures and qualitative information from raters' metadiscourse, are discussed, and it is suggested that such procedure can contribute to the process of validation itself, towards attaining reliability of research results, as well as indicate some constraints of the adopted research methodology.

The Development, Reliability, and Validity of the OARS Multidimensional Functional Assessment Questionnaire

This report outlines the development, validity, and reliability of Part A of the OARS Multidimensional Functional Assessment Questionnaire. Part A permits assessment of individuals' functioning on each of five dimensions (social, economic, mental health, physical health and self-care capacity), the detailed information in each area being summarized on a 6-point rating scale by a rater. Content and consensual validity were ensured by the manner of construction. Information on criterion validity was obtained for all dimensions except social. The criterion used and their associated Kendall's Tau values were: an objective economic scale (.62); ratings based on personal interviews by geropsychiatrists (.60); physician's associates (.82); and physical therapists (.89). For 11 geographically dispersed raters from research and clinic settings, intraclass correlational coefficients, based on 30 subjects, ranged from .66 on physical health to .87 in self-care capacity; 74% of the ratings were in complete agreement, 24% differed by one point.

Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning

Background: The design of Virtual Patients (VPs) is essential. So far there are no validated evaluation instruments for VP design published. Summary of work: We examined three sources of validity evidence of an instrument to be filled out by students aimed at measuring the quality of VPs with a special emphasis on fostering clinical reasoning: (1) Content was examined based on theory of clinical reasoning and an international VP expert team. (2) Response process was explored in think aloud pilot studies with students and content analysis of free text questions accompanying each item of the instrument. (3) Internal structure was assessed by confirmatory factor analysis (CFA) using 2547 student evaluations and reliability was examined utilizing generalizability analysis. Summary of results: Content analysis was supported by theory underlying Gruppen and Frohna’s clinical reasoning model on which the instrument is based and an international VP expert team. The pilot study and analysis of free text comments supported the validity of the instrument. The CFA indicated that a three factor model comprising 6 items showed a good fit with the data. Alpha coefficients per factor were 0,74 - 0,82. The findings of the generalizability studies indicated that 40-200 student responses are needed in order to obtain reliable data on one VP. Conclusions: The described instrument has the potential to provide faculty with reliable and valid information about VP design. Take-home messages: We present a short instrument which can be of help in evaluating the design of VPs.

Content of prenatal care and spontaneous preterm birth: Findings from the 1988 National Maternal and Infant Health Survey

The purposes of this study were to examine (1) the relationship between selected components of the content of prenatal care and spontaneous preterm birth; and (2) the degree of comparability between maternal and caregivers' responses regarding the number of prenatal care visits, selected components of the content of prenatal care, and gestational age, based on analyses of the 1988 National Maternal and Infant Health Survey conducted by the National Centers for Health Statistics. Spontaneous preterm birth was subcategorized into very preterm and moderately preterm births, with term birth as the controls. The study population was limited to non-Hispanic Anglo- and African-American mothers. The racial differences in terms of birth outcomes were also compared.^ This study concluded that: (1) there was not a high degree of comparability (less than 80%) between maternal and prenatal care provider's responses regarding the number of prenatal care visits and the content of prenatal care; (2) there was a low degree of comparability (less than 50%) between maternal and infant's hospital of delivery responses regarding gestational age at birth; (3) there were differences in selected components of the content of prenatal care between the cases and controls, overall and stratified by ethnicity (i.e., hemoglobin/hematocrit test, weight measurement, and breast-feeding counseling), but they were confounded with missing values and associated preterm delivery bias; (4) there were differences in selected components of the content of prenatal care between Anglo- and African-American cases (i.e., vitamin/mineral supplement advice, weight measurement, smoking cessation and drug abuse counseling), but they, too, were difficult to interpret definitively due to item nonresponse and preterm delivery biases; (5) no significant predictive association between selected components of the content of prenatal care and spontaneous preterm birth was found; and (6) inadequate/intermediate prenatal care and birth out of wedlock were found to be associated with moderately preterm birth.^ Future research is needed to examine the validity of maternal and prenatal care providers' responses and identify the sources of disagreement between their responses. In addition, further studies are needed to examine the relationship between the quality of prenatal care and preterm birth. Finally, the completeness and quality of patient and provider data on the utilization and content of prenatal care needs to be strengthened in subsequent studies. ^

Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning

Background: Virtual patients (VPs) are increasingly used to train clinical reasoning. So far, no validated evaluation instruments for VP design are available. Aims: We examined the validity of an instrument for assessing the perception of VP design by learners. Methods: Three sources of validity evidence were examined: (i) Content was examined based on theory of clinical reasoning and an international VP expert team. (ii) The response process was explored in think-aloud pilot studies with medical students and in content analyses of free text questions accompanying each item of the instrument. (iii) Internal structure was assessed by exploratory factor analysis (EFA) and inter-rater reliability by generalizability analysis. Results: Content analysis was reasonably supported by the theoretical foundation and the VP expert team. The think-aloud studies and analysis of free text comments supported the validity of the instrument. In the EFA, using 2547 student evaluations of a total of 78 VPs, a three-factor model showed a reasonable fit with the data. At least 200 student responses are needed to obtain a reliable evaluation of a VP on all three factors. Conclusion: The instrument has the potential to provide valid information about VP design, provided that many responses per VP are available.

AN INVESTIGATION OF THE RELIABILITY AND VALIDITY OF THE CENTER FOR EPIDEMIOLOGIC STUDIES DEPRESSION SCALE IN THREE ETHNIC GROUPS

Epidemiologic studies of mental disorder have called attention to the need for identifying untreated cases and to the inadequacies of the instruments available for this purpose. Accurate case ascertainment devices are the basis of sound epidemiology. Without these, neither case classification nor analytic studies of risk factors is possible.^ The purpose of this research was to examine the reliability and validity of an instrument designed to measure depressive symptoms in community populations--the Center for Epidemiologic Studies Depression Scale (CES-D Scale). Two particular foci of the study were whether or not the scale had the same statistical structure across three ethnic groups and whether or not the magnitude and pattern of rates of symptoms for these groups were affected by one source of response error, that due to response tendencies. The effects of age and education on the pattern and magnitude of rates also were examined. In addition, the reliability and validity of the measures of response tendencies were assessed.^ The study population consisted of residents of Alameda County, California. A stratified sample of approximately 700 whites, blacks and Mexican-Americans was interviewed in the summer and fall of 1978.^ The results of the analysis indicated that the scale was reliable and measured a similar content domain across the three ethnic groups. The unadjusted sex- and ethnic-specific rates of depressive symptoms showed an ethnic pattern for both sexes: rates for whites were lowest, those for Mexican-Americans were highest, and those for blacks were intermediate. Measures of response tendencies--need for social approval, trait desirability, and acquiescence--affected the magnitude of the rates for most comparisons. Likewise, the pattern of rates changed somewhat from that originally observed. The one fairly consistent observation was that rates for Mexican-American women were higher than those for the other two female subgroups in most of the comparisons. These results must be considered in the context of the reliability and validity assessment of the measures of response tendencies which indicated the tenuousness of these measures.^ Age affected the ethnic pattern of rates for men in an inconsistent way; for women, Mexican-Americans continued to have higher rates than whites or blacks in all age categories. Education affected the magnitude of rates for women but not for men. For both men and women, Mexican-Americans had higher rates in all educational strata. Rates for women showed an inverse association with education while those for men did not. ^

Index properties, calcium carbonate content and opal concentration of ODP Core 167-1016B-17H sediments

Physical properties measurements provide a relatively inexpensive and fast way to obtain high-resolution estimates of the variations in sedimentological properties. To better resolve the validity and cause of the geophysical signals measured by the Ocean Drilling Program (ODP) shipboard multisensor track (MST) instruments, 223 x 10 cm**3 core samples were collected at 4 cm intervals in Core 167-1016B-17H at the California Margin Conception Transect for the measurements of index properties, carbonate content, and opal content. This core was chosen because hole-to-hole stratigraphic correlation of MST data suggested that Core 17H corresponds to a depth interval that displays the greatest range of amplitude of many physical properties.

E-MIS validity applied to TELMA enhanced learning environment

Enhanced learning environments are arising with great success within the field of cognitive skills training in minimally invasive surgery (MIS) because they provides multiple benefits since they avoid time, spatial and cost constraints. TELMA [1,2] is a new technology enhanced learning platform that promotes collaborative and ubiquitous training of surgeons. This platform is based on four main modules: an authoring tool, a learning content and knowledge management system, an evaluation module and a professional network. TELMA has been designed and developed focused on the user; therefore it is necessary to carry out a user validation as final stage of the development. For this purpose, e-MIS validity [3] has been defined. This validation includes usability, contents and functionality validities both for the development and production stages of any e-Learning web platform. Using e-MIS validity, the e-Learning is fully validated since it includes subjective and objective metrics. The purpose of this study is to specify and apply a set of objective and subjective metrics using e-MIS validity to test usability, contents and functionality of TELMA environment within the development stage.

Using the R-PAS' Aggressive Content Score For the Evaluation of Aggressive Behaviors in Children

The Rorschach Performance Assessment System (R-PAS) was developed in 2011 as analternative to the previous Comprehensive System. The goal was to improve the psychometrics,and particularly the validity, of this assessment method. The norms for children werequestionable in the Comprehensive system (e.g., outdated, low numbers of subjects) and validitystudies for children were sparse. One of the indicators included in the R-PAS system, theaggressive content indicator (AgC), is intended to reflect aggressive behavior, but few studieshave examined the validity of this indicator. This study examined the validity of AgC in asample of 32 children and adolescents receiving services at a residential treatment center.Subjects' AgC scores were analyzed in relation to demographics and diagnosis, as well as ratingsof aggression and conduct problems from the Behavioral Assessment System for Children-2(BASC-2) Parent and Teacher Reports. Correlations between the AgC score and BASC-2aggression and conduct problems scores were not statistically significant. None of thecorrelations between AgC score and a diagnosis of Conduct Disorder, Oppositional DefiantDisorder, Posttraumatic Stress Disorder, or Mood Disorders were significant either. Given thesmall sample size, null results may be a result of power concerns. The lack of significantcorrelations may however, indicate that operational definitions of aggression used in variousforms of measurement reflect different constructs.

(Table 1) Deuterium ratios, chloride content and hydrate volume of pore water and hydrate meltwater from ODP Site 164-996

Site 996 is located above the Blake Diapir where numerous indications of vertical fluid migration and the presence of hydrate existed prior to Ocean Drilling Program (ODP) Leg 164. Direct sampling of hydrates and visual observations of hydrate-filled veins that could be traced 30-40 cm along cores suggest a connection between fluid migration and hydrate formation. The composition of pore water squeezed from sediment cores showed large variations due to melting of hydrate during core recovery and influence of saline water from the evaporitic diapir below. Analysis of water released during hydrate decomposition experiments showed that the recovered hydrates contained significant amounts of pore water. Solutions of the transport equations for deuterium (d2H) and chloride (Cl-) were used to determine maximum (d2H) and minimum (Cl-) in situ concentrations of these species. Minimum in situ concentrations of hydrate were estimated by combining these results with Cl- and d2H values measured on hydrate meltwaters and pore waters obtained by squeezing of sediments, by the means of a method based on analysis of distances in the two-dimensional Cl- d2H space. The computed Cl- and d2H distribution indicates that the minimum hydrate amount solutions are representative of the actual hydrate amount. The highest and mean hydrate concentrations estimates from our model are 31% and 10% of the pore space, respectively. These concentrations agree well with visual core observations, supporting the validity of the model assumptions. The minimum in situ Cl- concentrations were used to constrain the rates of upward fluid migration. Simulation of all available data gave a mean flow rate of 0.35 m/k.y. (range: 0.125-0.5 m/k.y.).

Content-based VLE designs improve learning efficiency in constructivist statistics education

Background: We introduced a series of computer-supported workshops in our undergraduate statistics courses, in the hope that it would help students to gain a deeper understanding of statistical concepts. This raised questions about the appropriate design of the Virtual Learning Environment (VLE) in which such an approach had to be implemented. Therefore, we investigated two competing software design models for VLEs. In the first system, all learning features were a function of the classical VLE. The second system was designed from the perspective that learning features should be a function of the course's core content (statistical analyses), which required us to develop a specific-purpose Statistical Learning Environment (SLE) based on Reproducible Computing and newly developed Peer Review (PR) technology. Objectives: The main research question is whether the second VLE design improved learning efficiency as compared to the standard type of VLE design that is commonly used in education. As a secondary objective we provide empirical evidence about the usefulness of PR as a constructivist learning activity which supports non-rote learning. Finally, this paper illustrates that it is possible to introduce a constructivist learning approach in large student populations, based on adequately designed educational technology, without subsuming educational content to technological convenience. Methods: Both VLE systems were tested within a two-year quasi-experiment based on a Reliable Nonequivalent Group Design. This approach allowed us to draw valid conclusions about the treatment effect of the changed VLE design, even though the systems were implemented in successive years. The methodological aspects about the experiment's internal validity are explained extensively. Results: The effect of the design change is shown to have substantially increased the efficiency of constructivist, computer-assisted learning activities for all cohorts of the student population under investigation. The findings demonstrate that a content-based design outperforms the traditional VLE-based design. © 2011 Wessa et al.

A rehabilitation intervention to promote physical recovery following intensive care: a detailed description of construct development, rationale and content together with proposed taxonomy to capture processes in a randomised controlled trial.

Background: increasing numbers of patients are surviving critical illness, but survival may be associated with a constellation of physical and psychological sequelae that can cause on going disability and reduced health-related quality of life. Limited evidence currently exists to guide the optimum structure, timing, and content of rehabilitation programmes. There is a need to both develop and evaluate interventions to support and expedite recovery during the post-ICU discharge period. This paper describes the construct development for a complex rehabilitation intervention intended to promote physical recovery following critical illness. The intervention is currently being evaluated in a randomised trial (ISRCTN09412438; funder Chief Scientists Office, Scotland). Methods: the intervention was developed using the Medical Research Council (MRC) framework for developing complex healthcare interventions. We ensured representation from a wide variety of stakeholders including content experts from multiple specialties, methodologists, and patient representation. The intervention construct was initially based on literature review, local observational and audit work, qualitative studies with ICU survivors, and brainstorming activities. Iterative refinement was aided by the publication of a National Institute for Health and Care Excellence guideline (No. 83), publicly available patient stories (Healthtalkonline), a stakeholder event in collaboration with the James Lind Alliance, and local piloting. Modelling and further work involved a feasibility trial and development of a novel generic rehabilitation assistant (GRA) role. Several rounds of external peer review during successive funding applications also contributed to development. Results: the final construct for the complex intervention involved a dedicated GRA trained to pre-defined competencies across multiple rehabilitation domains (physiotherapy, dietetics, occupational therapy, and speech/language therapy), with specific training in post-critical illness issues. The intervention was from ICU discharge to 3 months post-discharge, including inpatient and post-hospital discharge elements. Clear strategies to provide information to patients/families were included. A detailed taxonomy was developed to define and describe the processes undertaken, and capture them during the trial. The detailed process measure description, together with a range of patient, health service, and economic outcomes were successfully mapped on to the modified CONSORT recommendations for reporting non-pharmacologic trial interventions. Conclusions: the MRC complex intervention framework was an effective guide to developing a novel post-ICU rehabilitation intervention. Combining a clearly defined new healthcare role with a detailed taxonomy of process and activity enabled the intervention to be clearly described for the purpose of trial delivery and reporting. These data will be useful when interpreting the results of the randomised trial, will increase internal and external trial validity, and help others implement the intervention if the intervention proves clinically and cost effective.