Spurious precision : variability in procedural validity of diagnostic procedures in psychotic disorders and implications for research and treatment

Very few studies have quantified the level of agreement among alternative diagnostic procedures that use a common set of fixed operational criteria. The authors examined the procedural validity of four independent methods of assigning DSM-III-R diagnoses of psychotic disorders. METHOD: The research was conducted as a satellite study to the DSM-IV Field Trial for Schizophrenia and Related Psychotic Disorders. The setting was the National Health and Medical Research Council Schizophrenia Research Unit's Early Psychosis Prevention and Intervention Centre, which focuses on first-episode psychosis. Consecutively admitted patients (N = 50) were assessed by independent raters who used four different procedures to determine a DSM-III-R diagnosis. These procedures were 1) the diagnostic instrument developed for the DSM-IV field trial, 2) the Royal Park Multidiagnostic Instrument for Psychosis, 3) the Munich Diagnostic Checklists, and 4) a consensus DSM-III-R diagnosis assigned by a team of clinician researchers who were expert in the use of diagnostic criteria. RESULTS: Concordance between pairs of diagnostic procedures was only moderate. Corresponding levels of percent agreement, however, ranged from 66% to 76%, with converse misclassification rates of 24%-34% (assuming one procedure to be "correct"). CONCLUSIONS: These findings have significant research and clinical implications. Despite the introduction of operationally defined diagnoses, there remained an appreciable level of differential classification or misclassification arising from variability in the method of assigning the diagnostic criteria rather than the criteria themselves. Such misclassification may impede neurobiological research and have harmful clinical effects on patients with first-episode psychosis.

The procedural validity of retrospective case note diagnosis

Objective: Deriving diagnoses from retrospective case note examination is a common practice in psychiatric research. The Operational Criteria (OPCRIT) diagnostic checklist is essentially a checklist built up of operational criteria defined by a comprehensive glossary and is designed to assign reliable diagnoses from case notes. However, the validity of such a procedure compared with procedures involving prospective assessment has never been tested. We examined the procedural validity of the OPCRIT diagnostic system in relation to four other diagnostic procedures mostly employing prospectively gathered information.

Method: Three experienced psychopathology raters rated the case notes and clinical abstracts, using the OPCRIT method of diagnostic assignment, of 50 subjects who had participated in an early procedural validity study as an adjunct to the DSM-IV Field Trial for psychotic disorders. The setting was the Early Psychosis Prevention and Intervention Centre (EPPIC), which focuses on first episode psychosis.

Results: The pairwise concordance with the other procedures for DSM-III-R diagnoses assigned by OPCRIT using ratings derived from either the clinical abstracts or the case notes was found to be only poor to moderate when compared with the pairwise concordance of the four other procedures. The per cent agreement between OPCRIT clinical abstracts diagnoses and the other procedures ranged from 49% to 60% with kappa values between 0.30 and 0.45, and for OPCRIT case note diagnoses and the other procedures the per cent agreement range was between 44% and 57% and the kappa values were between 0.35 and 0.49.

Conclusions: The procedural validity of diagnoses assigned via the application of checklists of operational criteria to case notes and clinical abstracts alone is unacceptably poor. Such sources need to be buttressed by other data, particularly direct patient interview and informant material.

Assessing the validity of impact pathways for child labour and well-being in social life cycle assessment

Background, aim and scope: Assuming that the goal of social life cycle assessment (SLCA) is to assess damage and benefits on its ‘area of protection’ (AoP) as accurately as possible, it follows that the impact pathways, describing the cause effect relationship between indicator and the AoP, should have a consistent theoretical foundation so the inventory results can be associated with a predictable damage or benefit to the AoP. This article uses two concrete examples from the work on SLCA to analyse to what extent this is the case in current practice. One considers whether indicators included in SLCA approaches can validly assess impacts on the well-being of the stakeholder, whereas the other example addresses whether the ‘incidence of child labour’ is a valid measure for impacts on the AoPs.

Materials and methods: The theoretical basis for the impact pathway between the relevant indicators and the AoPs is analysed drawing on research from relevant scientific fields.

Results:   The examples show a lack of valid impact pathways in both examples. The first example shows that depending on the definition of ‘well-being’, the assessment of impacts on well-being of the stakeholder cannot be performed exclusively with the type of indicators which are presently used in SLCA approaches. The second example shows that the mere fact that a child is working tells little about how this may damage or benefit the AoPs, implying that the normally used indicator; ‘incidence of child labour’ lacks validity in relation to predicting damage or benefit on the AoPs of SLCA.

Discussion: New indicators are proposed to mitigate the problem of invalid impact pathways. However, several problems arise relating to difficulties in getting data, the usability of the new indicators in management situations, and, in relation to example one, boundary setting issues.

Conclusions: The article shows that it is possible to assess the validity of the impact pathways in SLCA. It thereby point to the possibility of utilising the same framework that underpins the environmental LCA in this regard. It also shows that in relation to both of the specific examples investigated, the validity of the impact pathways may be improved by adopting other indicators, which does, however, come with a considerable ‘price’.

Recommendations and perspectives: It is argued that there is a need for analysing impact pathways of other impact categories often included in SLCA in order to establish indicators that better reflect actual damage or benefit to the AoPs.

The relationship among malingering, psychopathy, and the MMPI-2 validity scales in maximum security forensic psychiatric inpatients

The records of 392 men hospitalized in a maximum security forensic psychiatric hospital were reviewed. Demographic information was collected as well as data from the men's performance on the Psychopathy Checklist-Revised (PCL-R) and Minnesota Multiphasic Personality Inventory-2 (MMPI-2). Prevalence rates for malingering were low across the sample. However, results of chi-square analysis revealed that those who scored high on the PCL-R received a diagnosis of malingering significantly more frequently than those who scored low on the PCL-R. Clinical applications and theoretical implications of the results are discussed.

Reliability and validity of the system for observing children's activity and relationships during play (SOCARP)

Background: Children frequently engage in diverse activities that are broadly defined as play, but little research has documented children’s activity levels during play and how they are influenced by social contexts. Assessing potentially modifiable conditions that influence play behavior is needed to design optimal physical activity interventions.

Methods: System for Observing Children’s Activity and Relationships during Play (SOCARP) was developed to simultaneously assess children’s physical activity, social group sizes, activity type, and social behavior during play. One hundred and fourteen children (48 boys, 66 girls; 42% overweight) from 8 elementary schools were observed during recess over 24 days, with 12 days videotaped for reliability purposes. Ninety-nine children wore a uni-axial accelerometer during their observation period.

Results: Estimated energy expenditure rates from SOCARP observations and mean accelerometer counts were significantly correlated (r = .67; P < .01), and interobserver reliabilities (ie, percentage agreement) for activity level (89%), group size (88%), activity type (90%) and interactions (88%) met acceptable criteria. Both physical activity and social interactions were influenced by group size, activity type, and child gender and body weight status.

Conclusions: SOCARP is a valid and reliable observation system for assessing physical activity and play behavior in a recess context.

The development and validation of Australian indices of child development — part II : validity support

The two outcome indices described in a companion paper (Sanson et al., Child Indicators Research, 2009) were developed using data from the Longitudinal Study of Australian Children (LSAC). These indices, one for infants and the other for 4 year to 5 year old children, were designed to fill the need for parsimonious measures of children’s developmental status to be used in analyses by a broad range of data users and to guide government policy and interventions to support young children’s optimal development. This paper presents evidence from Wave 1data from LSAC to support the validity of these indices and their three domain scores of Physical, Social/Emotional, and Learning. Relationships between the indices and child, maternal, family, and neighborhood factors which are known to relate concurrently to child outcomes were examined. Meaningful associations were found with the selected variables, thereby demonstrating the usefulness of the outcome indices as tools for understanding children’s development in their family and socio-cultural contexts. It is concluded that the outcome indices are valuable tools for increasing understanding of influences on children’s development, and for guiding policy and practice to optimize children’s life chances.

Validity and reliability of instruments to assess potential mediators of children's physical activity : a systematic review

This paper aimed to (1) identify potential mediators reported in children's physical activity interventions; and (2) review the psychometric properties of measures of potential mediators included in such interventions. A systematic search of the literature was conducted and studies that reported potential mediators and/or the psychometric properties of measures of potential mediators were retrieved and reviewed. The studies were reviewed for the following psychometric properties: face/logical, construct, and criterion validity; and test–retest and internal consistency (factor analysis and Cronbach's α) reliability. Twenty-one potential mediators were identified from 24 published children's physical activity intervention studies; 17 individual, three social, and one physical environmental. Overall, there was considerable variation in the reporting of the psychometric properties of these measures. Reliability properties (specifically Cronbach's α) were more commonly reported than validity properties; however, for many potential mediators, the quality of these measures was low. Future studies should examine the psychometric properties of measures of potential mediators to ensure appropriate, valid and reliable instruments are used. This will assist in the ability to accurately determine the efficacy of physical activity interventions among children.

The convergent validity of two sensory processing scales used with school - age children : comparing the sensory profile and the sensory processing measure

Abstract: Aim: To investigate the convergent validity between the Sensory Profile, the Sensory Profile School Companion, and the Home and Main Classroom Forms of the Sensory Processing Measure.

Method: Thirty mothers completed the Sensory Profile and the Sensory Processing Measure - Home Form on one child each. Nineteen teachers of the same children completed the Sensory Profile School Companion and the Sensory Processing Measure - Main Classroom Form.

Results: The Sensory Profile and the Sensory Processing Measure - Home Form were significantly correlated (rho=0.86, p less-than .01). The Sensory Profile School Companion and Sensory Processing Measure - Main Classroom Form were also significantly correlated (rho=.74, p less-than .01).

Conclusion: The two sets of sensory processing scales had moderate levels of convergent validity.

Validity and relevance of the pack hike wildland firefighter work capacity test: a review

Fighting wildland fire is a physically demanding occupation. Wildland firefighters need to be physically fit to work safely and productively. To determine whether personnel are fit for duty, many firefighting agencies employ physical competency tests, such as the pack hike test (PHT). The PHT involves a 4.83-km hike over level terrain carrying a 20.4-kg pack within a 45-min period. The PHT was devised to test the job readiness of US wildland firefighters but is also currently used by some fire agencies in Australia and Canada. This review discusses the history and development of the PHT with emphasis on the process of test validation. Research-based training advice for the PHT is given, as well as discussion of the risks associated with completing the PHT. Different versions and modifications to the PHT have emerged in recent years and these are discussed with regard to their validity. Finally, this review addresses the relevance and validity of the PHT for Australian and Canadian wildland firefighters.

The validity and internal structure of the bipolar depression rating scale (BDRS): data from a clinical trial of N-acetylcysteine as adjunctive therapy in bipolar disorder

Background: The phenomenology of unipolar and bipolar disorders differ in a number of ways, such as the presence of mixed states and atypical features. Conventional depression rating instruments are designed to capture the characteristics of unipolar depression and have limitations in capturing the breadth of bipolar disorder.

Method: The Bipolar Depression Rating Scale (BDRS) was administered together with the Montgomery Asberg Rating Scale (MADRS) and Young Mania Rating Scale (YMRS) in a double-blind randomised placebo-controlled clinical trial of N-acetyl cysteine for bipolar disorder (N = 75).

Results: A factor analysis showed a two-factor solution: depression and mixed symptom clusters. The BDRS has strong internal consistency (Cronbach's alpha = 0.917), the depression cluster showed robust correlation with the MADRS (r = 0.865) and the mixed subscale correlated with the YMRS (r = 0.750).

Conclusion: The BDRS has good internal validity and inter-rater reliability and is sensitive to change in the context of a clinical trial.

The validity of the CGI severity and improvement scales as measures of clinical effectiveness suitable for routine clinical use

Objective The Clinical Global Impression Scale (CGI) is established as a core metric in psychiatric research. This study aims to test the validity of CGI as a clinical outcome measure suitable for routine use in a private inpatient setting.

Methods The CGI was added to a standard battery of routine outcome measures in a private psychiatric hospital. Data were collected on consecutive admissions over a period of 24 months, which included clinical diagnosis, demographics, service utilization and four routine measures (CGI, HoNOS, MHQ-14 and DASS-21) at both admission and discharge. Descriptive and comparative data analyses were performed.

Results Of 786 admissions in total, there were 624 and 614 CGI-S ratings completed at the point of admission and discharge, respectively, and 610 completed CGI-I ratings. The admission and discharge CGI-S scores were correlated (r = 0.40), and the indirect improvement measures obtained from their differences were highly correlated with the direct CGI-I scores (r = 0.71). The CGI results reflected similar trends seen in the other three outcome measures.

Conclusions The CGI is a valid clinical outcome measure suitable for routine use in an inpatient setting. It offers a number of advantages, including its established utility in psychiatric research, sensitivity to change, quick and simple administration, utility across diagnostic groupings, and reliability in the hands of skilled clinicians.

The validity of the 21-item version of the depression anxiety stress scales as a routine outcome measure

Objective: This study aimed to test the validity of the 21-item Depression Anxiety Stress Scales (DASS-21) as a routine clinical outcome measure in the private in-patient setting. We hypothesized that it would be a suitable routine outcome instrument in this setting.

Method: All in-patients treated at a private psychiatric hospital over a period of 24 months were included in the study. Data were collected on demographics, service utilization, diagnosis and a set of four routine measures both at admission and discharge. These measures consisted of the Clinical Global Impressions (CGI) scales, Health of the Nation Outcome Scales (HoNOS), the Mental Health Questionnaire (MHQ-14) and DASS-21. The results of these measures were compared.

Results: Of 786 admissions in total, the number of fully completed (ie paired admission and discharge) data sets for the DASS-21 depression, anxiety and stress subscales were 337, 328 and 347, respectively. All subscales showed statistically significant reductions in mean scores from admission to discharge (P < 0.001) and were significantly correlated with all MHQ-14 subscales and significantly related to CGI scale categories. The total DASS-21 and total HoNOS scores were also significantly correlated.

Conclusions: The findings from the present study support the validity of DASS-21 as a routine clinical outcome measure in the private in-patient setting.