Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning

Background: The design of Virtual Patients (VPs) is essential. So far there are no validated evaluation instruments for VP design published. Summary of work: We examined three sources of validity evidence of an instrument to be filled out by students aimed at measuring the quality of VPs with a special emphasis on fostering clinical reasoning: (1) Content was examined based on theory of clinical reasoning and an international VP expert team. (2) Response process was explored in think aloud pilot studies with students and content analysis of free text questions accompanying each item of the instrument. (3) Internal structure was assessed by confirmatory factor analysis (CFA) using 2547 student evaluations and reliability was examined utilizing generalizability analysis. Summary of results: Content analysis was supported by theory underlying Gruppen and Frohna’s clinical reasoning model on which the instrument is based and an international VP expert team. The pilot study and analysis of free text comments supported the validity of the instrument. The CFA indicated that a three factor model comprising 6 items showed a good fit with the data. Alpha coefficients per factor were 0,74 - 0,82. The findings of the generalizability studies indicated that 40-200 student responses are needed in order to obtain reliable data on one VP. Conclusions: The described instrument has the potential to provide faculty with reliable and valid information about VP design. Take-home messages: We present a short instrument which can be of help in evaluating the design of VPs.

Validity of the Adult ADHD Self-Report Scale (ASRS) as a screener for adult ADHD in treatment seeking substance use disorder patients

Background: To detect attention deficit hyperactivity disorder (ADHD) in treatment seeking substance use disorders (SUD) patients, a valid screening instrument is needed. Objectives: To test the performance of the Adult ADHD Self-Report Scale V 1.1(ASRS) for adult ADHD in an international sample of treatment seeking SUD patients for DSM-IV-TR; for the proposed DSM-5 criteria; in different subpopulations, at intake and 1–2 weeks after intake; using different scoring algorithms; and different externalizing disorders as external criterion (including adult ADHD, bipolar disorder, antisocial and borderline personality disorder). Methods: In 1138 treatment seeking SUD subjects, ASRS performance was determined using diagnoses based on Conner's Adult ADHD Diagnostic Interview for DSM-IV (CAADID) as gold standard. Results: The prevalence of adult ADHD was 13.0% (95% CI: 11.0–15.0%). The overall positive predictive value (PPV) of the ASRS was 0.26 (95% CI: 0.22–0.30), the negative predictive value (NPV) was 0.97 (95% CI: 0.96–0.98). The sensitivity (0.84, 95% CI: 0.76–0.88) and specificity (0.66, 95% CI: 0.63–0.69) measured at admission were similar to the sensitivity (0.88, 95% CI: 0.83–0.93) and specificity (0.67, 95% CI: 0.64–0.70) measured 2 weeks after admission. Sensitivity was similar, but specificity was significantly better in patients with alcohol compared to (illicit) drugs as the primary substance of abuse (0.76 vs. 0.56). ASRS was not a good screener for externalizing disorders other than ADHD. Conclusions: The ASRS is a sensitive screener for identifying possible ADHD cases with very few missed cases among those screening negative in this population.

Cross-cultural analysis of type D (distressed) personality in 6222 patients with ischemic heart disease: a study from the International HeartQoL Project

BACKGROUND Type D (distressed) personality, the conjoint effect of negative affectivity (NA) and social inhibition (SI), predicts adverse cardiovascular outcomes, and is assessed with the 14-item Type D Scale (DS14). However, potential cross-cultural differences in Type D have not been examined yet in a direct comparison of countries. AIM To examine the cross-cultural validity of the Type D construct and its relation with cardiovascular risk factors, cardiac symptom severity, and depression/anxiety. METHODS In 22 countries, 6222 patients with ischemic heart disease (angina, 33%; myocardial infarction, 37%; or heart failure, 30%) completed the DS14 as part of the International HeartQoL Project. RESULTS Type D personality was assessed reliably across countries (αNA>.80; αSI>.74; except Russia, which was excluded from further analysis). Cross-cultural measurement equivalence was established for Type D personality at all measurement levels, as the factor-item configuration, factor loadings, and error structure were not different across countries (fit: CFI=.91; NFI=.88; RMSEA=.018), as well as across gender and diagnostic subgroups. Type D personality was more prevalent in Southern (37%) and Eastern (35%) European countries compared to Northern (24%) and Western European and English-speaking (both 27%) countries (p<.001). Type D was not confounded by cardiac symptom severity, but was associated with a higher prevalence of hypertension, smoking, sedentary lifestyle, and depression. CONCLUSION Cross-cultural measurement equivalence was demonstrated for the Type D scale in 21 countries. There is a pan-cultural relationship between Type D personality and some cardiovascular risk factors, supporting the role of Type D personality across countries and cardiac conditions.

The structural substrate of brain function: analysis of brain structural networks based on diffusion-weighted imaging

The brain is a complex neural network with a hierarchical organization and the mapping of its elements and connections is an important step towards the understanding of its function. Recent developments in diffusion-weighted imaging have provided the opportunity to reconstruct the whole-brain structural network in-vivo at a large scale level and to study the brain structural substrate in a framework that is close to the current understanding of brain function. However, methods to construct the connectome are still under development and they should be carefully evaluated. To this end, the first two studies included in my thesis aimed at improving the analytical tools specific to the methodology of brain structural networks. The first of these papers assessed the repeatability of the most common global and local network metrics used in literature to characterize the connectome, while in the second paper the validity of further metrics based on the concept of communicability was evaluated. Communicability is a broader measure of connectivity which accounts also for parallel and indirect connections. These additional paths may be important for reorganizational mechanisms in the presence of lesions as well as to enhance integration in the network. These studies showed good to excellent repeatability of global network metrics when the same methodological pipeline was applied, but more variability was detected when considering local network metrics or when using different thresholding strategies. In addition, communicability metrics have been found to add some insight into the integration properties of the network by detecting subsets of nodes that were highly interconnected or vulnerable to lesions. The other two studies used methods based on diffusion-weighted imaging to obtain knowledge concerning the relationship between functional and structural connectivity and about the etiology of schizophrenia. The third study integrated functional oscillations measured using electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) as well as diffusion-weighted imaging data. The multimodal approach that was applied revealed a positive relationship between individual fluctuations of the EEG alpha-frequency and diffusion properties of specific connections of two resting-state networks. Finally, in the fourth study diffusion-weighted imaging was used to probe for a relationship between the underlying white matter tissue structure and season of birth in schizophrenia patients. The results are in line with the neurodevelopmental hypothesis of early pathological mechanisms as the origin of schizophrenia. The different analytical approaches selected in these studies also provide arguments for discussion of the current limitations in the analysis of brain structural networks. To sum up, the first studies presented in this thesis illustrated the potential of brain structural network analysis to provide useful information on features of brain functional segregation and integration using reliable network metrics. In the other two studies alternative approaches were presented. The common discussion of the four studies enabled us to highlight the benefits and possibilities for the analysis of the connectome as well as some current limitations.

University and school students' motivation for effortful thinking: Factor structure, reliability, and validity of the German Need for Cognition Scale

Need for cognition (NFC) reflects a relatively stable trait regarding the degree to which one enjoys and engages in cognitive endeavors. We examined whether the previously demonstrated one-dimensional structure of the German NFC Scale could be replicated in three samples of undergraduates and secondary school students. Moreover, we investigated the test-retest reliability of the German NFC Scale, which has not yet been tested. Further, we investigated whether the scale would be valid in a sample of secondary school students. Multigroup confirmatory factor analyses established the one-dimensional factor structure of the long form as well as the short form of the German NFC Scale for undergraduates (N = 559), students of academic track secondary schools (German Gymnasium; N = 555), and students of vocational track secondary schools (German Realschule; N = 486). The scale proved to have a high test-retest reliability in a university student sample (N = 43). For secondary school students, we again found a high test-retest reliability (N = 157), and also found the scale to be valid (N = 181).

Development and Criterion Validity of Differentiated and Elevated Vocational Interests in Adolescence

Interest differentiation and elevation are supposed to provide important information about a person’s state of interest development, yet little is known about their development and criterion validity. The present study explored these constructs among a group of Swiss adolescents. Study 1 applied a cross-sectional design with 210 students in 11th grade. Study 2 applied a 1-year longitudinal design with 289 students in 7th to 8th grade. gender, personality traits, and career exploration were significant predictors of state and development of differentiation and elevation. Increase in differentiation predicted increase in career decidedness above traits. elevation could not predict increase in exploration behavior over traits. The results provide support for differentiation and elevation as important aspects of adolescents’ vocational interests.

Validierung einer deutschsprachigen Version des Career Futures Inventory (CFI): Berufliche Anpassungsfähigkeit, beruflicher Optimismus, Arbeitsmarktwissen und Berufserfolg

Das Career Futures Inventory (CFI; Rottingaus, Day & Borgen, 2005) erfasst die drei laufbahnbezogenen Zukunftsfaktoren career adaptability (Berufliche Anpassungsfähigkeit), career optimism (Beruflicher Optimismus) und knowledge of job market (Arbeitsmarktwissen). Die Ziele dieser Studie waren (a) eine deutschsprachige Version des CFIs zu konstruieren und (b) an zwei Stichproben mit Erwerbstätigen mittels (c) unterschiedlicher Indikatoren des Berufserfolgs und anhand von Persönlichkeitseigenschaften zu validieren. Es konnte in beiden Stichproben (N1 = 153; N2 = 110) die 3-faktorielle Struktur des CFIs nachgewiesen werden. Zudem konnten Annahmen zur konvergenten und diskriminanten Validität in beiden Stichproben bestätigt werden, da die CFI-Faktoren in erwarteter Richtung mit Berufserfolg (z. B. Beförderungen, Karrierezufriedenheit) bzw. Persönlichkeitseigenschaften (d. h. Big Five) korrelierten bzw. nicht korrelierten. Eine Analyse der inkrementellen Validität zeigte, dass Arbeitsmarktwissen zusätzliche Varianz beim Gehalt über Kontrollvariablen und Persönlichkeit hinaus erklärte. Hinsichtlich Karrierezufriedenheit war dies für beruflichen Optimismus der Fall. Diese Validierungsstudie ermöglicht den Einsatz des CFIs im deutschsprachigen Raum.

The construct of career success: measurement issues and an empirical example

The aim of the present paper is twofold. We first give an overview of the multi-faceted construct of career success with its “objective” (real attainments) and “subjective” (perceived attainments) components and with the different methodologies measuring them (“subjectivist” vs. “objectivist” approaches). We also discuss the relationship between the different components of career success as revealed in the literature. Our second aim is to show that the interrelationship between the different facets of career success varies with the employment context. We analyze subjective and objective career success measures collected from professionals working either in the private sector, in the public sector, or that are self-employed. Our findings show that objective success and subjective success are more closely related in the private and self-employment sectors than in the public sector. Moreover, the interrelationship between objective and subjective success varies with the specific subjective success measure chosen. It is stronger for an “other-referent” subjective success measure than for a “self-referent” subjective success measure. Conclusions regarding the measurement of career success as well as regarding the relevance of subjective success for studying occupational careers are drawn.

Clinical validity of the nerve root sedimentation sign in patients with suspected lumbar spinal stenosis

BACKGROUND CONTEXT The nerve root sedimentation sign in transverse magnetic resonance imaging has been shown to discriminate well between selected patients with and without lumbar spinal stenosis (LSS), but the performance of this new test, when used in a broader patient population, is not yet known. PURPOSE To evaluate the clinical performance of the nerve root sedimentation sign in detecting central LSS above L5 and to determine its potential significance for treatment decisions. STUDY DESIGN Retrospective cohort study. PATIENT SAMPLE One hundred eighteen consecutive patients with suspected LSS (52% women, median age 62 years) with a median follow-up of 24 months. OUTCOME MEASURES Oswestry disability index (ODI) and back and leg pain relief. METHODS We performed a clinical test validation study to assess the clinical performance of the sign by measuring its association with health outcomes. Subjects were patients referred to our orthopedic spine unit from 2004 to 2007 before the sign had been described. Based on clinical and radiological diagnostics, patients had been treated with decompression surgery or nonsurgical treatment. Changes in the ODI and pain from baseline to 24-month follow-up were compared between sedimentation sign positives and negatives in both treatment groups. RESULTS Sixty-nine patients underwent surgery. Average baseline ODI in the surgical group was 54.7%, and the sign was positive in 39 patients (mean ODI improvement 29.0 points) and negative in 30 (ODI improvement 28.4), with no statistically significant difference in ODI and pain improvement between groups. In the 49 patients of the nonsurgical group, mean baseline ODI was 42.4%; the sign was positive in 18 (ODI improvement 0.6) and negative in 31 (ODI improvement 17.7). A positive sign was associated with a smaller ODI and back pain improvement than negative signs (both p<.01 on t test). CONCLUSIONS In patients commonly treated with decompression surgery, the sedimentation sign does not appear to predict surgical outcome. In nonsurgically treated patients, a positive sign is associated with more limited improvement. In these cases, surgery might be effective, but this needs investigation in prospective randomized trials (Australian New Zealand Clinical Trial Registry, number ACTRN12610000567022).

Measuring depression with a well-being index: Further evidence for the validity of the WHO Well-Being Index (WHO-5) as a measure of the severity of depression

BACKGROUND: In recent years, the WHO Wellbeing Index (WHO-5) has been used as a screening measure for depression. Nevertheless, research on the validity of this measure in the context of clinical depression is sparse. QUESTIONS: The aim of the present study was to investigate the measurement invariance of the WHO-5 across depressed and non-depressed individuals, as well as the shape and specificity of its relationship to measures of depression severity. METHOD: Of the 414 subjects who completed the WHO-5 and the Beck Depression Inventory-II (BDI-II), 207 had a diagnosis of a major depressive episode (MDE). A subsample also completed the Beck Anxiety Inventory (BAI) and was assessed by clinicians using the Hamilton Depression Rating Scale (HAM-D) and the Hamilton Anxiety Rating Scale (HAM-A). RESULTS: The WHO-5 demonstrated strong measurement invariance regarding the presence or absence of a current MDE. The WHO-5 showed a very high negative association with self- and observer-rated measures of depressive symptoms, especially in the range of mild to moderate symptoms. These associations were still substantial after controlling for measures of anxiety symptoms. LIMITATIONS: In addition to a diagnostic interview, only one measure for self- and observer-rated symptoms of depression was used. Furthermore, the observer-rated measure was only assessed in one subsample that exhibited a somewhat restricted range of depression severity. CONCLUSION: Although this index was originally designed as a measure of well-being, the results support the use of the WHO-5 in the context of depression research.

Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning

Background: Virtual patients (VPs) are increasingly used to train clinical reasoning. So far, no validated evaluation instruments for VP design are available. Aims: We examined the validity of an instrument for assessing the perception of VP design by learners. Methods: Three sources of validity evidence were examined: (i) Content was examined based on theory of clinical reasoning and an international VP expert team. (ii) The response process was explored in think-aloud pilot studies with medical students and in content analyses of free text questions accompanying each item of the instrument. (iii) Internal structure was assessed by exploratory factor analysis (EFA) and inter-rater reliability by generalizability analysis. Results: Content analysis was reasonably supported by the theoretical foundation and the VP expert team. The think-aloud studies and analysis of free text comments supported the validity of the instrument. In the EFA, using 2547 student evaluations of a total of 78 VPs, a three-factor model showed a reasonable fit with the data. At least 200 student responses are needed to obtain a reliable evaluation of a VP on all three factors. Conclusion: The instrument has the potential to provide valid information about VP design, provided that many responses per VP are available.

General distress, hopelessness—suicidal ideation and worrying in adolescence: Concurrent and predictive validity of a symptom-level bifactor model for clinical diagnoses

BACKGROUND: Clinical disorders often share common symptoms and aetiological factors. Bifactor models acknowledge the role of an underlying general distress component and more specific sub-domains of psychopathology which specify the unique components of disorders over and above a general factor. METHODS: A bifactor model jointly calibrated data on subjective distress from The Mood and Feelings Questionnaire and the Revised Children's Manifest Anxiety Scale. The bifactor model encompassed a general distress factor, and specific factors for (a) hopelessness-suicidal ideation, (b) generalised worrying and (c) restlessness-fatigue at age 14 which were related to lifetime clinical diagnoses established by interviews at ages 14 (concurrent validity) and current diagnoses at 17 years (predictive validity) in a British population sample of 1159 adolescents. RESULTS: Diagnostic interviews confirmed the validity of a symptom-level bifactor model. The underlying general distress factor was a powerful but non-specific predictor of affective, anxiety and behaviour disorders. The specific factors for hopelessness-suicidal ideation and generalised worrying contributed to predictive specificity. Hopelessness-suicidal ideation predicted concurrent and future affective disorder; generalised worrying predicted concurrent and future anxiety, specifically concurrent generalised anxiety disorders. Generalised worrying was negatively associated with behaviour disorders. LIMITATIONS: The analyses of gender differences and the prediction of specific disorders was limited due to a low frequency of disorders other than depression. CONCLUSIONS: The bifactor model was able to differentiate concurrent and predict future clinical diagnoses. This can inform the development of targeted as well as non-specific interventions for prevention and treatment of different disorders.