Do multiple true-false items beat the commonly used one-best-answers questions regarding to the Ottawa Criteria for Good Assessment? Results of a literature review

Background: Multiple True-False-Items (MTF-Items) might offer some advantages compared to one-best-answer-questions (TypeA) as they allow more than one correct answer and may better represent clinical decisions. However, in medical education assessment MTF-Items are seldom used. Summary of Work: With this literature review existing findings on MTF-items and on TypeA were compared along the Ottawa Criteria for Good Assessment, i.e. (1) reproducibility, (2) feasibility, (3) validity, (4) acceptance, (5) educational effect, (6) catalytic effects, and (7) equivalence. We conducted a literature research on ERIC and Google Scholar including papers from the years 1935 to 2014. We used the search terms “multiple true-false”, “true-false”, “true/false”, and “Kprim” combined with “exam”, “test”, and “assessment”. Summary of Results: We included 29 out of 33 studies. Four of them were carried out in the medical field Compared to TypeA, MTF-Items are associated with (1) higher reproducibility (2) lower feasibility (3) similar validity (4) higher acceptance (5) higher educational effect (6) no studies on catalytic effects or (7) equivalence. Discussion and Conclusions: While studies show overall good characteristics of MTF items according to the Ottawa criteria, this type of question seems to be rather seldom used. One reason might be the reported lower feasibility. Overall the literature base is still weak. Furthermore, only 14 % of literature is from the medical domain. Further studies to better understand the characteristics of MTF-Items in the medical domain are warranted. Take-home messages: Overall the literature base is weak and therefore further studies are needed. Existing studies show that: MTF-Items show higher reliability, acceptance and educational effect; MTF-Items are more difficult to produce

Assessment of in vivo genotoxicity of the rodent carcinogen furan: evaluation of DNA damage and induction of micronuclei in mouse splenocytes

In recent years, several surveys have highlighted the presence of the rodent carcinogen furan in a variety of food items. Even though the evidence of carcinogenicity of furan is unequivocal, the underlying mechanism has not been fully elucidated. In particular, the role of genotoxicity in furan carcinogenicity is still not clear, even though this information is considered pivotal for the assessment of the risk posed by the presence of low doses of furan in food. In this work, the genotoxic potential of furan in vivo has been investigated in mice, under exposure conditions similar to those associated with cancer onset in the National Toxicology Program long-term bioassay. To this aim, male B6C3F1 mice were treated by gavage for 4 weeks with 2, 4, 8 and 15 mg furan/kg b.w./day. Spleen was selected as the target organ for genotoxicity assessment, in view of the capability of quiescent splenocytes to accumulate DNA damage induced by repeat dose exposure. The induction of primary DNA damage in splenocytes was evaluated by alkaline single-cell gel electrophoresis (comet assay) and by the immunofluorescence detection of foci of phosphorylated histone H2AX (gamma-H2AX). The presence of cross-links was probed in a modified comet assay, in which cells were irradiated in vitro with gamma-rays before electrophoresis. Chromosome damage was quantitated through the detection of micronuclei in mitogen-stimulated splenocytes using the cytokinesis-block method. Micronucleus induction was also assessed with a modified protocol, using the repair inhibitor 1-beta-arabinofuranosyl-cytosine to convert single-strand breaks in micronuclei. The results obtained show a significant (P < 0.01) increase of gamma-H2AX foci in mitogen-stimulated splenocytes of mice treated with 8 and 15 mg furan/kg b.w. and a statistically significant (P < 0.001) increases of micronuclei in binucleated splenocytes cultured in vitro. Conversely, no effect of in vivo exposure to furan was observed when freshly isolated quiescent splenocytes were analysed by immunofluorescence and in comet assays, both with standard and radiation-modified protocols. These results indicate that the in vivo exposure to furan gives rise to pre-mutagenic DNA damage in resting splenocytes, which remains undetectable until it is converted in frank lesions during the S-phase upon mitogen stimulation. The resulting DNA strand breaks are visualized by the increase in gamma-H2AX foci and may originate micronuclei at the subsequent mitosis.

Comprehensive assessment of gesture production: a new test of upper limb apraxia (TULIA)

BACKGROUND: Only few standardized apraxia scales are available and they do not cover all domains and semantic features of gesture production. Therefore, the objective of the present study was to evaluate the reliability and validity of a newly developed test of upper limb apraxia (TULIA), which is comprehensive and still short to administer. METHODS: The TULIA consists of 48 items including imitation and pantomime domain of non-symbolic (meaningless), intransitive (communicative) and transitive (tool related) gestures corresponding to 6 subtests. A 6-point scoring method (0-5) was used (score range 0-240). Performance was assessed by blinded raters based on videos in 133 stroke patients, 84 with left hemisphere damage (LHD) and 49 with right hemisphere damage (RHD), as well as 50 healthy subjects (HS). RESULTS: The clinimetric findings demonstrated mostly good to excellent internal consistency, inter- and intra-rater (test-retest) reliability, both at the level of the six subtests and at individual item level. Criterion validity was evaluated by confirming hypotheses based on the literature. Construct validity was demonstrated by a high correlation (r = 0.82) with the De Renzi-test. CONCLUSION: These results show that the TULIA is both a reliable and valid test to systematically assess gesture production. The test can be easily applied and is therefore useful for both research purposes and clinical practice.

The Bern psychopathology scale for the assessment of system-specific psychotic symptoms

The translation from psychiatric core symptoms to brain functions and vice versa is a largely unresolved issue. In particular, the search for disorders of single brain regions explaining classical symptoms has not yielded the expected results. Based on the assumption that the psychopathology of psychosis is related to a functional imbalance of higher-order brain systems, the authors focused on three specific candidate brain circuitries, namely the language, and limbic and motor systems. These domains are of particular interest for understanding the disastrous communication breakdown during psychotic disorders. Core symptoms of psychosis were mapped on these domains by shaping their definitions in order to match the related brain functions. The resulting psychopathological assessment scale was tested for interrater reliability and internal consistency in a group of 168 psychotic patients. The items of the scale were reliable and a principal component analysis (PCA) was best explained by a solution resembling the three candidate systems. Based on the results, the scale was optimized as an instrument to identify patient subgroups characterized by a prevailing dysfunction of one or more of these systems. In conclusion, the scale is apt to distinguish symptom domains related to the activity of defined brain systems. PCA showed a certain degree of independence of the system-specific symptom clusters within the patient group, indicating relative subgroups of psychosis. The scale is understood as a research instrument to investigate psychoses based on a system-oriented approach. Possible immediate advantages in the clinical application of the understanding of psychoses related to system-specific symptom domains are also discussed.

The assessment of cognition in visually impaired older adults

Background: visual and cognitive impairments are common in later life. Yet there are very few cognitive screening tests for the visually impaired. Objective: to screen for cognitive impairment in the visually impaired. Methods: case-control study including 150 elderly participants with visual impairment (n = 74) and a control group without visual impairment (n = 76) using vision-independent cognitive tests and cognitive screening tests (MMSE and clock drawing tests (CDT)) which are in part vision dependent. Results: the scoring of the two groups did not differ in the vision-independent cognitive tests. Visually impaired patients performed poorer than controls in the vision-dependent items of the MMSE (T = 7.3; df: 148; P < 0.001) and in CDT (T = 3.1; df: 145; P = 0.003). No group difference was found when vision-independent items were added to MMSE and CDT. The test score gain by the use of vision-independent items correlated with the severity of visual impairment (P < 0.002). Conclusion: visually impaired patients benefit from cognitive tests, which do not rely on vision. The more visually impaired the greater the benefit.

A pain assessment scale for population-based studies: Development and validation of the Pain Module of the Standard Evaluation Questionnaire

The objectives of this study were to develop and validate a tool for assessing pain in population-based observational studies and to develop three subscales for back/neck, upper extremity and lower extremity pain. Based on a literature review, items were extracted from validated questionnaires and reviewed by an expert panel. The initial questionnaire consisted of a pain manikin and 34 items relating to (i) intensity of pain in different body regions (7 items), (ii) pain during activities of daily living (18 items) and (iii) various pain modalities (9 items). Psychometric validation of the initial questionnaire was performed in a random sample of the German-speaking Swiss population. Analyses included tests for reliability, correlation analysis, principal components factor analysis, tests for internal consistency and validity. Overall, 16,634 of 23,763 eligible individuals participated (70%). Test-retest reliability coefficients ranged from 0.32 to 0.97, but only three coefficients were below 0.60. Subscales were constructed combining four items for each of the subscales. Item-total coefficients ranged from 0.76 to 0.86 and Cronbach's alpha were 0.75 or higher for all subscales. Correlation coefficients between subscales and three validated instruments (WOMAC, SPADI and Oswestry) ranged from 0.62 to 0.79. The final Pain Standard Evaluation Questionnaire (SEQ Pain) included 28 items and the pain manikin and accounted for the multidimensionality of pain by assessing pain location and intensity, pain during activity, triggers and time of onset of pain and frequency of pain medication. It was found to be reliable and valid for the assessment of pain in population-based observational studies.

Development of the Davos Assessment of Cognitive Biases Scale (DACOBS)

Objective: Cognitive problems and biases play an important role in the development and continuation of psychosis. A self-report measure of these deficits and processes was developed (Davos Assessment of Cognitive Biases Scale: DACOBS) and is evaluated in this study. Methods: An item pool made by international experts was used to develop a self-report scale on a sample of 138 schizophrenia spectrum patients. Another sample of 71 patients was recruited to validate the subscales. A group of 186 normal control subjects was recruited to establish norms and examine discriminative validity. Results: Factor analyses resulted in seven factors, each with six items (jumping to conclusions, belief inflexibility bias, attention for threat bias, external attribution bias, social cognition problems, subjective cognitive problems and safety behavior). All factors independently explained the variance (eigenvalues > 2) and total explained variance was 45%. Reliability was good (Cronbach's alpha = .90; split-half reliability = .92; test–retest reliability = .86). The DACOBS discriminates between schizophrenia spectrum patients and normal control subjects. Validity was affirmed for five of seven subscales. The scale ‘Subjective Cognitive problems’ was not associated with objective cognitive functioning and ‘Social cognition problems’ was not associated with the Hinting task, but with the scale measuring ideas of social reference. Conclusions: The DACOBS scale, with seven independent subscales, is reliable and valid for use in clinical practice and research.

FAIR_OSCE – Feedback structure for assessment of interactive roleplay in Objective Structured Clinical Exams

Introduction Since the quality of patient portrayal of standardized patients (SPs) during an Objective Structured Clinical Exam (OSCE) has a major impact on the reliability and validity of the exam, quality control should be initiated. Literature about quality control of SPs’ performance focuses on feedback [1, 2] or completion of checklists [3, 4]. Since we did not find a published instrument meeting our needs for the assessment of patient portrayal, we developed such an instrument after being inspired by others [5] and used it in our high-stakes exam. Project description SP trainers from five medical faculties collected and prioritized quality criteria for patient portrayal. Items were revised twice, based on experiences during OSCEs. The final instrument contains 14 criteria for acting (i.e. adequate verbal and non-verbal expression) and standardization (i.e. verbatim delivery of the first sentence). All partners used the instrument during a high-stakes OSCE. SPs and trainers were introduced to the instrument. The tool was used in training (more than 100 observations) and during the exam (more than 250 observations). Outcome High quality of SPs’ patient portrayal during the exam was documented. More than 90% of SP performances were rated to be completely correct or sufficient. An increase in quality of performance between training and exam was noted. For example, the rate of completely correct reaction in medical tests increased from 88% to 95%. Together with 4% of sufficient performances these 95% add up to 99% of the reactions in medical tests meeting the standards of the exam. SP educators using the instrument reported an augmentation of SPs’ performance induced by the use of the instrument. Disadvantages mentioned were the high concentration needed to observe all criteria and the cumbersome handling of the paper-based forms. Discussion We were able to document a very high quality of SP performance in our exam. The data also indicates that our training is effective. We believe that the high concentration needed using the instrument is well invested, considering the observed enhancement of performance. The development of an iPad-based application for the form is planned to address the cumbersome handling of the paper.

Factor analysis of the North American Spine Society outcome assessment instrument: a study based on a spine registry of patients treated with lumbar and cervical disc arthroplasty.

Background context Studies involving factor analysis (FA) of the items in the North American Spine Society (NASS) outcome assessment instrument have revealed inconsistent factor structures for the individual items. Purpose This study examined whether the factor structure of the NASS varied in relation to the severity of the back/neck problem and differed from that originally recommended by the developers of the questionnaire, by analyzing data before and after surgery in a large series of patients undergoing lumbar or cervical disc arthroplasty. Study design/setting Prospective multicenter observational case series. Patient sample Three hundred ninety-one patients with low back pain and 553 patients with neck pain completed questionnaires preoperatively and again at 3 to 6 and 12 months follow-ups (FUs), in connection with the SWISSspine disc arthroplasty registry. Outcome measures North American Spine Society outcome assessment instrument. Methods First, an exploratory FA without a priori assumptions and subsequently a confirmatory FA were performed on the 17 items of the NASS-lumbar and 19 items of the NASS-cervical collected at each assessment time point. The item-loading invariance was tested in the German version of the questionnaire for baseline and FU. Results Both NASS-lumbar and NASS-cervical factor structures differed between baseline and postoperative data sets. The confirmatory analysis and item-loading invariance showed better fit for a three-factor (3F) structure for NASS-lumbar, containing items on “disability,” “back pain,” and “radiating pain, numbness, and weakness (leg/foot)” and for a 5F structure for NASS-cervical including disability, “neck pain,” “radiating pain and numbness (arm/hand),” “weakness (arm/hand),” and “motor deficit (legs).” Conclusions The best-fitting factor structure at both baseline and FU was selected for both the lumbar- and cervical-NASS questionnaires. It differed from that proposed by the originators of the NASS instruments. Although the NASS questionnaire represents a valid outcome measure for degenerative spine diseases, it is able to distinguish among all major symptom domains (factors) in patients undergoing lumbar and cervical disc arthroplasty; overall, the item structure could be improved. Any potential revision of the NASS should consider its factorial structure; factorial invariance over time should be aimed for, to allow for more precise interpretations of treatment success.

Food Recognition for Dietary Assessment Using Deep Convolutional Neural Networks

Diet management is a key factor for the prevention and treatment of diet-related chronic diseases. Computer vision systems aim to provide automated food intake assessment using meal images. We propose a method for the recognition of already segmented food items in meal images. The method uses a 6-layer deep convolutional neural network to classify food image patches. For each food item, overlapping patches are extracted and classified and the class with the majority of votes is assigned to it. Experiments on a manually annotated dataset with 573 food items justified the choice of the involved components and proved the effectiveness of the proposed system yielding an overall accuracy of 84.9%.

Evaluation of an inventory for the assessment of critical incidents involving football fans

Introduction: Fan violence is a frequent occurrence in Swiss football (Bundesamt für Polizei, 2015) leading to high costs for prevention and control (Mensch & Maurer, 2014). Various theories put forward an explanation of fan violence, such as the Elaborated Social Identity Model (Drury & Reicher, 2000)and the Aggravation Mitigation Model (Hylander & Guvå, 2010). Important observations from these theories are the multi-dimensional understanding of fan violence and the Dynamics occurring in the fan group. Nevertheless, none of them deal with critical incidents (CIs) which involve a tense atmosphere combined with a higher risk of fan violence. Schumacher Dimech, Brechbühl and Seiler (2015) tackled this gap in research and explored CIs where 43 defining criteria were identified and compiled in an integrated model of CIs. The defining criteria were categorised in four higher-order themes “antecedents” (e.g. a documented history of fan rivalry), “triggers” (e.g. the arrest of a fan), “reactions” (e.g. fans masking themselves) and “consequences” (e.g. fans avoiding communication with fan social workers). Methods: An inventory based on this model is being developed including these 43 criteria. In an exploratory phase, this inventory was presented as an online questionnaire and was completed by 143 individuals. Three main questions are examined: Firstly, the individual items are tested using descriptive analyses. An item analysis is conducted to test reliability, item difficulty and discriminatory power. Secondly, the model’s four higher-order themes are tested using exploratory factor analysis (EFA). Thirdly, differences between sub -groups are explored, such as gender and age-related differences. Results: Respondents rated the items’ importance as high and the quota of incomplete responses was not systematic. Two items were removed from the inventory because of low mean or a high rate of “don’t know”-responses. EFA produced a six-factor solution grouping items into match-related factors, repressive measures, fans’ delinquent behaviour, intra-group behaviour, communication and control and inter-group factors. The item “fans consume alcohol” could not be ordered into any category but was retained since literature accentuates this factor’s influence on fan violence. Analyses examining possible differences between groups are underway. Discussion: Results exploring the adequacy of this inventory assessing defining criteria of CIs in football are promising and thus further evaluative investigation is recommended. This inventory can be used in two ways: as a standardised instrument of assessment for experts evaluating specific CIs and as an instrument for exploring differences in perception and assessment of a CI e.g. gender and age differences, differences between interest groups and stakeholders.

Evaluation of an inventory for the assessment of critical incidents involving football fans

Introduction: Fan violence is a frequent occurrence in Swiss football (Bundesamt für Polizei, 2015) leading to high costs for prevention and control (Mensch & Maurer, 2014). Various theories put forward an explanation of fan violence, such as the Elaborated Social Identity Model (Drury & Reicher, 2000) and the Aggravation Mitigation Model (Hylander & Guvå, 2010). Important observations from these theories are the multi-dimensional understanding of fan violence and the dynamics occurring in the fan group. Nevertheless, none of them deal with critical incidents (CIs) which involve a tense atmosphere combined with a higher risk of fan violence. Schumacher Dimech, Brechbühl and Seiler (2015) tackled this gap in research and explored CIs where 43 defining criteria were identified and compiled in an integrated model of CIs. The defining criteria were categorised in four higher-order themes “antecedents” (e.g. a documented history of fan rivalry), “triggers” (e.g. the arrest of a fan), “reactions” (e.g. fans masking themselves) and “consequences” (e.g. fans avoiding communication with fan social workers). Methods: An inventory based on this model is being developed including these 43 criteria. In an exploratory phase, this inventory was presented as an online questionnaire and was completed by 143 individuals. Three main questions are examined: Firstly, the individual items are tested using descriptive analyses. An item analysis is conducted to test reliability, item difficulty and discriminatory power. Secondly, the model’s four higher-order themes are tested using exploratory factor analysis (EFA). Thirdly, differences between sub-groups are explored, such as gender and agerelated differences. Results: Respondents rated the items’ importance as high and the quota of incomplete responses was not systematic. Two items were removed from the inventory because of low mean or a high rate of “don’t know”-responses. EFA produced a six-factor solution grouping items into match-related factors, repressive measures, fans’ delinquent behaviour, intra-group behaviour, communication and control and inter-group factors. The item “fans consume alcohol” could not be ordered into any category but was retained since literature accentuates this factor’s influence on fan violence. Analyses examining possible differences between groups are underway. Discussion: Results exploring the adequacy of this inventory assessing defining criteria of CIs in football are promising and thus further evaluative investigation is recommended. This inventory can be used in two ways: as a standardised instrument of assessment for experts evaluating specific CIs and as an instrument for exploring differences in perception and assessment of a CI e.g. gender and age differences, differences between interest groups and stakeholders. References: Bundesamt für Polizei. (2015). Jahresbericht 2014. Kriminalitätsbekämpfung Bund. Lage, Massnahmen und Mittel [Electronic Version]. Drury, J., & Reicher, S. (2000). Collective action and psychological change. The emergence of new social identities. British Journal of Social Psychology, 39, 579-604. Hylander, I., & Guvå, G. (2010). Misunderstanding of out-group behaviour: Different interpretations of the same crowd events among police officers and demonstrators. Nordic Psychology, 62, 25-47. Schumacher-Dimech, A., Brechbühl, A. &, Seiler, R. (2016). Dynamics of critical incidents with potentially violent outcomes involving ultra fans: an explorative study. Sport in Society. Advance online publication. doi: 10.1080/17430437.2015.1133597