Assessment of medical courses in Brazil using student-completed questionnaires. Is it reliable?

INTRODUCTION: Debates about the quality of medical education have become more evident in the recent past, and as a result several different assessment methods have been refined for that purpose. The use of questionnaires filled out by medical students to assess the quality of lectures is one of the most common methods employed in our milieu. However, the reliability of this investigation method has not yet been systematically tested. The authors present the reliability of a specific form applied to the fourth grade medical students during the clinical psychiatry course. METHOD: Eighty-one fourth grade medical students were instructed to complete a form immediately after each clinical psychiatry lecture. Thirty-four students (42%) failed to turn in the forms after the final lecture. These students were given an identical form to assess the lectures in a retrospective fashion. The grades given by both groups of students for each performed lecture and the number of students who have graded an unperformed lecture were compared. Statistical significance for both groups was determined by means of the chi-square test (p< 0.05). RESULTS: Eighteen out of the 34 students who filled out the forms retrospectively (53%) rated the unperformed lecture, whereas only 5 out of the 47 students who filled out the forms during the course (11%) did so. This is statistically significant (p< 0.05). There was no statistical difference for the grades given to the lectures that were actually performed. DISCUSSION: The authors concluded the low reliability rate of the retrospective evaluation warrant a continuous assessment method during the course.

Reliability and validity of heart rate variability threshold assessment during an incremental shuttle-walk test in middle-aged and older adults

Studies on the assessment of heart rate variability threshold (HRVT) during walking are scarce. We determined the reliability and validity of HRVT assessment during the incremental shuttle walk test (ISWT) in healthy subjects. Thirty-one participants aged 57 ± 9 years (17 females) performed 3 ISWTs. During the 1st and 2nd ISWTs, instantaneous heart rate variability was calculated every 30 s and HRVT was measured. Walking velocity at HRVT in these tests (WV-HRVT1 and WV-HRVT2) was registered. During the 3rd ISWT, physiological responses were assessed. The ventilatory equivalents were used to determine ventilatory threshold (VT) and the WV at VT (WV-VT) was recorded. The difference between WV-HRVT1 and WV-HRVT2 was not statistically significant (median and interquartile range = 4.8; 4.8 to 5.4 vs4.8; 4.2 to 5.4 km/h); the correlation between WV-HRVT1 and WV-HRVT2 was significant (r = 0.84); the intraclass correlation coefficient was high (0.92; 0.82 to 0.96), and the agreement was acceptable (-0.08 km/h; -0.92 to 0.87). The difference between WV-VT and WV-HRVT2 was not statistically significant (4.8; 4.8 to 5.4 vs 4.8; 4.2 to 5.4 km/h) and the agreement was acceptable (0.04 km/h; -1.28 to 1.36). HRVT assessment during walking is a reliable measure and permits the estimation of VT in adults. We suggest the use of the ISWT for the assessment of exercise capacity in middle-aged and older adults.

Explicit discrimination and health: development and psychometric properties of an assessment instrument

OBJECTIVE: To develop an instrument to assess discrimination effects on health outcomes and behaviors, capable of distinguishing harmful differential treatment effects from their interpretation as discriminatory events. METHODS: Successive versions of an instrument were developed based on a systematic review of instruments assessing racial discrimination, focus groups and review by a panel comprising seven experts. The instrument was refined using cognitive interviews and pilot-testing. The final version of the instrument was administered to 424 undergraduate college students in the city of Rio de Janeiro, Southeastern Brazil, in 2010. Structural dimensionality, two types of reliability and construct validity were analyzed. RESULTS: Exploratory factor analysis corroborated the hypothesis of the instrument's unidimensionality, and seven experts verified its face and content validity. The internal consistency was 0.8, and test-retest reliability was higher than 0.5 for 14 out of 18 items. The overall score was higher among socially disadvantaged individuals and correlated with adverse health behaviors/conditions, particularly when differential treatments were attributed to discrimination. CONCLUSIONS: These findings indicate the validity and reliability of the instrument developed. The proposed instrument enables the investigation of novel aspects of the relationship between discrimination and health.

Short Assessment of Health Literacy for Portuguese-speaking Adults

OBJECTIVE: To develop and validate a short health literacy assessment tool for Portuguese-speaking adults. METHODS: The Short Assessment of Health Literacy for Portuguese-speaking Adults is an assessment tool which consists of 50 items that assess an individual's ability to correctly pronounce and understand common medical terms. We evaluated the instrument's psychometric properties in a convenience sample of 226 Brazilian older adults. Construct validity was assessed by correlating the tool scores with years of schooling, self-reported literacy, and global cognitive functioning. Discrimination validity was assessed by testing the tool's accuracy in detecting inadequate health literacy, defined as failure to fully understand standard medical prescriptions. RESULTS: Moderate to high correlations were found in the assessment of construct validity (Spearman's coefficients ranging from 0.63 to 0.76). The instrument showed adequate internal consistency (Cronbach's alpha=0.93) and adequate test-retest reliability (intraclass correlation coefficient=0.95). The area under the receiver operating characteristic curve for detection of inadequate health literacy was 0.82. A version consisting of 18 items was tested and showed similar psychometric properties. CONCLUSIONS: The instrument developed showed good validity and reliability in a sample of Brazilian older adults. It can be used in research and clinical settings for screening inadequate health literacy.

Validity and reliability of a health care service evaluation instrument for tuberculosis

OBJECTIVE To evaluate the validity and reliability of an instrument that evaluates the structure of primary health care units for the treatment of tuberculosis.METHODS This cross-sectional study used simple random sampling and evaluated 1,037 health care professionals from five Brazilian municipalities (Natal, state of Rio Grande do Norte; Cabedelo, state of Paraíba; Foz do Iguaçu, state of Parana; Sao José do Rio Preto, state of Sao Paulo, and Uberaba, state of Minas Gerais) in 2011. Structural indicators were identified and validated, considering different methods of organization of the health care system in the municipalities of different population sizes. Each structure represented the organization of health care services and contained the resources available for the execution of health care services: physical resources (equipment, consumables, and facilities); human resources (number and qualification); and resources for maintenance of the existing infrastructure and technology (deemed as the organization of health care services). The statistical analyses used in the validation process included reliability analysis, exploratory factor analysis, and confirmatory factor analysis.RESULTS The validation process indicated the retention of five factors, with 85.9% of the total variance explained, internal consistency between 0.6460 and 0.7802, and quality of fit of the confirmatory factor analysis of 0.995 using the goodness-of-fit index. The retained factors comprised five structural indicators: professionals involved in the care of tuberculosis patients, training, access to recording instruments, availability of supplies, and coordination of health care services with other levels of care. Availability of supplies had the best performance and the lowest coefficient of variation among the services evaluated. The indicators of assessment of human resources and coordination with other levels of care had satisfactory performance, but the latter showed the highest coefficient of variation. The performance of the indicators “training” and “access to recording instruments” was inferior to that of other indicators.CONCLUSIONS The instrument showed feasibility of application and potential to assess the structure of primary health care units for the treatment of tuberculosis.

Personal and Social Performance (PSP) scale for patients with schizophrenia: translation to Portuguese, cross-cultural adaptation and interrater reliability

INTRODUCTION: Schizophrenia is a chronic mental disorder associated with impairment in social functioning. The most widely used scale to measure social functioning is the GAF (Global Assessment of Functioning), but it has the disadvantage of measuring at the same time symptoms and functioning, as described in its anchors. OBJECTIVES:Translation and cultural adaptation of the PSP, proposing a final version in Portuguese for use in Brazil. METHODS: We performed five steps: 1) translation; 2) back translation; 3) formal assessment of semantic equivalence; 4) debriefing; 5) analysis by experts. Interrater reliability (Intraclass correlation, ICC) between two raters was also measured. RESULTS: The final version was applied by two independent investigators in 18 adults with schizophrenia (DSM-IV-TR). The interrater reliability (ICC) was 0.812 (p < 0.001). CONCLUSION: The translation and adaptation of the PSP had an adequate level of semantic equivalence between the Portuguese version and the original English version. There were no difficulties related to understanding the content expressed in the translated texts and terms. Its application was easy and it showed a good interrater reliability. The PSP is a valid instrument for the measurement of personal and social functioning in schizophrenia.

Psychometric properties of assessment instruments for autism spectrum disorder: a systematic review of Brazilian studies

Objective To systematically review the scientific literature on the psychometric properties of international instruments for the assessment of autism spectrum disorder (ASD) in the Brazilian population. Methods A search of bibliographic references was conducted in six electronic databases: PsycINFO, PubMed, IndexPsi, Lilacs, Capes (theses and dissertations) and SciELO. The studies were selected by two independent researchers. Results The procedure identified 11 studies of the Brazilian population that encompassed six ASD assessment tools. Given the information provided, the adaptation of the M-CHAT, a screening instrument, was the best conducted. All steps of the adaptation process were described and the changes made to the final version of the instrument were presented, which was not addressed in other studies. In terms of reliability, all of the instruments that assessed internal consistency showed adequate values. In addition, the ADI-R and the CARS adaptations also satisfactorily contemplated inter-rater reliability and test-retest indices, respectively. Finally, all studies aiming to validate instruments showed evidence of validity and sensitivity, and specificity values above 0.90 were observed in the ASQ, ADI-R and ABC. Conclusion Considering both the psychometric aspects and the copyright information, the screening instrument that currently appears to be best indicated for clinical and research use is the M-CHAT. It was also noticed that there are still no specific ASD diagnostic tools available for use in Brazil. This lack of diagnostic instruments consists in a critical situation for the improvement of clinical practice and the development of research in this area.

Validity and reliability of a questionnaire on knowledge of cardiovascular risk factors for use in Brazil

Using a sample of patients with coronary artery disease, this methodological study aimed to conduct a cross-cultural adaptation and validation of a questionnaire on knowledge of cardiovascular risk factors (Q-FARCS), lifestyle changes, and treatment adherence for use in Brazil. The questionnaire has three scales: general knowledge of risk factors (RFs); specific knowledge of these RFs; and lifestyle changes achieved. Cross-cultural adaptation included translation, synthesis, back-translation, expert committee review, and pretesting. Face and content validity, reliability, and construct validity were measured. Cronbach’s alpha for the total sample (n = 240) was 0.75. Assessment of psychometric properties revealed adequate face and content validity, and the construct revealed seven components. It was concluded that the Brazilian version of Q-FARCS had adequate reliability and validity for the assessment of knowledge of cardiovascular RFs.

Cultural adaptation of the scale Pain Assessment in Advanced Dementia – PAINAD to Brazil

Objective: To translate and culturally adapt to Brazil the scale Pain Assessment in Advanced Dementia(PAINAD).Method: The cultural adaptation process followed the methodology of a theorical reference, in five steps: translation to Brazilian Portuguese, consensual version of translations, back-translation to the original language, revision by a committee of specialists in the field and a equivalency pre-test. The instrument was assessed and applied by 27 health professionals in the last step. Results: The Escala de Avaliação de Dor em Demência Avançada was culturally adapted to Brazil and presented semantic equivalency to the original, besides clarity, applicability and easy comprehension of the instrument items. Conclusion: This process secured the psychometric properties as the reliability and content validity of the referred scale.


Development of the short version of the informal caregiver burden assessment questionnaire

OBJETIVE to create a reduced version of the QASCI, which is structurally equivalent to the long one and meets the criteria of reliability and validity. METHOD Through secondary data from previous studies, the participants were divided into two samples, one for the development of reduced version and the second for study of the factorial validity. Participants responded to QASCI, the SF 36, the ADHS and demographic questions. RESULTS A reduced version of 14 items showed adequate psychometric properties of validity and internal consistency, adapted to a heptadimensional structure that assesses positive and negative aspects of care. CONCLUSION Confirmatory factor analysis revealed a good fit with the advocated theoretical model.

Cross-cultural adaptation of the Caregiver Reaction Assessment for use in Brazil with informal caregivers of the elderly

This study aimed to carry out the cross-cultural adaptation of the Caregiver Reaction Assessment CRA for use in Brazil with informal caregivers of dependent elderly METHOD A methodological study, of five steps: initial translation, synthesis of translations, retro-translation, evaluation by a judge committee and a pre-test, with 30 informal caregivers of older persons in Fortaleza, Brazil. Content validity was assessed by five experts in gerontology and geriatrics. The cross-cultural adaptation was rigorously conducted, allowing for inferring credibility. RESULTS The Brazilian version of the CRA had a simple and fast application (ten minutes), easily understood by the target audience. It is semantically, idiomatically, experimentally and conceptually equivalent to the original version, with valid content to assess the burden of informal caregivers for the elderly (Content Validity Index = 0.883). CONCLUSION It is necessary that other psychometric properties of validity and reliability are tested before using in care practice and research.

Reliability and validity of the interactive observation scale for psychiatric inpatients applied by nursing aides in daily ward practice

The main characteristic of the nursing Interactive Observation Scale for Psychiatric Inpatients (IOSPI) is the necessity of interaction between raters and patients during assessment. The aim of this study was to evaluate the reliability and validity of the scale in the "real" world of daily ward practice and to determine whether the IOSPI can increase the interaction time between raters and patients and influence the raters' opinion about mental illness. All inpatients of a general university hospital psychiatric ward were assessed daily over a period of two months by 9 nursing aides during the morning and afternoon shifts, with 273 pairs of daily observations. Once a week the patients were interviewed by a psychiatrist who filled in the Brief Psychiatric Rating Scale (BPRS). The IOSPI total score was found to show significant test-retest reliability (interclass correlation coefficient = 0.83) and significant correlation with the BPRS total score (r = 0.69), meeting the criteria of concurrent validity. The instrument can also discriminate between patients in need of further inpatient treatment from those about to be discharged (negative predictive value for discharge = 0.91). Using this scale, the interaction time between nursing aides and patients increased significantly (t = 2.93, P<0.05) and their opinion about the mental illness changed. The "social restrictiveness" factor of the opinion scale about mental illness showed a significant reduction (t = 4.27, P<0.01) and the "interpersonal etiology" factor tended to increase (t = 1.98, P = 0.08). The IOSPI was confirmed as a reliable and valid scale and as an efficient tool to stimulate the therapeutic attitudes of the nursing staff.

Reliability and validity of a Portuguese version of the Young Mania Rating Scale

The reliability and validity of a Portuguese version of the Young Mania Rating Scale were evaluated. The original scale was translated into and adapted to Portuguese by the authors. Definitions of clinical manifestations, a semi-structured anchored interview and more explicit rating criteria were added to the scale. Fifty-five adult subjects, aged 18 to 60 years, with a diagnosis of Current Manic Episode according to DSM-III-R criteria were assessed using the Young Mania Rating Scale as well as the Brief Psychiatric Rating Scale in two sessions held at intervals from 7 to 10 days. Good reliability ratings were obtained, with intra-class correlation coefficient of 0.97 for total scores, and levels of agreement above 0.80 (P < 0.001) for all individual items. Internal consistency analysis resulted in an alpha = 0.67 for the scale as a whole, and an alpha = 0.72 for each standardized item (P < 0.001). For the concurrent validity, a correlation of 0.78 was obtained by the Pearson coefficient between the total scores of the Young Mania Rating Scale and Brief Psychiatric Rating Scale. The results are similar to those reported for the English version, indicating that the Portuguese version of the scale constitutes a reliable and valid instrument for the assessment of manic patients.

Development and applications of the SWAN rating scale for assessment of attention deficit hyperactivity disorder: a literature review

This study reviewed the use of the Strengths and Weaknesses of Attention-Deficit/Hyperactivity-symptoms and Normal-behaviors (SWAN) rating scale in diagnostic and evolutive approaches to attention deficit hyperactivity disorder (ADHD) and in correlational studies of the disorder. A review of articles published in indexed journals from electronic databases was conducted and 61 articles on the SWAN scale were analyzed. From these, 27 were selected to a) examine use of SWAN in research on attention disorders and b) verify evidence of its usefulness in the areas of genetics, neuropsychology, diagnostics, psychiatric comorbidities, neuroimaging, pharmacotherapy, and to examine its statistical reliability and validity in studies of diverse populations. This review of articles indicated a growing use of the SWAN scale for diagnostic purposes, for therapy, and in research on areas other than ADHD, especially when compared with other reliable scales. Use of the scale in ADHD diagnosis requires further statistical testing to define its psychometric properties.

Impact of specimen adequacy on the assessment of renal allograft biopsy specimens

The Banff classification was introduced to achieve uniformity in the assessment of renal allograft biopsies. The primary aim of this study was to evaluate the impact of specimen adequacy on the Banff classification. All renal allograft biopsies obtained between July 2010 and June 2012 for suspicion of acute rejection were included. Pre-biopsy clinical data on suspected diagnosis and time from renal transplantation were provided to a nephropathologist who was blinded to the original pathological report. Second pathological readings were compared with the original to assess agreement stratified by specimen adequacy. Cohen's kappa test and Fisher's exact test were used for statistical analyses. Forty-nine specimens were reviewed. Among these specimens, 81.6% were classified as adequate, 6.12% as minimal, and 12.24% as unsatisfactory. The agreement analysis among the first and second readings revealed a kappa value of 0.97. Full agreement between readings was found in 75% of the adequate specimens, 66.7 and 50% for minimal and unsatisfactory specimens, respectively. There was no agreement between readings in 5% of the adequate specimens and 16.7% of the unsatisfactory specimens. For the entire sample full agreement was found in 71.4%, partial agreement in 20.4% and no agreement in 8.2% of the specimens. Statistical analysis using Fisher's exact test yielded a P value above 0.25 showing that - probably due to small sample size - the results were not statistically significant. Specimen adequacy may be a determinant of a diagnostic agreement in renal allograft specimen assessment. While additional studies including larger case numbers are required to further delineate the impact of specimen adequacy on the reliability of histopathological assessments, specimen quality must be considered during clinical decision making while dealing with biopsy reports based on minimal or unsatisfactory specimens.