Self-Assessment in a Competency-Based Curriculum: Preliminary Findings

Introduction: The introduction of the ACGME core competency framework brought challenges of developing appropriate evaluation tools (i.e. self assessment) to provide evidence of competency. Baylor College of Medicine has 43 competency goals organized within the 6 ACGME domains, each domain having 4-10 goals. [See PDF for complete abstract]

A STUDY OF THE RELATIONSHIP BETWEEN SELF-ASSESSMENT AND CLINICAL EVALUATION OF HEARING STATUS

The relationship was explored between a subjective measure of hearing status, derived from a functional self-assessment expressed in terms of ability to hear and understand spoken words, and a comparable objective measure of hearing status, obtained from a speech reception test. The Augmentation Survey of the Health and Nutrition Examination Survey of the National Center for Health Statistics provided the necessary data for a sample of 3059 adults. Using chi-square tests for the subsample with the highest level of objectively assessed hearing status, favorable subjective assessments were found to be significantly associated with higher income, lower age group, higher level of educational attainment, greater psychological adjustment, fewer symptoms of depression, and higher self-ratings of overall health. In a linear regression with self-assessment of hearing status as the dependent variable, less than one-quarter of the variation could be explained by objective status and the six explanatory variables.^

Predictive validity of the personal background and preparation survey and effectiveness of targeted interventions among health science community college students

Existing data, collected from 1st-year students enrolled in a major Health Science Community College in the south central United States, for Fall 2010, Spring 2011, Fall 2011 and Spring 2012 semesters as part of the "Online Navigational Assessment Vehicle, Intervention Guidance, and Targeting of Risks (NAVIGATOR) for Undergraduate Minority Student Success" with CPHS approval number HSC-GEN-07-0158, was used for this thesis. The Personal Background and Preparation Survey (PBPS) and a two-question risk self-assessment subscale were administered to students during their 1st-year orientation. The PBPS total risk score, risk self-assessment total and overall scores, and Under Representative Minority Student (URMS) status were recorded. The purpose of this study is to evaluate and report the predictive validity of the indicators identified above for Adverse Academic Status Events (AASE) and Nonadvancement Adverse Academic Status Events (NAASE) as well as the effectiveness of interventions targeted using the PBPS among a diverse population of health science community college students. The predictive validity of the PBPS for AASE has previously been demonstrated among health science professions and graduate students (Johnson, Johnson, Kim, & McKee, 2009a; Johnson, Johnson, McKee, & Kim, 2009b). Data will be analyzed using binary logistic regression and correlation using SPSS 19 statistical package. Independent variables will include baseline- versus intervention-year treatments, PBPS, risk self-assessment, and URMS status. The dependent variables will be binary AASE and NAASE status. ^ The PBPS was the first reliable diagnostic and prescriptive instrument to establish documented predictive validity for student Adverse Academic Status Events (AASE) among students attending health science professional schools. These results extend the documented validity for the PBPS in predicting AASE to a health science community college student population. Results further demonstrated that interventions introduced using the PBPS were followed by approximately one-third reduction in the odds of Nonadvancement Adverse Academic Status Events (NAASE), controlling for URMS status and risk self-assessment scores. These results indicate interventions introduced using the PBPS may have potential to reduce AASE or attrition among URMS and nonURMS attending health science community colleges on a broader scale; positively impacting costs, shortages, and diversity of health science professionals.^

An assessment of self-concept and violence within a delinquent adolescent population

The purpose of this study was to provide further data on the relationship between self-concept and violence focusing on a delinquent adolescent population. Recent research has explored the relationship between self-concept and violence with most of the research being done with adult populations. Within the literature, there are two opposing views on the question of this relationship. The traditional view supports the idea that low self-esteem is a cause of violent behavior while the non-traditional view supports the idea that high self-esteem may be a contributor to violent behavior. ^ Using a sample of 200 delinquent adolescents 100 of whom had committed acts of violence and 100 who had not, a group comparison study was done which addressed the following questions, (1) within a delinquent population of violent and non-violent adolescents, is there a relationship between violence and self-concept? (2) what is that relationship; (3) using the Piers-Harris Children's Self-Concept Scale, can it be determined that attributes such as behavior, anxiety, popularity, happiness, and physical appearance as they relate to self-concept are more predictive than others in determining who within a delinquent population will commit acts of violence. For the purposes of this study, delinquent adolescents were those who had official records of misconduct with either the school or juvenile authorities. Adolescents classified as violent were those who had committed acts such as assault, use of a weapon, use of deadly force, and sexual assault while adolescents classified as non-violent had committed anti-social acts such as, truancy, talking back and rule breaking. ^ The study concluded that there is a relationship between adolescent violence and self-concept. However, there was insufficient statistical evidence that self-concept is a predictor of violence. ^

Association of self-reported physical activity and performance in the physical fitness assessment among Airmen

The U.S. Air Force, as with the other branches of military services, has physical fitness standards imposed on their personnel. These standards ensure a healthy and fit combat force. To meet these standards, Airmen have to maintain a certain level of physical activity in their lifestyle. Objective. This was a cross sectional (prevalence) study to evaluate the association of Airmen's self-reported physical activity and their performance in the Air Force Physical Fitness Assessment in 2007. Methods. The self-reported physical activity data were obtained from the Air Force Web Health Assessment (AF WEB HA), a web-based health questionnaire completed by the Airmen during their annual Preventive Health Assessment. The physical activity levels were categorized as having met or not having met the Centers for Disease Control and Prevention (CDC) and the American College of Sports Medicine (ACSM) physical activity recommendations. Physical Fitness scores were collected from the Air Force Fitness Management System (AFFMS), a repository of physical fitness test data. Results. There were 49,029 Airmen who answered the AF WEB HA in 2007 and also took their physical fitness test. 94.4% (n = 46,304) of Airmen met the recommended physical activity guidelines and 79.9% (n = 39,178) passed the fitness test. Total Airmen who both met the physical activity recommendations and passed the fitness test was 75.6% (n = 37,088). Airmen who did not meet the activity recommendations and also failed the fitness test totaled 635 or 1.3% of the study group. The Mantel-Haenszel Chi-Square analysis of the data on the activity levels and the physical fitness test relationship was the following χ2 = 18.52, df 1, and p = <0.0001. The Odds Ratio (OR) was 1.22 (95% CI 1.12, 1.34). Conclusion. The study determined that there was a positive association between Airmen's self-reported physical activity and their performance in the physical fitness assessment.^

FAMILY MEDICINE RESIDENT SURVEY ON HEALTH PROMOTION (PERSONAL EXPECTATIONS, BEHAVIORS, PERCEIVED SELF-EFFICACY, COGNITIVE ASSESSMENT, OUTCOME EXPECTANCY)

This study was designed to test the theoretical predictors of personal efficacy expectations among family medicine resident physicians for helping their patients change thirteen high risk health behaviors. A survey questionnaire was sent to 781 family medicine residents in the six state south central region. The response rate was 60 percent. The hypothesized relationship between lower levels of difficulty and higher personal efficacy expectations was supported by the data. Effort was a significant predictor of perceived self efficacy for health behaviors considered less difficult to change. Situational support did not prove to be a significant predictor for many of the health behaviors. Rate and pattern of success were consistent and significant predictors of perceived self efficacy for helping patients change all thirteen of the health behaviors. Modeling of effective methods by faculty was a significant predictor of efficacy expectations for several but not all of the behaviors. Personal modeling was a significant predictor of perceived efficacy for helping patients change behaviors related to alcohol misuse and exercise. The respondents personally modeled positive health behaviors more consistently than their older colleagues or the general population.^ The results of this study lend substantially to the usefulness of the cognitive-behavioral theory of perceived self efficacy and provide a mechanism for assessing the predictors of personal efficacy expectations of family medicine resident physicians. The findings are expected to have direct implications for faculty to institute systematic programs of interventions designed to increase residents' perceptions of efficacy in facilitating more positive health behaviors among their patients. ^

Assessment of executive function before and after vitamin D replacement in vulnerable adults who self-neglect

Purpose: Self-neglect (SN) is the inability to maintain self-care needs. It is thought that older adults who have impaired executive function (EF) develop the inability to do self-care and to protect themselves. The specific aims were to (1) determine the feasibility of using multiple EF measures with community-dwelling elders with SN, (2) identify changes in EF between baseline and 5-months in community-dwelling elders with SN who receive 50,000 IU or 400 IU of oral vitamin D monthly and (2) explore changes in specific dimensions of EF between the groups. ^ Methods: Fifty adults, 65 years of age and older, were recruited from Adult Protective Services with confirmed SN. A research nurse administered the following tests at baseline and five-months: Delis-Kaplan Card Sort Test (D-KEFS), Executive Interview (EXIT 25), CLOX Drawing Test (CLOX I, II), Trails Making Test A and B (TMT A & B) and the Mini-Mental State Examination (MMSE). Demographic data was collected at baseline and serum 25-OHD levels were collected at baseline and five-months. ^ Results: Older adults with SN were more likely to fail the CLOX1 and D-KEFS, while passing the MMSE, CLOX II, TMT A & B and the EXIT 25. At five-months, the only statistically significant difference between groups was in the TMT A & B test scores; the control group did better than the treatment group. There was a non-significant increase in serum vitamin D levels for both groups and no difference between groups. ^ Conclusions: Results from this study provide support that individuals who SN will complete a battery of EF tests and that they exhibit the following impairments consistent with executive dysfunction: 'concept generation', 'planning', 'inhibition', and 'spatial working memory'. Utilizing only one EF measure in individuals with intact cognition may result in unidentification of individuals with executive dysfunction, thus delaying necessary treatment. Future studies should attempt to determine different etiologies of executive dysfunction and determine if early treatment can prevent or reverse SN. ^ Key Words: Self-neglect, Executive Dysfunction, Executive Function, Adult Protective Services, Community-dwelling, Vitamin D ^

Multiple retinal detachment surgery: A functional outcome study. The Houston Vision Assessment Test-Retina (HVAT-Retina)

Retinal detachment is a common ophthalmologic procedure, and outcome is typically measured by a single factor-improvement in visual acuity. Health related functional outcome testing, which quantifies patient's self-reported perception of impairment, can be integrated with objective clinical findings. Based on the patient's self-assessed lifestyle impairment, the physician and patient together can make an informed decision on the treatment that is most likely to benefit the patient. ^ A functional outcome test (the Houston Vision Assessment Test-Retina; HVAT-Retina) was developed and validated in patients with multiple retinal detachments in the same eye. The HVAT-Retina divides an estimated total impairment into subcomponents: contribution of visual disability (potentially correctable by retinal detachment surgery) and nonvisual physical disabilities (co-morbidities not affected by retinal detachment surgery. ^ Seventy-six patients participated in this prospective multicenter study. Seven patients were excluded from the analysis because they were not certain of their answers. Cronbach's alpha coefficient was 0.91 for presurgery HVAT-Retina and 0.94 post-surgery. The item-to-total correlation ranged from 0.50 to 0.88. Visual impairment score improved by 9 points from pre-surgery (p = 0.0003). Physical impairment score also improved from pre-surgery (p = 0.0002). ^ In conclusion, the results of this study demonstrate that the instrument is reliable and valid in patients presenting with recurrent retinal detachments. The HVAT-Retina is a simple instrument and does not burden the patient or the health professional in terms of time or cost. It may be self-administrated, not requiring an interviewer. Because the HVAT-Retina was designed to demonstrate outcomes perceivable by the patient, it has the potential to guide the decision making process between patient and physician. ^

Emergency health care assessment in the Houston-Galveston region 2002--2006

Background. Previous studies show emergency rooms to be over crowded nation wide. With growing attention to this problem, the Houston-Galveston Area Council (H-GAC) initiated a study in 2005 to assess their region's emergency health care system, and continued this effort in 2007. The purpose of this study was to examine recent changes in volume, capacity and performance in the Houston-Galveston region's emergency health care system and determine whether the system has been able to effectively respond to the residents' demands. Methods. Data were collected by the Houston-Galveston Area Council and The Abaris Group using a self-administered 2002-2006 survey completed by administrators of the region's hospitals, EMS providers, and select fire departments that provide EMS services. Data from both studies were combined and matched to examine trends. Results. Volume increased among the reporting hospitals within the Houston-Galveston region from 2002 to 2006; however, capacity remained relatively unchanged. EMS providers reported higher average off load times in 2007 compared to 2005, but the increases were not statistically significant. Hospitals reported transferring a statistically significant greater percentage of patients in 2006 than 2004. There was no statistically significant change in any of the other measures. Conclusion. These findings indicate an increase in demand for the Houston-Galveston region's emergency healthcare services with no change in supply. Additional studies within the area are needed to fully assess and evaluate the impact of these changes on system performance. ^

Comparing experience of diabetes care with chronic illness care in the primary care clinic using the Patient Assessment of Chronic Illness Care (PACIC)

Purpose. The overall purpose of the study was to evaluate the patient experience relevant to the Chronic Care Model as measured by the PACIC. Chronic illness care of patients with diabetes was compared to those with other chronic illnesses. In addition, chronic illness care of Hispanics was compared to those of other race/ethnicity. ^ Methods. The setting of this study was 20 primary care practices located in San Antonio, TX. The subjects in this study were consecutive adult patients age >18 yrs. Data was collected via a survey (PACIC) administered to 40-60 consecutive adult patients in each primary care clinic who presented for a scheduled appointment. ^ Results. Patient experience of the Chronic Care Model is different among those with diabetes than those with other chronic diseases: those with diabetes report a higher PACIC score. (P = 0.012) Although Hispanic patients report a higher PACIC score, patient experience of the Chronic Care Model among Hispanic patients is not significantly different than that of patients of other race/ethnicity regardless of chronic disease. (P = 0.053) After controlling for the patient characteristics of age, education, health status, and race/ethnicity, the diabetes status of the patient remains significantly associated with the outcome, the PACIC score. (P = 0.033) ^ Conclusions. Diabetes is associated with a greater experience of the Chronic Care model. Contributing factors to diabetes patients’ greater experience of the Chronic Care Model include the greater heath care use and higher self-care needs unique to individuals with diabetes. Special consideration must be given to the specific needs diabetic patients to ensure effective interventions, higher patient education, greater patient compliance, and lower health care costs. ^

An examination of spirituality, self efficacy, and smoking cessation among adults in Houston, Texas

Over the past decade there has been a growing interest in the association between spirituality and health outcomes. Little is known about the role of spirituality among adult smokers who are motivated to stop smoking. The purpose of this dissertation is to examine the relations among immutable individual differences, spirituality, and self efficacy among adults motivated to stop smoking. The first paper of this dissertation systematically reviewed literature to measure the concordance between spirituality and smoking status among adults in the United States. The second paper of this dissertation explored the association between spirituality and smoking cessation. We hypothesized that higher levels of spirituality were positively associated with smoking cessation. The third paper of this dissertation examined the association between perceived self efficacy and spirituality. We hypothesized that both high levels of self efficacy and spirituality were positively associated with smoking cessation.^ A total of 152 citations were identified based on the preliminary search of databases and reference lists. After a preliminary title- and abstract-based review, 17 full text articles were retrieved for further assessment. Of these, eight met the criteria for inclusion. Results of the systematic review suggest that there is inconsistent evidence to support or refute an association between spirituality and smoking status among adults.^ Smokers (N = 200) at least 18 years of age enrolled in a minimal contact smoking cessation intervention in Houston, Texas completed questionnaires. To examine our hypotheses we conducted cross-sectional analyses of responses to questions included in selected baseline questions and the final in-person visit three weeks post-quit day. Results of the logistic regression analyses indicated that individuals with higher levels of spirituality and self efficacy were significantly more likely to abstain from smoking. The positive association is also evident when controlling for employment, income, race, education, and nicotine dependence. The interaction between self efficacy and spirituality was not statistically significant in predicting smoking abstinence.^ Recommendations for future research and implications for smoking cessation interventions are discussed. Further research in this area would benefit from using standard measures of abstinence, recruiting larger and more diverse populations, and using longitudinal study designs.^

Air Force Web Preventive Health Assessment (AF WEBPHA) mental health screening effectiveness

The U.S. Air Force assesses Active Duty Air Force (ADAF) health annually using the Air Force Web-based Preventative Health Assessment (AF WebPHA). The assessment is based on a self-administered survey used to determine the overall Air Force health and readiness, as well as, the individual health of each airman. Individual survey responses as well as groups of responses generate further computer generated assessment and result in a classification of 'Critical', 'Priority', or 'Routine', depending on the need and urgency for further evaluation by a health care provider. The importance of the 'Priority' and 'Critical' classifications is to provide timely intervention to prevent or limit unfavorable outcomes that may threaten an airman. Though the USAF has been transitioning from a paper form to the online WebPHA survey for the last three years it was not made mandatory for all airmen until 2009. The survey covers many health aspects including family history, tobacco use, exercise, alcohol use, and mental health. ^ Military stressors such as deployment, change of station, and the trauma of war can aggravate and intensify the common baseline worries experienced by the general population and place airmen at additional risks for mental health concerns and illness. This study assesses the effectiveness of the AF WebPHA mental health screening questions in predicting a mental health disorder diagnosis according to International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) codes generated by physicians or their surrogates. In order to assess the sensitivity, specificity, and positive predictive value of the AF WebPHA as a screening tool for mental health, survey results were compared to ascertain if they generated any mental health disorder related diagnosis for the period from January 1, 2009 to March 31, 2010. ^ Statistical analysis of the AF WebPHA mental health responses when compared with matching ICD-9-CM codes found that the sensitivity for 'Critical' or 'Priority' responses was only 3.4% and that it would correctly predict those who had the selected mental health diagnosis 9% of the time.^

A methodologic study of pain assessment in Anglo and Hispanic children with cancer

There is growing clinical evidence that even young children experience pain and accompanying anxiety. Few instruments have been validated to assess pain characteristics in children. The study of related demographic, illness, psychologic and parental factors in children has also been limited. This study examines the reliability and validity of pain assessment tools in an outpatient pediatric cancer population. A total of 78 children from three to fifteen years of age were observed and interviewed about the pain of invasive procedures. The effect of cultural factors and the stress of acculturation were examined by comparing data from two cultural groups, Anglo and Hispanic.^ Spielberger State-Trait Anxiety Scales were administered to children and parents prior to an invasive procedure. The Procedure Behavioral Checklist (PBCL) was used for observation of the child's response during the procedure. The Children's Procedural Interview (CPI) which contains items on the PBCL and visual analogues (scales of faces indicating varying degrees of pain and anxiety) was administered following the procedure.^ Reliability coefficients for Anglos were.78 on the PBCL,.79 on the CPI and.85 on the visual analogue scales. For Hispanics, the reliability for the PBCL was.54, while the CPI had a reliability of.72 and the visual analogue scales,.87. Construct validity was demonstrated by high correlations between the PBCL and CPI scores for both ethnic groups (.66 for Anglos and.64 for Hispanics) and by the significant correlation of State anxiety scores with both PBCL and CPI scores. Age was inversely correlated with PBCL and CPI scores for both ethnic groups. Hispanic parents' anxiety scores were higher than Anglo parents, but were not highly correlated with their child's PBCL, CPI or State-Trait anxiety scores. Caregivers' ratings were correlated with the PBCL scores for Anglos but not for Hispanics.^ The findings of this study indicate that pain responses may be reliably assessed using both observational and self-report methods in children, though differences in Anglo and Hispanic cultures exist. Differences in pain symptomatology and assessment in the two cultural groups warrant further study. ^

The feasibility of self-report advance directives among elderly patients with cognitive impairment

The central objective of this dissertation was to determine the feasibility of self-completed advance directives (AD) in older persons suffering from mild and moderate stages of dementia. This was accomplished by identifying differences in ability to complete AD among elderly subjects with increasing degrees of dementia and cognitive incompetence. Secondary objectives were to describe and compare advance directives completed by elders and identified proxy decision makers. Secondary objectives were accomplished by measuring the agreement between advance directives completed by proxy and elder, and comparing that agreement across groups defined by the elder's cognitive status. This cross-sectional study employed a structured interview to elicit AD, followed by a similar interview with a proxy decision maker identified by the elder. A stratified sampling scheme recruited elders with normal cognition, mild, and moderate forms of dementia using the Mini Mental-State Exam (MMSE). The Hopkins Competency Assessment Test (HCAT) was used for evaluation of competency to make medical decisions. Analysis was conducted on "between group" (non-demented $\leftrightarrow$ mild dementia $\leftrightarrow$ moderate dementia, and competent $\leftrightarrow$ incompetent) and "within group" (elder $\leftrightarrow$ family member) variation.^ The 118 elderly subjects interviewed were generally male, Caucasian, and of low socioeconomic status. Mean age was 77. Overall, elders preferred a "trial of therapy" regarding AD rather than to "always receive the therapy". No intervention was refused outright more often than it was accepted. A test-retest of elders' AD revealed stable responses. Eleven logic checks measured appropriateness of AD responses independent of preference. No difference was found in logic error rates between elders grouped by MMSE or HCAT. Agreement between proxy and elder responses showed significant dissimilarity, indicating that proxies were not making the same medical decisions as the elders.^ Conclusions based on these data are: (1) Self reporting AD is feasible among elders showing signs of cognitive impairment and they should be given all opportunities to complete advance directives, (2) variation in preferences for advance directives in cognitively impaired elders should not be assumed to be the effects of their impairment alone, (3) proxies do not appear to forego life-prolonging interventions in the face of increasing impairment in their ward, however, their advance directives choices are frequently not those of the elder they represent. ^

COSTS AND EFFECTS OF ARKANSAS' LONG-TERM CARE DEMONSTRATION PROJECT (MEDICAID, CHANELLING, COMMUNITY ASSESSMENT)

The purpose of this study was to assess the impact of the Arkansas Long-Term Care Demonstration Project upon Arkansas' Medicaid expenditures and upon the clients it serves. A Retrospective Medicaid expenditure study component used analyses of variance techniques to test for the Project's effects upon aggregated expenditures for 28 demonstration and control counties representing 25 percent of the State's population over four years, 1979-1982.^ A second approach to the study question utilized a 1982 prospective sample of 458 demonstration and control clients from the same 28 counties. The disability level or need for care of each patient was established a priori. The extent to which an individual's variation in Medicaid utilization and costs was explained by patient need, presence or absence of the channeling project's placement decision or some other patient characteristic was examined by multiple regression analysis. Long-term and acute care Medicaid, Medicare, third party, self-pay and the grand total of all Medicaid claims were analyzed for project effects and explanatory relationships.^ The main project effect was to increase personal care costs without reducing nursing home or acute care costs (Prospective Study). Expansion of clients appeared to occur in personal care (Prospective Study) and minimum care nursing home (Retrospective Study) for the project areas. Cost-shifting between Medicaid and Medicare in the project areas and two different patterns of utilization in the North and South projects tended to offset each other such that no differences in total costs between the project areas and demonstration areas occurred. The project was significant ((beta) = .22, p < .001) only for personal care costs. The explanatory power of this personal care regression model (R('2) = .36) was comparable to other reported health services utilization models. Other variables (Medicare buy-in, level of disability, Social Security Supplemental Income (SSI), net monthly income, North/South areas and age) explained more variation in the other twelve cost regression models. ^