A protocol for advanced psychometric assessment of surveys

Background and Purpose. In this paper, we present a protocol for advanced psychometric assessments of surveys based on the Standards for Educational and Psychological Testing. We use the Alberta Context Tool (ACT) as an exemplar survey to which this protocol can be applied.

Methods. Data mapping, acceptability, reliability, and validity are addressed. Acceptability is assessed with missing data frequencies and the time required to complete the survey. Reliability is assessed with internal consistency coefficients and information functions. A unitary approach to validity consisting of accumulating evidence based on instrument content, response processes, internal structure, and relations to other variables is taken. We also address assessing performance of survey data when aggregated to higher levels (e.g., nursing unit).

Discussion. In this paper we present a protocol for advanced psychometric assessment of survey data using the Alberta Context Tool (ACT) as an exemplar survey; application of the protocol to the ACT survey is underway. Psychometric assessment of any survey is essential to obtaining reliable and valid research findings. This protocol can be adapted for use with any nursing survey.

Critical appraisal of health literacy indices revealed variable underlying constructs, narrow content and psychometric weaknesses

Objective
Health literacy refers to an individual's ability to seek, understand, and use health information. A range of indices exist that purport to measure health literacy across individuals and populations. This study aimed to review the development and content of existing indices and to critically appraise their properties.

Study Design and Setting
Using standardized search terms, published generic health literacy indices (1990–2008) were identified. Using a methodological framework, each was evaluated for purpose, validity (face, content, construct), reliability, responsiveness, feasibility, and generalizability.

Results
Nineteen instruments were evaluated. Three measurement approaches were identified: direct testing of individual abilities, self-report of abilities, and population-based proxy measures. Composition of underlying constructs and content varied widely across instruments, and none appeared to fully measure a person's ability to seek, understand, and use health information. The content was focused primarily on reading comprehension and numeracy; scoring categories were poorly defined and may not be mutually exclusive, and few indices had been assessed for reliability.

Conclusion
Health literacy is not consistently measured, making it difficult to interpret and compare health literacy at individual and population levels. Empirical evidence demonstrating validity and reliability of existing indices is required, and more comprehensive health literacy instruments need to be developed.

Measuring nursing competencies in the operating theatre: instrument development and psychometric analysis using Item Response Theory

BACKGROUND: Concern about the process of identifying underlying competencies that contribute to effective nursing performance has been debated with a lack of consensus surrounding an approved measurement instrument for assessing clinical performance. Although a number of methodologies are noted in the development of competency-based assessment measures, these studies are not without criticism. RESEARCH AIM: The primary aim of the study was to develop and validate a Performance Based Scoring Rubric, which included both analytical and holistic scales. The aim included examining the validity and reliability of the rubric, which was designed to measure clinical competencies in the operating theatre. RESEARCH METHOD: The fieldwork observations of 32 nurse educators and preceptors assessing the performance of 95 instrument nurses in the operating theatre were used in the calibration of the rubric. The Rasch model, a particular model among Item Response Models, was used in the calibration of each item in the rubric in an attempt at improving the measurement properties of the scale. This is done by establishing the 'fit' of the data to the conditions demanded by the Rasch model. RESULTS: Acceptable reliability estimates, specifically a high Cronbach's alpha reliability coefficient (0.940), as well as empirical support for construct and criterion validity for the rubric were achieved. Calibration of the Performance Based Scoring Rubric using Rasch model revealed that the fit statistics for most items were acceptable. CONCLUSION: The use of the Rasch model offers a number of features in developing and refining healthcare competency-based assessments, improving confidence in measuring clinical performance. The Rasch model was shown to be useful in developing and validating a competency-based assessment for measuring the competence of the instrument nurse in the operating theatre with implications for use in other areas of nursing practice.

Psychometric evaluation of the body change inventory: an assessment instrument for adolescent boys and girls

A new instrument, the Body Change Inventory, was developed to provide an assessment of body change strategies that are used by both adolescent girls and boys. The novel aspect of this instrument is that it evaluates strategies to increase body size and increase muscle size, as well strategies to decrease body size. Independent samples of adolescent girls and boys aged between 11 and 17 years (N=1732) participated in four studies. The revised instrument consisted of three body change scales—Strategies to Decrease Body Size, Strategies to Increase Body Size, and Strategies to Increase Muscle Size. The studies demonstrated content validity, construct validity, internal consistency, and concurrent and discriminant validity for the new scales. The new scales provide a valuable addition in the literature for assessing three global body change strategies among adolescent girls and boys. They are needed in order to examine further the normative development of different kinds of body change strategies and how these may lead to behavioural problems such as disordered eating, exercise dependence, and steroid use.

Quality of life instruments for caregivers of patients with cancer : A review of their psychometric properties

The quality of life of caregivers of patients with cancer is an important construct given the substantial impact of caring on the physical, psychological, social, and financial well-being of caregivers. Moreover, caring for patients with cancer also affects family functioning and places burdens on caregivers. The reliability and validity of instruments used to assess the quality of life of caregivers of patients with cancer were reviewed to aid in the selection of the most appropriate measures For research and practice. MEDLINE (1980-2000) and c/NAHL (1982-2000) searches located relevant quality of life instruments using the keywords "cancer and quality of Iife" and "caregiver or spouse or partner." The search identified the following instruments: the Caregiver Quality of life Index-Cancer Scale, the 'Caregiver Quality of Life Index, the Quality of Life Tool, and the Quality of life Index-Cancer Version. Quality of life instruments developed specifically to measure the quality of life of caregivers of patients with cancer had the best psychometric properties. The Caregiver Quality of Life Index-Cancer Scale, in particular, met or exceeded minimum psychometric criteria for reliability and validity. The development of reliable and valid caregiver quality of life measures is an important factor in developing interventions to enhance quality of life of caregivers of patients with cancer.

Psychometric data on the mathematics self-efficacy scale

Descriptive measurement reliability and validity data are reported on scores from the Mathematics Self-Efficacy Scale (MSES) in two contexts—class and test—for a sample of 302 Australian high school students. Summated scores on the MSES correlated r = .74, and together these items yielded one component that explained 49% of the variance. MSES scores demonstrated internal reliability for both class and test (Cronbach alphas = .86 and .90). Statistically significant correlations between MSES scores, past mathematics grades, and Marsh’s Self-Description Questionnaire III (Math) were used to indicate the convergent validity of scores for the MSES. Discriminant validity of the MSES scores was suggested by the lack of significant correlations between the MSES measures and students’ desired English grades. Statistically significant correlations between MSES scores and students’ desired mathematics grades indicated concurrent validity of the MSES measures. A statistically significant difference between class and test scores supported Bandura’s prescription for context specificity in efficacy research.

Psychometric properties of the quality of life questionnaire for children with CP

This paper describes the development and psychometric properties of a condition-specific quality of life instrument for children with cerebral palsy (CP QOL-Child). A sample of 205 primary caregivers of children with CP aged 4 to 12 years (mean 8y 5mo) and 53 children aged 9 to 12 years completed the CP QOL-Child. The children (112 males, 93 females) were sampled across Gross Motor Function Classification System (GMFCS) levels (Level I=18%, II=28%, III=14%, IV=11%, V=27%). Primary caregivers also completed other measures of child health (Child Health Questionnaire; CHQ), QOL (KIDSCREEN), and functioning (GMFCS). Internal consistency ranged from 0.74 to 0.92 for primary caregivers and from 0.80 to 0.90 for child self-report. For primary caregivers, 2-week test-retest reliability ranged from 0.76 to 0.89. The validity of the CP QOL is supported by the pattern of correlations between CP QOL-Child scales with the CHQ, KIDSCREEN, and GMFCS. Preliminary statistics suggest that the child self-report questionnaire has acceptable psychometric properties. The questionnaire can be freely accessed at http://www.deakin.edu.ac/hmnbs/chase/cerebralpalsy/cp_qol_home.php.

Addressing the potential adverse effects of school-based BMI assessments on children

Introduction. Do child obesity prevention research and intervention measures have the potential to generate adverse concerns about body image by focussing on food, physical activity and body weight? Research findings now demonstrate the emergence of body image concerns in children as young as 5 years. In the context of a large school-community-based child health promotion and obesity prevention study, we aimed to address the potential negative effects of height and weight measures on child wellbeing by developing and implementing an evidence-informed protocol to protect and prevent body image concerns. fun 'n healthy in Moreland! is a cluster randomised controlled trial of a child health promotion and obesity prevention intervention in 23 primary schools in an inner urban area of Melbourne, Australia. Body image considerations were incorporated into the study philosophies, aims, methods, staff training, language, data collection and reporting procedures of this study. This was informed by the published literature, professional body image expertise, pilot testing and implementation in the conduct of baseline data collection and the intervention. This study is the first record of a body image protection protocol being an integral part of the research processes of a child obesity prevention study. Whilst we are yet to measure its impact and outcome, we have developed and tested a protocol based on the evidence and with support from stakeholders in order to minimise the adverse impact of study processes on child body image concerns.

Psychometric properties of the modified Emory functional ambulation profile in stroke patients

Objective: To examine the test – retest reliability, convergent and predictive validity and responsiveness of the modified Emory Functional Ambulation Profile (mEFAP) in assessing gait function in stroke patients undergoing rehabilitation. Subjects: Forty subacute stroke inpatients were enrolled in the validity and responsiveness study. Twenty chronic patients participated in the reliability study. Setting: Rehabilitation department of a medical centre. Design: In the validity and responsiveness study, patients were tested using the mEFAP, 10-m walking speed test, Barthel Index and Rivermead Mobility Index at admission and at discharge. For the reliability study, the patients completed the mEFAP twice one week apart. Results: The patients' performances on the mEFAP were moderately to highly correlated with results of the 10-m walking speed test and Rivermead Mobility Index (absolute correlation coefficients ≥ 0.67), indicating good convergent validity. Patients' performance on the mEFAP at admission was moderately correlated with the Barthel Index and Rivermead Mobility Index scores at discharge (Spearman's rank correlation coefficients = -0.52 and -0.78, respectively), indicating good predictive validity. The standardized response mean of the mEFAP was 1.1 (P < 0.0001), suggesting good responsiveness. The intraclass correlation coefficient ICC(3,1) for the mEFAP was 0.997, indicating excellent test – retest reliability. Conclusions: Our results provide strong evidence that the mEFAP has good reliability, validity and responsiveness for assessing stroke patients undergoing rehabilitation. The mEFAP is a useful scale for measuring walking function and recovery in stroke patients.

Fire fighters' assessments of fireground risk

Fire fighters are often required to work in dynamic and hazardous environments involving a high level of uncertainty. The present study investigated 110 volunteer fire fighters’ assessments of levels of risk associated with a photographic depiction of a typical grassland fire situation. The fire fighters used a standard fire agency risk-rating matrix procedure requiring them to specify the severity of the hazards depicted and the probability of a mishap in order to rate overall level of risk (1 = Low; 4 = Extreme). The risk ratings made by the fire fighters varied greatly. The overall rate of agreement with the risk level rating of the situation made by a panel of expert fire officers (=1, Low) was only 27%. It seems that use of a standard risk-rating matrix procedure by fire fighters at incidents, as recommended currently by many fire agencies, is likely to result in unreliable risk assessments, at least in the absence of effective training in the risk assessment procedure. The 110 volunteers were also asked to identify the total number of potential hazards apparent in five photographs depicting different kinds of emergency incidents. Identifying more hazards was found to be associated with (a) previous personal experience of a ‘near-miss’; and (b) higher levels of education. The findings imply that when faced with identical fire ground situations, individual fire fighters are likely to differ in their situational awareness of hazards and consequent risk assessments.

The impact of consumer ethnocentrism and country of origin sub-components for high involvement products on young Chinese consumers' product assessments

Purpose – The purpose of this paper is to examine the effect of country of origin (COO) sub-components (i.e. design, assembly and parts), as well as the extent to which consumer ethnocentrism tendencies interact with these COO sub-components for young Chinese consumers with regards to product quality assessments and purchase intentions.

Design/methodology/approach – A 2?×?2?×?2 factorial experimental design was used to examine the effects of the three sub-components of COO with two levels of sourcing location – Home (China) and Foreign (Germany), for two high involvement products (an automobile and a digital camera). Chinese students in China represented the sample of 272 respondents. MANOVA was used to examine the direct effects and interactions of the three COO components, as well as ethnocentrism, measured using the CETSCALE.

Findings – It was found that the three COO sub-components did not influence young Chinese consumers’ evaluation of product quality or purchase intentions. In addition, consumers’ level of ethnocentrism also did not have a direct effect on perceived product quality or purchase intentions. There was only one statistically significant interaction effect between ethnocentrism and country of parts for one of the two products. As such, COO dimensions and young Chinese consumers’ ethnocentrism appears to have limited influence on their assessments of product quality or purchase intentions. This may occur because young Chinese consumers perceive that hybrid products are the norm for high involvement products in China as these products are all these consumers have experienced.

Originality/value – The findings of this research dispute the commonly held belief and evidence that sub-components of COO have an impact on the perceptions of product quality and purchase intentions. Young Chinese consumers may be different to consumers from western countries because they have been extensively exposed to hybrid products. Given the size and growth potential of China, young Chinese are an important, under-researched segment within the Chinese market.