A protocol for advanced psychometric assessment of surveys

Background and Purpose. In this paper, we present a protocol for advanced psychometric assessments of surveys based on the Standards for Educational and Psychological Testing. We use the Alberta Context Tool (ACT) as an exemplar survey to which this protocol can be applied.

Methods. Data mapping, acceptability, reliability, and validity are addressed. Acceptability is assessed with missing data frequencies and the time required to complete the survey. Reliability is assessed with internal consistency coefficients and information functions. A unitary approach to validity consisting of accumulating evidence based on instrument content, response processes, internal structure, and relations to other variables is taken. We also address assessing performance of survey data when aggregated to higher levels (e.g., nursing unit).

Discussion. In this paper we present a protocol for advanced psychometric assessment of survey data using the Alberta Context Tool (ACT) as an exemplar survey; application of the protocol to the ACT survey is underway. Psychometric assessment of any survey is essential to obtaining reliable and valid research findings. This protocol can be adapted for use with any nursing survey.

“Just try harder and you will shine” : a Study of 20 lazy children

Attributions of laziness, reflected in teacher comments such as “just try harder and you will shine” may mask specific cognitive, learning, attentional or emotional problems that could explain low motivation in some children. This paper reports findings from an investigation of 20 children, aged 7 to 10 years, who were regarded as lazy by their parents and teachers. Questionnaire measures provided evidence of low levels of motivation and classroom engagement. Psychometric assessments revealed the presence of a range of difficulties including phonologically-based learning disabilities and significant problems with attention in 17 of the 20 children. The paper concludes that the special needs of an unknown number of children may be overlooked because they are simply presumed to be lazy.

Variable phenotype in 16p duplication within a family

Background More individuals are now being identified with very rare genetic syndromes. We present a family with an inherited duplication of 16p11.2 to 16q12.1 in ring formation. Three of the four children, (aged 15 months to 10 years), mother, uncle, and grandmother are affected. Our aim was to provide preliminary evidence of possible phenotypic patterns of learning and behaviour associated with this chromosome anomaly. Method Psychometric assessments were undertaken with all four children. The mother and uncle also agreed to participate in the study. Measures of development (Bayley or Mullen), intellectual ability (WISC-IV or WAIS-III), academic achievement (WIAT-II), adaptive behaviour (Vinelands), and other relevant aspects of functioning (e.g., Children’s Memory Scale) were administered. Results. The first-born child is the only one who is unaffected. Her intellectual ability was assessed as being within the superior range. The second child experienced early difficulties with speech and motor skills. Although his intelligence is average, he has learning difficulties and significant auditory memory problems. The third child’s speech and motor milestones were markedly delayed. He has a complex medical history that includes a vitamin B12 deficiency. On the Mullen Scales at age 4 his scores ranged from average to very low. The development of the youngest child (aged 15 months), who also had a B12 deficiency but was treated early, was assessed as being within typical limits. Conclusions There is considerable developmental variability among the three children with this inherited 16p duplication. We discuss the intriguing similarities and differences, considering common features that may reflect phenotypic patterns and speculating about possible explanations for the variable presentations.

EEG marker of inhibitory brain activity correlates with resting-state cerebral blood flow in the reward system in major depression

Frontal alpha band asymmetry (FAA) is a marker of altered reward processing in major depressive disorder (MDD), associated with reduced approach behavior and withdrawal. However, its association with brain metabolism remains unclear. The aim of this study is to investigate FAA and its correlation with resting – state cerebral blood flow (rCBF). We hypothesized an association of FAA with regional rCBF in brain regions relevant for reward processing and motivated behavior, such as the striatum. We enrolled 20 patients and 19 healthy subjects. FAA scores and rCBF were quantified with the use of EEG and arterial spin labeling. Correlations of the two were evaluated, as well as the association with FAA and psychometric assessments of motivated behavior and anhedonia. Patients showed a left – lateralized pattern of frontal alpha activity and a correlation of FAA lateralization with subscores of Hamilton Depression Rating Scale linked to motivated behavior. An association of rCBF and FAA scores was found in clusters in the dorsolateral prefrontal cortex bilaterally (patients) and in the left medial frontal gyrus, in the right caudate head and in the right inferior parietal lobule (whole group). No correlations were found in healthy controls. Higher inhibitory right – lateralized alpha power was associated with lower rCBF values in prefrontal and striatal regions, predominantly in the right hemisphere, which are involved in the processing of motivated behavior and reward. Inhibitory brain activity in the reward system may contribute to some of the motivational problems observed in MDD.

Psychometric properties of the parent-infant caregiving touch scale

Recent work in animals suggests that the extent of early tactile stimulation by parents of offspring is an important element in early caregiving. We evaluate the psychometric properties of a new parent-report measure designed to assess frequency of tactile stimulation across multiple caregiving domains in infancy. We describe the full item set of the Parent-Infant Caregiving Touch Scale (PICTS) and, using data from a UK longitudinal Child Health and Development Study, the response frequencies and factor structure and whether it was invariant over two time points in early development (5 and 9 weeks). When their infant was 9 weeks old, 838 mothers responded on the PICTS while a stratified subsample of 268 mothers completed PICTS at an earlier 5 week old assessment (229 responded on both occasions). Three PICTS factors were identified reflecting stroking, holding and affective communication. These were moderately to strongly correlated at each of the two time points of interest and were unrelated to, and therefore distinct from, a traditional measure of maternal sensitivity at 7-months. A wholly stable psychometry over 5 and 9-week assessments was not identified which suggests that behavior profiles differ slightly for younger and older infants. Tests of measurement invariance demonstrated that all three factors are characterized by full configural and metric invariance, as well as a moderate degree of evidence of scalar invariance for the stroking factor. We propose the PICTS as a valuable new measure of important aspects of caregiving in infancy.

Critical appraisal of health literacy indices revealed variable underlying constructs, narrow content and psychometric weaknesses

Objective
Health literacy refers to an individual's ability to seek, understand, and use health information. A range of indices exist that purport to measure health literacy across individuals and populations. This study aimed to review the development and content of existing indices and to critically appraise their properties.

Study Design and Setting
Using standardized search terms, published generic health literacy indices (1990–2008) were identified. Using a methodological framework, each was evaluated for purpose, validity (face, content, construct), reliability, responsiveness, feasibility, and generalizability.

Results
Nineteen instruments were evaluated. Three measurement approaches were identified: direct testing of individual abilities, self-report of abilities, and population-based proxy measures. Composition of underlying constructs and content varied widely across instruments, and none appeared to fully measure a person's ability to seek, understand, and use health information. The content was focused primarily on reading comprehension and numeracy; scoring categories were poorly defined and may not be mutually exclusive, and few indices had been assessed for reliability.

Conclusion
Health literacy is not consistently measured, making it difficult to interpret and compare health literacy at individual and population levels. Empirical evidence demonstrating validity and reliability of existing indices is required, and more comprehensive health literacy instruments need to be developed.

The dementia cognitive fluctuation scale, a new psychometric test for clinicians to identify cognitive fluctuations in people with dementia.

OBJECTIVES Cognitive fluctuation (CF) is a common feature of dementia and a core diagnostic symptom for dementia with Lewy bodies (DLB). CF remains difficult to accurately and reliably detect clinically. This study aimed to develop a psychometric test that could be used by clinicians to facilitate the identification of CF and improve the recognition and diagnosis of DLB and Parkinson disease, and to improve differential diagnosis of other dementias. METHODS We compiled a 17-item psychometric test for identifying CF and applied this measure in a cross-sectional design. Participants were recruited from the North East of England, and assessments were made in individuals' homes. We recruited people with four subtypes of dementia and a healthy comparison group, and all subjects were administered this pilot scale together with other standard ratings. The psychometric properties of the scale were examined with exploratory factor analysis. We also examined the ability of individual items to identify CF to discriminate between dementia subtypes. The sensitivity and specificity of discriminating items were explored along with validity and reliability analyses. RESULTS Participants comprised 32 comparison subjects, 30 people with Alzheimer disease, 30 with vascular dementia, 29 with DLB, and 32 with dementia associated with Parkinson disease. Four items significantly discriminated between dementia groups and showed good levels of sensitivity (range: 78.6%-80.3%) and specificity (range: 73.9%-79.3%). The scale had very good levels of test-retest (Cronbach's alpha: 0.82) and interrater (0.81) reliabilities. The four items loaded onto three different factors. These items were: 1) marked differences in functioning during the daytime; 2) daytime somnolence; 3) daytime drowsiness; and 4) altered levels of consciousness during the day. CONCLUSIONS We identified four items that provide valid, sensitive, and specific questions for reliably identifying CF and distinguishing the Lewy body dementias from other major causes of dementia (Alzheimer disease and vascular dementia).

The Psychometric Properties of the Developmental Reading Assessment

ABSTRACT The purpose of this study was to examine the technical adequacy of the Developmental Reading Assessment (Beaver & Carter, 2004). Internal consistency analysis, factor analysis, and linear regression analyses were used to test whether the DRA is a statistically reliable measuring of reading comprehension for Grades 7 and 8 students. Correlational analyses, decision consistency analyses, and a focus group of experienced Intermediate (Grades 7 and 8) teachers examined whether there is evidence that the results from the DRA provide valid interpretations regarding students’ reading skills and comprehension. Results indicated that, as currently scored, internal consistency is low and skewness of distribution is high. Factor analyses did not replicate those cited by the DRA developers to prove construct validity. Two-way contingency analyses determined that decision consistency did not vary greatly between the DRA, EQAO, scores and report card marks. Views expressed during the focus group echoed many of the challenges to validity found in the statistical analysis. The teachers found that the DRA was somewhat useful, as there were limited alternative reading assessments available for the classroom, but did not endorse it strongly. The study found little evidence that the DRA provides valid interpretations regarding Intermediate students’ reading skills. Indicated changes to the structure and administration procedures of the DRA may ameliorate some of these issues.

Measuring nursing competencies in the operating theatre: instrument development and psychometric analysis using Item Response Theory

BACKGROUND: Concern about the process of identifying underlying competencies that contribute to effective nursing performance has been debated with a lack of consensus surrounding an approved measurement instrument for assessing clinical performance. Although a number of methodologies are noted in the development of competency-based assessment measures, these studies are not without criticism. RESEARCH AIM: The primary aim of the study was to develop and validate a Performance Based Scoring Rubric, which included both analytical and holistic scales. The aim included examining the validity and reliability of the rubric, which was designed to measure clinical competencies in the operating theatre. RESEARCH METHOD: The fieldwork observations of 32 nurse educators and preceptors assessing the performance of 95 instrument nurses in the operating theatre were used in the calibration of the rubric. The Rasch model, a particular model among Item Response Models, was used in the calibration of each item in the rubric in an attempt at improving the measurement properties of the scale. This is done by establishing the 'fit' of the data to the conditions demanded by the Rasch model. RESULTS: Acceptable reliability estimates, specifically a high Cronbach's alpha reliability coefficient (0.940), as well as empirical support for construct and criterion validity for the rubric were achieved. Calibration of the Performance Based Scoring Rubric using Rasch model revealed that the fit statistics for most items were acceptable. CONCLUSION: The use of the Rasch model offers a number of features in developing and refining healthcare competency-based assessments, improving confidence in measuring clinical performance. The Rasch model was shown to be useful in developing and validating a competency-based assessment for measuring the competence of the instrument nurse in the operating theatre with implications for use in other areas of nursing practice.

The use of technology-based self-service on assessments of consumer satisfaction and consumer commitment

Technology-based self-service (TBSS) enables consumers to complete services themselves using a technological interface. As evaluations of consumer satisfaction and commitment have typically focused on interpersonal interactions, the effect of TBSS on these is under researched . This paper explores the impact of TBSS on consumer satisfaction and on a multidimensional measure of consumer commitment.Data are collected from 241 hotel guests. The results suggest personal-service is more important for satisfaction and commitment. This has implications for marketing as the benefits of adopting TBSS are not clear. Multi-dimensional commitment provides some interesting findings and suggests the need for further research into TBSS and commitment.

Towards integrated assessments for urban development

The role of sustainability in urban design is becoming increasingly important as Australia’s cities continue to grow, putting pressure on existing infrastructure such as water, energy and transport. To optimise an urban design many different aspects such as water, energy, transport, costs need to be taken into account integrally. Integrated software applications assessing urban designs on a large variety of aspects are hardly available. With the upcoming next generation of the Internet often referred to as the Semantic Web, data can become more machine-interpretable by developing ontologies that can support the development of integrated software systems. Software systems can use these ontologies to perform an intelligent task such as assessing an urban design on a particular aspect. When ontologies of different applications are aligned, they can share information resulting in interoperability. Inference such as compliancy checks and classifications can support aligning the ontologies. A proof of concept implementation has been made to demonstrate and validate the usefulness of machine interpretable ontologies for urban designs.

How can flexible learning centres use social networking to improve their ongoing educational assessments?

Most online assessment systems now incorporate social networking features, and recent developments in social media spaces include protocols that allow the synchronisation and aggregation of data across multiple user profiles. In light of these advances and the concomitant fear of data sharing in secondary school education this papers provides important research findings about generic features of online social networking, which educators can use to make sound and efficient assessments in collaboration with their students and colleagues. This paper reports on a design experiment in flexible educational settings that challenges the dichotomous legacy of success and failure evident in many assessment activities for at-risk youth. Combining social networking practices with the sociology of education the paper proposes that assessment activities are best understood as a negotiable field of exchange. In this design experiment students, peers and educators engage in explicit, "front-end" assessment (Wyatt-Smith, 2008) to translate digital artefacts into institutional, and potentiality economic capital without continually referring to paper based pre-set criteria. This approach invites students and educators to use social networking functions to assess “work in progress” and final submissions in collaboration, and in doing so assessors refine their evaluative expertise and negotiate the value of student’s work from which new criteria can emerge. The mobile advantages of web-based technologies aggregate, externalise and democratise this transparent assessment model for most, if not all, student work that can be digitally represented.

Psychometric evaluation of the Chinese version of the diabetes coping measure scale

Purpose: To evaluate the psychometric properties of a Chinese version of the Diabetes Coping Measure (DCM-C) scale.----- Methods: A self-administered questionnaire was completed by 205 people with type 2 diabetes from the endocrine outpatient departments of three hospitals in Taiwan. Confirmatory factor analysis, criterion validity, and internal consistency reliability were conducted to evaluate the psychometric properties of the DCM-C.----- Findings: Confirmatory factor analysis confirmed a four-factor structure (χ2 /df ratio=1.351, GFI=.904, CFI=.902, RMSEA=.041). The DCM-C was significantly associated with HbA1c and diabetes self-care behaviors. Internal consistency reliability of the total DCM-C scale was .74. Cronbach’s alpha coefficients for each subscale of the DCM-C ranged from .37 (tackling spirit) to .66 (diabetes integration).----- Conclusions: The DCM-C demonstrated satisfactory reliability and validity to determine the use of diabetes coping strategies. The tackling spirit dimension needs further refinement when applies this scale to Chinese populations with diabetes.----- Clinical Relevance: Healthcare providers who deal with Chinese people with diabetes can use the DCM-C to implement an early determination of diabetes coping strategies.