Assessment of statistical change criteria used to define significant change in neuropsychological test performance following cardiac surgery

Objective: This paper compares four techniques used to assess change in neuropsychological test scores before and after coronary artery bypass graft surgery (CABG), and includes a rationale for the classification of a patient as overall impaired. Methods: A total of 55 patients were tested before and after surgery on the MicroCog neuropsychological test battery. A matched control group underwent the same testing regime to generate test–retest reliabilities and practice effects. Two techniques designed to assess statistical change were used: the Reliable Change Index (RCI), modified for practice, and the Standardised Regression-based (SRB) technique. These were compared against two fixed cutoff techniques (standard deviation and 20% change methods). Results: The incidence of decline across test scores varied markedly depending on which technique was used to describe change. The SRB method identified more patients as declined on most measures. In comparison, the two fixed cutoff techniques displayed relatively reduced sensitivity in the detection of change. Conclusions: Overall change in an individual can be described provided the investigators choose a rational cutoff based on likely spread of scores due to chance. A cutoff value of ≥20% of test scores used provided acceptable probability based on the number of tests commonly encountered. Investigators must also choose a test battery that minimises shared variance among test scores.

OMERACT-OARSI Initiative: Osteoarthritis Research Society International set of responder criteria for osteoarthritis clinical trials revisited.

Background: The OARSI Standing Committee for Clinical Trials Response Criteria Initiative had developed two sets of responder criteria to present the results of changes after treatment in three symptomatic domains (pain, function, and patient's global assessment) as a single variable for clinical trials (1). For each domain, a response was defined by both a relative and an absolute change, with different cut-offs with regard to the drug, the route of administration and the OA localization. Objective: To propose a simplified set of responder criteria with a similar cut-off, whatever the drug, the route or the OA localization. Methods: Data driven approach: (1) Two databases were considered The 'elaboration' database with which the formal OARSI sets of responder criteria were elaborated and The 'revisit' database. (2) Six different scenarios were evaluated: The two formal OARSI sets of criteria Four proposed scenarios of simplified sets of criteria Data from clinical randomized blinded placebo controlled trials were used to evaluate the performances of the two formal scenarios with two different databases ('elaboration' versus 'revisit') and those of the four proposed simplified scenarios within the 'revisit' database. The placebo effect, active effect, treatment effect, and the required sample arm size to obtain the placebo effect and the active treatment effect observed were the performances evaluated for each of the six scenarios. Experts' opinion approach: Results were discussed among the participants of the OMERACT VI meeting, who voted to select the definite OMERACT-OARSI set of criteria (one of the six evaluated scenarios). Results: Data driven approach: Fourteen trials totaling 1886 CA patients and fifteen studies involving 8164 CA patients were evaluated in the 'elaboration' and the 'revisit' databases respectively. The variability of the performances observed in the 'revisit' database when using the different simplified scenarios was similar to that observed between the two databases ('elaboration' versus 'revisit') when using the formal scenarios. The treatment effect and the required sample arm size were similar for each set of criteria. Experts' opinion approach: According to the experts, these two previous performances were the most important of an optimal set of responder criteria. They chose the set of criteria considering both pain and function as evaluation domain and requiring an absolute change and a relative change from baseline to define a response, with similar cut-offs whatever the drug, the route of administration or the CA localization. Conclusion: This data driven and experts' opinion approach is the basis for proposing an optimal simplified set of responder criteria for CA clinical trials. Other studies, using other sets of CA patients, are required in order to further validate this proposed OMERACT - OARSI set of criteria. (C) 2004 OsteoArthritis Research Society International. Published by Elsevier Ltd. All rights reserved.

The effectiveness of hylan G-F 20 in patients with knee osteoarthritis: an application of two sets of response criteria developed by the OARSI and one set developed by OMERACT-OARSI

Objective: Secondary analyses of a previously conducted 1-year randomized controlled trial were performed to assess the application of responder criteria in patients with knee osteoarthritis (OA) using different sets of responder criteria developed by the Osteoarthritis Research Society International (OARSI) (Propositions A and B) for intra-articular drugs and Outcome Measures in Arthritis Clinical Trials (OMERACT)-OARSI (Proposition D). Methods: Two hundred fifty-five patients with knee OA were randomized to appropriate care with hylan G-F 20 (AC + H) or appropriate care without hylan G-F 20 (AC). A patient was defined as a responder at month 12 based on change in Western Ontario and McMaster Universities Osteoarthritis Index pain and function (0-100 normalized scale) and patient global assessment of OA in the study knee (at least one-category improvement in very poor, poor, fair, good and very good). All propositions incorporate both minimum relative and absolute changes. Results: Results demonstrated that statistically significant differences in responders between treatment groups, in favor of hylan G-F 20, were detected for Proposition A (AC + H = 53.5%, AC = 25.2%), Proposition B (AC + H = 56.7%, AC = 32.3%) and Proposition D (AC + H = 66.9%, AC = 42.5%). The highest effectiveness in both treatment groups was observed with Proposition D, whereas Proposition A resulted in the lowest effectiveness in both treatment groups. The treatment group differences always exceeded the required 20% minimum clinically important difference between groups established a priori, and were 28.3%, 24.4% and 24.4% for Propositions A, B and D, respectively. Conclusion: This analysis provides evidence for the capacity of OARSI and OMERACT-OARSI responder criteria to detect clinically important statistically detectable differences between treatment groups. (C) 2004 OsteoArthritis Research Society International. Published by Elsevier Ltd. All rights reserved.

Science of assessment

Psoriatic arthritis is a multisystem disorder which, from a measurement standpoint, demands consideration of its cutaneous manifestations and both axial and peripheral musculoskeletal involvement. Measurements of various aspects of impairment, ability/disability, and participation/ handicap are feasible using existing measurement techniques, which are for the most part valid, reliable, and responsive. Nevertheless, there remain opportunities for the further development of consensus around core set measures and responder criteria, as well as for instrument development and refinement, standardised assessor training, cross-cultural adaptation of health status questionnaires, electronic data capture, and the introduction of standardised quantitative measurement into routine clinical care.

Assessment of disease progression in motor neuron disease

Motor neuron disease (MND) is characterised by progressive deterioration of the corticospinal tract, brainstem, and anterior horn cells of the spinal cord. There is no pathognomonic test for the diagnosis of MND, and physicians rely on clinical criteria-upper and lower motor neuron signs-for diagnosis. The presentations, clinical phenotypes, and outcomes of MND are diverse and have not been combined into a marker of disease progression. No single algorithm combines the findings of functional assessments and rating scales, such as those that assess quality of life, with biological markers of disease activity and findings from imaging and neurophysiological assessments. Here, we critically appraise developments in each of these areas and discuss the potential of such measures to be included in the future assessment of disease progression in patients with MND.

Using student feedback to construct an assessment rubric for a concept map in physiology

Constructing quality assessment rubrics can be challenging, especially when they are used for integrated, group-centered, applied learning. We describe a collaborative assessment task in which groups of second-year dentistry students developed a complex concept map. In groups of four, the students were given a written, simulated, medical history of a patient and required to construct a concept map illustrating relevant pathophysiological concepts and pharmacological interventions. This report describes a research project aimed at making educational goals of the task more explicit through investigating student and faculty member understandings of the criteria that might be used to assess the concept map. Information was gathered about the perceptions of students in relation to the learning goals associated with the task. These were compared with faculty member perceptions. The findings were used to develop an assessment rubric intended to be more accessible to learners. The new rubric used the language of both faculty members and students to more clearly represent expectations of each criterion and standard. This assessment rubric will be used in 2005 for the next phase of the project.

An Internet-based telerehabilitation system for the assessment of motor speech disorders: A pilot study

Purpose: This pilot study explored the feasibility and effectiveness of an Internet-based telerehabilitation application for the assessment of motor speech disorders in adults with acquired neurological impairment. Method: Using a counterbalanced, repeated measures research design, 2 speech-language pathologists assessed 19 speakers with dysarthria on a battery of perceptual assessments. The assessments included a 19-item version of the Frenchay Dysarthria Assessment (FDA; P. Enderby, 1983), the Assessment of Intelligibility of Dysarthric Speech (K. M. Yorkston & D. R. Beukelman, 1981), perceptual analysis of a speech sample, and an overall rating of severity of the dysarthria. One assessment was conducted in the traditional face-to-face manner, whereas the other assessment was conducted using an online, custom-built telerehabilitation application. This application enabled real-time videoconferencing at 128 kb/s and the transfer of store-and-forward audio and video data between the speaker and speech-language pathologist sites. The assessment methods were compared using the J.M.Bland and D.G.Altman (1986, 1999) limits-of-agreement method and percentage level of agreement between the 2 methods. Results: Measurements of severity of dysarthria, percentage intelligibility in sentences, and most perceptual ratings made in the telerehabilitation environment were found to fall within the clinically acceptable criteria. However, several ratings on the FDA were not comparable between the environments, and explanations for these results were explored. Conclusions: The online assessment of motor speech disorders using an Internet-based telerehabilitation system is feasible. This study suggests that with additional refinement of the technology and assessment protocols, reliable assessment of motor speech disorders over the Internet is possible. Future research methods are outlined.

Science-policy in environmental and health risk assessment: if we cannot do without, can we do better?

How can empirical evidence of adverse effects from exposure to noxious agents, which is often incomplete and uncertain, be used most appropriately to protect human health? We examine several important questions on the best uses of empirical evidence in regulatory risk management decision-making raised by the US Environmental Protection Agency (EPA)'s science-policy concerning uncertainty and variability in human health risk assessment. In our view, the US EPA (and other agencies that have adopted similar views of risk management) can often improve decision-making by decreasing reliance on default values and assumptions, particularly when causation is uncertain. This can be achieved by more fully exploiting decision-theoretic methods and criteria that explicitly account for uncertain, possibly conflicting scientific beliefs and that can be fully studied by advocates and adversaries of a policy choice, in administrative decision-making involving risk assessment. The substitution of decision-theoretic frameworks for default assumption-driven policies also allows stakeholder attitudes toward risk to be incorporated into policy debates, so that the public and risk managers can more explicitly identify the roles of risk-aversion or other attitudes toward risk and uncertainty in policy recommendations. Decision theory provides a sound scientific way explicitly to account for new knowledge and its effects on eventual policy choices. Although these improvements can complicate regulatory analyses, simplifying default assumptions can create substantial costs to society and can prematurely cut off consideration of new scientific insights (e.g., possible beneficial health effects from exposure to sufficiently low 'hormetic' doses of some agents). In many cases, the administrative burden of applying decision-analytic methods is likely to be more than offset by improved effectiveness of regulations in achieving desired goals. Because many foreign jurisdictions adopt US EPA reasoning and methods of risk analysis, it may be especially valuable to incorporate decision-theoretic principles that transcend local differences among jurisdictions.

Comparison of ASAS improvement criteria for ankylosing spondylitis with clinically relevant improvement according to an expert panel

Objective: To investigate whether the recently developed (statistically derived) "ASsessment in Ankylosing Spondylitis Working Group" improvement criteria (ASAS-IC) for ankylosing spondylitis (AS) reflect clinically relevant improvement according to the opinion of an expert panel. Methods: The ASAS-IC consist of four domains: physical function, spinal pain, patient global assessment, and inflammation. Scores on these four domains of 55 patients with AS, who had participated in a non-steroidal anti-inflammatory drug efficacy trial, were presented to an international expert panel (consisting of patients with AS and members of the ASAS Working Group) in a three round Delphi exercise. The number of (non-) responders according to the ASAS-IC was compared with the final-consensus of the experts. The most important domains in the opinion of the experts were identified, and also selected with discriminant analysis. A number of provisional criteria sets that best represented the consensus of the experts were defined. Using other datasets, these clinically derived criteria sets as well as the statistically derived ASAS-IC were then tested for discriminative properties and for agreement with the end of trial efficacy by patient and doctor. Results: Forty experts completed the three Delphi rounds. The experts considered twice as many patients to be responders than the ASAS-IC (42 v 21). Overall agreement between experts and ASAS-IC was 62%. Spinal pain was considered the most important domain by most experts and was also selected as such by discriminant analysis. Provisional criteria sets with an agreement of greater than or equal to 80% compared with the consensus of the experts showed high placebo response rates (27-42%), in contrast with the ASAS-IC with a predefined placebo response rate of 25%. All criteria sets and the ASAS-IC discriminated well between active and placebo treatment (chi(2) = 36-45; p < 0.001). Compared with the end of trial efficacy assessment, the provisional criteria sets showed an agreement of 71-82%, sensitivity of 67-83%, and specificity of 81-88%. The ASAS-IC showed an agreement of 70%, sensitivity of 62%, and specificity of 89%. Conclusion: The ASAS-IC are strict in defining response, are highly specific, and consequently show lower sensitivity than the clinically derived criteria sets. However, those patients who are considered as responders by applying the ASAS-IC are acknowledged as such by the expert panel as well as by. patients' and doctors' judgments, and are therefore likely to be true responders.