A Benefit-Risk Assessment Framework for Development of Clinical Guidelines in Diagnostic Radiology

Thesis (Ph.D.)--University of Washington, 2015-12

Background The body of this dissertation focuses on benefit-risk assessment in diagnostic radiology guideline development. We look specifically at structuring and making more transparent the role of expert consensus in evidence-based practice guidelines like those developed by the American College of Radiology (ACR). While several frameworks for structured benefit-risk assessment for pharmaceutical products are available, their applicability in guiding guideline development of diagnostic imaging is not well characterized.2 The most prominent are the (BRAT) Benefit-Risk Action Team and ProACT-URL (Problem, Outcomes, Alternatives, Consequence and Tradeoffs Uncertainty Risk and Linked decisions) frameworks. The six steps of the BRAT framework include, (1) defining the decision context (selecting stakeholder perspective and time horizon); (2) identifying benefit and risk outcomes visually with use of a value tree (one branch exploding all benefits and the other, all harms); (3) identifying data sources and creating a data source table; (4) tuning or customizing the framework, i.e. aligning outcomes in value tree criteria with outcomes for which data are available; (5) assessing the importance or weight of each outcome under consideration; (6) and displaying or communicating the metrics using visual and tabular formats. Likewise, the PrOACT-URL decision-making framework comprises five core elements (Problem, Outcomes, Alternatives, Consequences and Tradeoffs) and three elements relevant to evolving or volatile settings: Uncertainty, Risk attitude, and Linked decisions.3 The ACR expert panels follow a structured process in developing appropriateness criteria (AC) but do not employ a benefit-risk framework.4-7 In many cases benefit-risk assessment is limited to comparing the diagnostic accuracy of a test against test procedural risks (e.g., exposure to radiation, invasiveness). Unlike endpoints of pharmaceutical product trials, the endpoint of diagnostic accuracy does not easily fit within the benefit-risk framework and needs to be translated into other endpoints. Endpoints that are easily identified as benefits or risks — effects of test information on the provider, on patient management, and on the patient — are rarely known. It is for this reason that the ACR AC are developed using a combined approach, relying in part on the body of evidence, and in part on expert consensus.8 However, the ACR AC lack key framework elements that ensure the transparent contribution of expert consensus: making explicit relevant benefits and risks, their prioritization and respective data sources or lack thereof. Independently, several frameworks for assessing the value of diagnostic imaging are available.9-12 These frameworks are of limited value in this setting as few new studies are conducted specifically to inform clinical guideline development. At the same time, there are calls for expanding the way imaging tests are valued.13 Lee et al. proposed a three dimensional value framework (medical, psychic, and planning) for health technology assessment, proposing that the value of diagnostic tests may be underestimated in cost-effectiveness analyses that limit the scope to medical impacts.9 Staub et al. promote inclusion of patient management measures as proxies for patient health outcomes.14,15 Otero et al. discuss the lack of inclusion in cost-utility analyses of intrinsic value elements (non-clinical impacts of test information), similar to the Lee et al. psychic and planning dimensions but incorporating both provider and patient intrinsic value.16 Bossuyt and McCaffery present a framework incorporating dimensions independent of clinical outcome (emotional, social, cognition, behavior) with incomplete overlap with Otero’s and Lee’s.17 Chapter 1. Toward a Framework for Benefit Risk Assessment in Diagnostic Imaging: Identifying Scenario-Specific Criteria In the first paper, we work toward creating a unified framework incorporating elements of structured benefit-risk and elements of diagnostic imaging frameworks. To address the question of clearly defined criteria, we abstract from the literature measures of diagnostic imaging value and translate these into benefit-risk criteria (BRC). To further refine the BRC, we cross-reference our literature findings by surveying radiologist and non-radiologist perceptions of the benefits and risks in diagnostic radiology. Within the survey, we operationalize the initial broad list of BRC across four clinical use case scenarios. For each use case, we compare BRC selections between radiologists and non-radiologists. We arrive at thirty six criteria, organized into three domains: 1) those that account for differences among tests, attributable only to the test or device (n=17); 2) those that account for clinical management and provider experience effects (n=12); and 3) and those that measure distant, less direct effects of imaging tests on patients (n=7). Our results suggest that radiologist considerations do not dramatically differ from those of non-radiologists but the addition of non-radiologist selections may help guideline developers reach the goal of a broader set of priorities. These results can inform future hypotheses and studies of effects of increased clinical diversity on guideline quality and adoption. Chapter 2. A Proposed Approach for Quantitative Benefit-Risk Assessment in Diagnostic Radiology Guideline Development: The American College of Radiology Appropriateness Criteria Example In the second paper, we continue building a framework for benefit-risk assessment with a critical appraisal of quantitative benefit-risk assessment (QBRA) methodology. As there is only limited guidance on method selection and it is not clear whether these methods are well suited to the clinical guideline development process, further exploration of the potential for benefit-risk methodology to meet the needs of the ACR AC process is warranted.18-20 Thus, we review the benefit-risk methodology literature and propose several steps for selection of comparators and criteria. These steps include investigation of weak evidence and disagreement. We identify a set of benefit-risk methods addressing one or more of these needs and build a decision aid for selecting among these methods. Our results suggest there is opportunity to use multi-criteria decision analysis and incremental net health benefit methods for some decision problems the ACR faces when creating AC ratings. The process leading to the decision aid facilitates transparent contribution of expert opinion to ratings. Since the structure of the decision aid is based on clinicians’ input but also requires the skills of methods experts, the decision aid represents a key component to uniting clinical experts and methodologists. Chapter 3. Analytic Hierarchy Process for Prioritizing Imaging Tests in Diagnosis of Suspected Appendicitis In the third paper, to complement inclusion of QBRA in the framework, we evaluate empirically one of the QBRA methods for use in guideline development. There is prevailing skepticism that regulatory approval and clinical guideline decision-making is far too complicated, and too multi-dimensional for quantitative methods.21 To investigate this position further, we compare the results of a multi-criteria decision analysis approach, analytic hierarchy process, evaluating computed tomography against magnetic resonance imaging and ultrasound for classic presentation of suspected appendicitis, to the ACR AC ratings. We ask those who participated whether the process is manageable, transparent, and improves shared-understanding of the decision problem. This is, to our knowledge, the first study to show that a quantitative method produces comparable results to ratings of ACR AC guidelines and that this QBRA was found, among study participants, to facilitate shared understanding and transparency. Conclusions Structured benefit-risk assessment promises to improve the transparency of the contribution of expert consensus to clinical guideline development in diagnostic radiology. In this body of work, we propose a step-wise process resembling existing benefit-risk frameworks, but tailored to the unique needs of diagnostic radiology and demonstrate the feasibility of these recommendations. Specifically, we provide a systematic approach to selection of benefits and risks in evaluating the value of diagnostic imaging. We show that diversity of participants in selection of BRC expands the decision problem and suggests that a comprehensive definition of value is best accomplished using a multi-disciplinary perspective. Likewise, we reduce the number of QBRA for ACR to consider and provide guidance for when and how to use QBRA. Lastly, we demonstrate that MCDA can add transparency to a process like the ACR AC guideline development. Future work will entail field testing the BRC, the decision aid for QBRA selection as well as the QBRA in a setting like the ACR AC.