Comparison of clinical knowledge management capabilities of commercially-available and leading internally-developed electronic health records.

BACKGROUND: We have carried out an extensive qualitative research program focused on the barriers and facilitators to successful adoption and use of various features of advanced, state-of-the-art electronic health records (EHRs) within large, academic, teaching facilities with long-standing EHR research and development programs. We have recently begun investigating smaller, community hospitals and out-patient clinics that rely on commercially-available EHRs. We sought to assess whether the current generation of commercially-available EHRs are capable of providing the clinical knowledge management features, functions, tools, and techniques required to deliver and maintain the clinical decision support (CDS) interventions required to support the recently defined "meaningful use" criteria. METHODS: We developed and fielded a 17-question survey to representatives from nine commercially available EHR vendors and four leading internally developed EHRs. The first part of the survey asked basic questions about the vendor's EHR. The second part asked specifically about the CDS-related system tools and capabilities that each vendor provides. The final section asked about clinical content. RESULTS: All of the vendors and institutions have multiple modules capable of providing clinical decision support interventions to clinicians. The majority of the systems were capable of performing almost all of the key knowledge management functions we identified. CONCLUSION: If these well-designed commercially-available systems are coupled with the other key socio-technical concepts required for safe and effective EHR implementation and use, and organizations have access to implementable clinical knowledge, we expect that the transformation of the healthcare enterprise that so many have predicted, is achievable using commercially-available, state-of-the-art EHRs.

Clinical decision support capabilities of commercially-available clinical information systems.

BACKGROUND: The most effective decision support systems are integrated with clinical information systems, such as inpatient and outpatient electronic health records (EHRs) and computerized provider order entry (CPOE) systems. Purpose The goal of this project was to describe and quantify the results of a study of decision support capabilities in Certification Commission for Health Information Technology (CCHIT) certified electronic health record systems. METHODS: The authors conducted a series of interviews with representatives of nine commercially available clinical information systems, evaluating their capabilities against 42 different clinical decision support features. RESULTS: Six of the nine reviewed systems offered all the applicable event-driven, action-oriented, real-time clinical decision support triggers required for initiating clinical decision support interventions. Five of the nine systems could access all the patient-specific data items identified as necessary. Six of the nine systems supported all the intervention types identified as necessary to allow clinical information systems to tailor their interventions based on the severity of the clinical situation and the user's workflow. Only one system supported all the offered choices identified as key to allowing physicians to take action directly from within the alert. Discussion The principal finding relates to system-by-system variability. The best system in our analysis had only a single missing feature (from 42 total) while the worst had eighteen.This dramatic variability in CDS capability among commercially available systems was unexpected and is a cause for concern. CONCLUSIONS: These findings have implications for four distinct constituencies: purchasers of clinical information systems, developers of clinical decision support, vendors of clinical information systems and certification bodies.

Measuring Consumer Satisfaction in Family Preservation Services: Identifying Instrument Domains

Measuring consumer satisfaction in the social services has become an important source of feedback for the improvement of service delivery. Consumer satisfaction has recently been incorporated into family preservation evaluation. This article reviews instruments used to measure consumer satisfaction in family preservation services and other related areas. Trends in current practice are examined and instrument dimensions are identified. Finally, some recommendations are made about the application of consumer satisfaction measurement in family preservation services.

Patient classification system: An integrated method for measuring nursing intensity and optimizing resource allocation

A patient classification system was developed integrating a patient acuity instrument with a computerized nursing distribution method based on a linear programming model. The system was designed for real-time measurement of patient acuity (workload) and allocation of nursing personnel to optimize the utilization of resources.^ The acuity instrument was a prototype tool with eight categories of patients defined by patient severity and nursing intensity parameters. From this tool, the demand for nursing care was defined in patient points with one point equal to one hour of RN time. Validity and reliability of the instrument was determined as follows: (1) Content validity by a panel of expert nurses; (2) predictive validity through a paired t-test analysis of preshift and postshift categorization of patients; (3) initial reliability by a one month pilot of the instrument in a practice setting; and (4) interrater reliability by the Kappa statistic.^ The nursing distribution system was a linear programming model using a branch and bound technique for obtaining integer solutions. The objective function was to minimize the total number of nursing personnel used by optimally assigning the staff to meet the acuity needs of the units. A penalty weight was used as a coefficient of the objective function variables to define priorities for allocation of staff.^ The demand constraints were requirements to meet the total acuity points needed for each unit and to have a minimum number of RNs on each unit. Supply constraints were: (1) total availability of each type of staff and the value of that staff member (value was determined relative to that type of staff's ability to perform the job function of an RN (i.e., value for eight hours RN = 8 points, LVN = 6 points); (2) number of personnel available for floating between units.^ The capability of the model to assign staff quantitatively and qualitatively equal to the manual method was established by a thirty day comparison. Sensitivity testing demonstrated appropriate adjustment of the optimal solution to changes in penalty coefficients in the objective function and to acuity totals in the demand constraints.^ Further investigation of the model documented: correct adjustment of assignments in response to staff value changes; and cost minimization by an addition of a dollar coefficient to the objective function. ^

PREPARATION AND CHARACTERIZATION OF COVALENTLY LINKED ENZYME-ANTIBODY CONJUGATES AND THEIR USE IN MEASURING MINUTE QUANTITIES OF PROTEIN ANTIGENS

The discovery and characterization of oncofetal proteins have led to significant advances in early cancer diagnosis and therapeutic monitoring of patients undergoing cancer chemotherapy. These tumor-associated antigens are presently measured by sensitive, specific immunoassay techniques based on the detection of minute amounts of labeled antigen or antibody incorporated into immune complexes, which must be isolated from free antigen and antibody.^ Since there are several disadvantages with using radioisotopes, the most common immunolabel, one major objective was to prepare covalently coupled enzyme-antibody conjugates and evaluate their use as a practical alternative to radiolabeled immune reagents. An improved technique for the production of enzyme-antibody conjugates was developed that involves oxidizing the carbohydrate moieties on a glycoprotein enzyme, then introducing antibody in the presence of polyethylene glycol (PEG). Covalent enzyme-antibody conjugates involving alkaline phosphatase and amyloglucosidase were produced and characterized.^ In order to increase the sensitivity of detecting the amyloglucosidase-antibody conjugate, an enzyme cycling assay was developed that measures glucose, the product of maltose cleavage by amyloglucosidase, in the picomole range. The increased sensitivity obtained by combined usage of the amyloglucosidase-antibody conjugate and enzyme cycling assay was then compared to that of conventional enzyme immunoassay (EIA).^ For immune complex isolation, polystyrene tubes and protein A-bearing Staphylococcus aureus were evaluated as solid phase matrices, upon which antibodies can be immobilized. A sandwich-type EIA, using antibody-coated S. aureus, was developed that measures human albumin (HSA) in the nanogram range. The assay, using an alkaline phosphatase-anti-HSA conjugate, was applied to the determination of HSA in human urine and evaluated extensively for its clinical applicability.^ Finally, in view of the clinical significance of alpha-fetoprotein (AFP) as an oncofetal antigen and the difficulty with its purification for use as an immunogen and assay standard, a chemical purification protocol was developed that resulted in a high yield of immunochemically pure AFP. ^

Measuring health state preferences for hemophilia: Development of a disease-specific utility instrument

Research has shown that disease-specific health related quality of life (HRQoL) instruments are more responsive than generic instruments to particular disease conditions. However, only a few studies have used disease-specific instruments to measure HRQoL in hemophilia. The goal of this project was to develop a disease-specific utility instrument that measures patient preferences for various hemophilia health states. The visual analog scale (VAS), a ranking method, and the standard gamble (SG), a choice-based method incorporating risk, were used to measure patient preferences. Study participants (n = 128) were recruited from the UT/Gulf States Hemophilia and Thrombophilia Center and stratified by age: 0–18 years and 19+. ^ Test retest reliability was demonstrated for both VAS and SG instruments: overall within-subject correlation coefficients were 0.91 and 0.79, respectively. Results showed statistically significant differences in responses between pediatric and adult participants when using the SG (p = .045). However, no significant differences were shown between these groups when using the VAS (p = .636). When responses to VAS and SG instruments were compared, statistically significant differences in both pediatric (p < .0001) and adult (p < .0001) groups were observed. Data from this study also demonstrated that persons with hemophilia with varying severity of disease, as well as those who were HIV infected, were able to evaluate a range of health states for hemophilia. This has important implications for the study of quality of life in hemophilia and the development of disease-specific HRQoL instruments. ^ The utility measures obtained from this study can be applied in economic evaluations that analyze the cost/utility of alternative hemophilia treatments. Results derived from the SG indicate that age can influence patients' preferences regarding their state of health. This may have implications for considering treatment options based on the mean age of the population under consideration. Although both instruments independently demonstrated reliability and validity, results indicate that the two measures may not be interchangeable. ^

Measuring the quality of life of patients with diabetes: Validating the tool and determining the effects on quality of life

There are several tools for measuring quality of life (QoL) and specifically, health-related quality of life (HRQoL) for persons with diabetes. A commonly-used measure, the Diabetes Quality of Life (DQOL) Survey, developed for the Diabetes Control and Complications Trial (DCCT), has been used in several experimental settings, and its reliability and validity are well-established. However, it is considered too long to be of practical use in clinical settings. Because of this, a shortened version of the tool was used recently in the Community Diabetes Education (CoDE) Project in Dallas, Texas, a clinic-based patient education program that uses a specially-trained community healthcare worker to provide patient education. However, the modified scale has never been tested for reliability and validity. Thus, one goal of this thesis was to measure these psychometric properties of the scale. After establishing the reliability and validity, the results of the scale were analyzed to determine the effects of the intervention on the subjects’ quality of life. The changes in QoL scales were compared with changes in physiologic measures which are most closely allied with diabetes, including blood glucose levels, weight/BMI, co-morbidities and health beliefs in order to determine if there is a relationship between such measures and quality of life. The results of the reliability and validity testing were not conclusive. Measures of reliability and criterion validity were established, but these contrasted with poor measures of repeatability and content validity. The effect of the intervention on quality of life, however, was more significant, particularly regarding the impact of diabetes. Those who received the counseling had significantly higher scores on the Impact scale than those who did not, and the former group had much greater improvement in scores over the twelve month period than the latter group. ^

The imaging and dosimetric capabilities of a CT/MR-suitable, anatomically adaptive, shielded intracavitary brachytherapy applicator for the treatment of cervical cancer

Introduction. Investigations into the shortcomings of current intracavitary brachytherapy (ICBT) technology has lead us to design an Anatomically Adaptive Applicator (A3). The goal of this work was to design and characterize the imaging and dosimetric capabilities of this device. The A3 design incorporates a single shield that can both rotate and translate within the colpostat. We hypothesized that this feature, coupled with specific A3 component construction materials and imaging techniques, would facilitate artifact-free CT and MR image acquisition. In addition, by shaping the delivered dose distribution via the A3 movable shield, dose delivered to the rectum will be less compared to equivalent treatments utilizing current state-of-the-art ICBT applicators. ^ Method and materials. A method was developed to facilitate an artifact-free CT imaging protocol that used a "step-and-shoot" technique: pausing the scanner midway through the scan and moving the A 3 shield out of the path of the beam. The A3 CT imaging capabilities were demonstrated acquiring images of a phantom that positioned the A3 and FW applicators in a clinically-applicable geometry. Artifact-free MRI imaging was achieved by utilizing MRI-compatible ovoid components and pulse-sequences that minimize susceptibility artifacts. Artifacts were qualitatively compared, in a clinical setup. For the dosimetric study, Monte-Carlo (MC) models of the A3 and FW (shielded and unshielded) applicators were validated. These models were incorporated into a MC model of one cervical cancer patient ICBT insertion, using 192Ir (mHDR v2 source). The A3 shield's rotation and translation was adjusted for each dwell position to minimize dose to the rectum. Superposition of dose to rectum for all A3 dwell sources (4 per ovoid) was applied to obtain a comparison of equivalent FW treatments. Rectal dose-volume histograms (absolute and HDR/PDR biologically effective dose (BED)) and BED to 2 cc (BED2cc ) were determined for all applicators and compared. ^ Results. Using a "step-and-shoot" CT scanning method and MR compliant materials and optimized pulse-sequences, images of the A 3 were nearly artifact-free for both modalities. The A3 reduced BED2cc by 18.5% and 7.2% for a PDR treatment and 22.4% and 8.7% for a HDR treatment compared to treatments delivered using an uFW and sFW applicator, respectively. ^ Conclusions. The novel design of the A3 facilitated nearly artifact-free image quality for both CT and MR clinical imaging protocols. The design also facilitated a reduction in BED to the rectum compared to equivalent ICBT treatments delivered using current, state-of-the-art applicators. ^

The Open Door Mission: Measuring and predicting outcomes of one community-based substance abuse treatment program

The objectives of this study were to identify and measure the average outcomes of the Open Door Mission's nine-month community-based substance abuse treatment program, identify predictors of successful outcomes, and make recommendations to the Open Door Mission for improving its treatment program.^ The Mission's program is exclusive to adult men who have limited financial resources: most of which were homeless or dependent on parents or other family members for basic living needs. Many, but not all, of these men are either chemically dependent or have a history of substance abuse.^ This study tracked a cohort of the Mission's graduates throughout this one-year study and identified various indicators of success at short-term intervals, which may be predictive of longer-term outcomes. We tracked various levels of 12-step program involvement, as well as other social and spiritual activities, such as church affiliation and recovery support.^ Twenty-four of the 66 subjects, or 36% met the Mission's requirements for success. Specific to this success criteria; Fifty-four, or 82% reported affiliation with a home church; Twenty-six, or 39% reported full-time employment; Sixty-one, or 92% did not report or were not identified as having any post-treatment arrests or incarceration, and; Forty, or 61% reported continuous abstinence from both drugs and alcohol.^ Five research-based hypotheses were developed and tested. The primary analysis tool was the web-based non-parametric dependency modeling tool, B-Course, which revealed some strong associations with certain variables, and helped the researchers generate and test several data-driven hypotheses. Full-time employment is the greatest predictor of abstinence: 95% of those who reported full time employment also reported continuous post-treatment abstinence, while 50% of those working part-time were abstinent and 29% of those with no employment were abstinent. Working with a 12-step sponsor, attending aftercare, and service with others were identified as predictors of abstinence.^ This study demonstrates that associations with abstinence and the ODM success criteria are not simply based on one social or behavioral factor. Rather, these relationships are interdependent, and show that abstinence is achieved and maintained through a combination of several 12-step recovery activities. This study used a simple assessment methodology, which demonstrated strong associations across variables and outcomes, which have practical applicability to the Open Door Mission for improving its treatment program. By leveraging the predictive capability of the various success determination methodologies discussed and developed throughout this study, we can identify accurate outcomes with both validity and reliability. This assessment instrument can also be used as an intervention that, if operationalized to the Mission’s clients during the primary treatment program, may measurably improve the effectiveness and outcomes of the Open Door Mission.^

MEASURING THE EFFECT OF ILLNESS ON THE EXPECTED REMAINING SERVICE TIME FOR THE ACTIVE DUTY ARMY: A LIFE TABLE APPROACH

A life table methodology was developed which estimates the expected remaining Army service time and the expected remaining Army sick time by years of service for the United States Army population. A measure of illness impact was defined as the ratio of expected remaining Army sick time to the expected remaining Army service time. The variances of the resulting estimators were developed on the basis of current data. The theory of partial and complete competing risks was considered for each type of decrement (death, administrative separation, and medical separation) and for the causes of sick time.^ The methodology was applied to world-wide U.S. Army data for calendar year 1978. A total of 669,493 enlisted personnel and 97,704 officers were reported on active duty as of 30 September 1978. During calendar year 1978, the Army Medical Department reported 114,647 inpatient discharges and 1,767,146 sick days. Although the methodology is completely general with respect to the definition of sick time, only sick time associated with an inpatient episode was considered in this study.^ Since the temporal measure was years of Army service, an age-adjusting process was applied to the life tables for comparative purposes. Analyses were conducted by rank (enlisted and officer), race and sex, and were based on the ratio of expected remaining Army sick time to expected remaining Army service time. Seventeen major diagnostic groups, classified by the Eighth Revision, International Classification of Diseases, Adapted for Use In The United States, were ranked according to their cumulative (across years of service) contribution to expected remaining sick time.^ The study results indicated that enlisted personnel tend to have more expected hospital-associated sick time relative to their expected Army service time than officers. Non-white officers generally have more expected sick time relative to their expected Army service time than white officers. This racial differential was not supported within the enlisted population. Females tend to have more expected sick time relative to their expected Army service time than males. This tendency remained after diagnostic groups 580-629 (Genitourinary System) and 630-678 (Pregnancy and Childbirth) were removed. Problems associated with the circulatory system, digestive system and musculoskeletal system were among the three leading causes of cumulative sick time across years of service. ^

THE ALCOHOL TREATMENT PROFILE SYSTEM: ITS DEVELOPMENT AND AN EVALUATION OF ITS CAPABILITIES

The development of the Alcohol Treatment Profile System (ATPS) was described and an evaluation of its perceived value by various States was undertaken, The ATPS is a treatment needs assessment tool based on the unification of several large national epidemiologic and treatment data sets. It was developed by the National Institute on Alcohol Abuse and Alcoholism (NIAAA) and responsibility for its creation was given to the NIAAA's Alcohol Epidemiologic Data System (AEDS). The ATPS merges county-level measures of alcohol problem prevalence (the specially constructed AEDS Alcohol Problem Indicators), indicating "need" for treatment, and treatment utilization measures (the National Drug and Alcohol Treatment Utilization Survey), indicating treatment "demand." The capabilities of the ATPS in the unique planning and policy-making settings of several States were evaluated.^

The Need for Balanced Methods of Measuring Performance

Michelle Rhee weighs in on measuring success in pubic education.

Measuring Success in Public Education: Whether Wanted or Not

Robert Sanborn and Angelo Giardino's introduction for Volume 3, Issue 2: Measuring Success in Public Education.

Measuring Service Effectiveness for Families

While most professionals do not dispute the fact that evaluation is necessary to determine whether agencies and practitioners are truly providing services that meet clients’ needs, information regarding consistent measures on service effectiveness in human service organizations is sparse. A national survey of 250 not-for-profit family service organizations in the United States (52.8% return rate) yielded results relevant to client identified needs and agency effectiveness measures in serving today’s families. On an open-ended survey item, 52.3% agencies indicated that poverty represented the most pressing problem among today’s families because other psychological needs also take priority. Over two thirds of these agencies used multiple methods to evaluate their services. Clients’ feedback and outcome measures are the most popular methods. The findings reveal agencies' difficulties in determining what or who decides if the most appropriate services are being provided for the target population. Limited data collected on outcomes and impact may impose additional difficulties in program design and planning.

Measuring value in cancer-related health care: A case study in head and neck cancer

In the current climate of escalating health care costs, defining value and accurately measuring it are two critical issues affecting not only the future of cancer care in particular but also the future of health care in general. Specifically, measuring and improving value in cancer-related health care are critical for continued advancements in research, management, and overall delivery of care. However, in oncology, most of this research has focused on value as it relates to insurance industry and payment reform, with little attention paid to value as the output of clinical interventions that encompass integrated clinical teams focusing on the entire cycle of care and measuring objective outcomes that are most relevant to patients. ^ In this study, patient-centered value was defined as health outcomes achieved per dollar spent, and calculated using objective functional outcomes and total care costs. The analytic sample comprised patients diagnosed with three common head and neck cancers—cancer of the larynx, oral cavity, and oropharynx—who were treated in an integrated tertiary care center over an approximately 10-year period. The results of this study provide initial empirical data that can be used to assess and ultimately to help improve the quality and value of head and neck cancer care, and more importantly they can be used by patients and clinicians to make better-informed decisions about care, particularly what therapeutic services and outcomes matter the most to patients.^