Measuring Service Effectiveness for Families

While most professionals do not dispute the fact that evaluation is necessary to determine whether agencies and practitioners are truly providing services that meet clients’ needs, information regarding consistent measures on service effectiveness in human service organizations is sparse. A national survey of 250 not-for-profit family service organizations in the United States (52.8% return rate) yielded results relevant to client identified needs and agency effectiveness measures in serving today’s families. On an open-ended survey item, 52.3% agencies indicated that poverty represented the most pressing problem among today’s families because other psychological needs also take priority. Over two thirds of these agencies used multiple methods to evaluate their services. Clients’ feedback and outcome measures are the most popular methods. The findings reveal agencies' difficulties in determining what or who decides if the most appropriate services are being provided for the target population. Limited data collected on outcomes and impact may impose additional difficulties in program design and planning.

The Effectiveness of Court Mandated Intervention Versus Voluntary Services in Child Protective Services:

The general objective of this research was to compare the relative effectiveness of court mandated services versus a voluntary service plan in preventing in child maltreatment recidivism. Four-thirty-two children were selected at random from among children in a large California County who were receiving in-home services under a court mandate or a voluntary plan. Protective services files of study children were reviewed to derive study data. Type of plan did not make a difference on case outcome. Children were more likely to remain in the home at the end of the service delivery period in families that received voluntary plans. However, when other factors are controlled, the advantage of a voluntary plan disappears. Moreover, similar rates of recidivism were noted between both types of plans after the case was closed.

The effectiveness of a needleless intravenous system in prevention of percutaneous injury in two hospitals

This study assessed if hospital-wide implementation of a needleless intravenous connection system reduces the number of reported percutaneous injuries, overall and those specifically due to intravenous connection activities.^ Incidence rates were compared before and after hospital-wide implementation of a needleless intravenous system at two hospitals, a full service general hospital and a pediatric hospital. The years 1989-1991 were designated as pre-implementation and 1993 was designated as post-implementation. Data from 1992 were not included in the effectiveness evaluation to allow employees to become familiar with use of the new device. The two hospitals showed rate ratios of 1.37 (95% CI = 1.22-1.54, p $\le$.0001) and 1.63 (95% CI = 1.34-1.97, p $\le$.0001), or a 27.1% and a 38.6% reduction in overall injury rate, respectively. Rate ratios for intravenous connection injuries were 2.67 (95% CI = 1.89-3.78, p $\le$.0001) and 3.35 (95% CI = 1.87-6.02, p $\le$.0001), or a 62.5% and a 69.9% reduction in injury rate, respectively. Rate ratios for all non-intravenous connection injuries were calculated to control for factors other than device implementation that may have been operating to reduce the injury rate. These rate ratios were lower, 1.21 and 1.44, demonstrating the magnitude of injury reduction due to factors other than device implementation. It was concluded that the device was effective in reduction of numbers of reported percutaneous injuries.^ Use-effectiveness of the system was also assessed by a survey of randomly selected device users to determine satisfaction with the device, frequency of use and barriers to use. Four hundred seventy-eight surveys were returned for a response rate of 50.9%. Approximately 94% of respondents at both hospitals expressed satisfaction with the needleless system and recommended continued use. The survey also revealed that even though over 50% of respondents report using the device "always" or "most of the time" for intravenous medication administration, flushing lines, and connecting secondary intravenous lines, needles were still being used for these same activities. Compatibility, accessibility and other technical problems were reported as reasons for using needles for these activities. These problems must be addressed, by both manufacturers and users, before the needleless system will be effective in prevention of all intravenous connection injuries. ^

PROJECTION OF HEALTH DEMAND IN TERMS OF HOSPITAL ADMISSIONS FOR SELECTED DIAGNOSES IN HEALTH SYSTEMS AGENCY SERVICE AREAS

The National Health Planning and Resources Development Act of 1974 (Public Law 93-641) requires that health systems agencies (HSAs) plan for their health service areas by the use of existing data to the maximum extent practicable. Health planning is based on the identificaton of health needs; however, HSAs are, at present, identifying health needs in their service areas in some approximate terms. This lack of specificity has greatly reduced the effectiveness of health planning. The intent of this study is, therefore, to explore the feasibility of predicting community levels of hospitalized morbidity by diagnosis by the use of existing data so as to allow health planners to plan for the services associated with specific diagnoses.^ The specific objectives of this study are (a) to obtain by means of multiple regression analysis a prediction equation for hospital admission by diagnosis, i.e., select the variables that are related to demand for hospital admissions; (b) to examine how pertinent the variables selected are; and (c) to see if each equation obtained predicts well for health service areas.^ The existing data on hospital admissions by diagnosis are those collected from the National Hospital Discharge Surveys, and are available in a form aggregated to the nine census divisions. When the equations established with such data are applied to local health service areas for prediction, the application is subject to the criticism of the theory of ecological fallacy. Since HSAs have to rely on the availability of existing data, it is imperative to examine whether or not the theory of ecological fallacy holds true in this case.^ The results of the study show that the equations established are highly significant and the independent variables in the equations explain the variation in the demand for hospital admission well. The predictability of these equations is good when they are applied to areas at the same ecological level but become poor, predominantly due to ecological fallacy, when they are applied to health service areas.^ It is concluded that HSAs can not predict hospital admissions by diagnosis without primary data collection as discouraged by Public Law 93-641. ^

Effectiveness of Community Case Management in Family Risk Reduction

This study evaluated a modified home-based model of family preservation services, the long-term community case management model, as operationalized by a private child welfare agency that serves as the last resort for hard-to-serve families with children at severe risk of out-of-home placement. The evaluation used a One-Group Pretest-Posttest design with a modified time-series design to determine if the intervention would produce a change over time in the composite score of each family's Child Well-Being Scales (CWBS). A comparison of the mean CWBS scores of the 208 families and subsets of these families at the pretest and various posttests showed a statistically significant decrease in the CWBS scores, indicating decreased risk factors. The longer the duration of services, the greater the statistically significant risk reduction. The results support the conclusion that the families who participate in empowerment-oriented community case management, with the option to extend service duration to resolve or ameliorate chronic family problems, have experienced effective strengthening in family functioning.

Learning What Works: Demonstrating Practice Effectiveness with Children and Families Through Retrospective Investigation

Intensive family preservation services (IFPS), designed to stabilize at-risk families and avert out-of-home care, have been the focus of many randomized, experimental studies. Employing a retrospective “clinical data-mining” (CDM) methodology (Epstein, 2001), this study makes use of available information extracted from client records in one IFPS agency over the course of two years. The primary goal of this descriptive and associational study was to gain a clearer understanding of IFPS service delivery and effectiveness. Interventions provided to families are delineated and assessed for their impact on improved family functioning, their impact on the reduction of family violence, as well as placement prevention. Findings confirm the use of a wide range of services consistent with IFPS program theory. Because the study employs a quasi-experimental, retrospective use of available information, clinical outcomes described cannot be causally attributed to interventions employed as with randomized controlled trials. With regard to service outcomes, findings suggest that family education, empowerment services and advocacy are most influential in placement prevention and in ameliorating unmanageable behaviors in children as well as the incidence of family violence.

Family Preservation: Perceptions of Effectiveness

This qualitative study examines the attributes or perceptions of service providers and overseers as to the effectiveness of intensive family preservation services provided by a social services agency in Tucson, Arizona. The services provided are patterned after the Homebuilders' model developed in 1974 in Tacoma, Washington. Data collection was generated from interviews and focus groups with the in-home service providers, the program supervisor, and investigators and case managers with Child Protective Services (CPS). Although placement prevention rates (PPR) are the dependent variable in most studies on this form of intervention, this study seeks to understand those characteristics of the model that contribute to successful outcomes with client families. Those appear to be the short-term intervention coupled with a non-judgmental approach to client families and the clinical supervision provided by the program supervisor.

Cost -effectiveness of services and treatments for ambulatory dually diagnosed CMHC clients: Implications for managed care

This paper reports a cost-effectiveness analysis of standard therapeutic interventions received by ambulatory dually diagnosed clients of a Community Mental Health Center (CMHC). For the purposes of this study dually diagnosed was defined as a DSM-III-R or IV diagnosis of a major mental disorder and a concomitant substance abuse disorder. The prevalence of dually diagnosed people among the mentally ill and their unique and problematic nature continues to challenge and encumber CMHCs and poses grave public health risks. An absence of research on these clients in community-based settings and the cost-effectiveness of their standard CMHC care has hindered the development of effective community-based intervention strategies. This exploratory and descriptive effort is a first step toward providing information on which to base programmatic management decisions.^ Data for this study were derived from electronic client records of a CMHC located in a large Southwestern, Sun-belt metropolitan area. A total of 220 records were collected on clients consecutively admitted during a two-and-one-half year period. Information was gathered profiling the clients' background characteristics, receipt of standard services and treatments, costs of the care they received, and length of CMHC enrollment and subsequent psychiatric hospitalizations. The services and treatments were compared with regard to their costs and predicted contributions toward maintaining clients in the community and out of public psychiatric hospitals.^ This study investigated: (1) the study groups' background, mental illness, and substance abuse characteristics; (2) types, extent, and patterns of their receipt of standard services and treatments; (3) associations between the receipt of services and treatments, community tenure, and risk of psychiatric hospitalization; and, (4) comparisons of average costs for services and treatments in terms of their contributions toward maintaining the clients in the community.^ The results suggest that substance abuse and other lifestyle factors were related to the dually diagnosed clients' admissions to the CMHC. The dually diagnosed clients' receipt of care was associated strongly with their insurability and global functioning. Medication Services were the most expensive yet effective service or treatment. Supported Education was the third most expensive and second most effective. Psychosocial Services, the second most expensive, were only effective in terms of maintaining clients in the community. Group Counseling, the fourth most expensive, had no effect on community maintenance and increased the risk of hospitalization when accompanied by Medication Services. Individual Counseling, the least expensive, had no effect on community maintenance. But it reduced the risk of hospitalization when accompanied by Medication Services. Networking/Referral, the fifth most expensive service or treatment, was ineffective.^ The study compared the results with findings in the literature. Implications are discussed regarding further research, study limitations, practical applications and benefits, and improvements to theoretical understandings, in particular, concepts underscoring Managed Care. ^