The Comprehensive Assessment of Leadership for Learning: A Next Generation Formative Evaluation and Feedback System

Recent developments in federal policy have prompted the creation of state evaluation frameworks for principals and teachers that hold educators accountable for effective practices and student outcomes. These changes have created a demand for formative evaluation instruments that reflect current accountability pressures and can be used by schools to focus school improvement and leadership development efforts. The Comprehensive Assessment of Leadership for Learning (CALL) is a next generation, 360-degree on-line assessment and feedback system that reflect best practices in feedback design. Some unique characteristics of CALL include a focus on: leadership distributed throughout the school rather than as carried out by an individual leader; assessment of leadership tasks rather than perceptions of leadership practice; a focus on larger complex systems of middle and high school; and transparency of assessment design. This paper describes research contributing to the design and validation of the CALL survey instrument.

Idiographic Self-Monitoring Instruments to Empower Client Participation and Evaluate Outcome in Intensive Family Preservation Services

Intensive Family Preservation Services seek to reflect the values of focusing on client strengths and viewing clients as colleagues. To promote those values, Intensive Family Preservation Programs should include a systematic form of client self monitoring in their packages of outcome measures. This paper presents a model of idiographic self-monitoring used in time series, single system research design developed for Family Partners, a family preservation program of the School for Contemporary Education in Annandale, Virginia. The evaluation model provides a means of empowering client families to utilize their strengths and promote their status as colleague in determining their own goals, participating in the change process, and measuring their own progress.

AN OCCUPATIONAL INJURY MANAGEMENT INFORMATION SYSTEM FOR THE UNIVERSITY OF TEXAS HEALTH SCIENCE CENTER AT HOUSTON

The purpose of this research and development project was to develop a method, a design, and a prototype for gathering, managing, and presenting data about occupational injuries.^ State-of-the-art systems analysis and design methodologies were applied to the long standing problem in the field of occupational safety and health of processing workplace injuries data into information for safety and health program management as well as preliminary research about accident etiologies. The top-down planning and bottom-up implementation approach was utilized to design an occupational injury management information system. A description of a managerial control system and a comprehensive system to integrate safety and health program management was provided.^ The project showed that current management information systems (MIS) theory and methods could be applied successfully to the problems of employee injury surveillance and control program performance evaluation. The model developed in the first section was applied at The University of Texas Health Science Center at Houston (UTHSCH).^ The system in current use at the UTHSCH was described and evaluated, and a prototype was developed for the UTHSCH. The prototype incorporated procedures for collecting, storing, and retrieving records of injuries and the procedures necessary to prepare reports, analyses, and graphics for management in the Health Science Center. Examples of reports, analyses, and graphics presenting UTHSCH and computer generated data were included.^ It was concluded that a pilot test of this MIS should be implemented and evaluated at the UTHSCH and other settings. Further research and development efforts for the total safety and health management information systems, control systems, component systems, and variable selection should be pursued. Finally, integration of the safety and health program MIS into the comprehensive or executive MIS was recommended. ^

A MANAGEMENT INFORMATION SYSTEM AND EVALUATION SCHEME FOR THE UNIVERSITY OF TEXAS EMPLOYEE ASSISTANCE PROGRAMS

Early Employee Assistance Programs (EAPs) had their origin in humanitarian motives, and there was little concern for their cost/benefit ratios; however, as some programs began accumulating data and analyzing it over time, even with single variables such as absenteeism, it became apparent that the humanitarian reasons for a program could be reinforced by cost savings particularly when the existence of the program was subject to justification.^ Today there is general agreement that cost/benefit analyses of EAPs are desirable, but the specific models for such analyses, particularly those making use of sophisticated but simple computer based data management systems, are few.^ The purpose of this research and development project was to develop a method, a design, and a prototype for gathering managing and presenting information about EAPS. This scheme provides information retrieval and analyses relevant to such aspects of EAP operations as: (1) EAP personnel activities, (2) Supervisory training effectiveness, (3) Client population demographics, (4) Assessment and Referral Effectiveness, (5) Treatment network efficacy, (6) Economic worth of the EAP.^ This scheme has been implemented and made operational at The University of Texas Employee Assistance Programs for more than three years.^ Application of the scheme in the various programs has defined certain variables which remained necessary in all programs. Depending on the degree of aggressiveness for data acquisition maintained by program personnel, other program specific variables are also defined. ^

The Effect of Proximity, Explicitness, and Representation of Basic Science Information on Student Clinical Problem-Solving

Problem: Medical and veterinary students memorize facts but then have difficulty applying those facts in clinical problem solving. Cognitive engineering research suggests that the inability of medical and veterinary students to infer concepts from facts may be due in part to specific features of how information is represented and organized in educational materials. First, physical separation of pieces of information may increase the cognitive load on the student. Second, information that is necessary but not explicitly stated may also contribute to the student’s cognitive load. Finally, the types of representations – textual or graphical – may also support or hinder the student’s learning process. This may explain why students have difficulty applying biomedical facts in clinical problem solving. Purpose: To test the hypothesis that three specific aspects of expository text – the patial distance between the facts needed to infer a rule, the explicitness of information, and the format of representation – affected the ability of students to solve clinical problems. Setting: The study was conducted in the parasitology laboratory of a college of veterinary medicine in Texas. Sample: The study subjects were a convenience sample consisting of 132 second-year veterinary students who matriculated in 2007. The age of this class upon admission ranged from 20-52, and the gender makeup of this class consisted of approximately 75% females and 25% males. Results: No statistically significant difference in student ability to solve clinical problems was found when relevant facts were placed in proximity, nor when an explicit rule was stated. Further, no statistically significant difference in student ability to solve clinical problems was found when students were given different representations of material, including tables and concept maps. Findings: The findings from this study indicate that the three properties investigated – proximity, explicitness, and representation – had no statistically significant effect on student learning as it relates to clinical problem-solving ability. However, ad hoc observations as well as findings from other researchers suggest that the subjects were probably using rote learning techniques such as memorization, and therefore were not attempting to infer relationships from the factual material in the interventions, unless they were specifically prompted to look for patterns. A serendipitous finding unrelated to the study hypothesis was that those subjects who correctly answered questions regarding functional (non-morphologic) properties, such as mode of transmission and intermediate host, at the family taxonomic level were significantly more likely to correctly answer clinical case scenarios than were subjects who did not correctly answer questions regarding functional properties. These findings suggest a strong relationship (p < .001) between well-organized knowledge of taxonomic functional properties and clinical problem solving ability. Recommendations: Further study should be undertaken investigating the relationship between knowledge of functional taxonomic properties and clinical problem solving ability. In addition, the effect of prompting students to look for patterns in instructional material, followed by the effect of factors that affect cognitive load such as proximity, explicitness, and representation, should be explored.

A geographic information science (GIS) approach to conducting comparative area-level community assessments in the Texas CORD Study

This study assessed and compared sociodemographic and income characteristics along with food and physical activity assets (i.e. grocery stores, fast food restaurants, and park areas) in the Texas Childhood Obesity Research Demonstration (CORD) Study intervention and comparison catchment areas in Houston and Austin, Texas. The Texas CORD Study used a quasi-experimental study design, so it is necessary to establish the interval validity of the study characteristics by confirming that the intervention and comparison catchment areas are statistically comparable. In this ecological study, ArcGIS and Esri Business Analyst were used to spatially relate U.S. Census Bureau and other business listing data to the specific school attendance zones within the catchment areas. T-tests were used to compare percentages of sociodemographic and income characteristics and densities of food and physical activity assets between the intervention and comparison catchment areas.^ Only five variables were found to have significant differences between the intervention and comparison catchment areas: Age groups 0-4 and 35-64, the percentage of owner-occupied and renter-occupied households, and the percentage of Asian and Pacific Islander residents. All other variables showed no significant differences between the two groups. This study shows that the methodology used to select intervention and comparison catchment areas for the Texas CORD Study was effective and can be used in future studies. The results of this study can be used in future Texas CORD studies to confirm the comparability of the intervention and comparison catchment areas. In addition, this study demonstrates a methodology for describing detailed characteristics about a geographic area that practitioners, researchers, and educators can use.^

The Houston Academy of Medicine--Texas Medical Center Library management information system.

A management information system (MIS) provides a means for collecting, reporting, and analyzing data from all segments of an organization. Such systems are common in business but rare in libraries. The Houston Academy of Medicine-Texas Medical Center Library developed an MIS that operates on a system of networked IBM PCs and Paradox, a commercial database software package. The data collected in the system include monthly reports, client profile information, and data collected at the time of service requests. The MIS assists with enforcement of library policies, ensures that correct information is recorded, and provides reports for library managers. It also can be used to help answer a variety of ad hoc questions. Future plans call for the development of an MIS that could be adapted to other libraries' needs, and a decision-support interface that would facilitate access to the data contained in the MIS databases.

Clinical decision support capabilities of commercially-available clinical information systems.

BACKGROUND: The most effective decision support systems are integrated with clinical information systems, such as inpatient and outpatient electronic health records (EHRs) and computerized provider order entry (CPOE) systems. Purpose The goal of this project was to describe and quantify the results of a study of decision support capabilities in Certification Commission for Health Information Technology (CCHIT) certified electronic health record systems. METHODS: The authors conducted a series of interviews with representatives of nine commercially available clinical information systems, evaluating their capabilities against 42 different clinical decision support features. RESULTS: Six of the nine reviewed systems offered all the applicable event-driven, action-oriented, real-time clinical decision support triggers required for initiating clinical decision support interventions. Five of the nine systems could access all the patient-specific data items identified as necessary. Six of the nine systems supported all the intervention types identified as necessary to allow clinical information systems to tailor their interventions based on the severity of the clinical situation and the user's workflow. Only one system supported all the offered choices identified as key to allowing physicians to take action directly from within the alert. Discussion The principal finding relates to system-by-system variability. The best system in our analysis had only a single missing feature (from 42 total) while the worst had eighteen.This dramatic variability in CDS capability among commercially available systems was unexpected and is a cause for concern. CONCLUSIONS: These findings have implications for four distinct constituencies: purchasers of clinical information systems, developers of clinical decision support, vendors of clinical information systems and certification bodies.

Design and implementation of a multicriteria medical decision support system for diagnosis and treatment

Health care providers face the problem of trying to make decisions with inadequate information and also with an overload of (often contradictory) information. Physicians often choose treatment long before they know which disease is present. Indeed, uncertainty is intrinsic to the practice of medicine. Decision analysis can help physicians structure and work through a medical decision problem, and can provide reassurance that decisions are rational and consistent with the beliefs and preferences of other physicians and patients. ^ The primary purpose of this research project is to develop the theory, methods, techniques and tools necessary for designing and implementing a system to support solving medical decision problems. A case study involving “abdominal pain” serves as a prototype for implementing the system. The research, however, focuses on a generic class of problems and aims at covering theoretical as well as practical aspects of the system developed. ^ The main contributions of this research are: (1) bridging the gap between the statistical approach and the knowledge-based (expert) approach to medical decision making; (2) linking a collection of methods, techniques and tools together to allow for the design of a medical decision support system, based on a framework that involves the Analytic Network Process (ANP), the generalization of the Analytic Hierarchy Process (AHP) to dependence and feedback, for problems involving diagnosis and treatment; (3) enhancing the representation and manipulation of uncertainty in the ANP framework by incorporating group consensus weights; and (4) developing a computer program to assist in the implementation of the system. ^

INFLUENCE OF A CLINIC BASED IMMUNIZATION INFORMATION SYSTEM ON INFANT IMMUNIZATIONS

This study focuses on the impact of a clinic-based intervention program on the immunization status of limited-income urban children. The intervention program consisted of an information session for clinic health care providers and the placement of individualized immunization information labels on clinic notes at the time of each visit. The degree of impact of the intervention on immunization administration was ascertained through a comparison of two similar groups of infants born in the same months of the year immediately before (N = 201) and after (N = 203) the information session and initiation of the labeling system. The timeliness of administration of each diphtheria, pertussis, tetanus and trivalent oral polio vaccine (DPT/TOPV) in the first year series of three was compared pre- to postintervention. Significantly more third immunizations were given the postintervention subjects within ten days of the recommended time of application ( p = .0361). Life table analysis indicated that the probability of an infant's passing one year of age without the administration of the third immunization decreased for postintervention infants (p = .0515). The intervention was most successful in assuring administration of the series of immunizations in those infants who were seen by the health care provider for at least 50% of their first year visits. Results indicate that minor changes in the format of information given a relatively continuous provider can increase completion of immunization series in infants. ^

Atlas of cancer mortality in the United Arab Emirates aided by application of geographic information system methods

The aim of this study was to determine cancer mortality rates for the United Arab Emirates (UAE) and to create an atlas of cancer mortality for the UAE. This atlas is the first of its kind in the Gulf country and the Middle East. Death certificates were reviewed for a period from January 1, 1990 to December 31, 1999 and cancer deaths were identified. Cancer mortality cases were verified by comparing with medical records. Age-adjusted cancer mortality rates were calculated by gender, emirate/medical district and nationality (UAE nationals and overall UAE population). Individual rates for each emirate were compared to the overall rate of the corresponding population for the same cancer site and gender. Age-adjusted rates were mapped using MapInfo software. High rates for liver, lung and stomach cancer were observed in Abu Dhabi, Dubai and the northern emirates, respectively. Rates for UAE nationals were greater compared to the overall UAE population. Several factors were suggested that may account for high rates of specific cancers observed in certain emirates. It is hoped that this atlas will provide leads that will guide further epidemiologic and public health activities aimed at preventing cancer. ^

Continuous safety screens for randomized controlled clinical trials with blinded treatment information

Standard methods for testing safety data are needed to ensure the safe conduct of clinical trials. In particular, objective rules for reliably identifying unsafe treatments need to be put into place to help protect patients from unnecessary harm. DMCs are uniquely qualified to evaluate accumulating unblinded data and make recommendations about the continuing safe conduct of a trial. However, it is the trial leadership who must make the tough ethical decision about stopping a trial, and they could benefit from objective statistical rules that help them judge the strength of evidence contained in the blinded data. We design early stopping rules for harm that act as continuous safety screens for randomized controlled clinical trials with blinded treatment information, which could be used by anyone, including trial investigators (and trial leadership). A Bayesian framework, with emphasis on the likelihood function, is used to allow for continuous monitoring without adjusting for multiple comparisons. Close collaboration between the statistician and the clinical investigators will be needed in order to design safety screens with good operating characteristics. Though the math underlying this procedure may be computationally intensive, implementation of the statistical rules will be easy and the continuous screening provided will give suitably early warning when real problems were to emerge. Trial investigators and trial leadership need these safety screens to help them to effectively monitor the ongoing safe conduct of clinical trials with blinded data.^

Cost analysis of opt-in consent procedures for the Texas statewide immunization information system, ImmTrac

Background. Childhood immunization programs have dramatically reduced the morbidity and mortality associated with vaccine-preventable diseases. Proper documentation of immunizations that have been administered is essential to prevent duplicate immunization of children. To help improve documentation, immunization information systems (IISs) have been developed. IISs are comprehensive repositories of immunization information for children residing within a geographic region. The two models for participation in an IIS are voluntary inclusion, or "opt-in," and voluntary exclusion, or "opt-out." In an opt-in system, consent must be obtained for each participant, conversely, in an opt-out IIS, all children are included unless procedures to exclude the child are completed. Consent requirements for participation vary by state; the Texas IIS, ImmTrac, is an opt-in system.^ Objectives. The specific objectives are to: (1) Evaluate the variance among the time and costs associated with collecting ImmTrac consent at public and private birthing hospitals in the Greater Houston area; (2) Estimate the total costs associated with collecting ImmTrac consent at selected public and private birthing hospitals in the Greater Houston area; (3) Describe the alternative opt-out process for collecting ImmTrac consent at birth and discuss the associated cost savings relative to an opt-in system.^ Methods. Existing time-motion studies (n=281) conducted between October, 2006 and August, 2007 at 8 birthing hospitals in the Greater Houston area were used to assess the time and costs associated with obtaining ImmTrac consent at birth. All data analyzed are deidentified and contain no personal information. Variations in time and costs at each location were assessed and total costs per child and costs per year were estimated. The cost of an alternative opt-out system was also calculated.^ Results. The median time required by birth registrars to complete consent procedures varied from 72-285 seconds per child. The annual costs associated with obtaining consent for 388,285 newborns in ImmTrac's opt-in consent process were estimated at $702,000. The corresponding costs of the proposed opt-out system were estimated to total $194,000 per year. ^ Conclusions. Substantial variation in the time and costs associated with completion of ImmTrac consent procedures were observed. Changing to an opt-out system for participation could represent significant cost savings. ^