A comparison of labor and delivery management activities in two settings

Similarities and differences in management activities and patient health outcomes between a traditional physician staffed labor and delivery setting and a certified nurse-midwife staffed Birth Center within the same hospital were described. The 950 study subjects, low income, minority women, were classified as low obstetrical risk by a POPRAS score of 25 points or less at time of admission for labor and delivery. The study subjects were similar in demographic, antepartum and intrapartum characteristics; the labor course was problem free for the majority in both settings. There were no remarkable differences in health outcomes between the groups. Management activities varied between settings; these variations were policy related rather than health related. The POPRAS rating system was an accurate predictor for 93% of BC subjects and 85% of LDU subjects. Charge for service was approximately $600 less for BC women; length of stay did not contribute to the difference in charge. Overall, BC respondents to the attitude survey were more satisfied with their labor and delivery experience than L\&DU women. ^

Use of empowerment evaluation to determine a sustainability plan for a community-based preconception care program

Women With IMPACT (WWI) is a community-based preconception care educational intervention. WWI is being implemented by the Impacting Maternal and Prenatal Care Together (IMPACT) Collaborative and targets zip codes in Harris County, Texas at high risk for infant mortality, low birthweight, and preterm birth. WWI started March 2012 and continues through August 2013. Three workshop series are planned. This study was conducted with participants and facilitators from the first workshop series. This study aimed to 1) evaluate the WWI program using empowerment evaluation, 2) engage all WWI stakeholders in an empowerment evaluation so the method could be adopted as a participatory evaluation process for future IMPACT activities, and 3) develop recommendations for sustainability of the WWI intervention, based on empowerment evaluation findings and results from the pre/post program evaluation completed by WWI participants. Study participants included WWI participants and facilitators and IMPACT Collaborative Steering Committee members. WWI participants were female, 18-35 year-old, non-pregnant residents of zip codes at high risk of adverse birth outcomes. All other study participants were 18 years or older. A two-phased empowerment evaluation (EE) was utilized in this study. Sessions 1-4 were conducted independently of one another – 3 with participants at different sites and one with the facilitators. The fifth session included WWI participant and facilitator representatives, and IMPACT Steering Committee members. Session 5 built upon the work of the other sessions. Observation notes were recorded during each session. Thematic content analysis was conducted on all EE tables and observation notes. Mission statements drafted by each group focused on improvement of physical and mental health through behavior change and empowerment of all participants. The top 5 overall program components were: physical activity, nutrition, self-worth, in-class communication, and stress. Goals for program improvement were set by EE participants for each of these components. Through thematic content analysis of the tables and observation notes, social support emerged as an important theme of the program among all participant groups. Change to a healthy lifestyle emerged as an important theme in terms of program improvement. Two-phased EE provided an opportunity for all program stakeholders to provide feedback regarding important program components and provide suggestions for program improvement. EE, thematic content analysis, pre/post evaluation results, and inherent program knowledge were triangulated to make recommendations to sustain the program once the initial funding ends. ^

An evaluation of the personal response system program for the elderly in Harris County, Texas

The Personal Response System Program at Huffington Center on Aging, Baylor College of Medicine, provides emergency call systems for elderly people living independently in Houston, Texas. The goal of the project was to complete a formative evaluation of the Personal Response System Program. The specific aims of the evaluation were three-fold. One aim was to evaluate participant health status and level of disability. The second aim was to develop a health care cost estimation strategy. Finally, a preliminary cost-effectiveness analysis was completed to evaluate the program's impact on health care costs and health status among the elderly target population. ^ The evaluation was a longitudinal, randomized experimental design. After the screening of 120 volunteers for eligibility, clients were asked to complete a written questionnaire and a monthly health service contact diary. Volunteers were contacted by telephone interviewers to collect health status information from 100 eligible clients (83%) on three occasions during the six months of follow-up. ^ Initially, volunteers were randomized to two experimental groups. The two groups were found to be comparable at the beginning of the study. No significant differences were detected related to health status, level of disability, or history of physician visits at baseline. However, the group with the Personal Response System (PRS) device had more adverse health events, higher IADL scores, more frequent use of walkers, lower average health status scores, and fewer community volunteers hours than the usual care comparison group. ^ The health care costs were estimated based on an algorithm adapted from the American Medical Association guidelines. Average total health care costs for the group with the PRS device ($912) were greater than the usual care group ($464). However, median health care values for the PRS group ($263) were similar to the usual care comparison group ($234). The preliminary findings indicated that the use of the PRS device was not associated with health care cost savings. ^ In the preliminary cost-effectiveness analysis, use of the personal response system was found to be associated with increased mental health status among elderly clients. The cost-effectiveness evaluation indicated that the associated cost for six months was $710 per unit increase in mental component score when the $150 PRS subscription was included. However, clients with the PRS device were found to have a greater decline in physical health status during the six-month follow-up. The beneficial effect on mental health status was found to be in contrast to negative findings associated with changes in physical health status. The implications for future research relate to the need to identify risk factors among geriatric populations to better target groups that would most likely benefit from PRS Program enrollment. ^

An evaluation of Employee Assistance Programs: An overview and case study

Methodologies for the analysis and cross-evaluations of Employee Assistance Programs (EAPs) are necessary to indicate program weaknesses and strengths for both present and future EAP services.^ As EAPs have matured from simplistic industrial alcoholism programs to all problem-encompassing, complex programs, an aggressive group of entrepreneurs has developed. Business and industrial managements need established criteria to select the most suitable professional services for their needs, as well as the best investment value for cost containments.^ Four major EAPs were compared to determine critical areas necessary for cross evaluation. Future standardization of terminology and definitions of services will facilitate both the evaluation of a specific program and its comparison to problems or successes of other EAPs.^ Past evaluation difficulties have been caused by the lack of EAP program accreditations and professional personnel licensure requirements. The Association of Labor Management Administrators and Consultants on Alcoholism (ALMACA) has recently set certification criteria for personnel, which will enhance the professional standards for EAPs. ^

Evaluation and Model of a Clinical Librarian Program

Clinical medical librarianship is entering its second decade, but little evaluative data has accrued in the literature. Variations from the original programs and novel new approaches have insured the survival of the program so far. The clinical librarian (CL) forms a vital link between the library and the health care professional, operating as an important information transfer agent. However, to further insure the survival of these vital programs, hard evaluative evidence is needed. The University of Texas Medical Branch (UTMB) at Galveston began a CL Program in 1978/79. An extensive three-year pre/post evaluation study was conducted using a specifically developed evaluation model, which, if adopted by others, will provide the needed comparative data. Both a pilot study, or formative evaluation, and a summative evaluation were conducted. The results of this evaluation confirmed many of the conclusions reported by other CL programs. Eight hypotheses were proposed at the beginning of this study. Data were collected and used to support acceptance or rejection of the null hypotheses, and conclusions were drawn according to the results. Implications relevant to the study conclusions and future trends in medical librarianship are also discussed in the closing chapter.

Clinical decisions related to CBC screening of asymptomatic full-term infants at risk for group B streptococcus disease

Objective. Although complete blood count (CBC) changes occur with the development of clinical sepsis in newborns, the CBC has not been reported to be a sensitive predictor of sepsis in asymptomatic full-term newborn infants, nor has it been reported to be related to risk factors for sepsis or clinical decisions. The objective of this study was to evaluate the relationship between the WBC/I:T (immature:total neutrophil) ratio and maternal group B streptococcal (GBS) risk factors (rupture of membranes ≥18 hours, maternal temperature ≥100.4°F, maternal age ≤20 years, previous infant with invasive GBS disease, maternal GBS bacteriuria, and black ethnicity); and to evaluate the relationship between the WBC/I:T ratios and providers' clinical decisions (observe versus repeat the CBC or complete sepsis evaluation) in the asymptomatic full-term newborn at risk for early-onset GBS sepsis. ^ Methods. Medical records of infants admitted to the well baby nursery at a tertiary care teaching hospital in Houston, TX between 1/1/99 and 12/31/00 whose gestational ages were ≥35 weeks; who had mothers with GBS positive or unknown culture status and inadequate intrapartum antibiotic prophylaxis; and who had screening CBCs performed in the first 30 hours of life because of GBS risk were reviewed (n = 412). Demographic information, maternal GBS risk factors, CBC results, clinical decisions, and rationales for clinical decisions were collected. ^ Results. With the exception of black ethnicity (p = .0000, odds ratio = 0.213), no statistically significant differences in risk factors between infants with normal and abnormal WBC counts or normal and abnormal I:T ratios were found. Infants with abnormal WBCs had a significantly higher likelihood of having a CBC repeated (p = 0.002 for WBC). Providers documented the CBC result in the rationale for clinical decisions in 62% of the cases. ^ Conclusion. The CBC results were not related to maternal risk factors for GBS except for ethnicity. Black infants had significantly lower WBC levels than infants of other ethnicities, although this difference was clinically insignificant. Infants with abnormal WBCs had a significantly higher likelihood of undergoing repeat CBCs but not sepsis evaluations. Provider rationale was difficult to evaluate due to insufficient documentation. The screening CBC result did not impact the clinicians' decisions to initiate sepsis evaluations in this population. ^

The School Sealant Program: An evaluation of the oral health education component

Dental caries is a common preventable childhood disease leading to severe physical, mental and economic repercussions for children and their families if left untreated. A needs assessment in Harris County reported that 45.9% of second graders had untreated dental caries. In order to address this growing problem, the School Sealant Program (SSP), a primary preventive initiative, was launched by the Houston Department of Health and Human Services (HDHHS) to provide oral health education, and underutilized dental preventive services to second grade children from participating Local School Districts (LSDs). ^ To determine the effectiveness and efficiency of the SSP, a program evaluation was conducted by the HDHHS between September 2007 and June 2008 for the Oral Health Education (OHE) component of the SSP. The objective of the evaluation was to assess short term changes in oral health knowledge of the participants and determine if these changes, if any, were due to the OHE sessions. An 8-item multiple choice pre/post test was developed for this purpose and administered to the participants before and immediately after the OHE sessions. ^ The present project analyzed pre and post test data of 1,088 second graders from 22 participating schools. Changes in overall and topic-specific knowledge of the program participants before and after the OHE sessions were analyzed using the Wilcoxon's signed rank test. ^ Results. The overall knowledge assessment showed a statistically significant (p <0.001) increase in the dental health knowledge of the participants after the oral health education sessions. Participants in the higher scoring category (7-8 correct responses) increased from 9.5% at baseline to 60.8% after the education sessions. Overall knowledge increased in all school regions with the highest knowledge gains seen in the Central and South regions. Males and females had similar knowledge gains. Significant knowledge differences were also found for each of the topic specific categories (functions of teeth, healthy diet, healthy habits, dental sealants; p<0.001) indicating an increase in topic specific knowledge of the participants post-health education sessions. ^ Conclusions. The OHE sessions were successful in increasing the short term oral health knowledge of the participants. ^

A formative evaluation of the American Diabetes Association's Choosing Health and Sensible Exercise childhood obesity intervention

Background. The increasing prevalence of overweight among youth in the United States, and the parallel rise in related medical comorbidities has led to a growing need for efficient weight-management interventions. Purpose. The aim of this study was to evaluate the effects of the Choosing Health and Sensible Exercise (C.H.A.S.E.) childhood obesity prevention program on Body Mass Index (BMI), physical activity and dietary behaviors. Methods. This study utilized de-identified data collected during the fall 2006 session of the C.H.A.S.E. program. A total of 65 students at Woodview Elementary School and Deepwater Elementary School participated in this intervention. The C.H.A.S.E. program is a 10-week obesity prevention program that focuses on nutrition and physical activity education. Collection of height and weight data, and a health behavior survey was conducted during the first and last week of the intervention. Paired t-tests were used to determine statistically significant differences between pre- and post-intervention measurements. One-way analysis of variance was used to adjust for potential confounders, such as gender, age, BMI category ("normal weight", "at risk overweight", or "overweight"), and self-reported weight loss goals. Data were analyzed using STATA, v. 9.2. Results. A significant decrease in mean BMI (p< 0.05) was found after the 10-week intervention. While the results were statistically significant for the group as a whole, changes in BMI were not significant when stratified by age, sex, or ethnicity. The mean overall scores for the behavior survey did not change significantly pre- and post-intervention; however, significant differences were found in the dietary intention scale, indicating that students were more likely to intend to make healthier food choices (p<0.05). No statistically significant decreases in BMI were found when stratified for baseline BMI-for-age percentiles or baseline weight loss efforts (p>0.05). Conclusion. The results of this evaluation provide information that will be useful in planning and implementing an effective childhood obesity intervention in the future. Changes in the self-reported dietary intentions and BMI show that the C.H.A.S.E. program is capable of modifying food choice selection and decreasing BMI. Results from the behavior questionnaire indicate that students in the intervention program were making changes in a positive direction. Future implementation of the C.H.A.S.E. program, as well as other childhood obesity interventions, may want to consider incorporating additional strategies to increase knowledge and other behavioral constructs associated with decreased BMI. In addition, obesity prevention programs may want to increase parental involvement and increase the dose or intensity of the intervention. ^

An investigation into the relationship between screen time, consumption of advertised foods, and physical activity among Texas 4th grade elementary school children

Background. The high prevalence of obesity among children has spurred creation of a list of possible causative factors, including the advertising of foods of minimal nutritional value, a decrease in physical activity, and increased media use. Few studies show prevalence rates of these factors among large cohorts of children. ^ Methods. Using data from the 2004-2005 School Physical Activity and Nutrition project (SPAN), a secondary analysis of 7907 4th-grade children (mean age 9.74 years) was conducted. In addition, a comic-book–based intervention that addressed advertised food consumption, physical activity, and media use was developed and evaluated using a pre-post test design among 4th-grade children in an urban school district. ^ Results. Among a cohort of 4th-grade children across the state of Texas, children who had more than 2 hours of video game or computer time the previous day were more than twice as likely to drink soda and eat candy or pastries. In addition, children who watched more than 2 hours of TV the previous day were more than three times as likely to consume chips, punch, soda, candy, frozen desserts, or pastries (AOR 3.41, 95% CI: 1.58, 7.37). A comic-book based intervention held great promise and acceptance among 4th-grade children. Outcome evaluation showed that while results moved in a positive direction, they were not statistically significant. ^ Conclusion. Statistically significant associations were found between screen time and eating various types of advertised food. The comic book intervention was widely accepted by the children exposed to it, and pre-post surveys indicated they moved constructs in a positive direction. Further research is needed to look at more specific ways in which children are exposed to TV, and the relationship of the TV viewing time with their consumption of advertised foods. In addition, researchers should look at comic book interventions more closely and attempt to utilize them in more in studies with a longer follow-up time. ^

The role of workplace absence as an organizational mechanism in predicting employee alcohol and drug disorders

Background. EAP programs for airline pilots in companies with a well developed recovery management program are known to reduce pilot absenteeism following treatment. Given the costs and safety consequences to society, it is important to identify pilots who may be experiencing an AOD disorder to get them into treatment. ^ Hypotheses. This study investigated the predictive power of workplace absenteeism in identifying alcohol or drug disorders (AOD). The first hypothesis was that higher absenteeism in a 12-month period is associated with higher risk that an employee is experiencing AOD. The second hypothesis was that AOD treatment would reduce subsequent absence rates and the costs of replacing pilots on missed flights. ^ Methods. A case control design using eight years (time period) of monthly archival absence data (53,000 pay records) was conducted with a sample of (N = 76) employees having an AOD diagnosis (cases) matched 1:4 with (N = 304) non-diagnosed employees (controls) of the same profession and company (male commercial airline pilots). Cases and controls were matched on the variables age, rank and date of hire. Absence rate was defined as sick time hours used over the sum of the minimum guarantee pay hours annualized using the months the pilot worked for the year. Conditional logistic regression was used to determine if absence predicts employees experiencing an AOD disorder, starting 3 years prior to the cases receiving the AOD diagnosis. A repeated measures ANOVA, t tests and rate ratios (with 95% confidence intervals) were conducted to determine differences between cases and controls in absence usage for 3 years pre and 5 years post treatment. Mean replacement costs were calculated for sick leave usage 3 years pre and 5 years post treatment to estimate the cost of sick leave from the perspective of the company. ^ Results. Sick leave, as measured by absence rate, predicted the risk of being diagnosed with an AOD disorder (OR 1.10, 95% CI = 1.06, 1.15) during the 12 months prior to receiving the diagnosis. Mean absence rates for diagnosed employees increased over the three years before treatment, particularly in the year before treatment, whereas the controls’ did not (three years, x = 6.80 vs. 5.52; two years, x = 7.81 vs. 6.30, and one year, x = 11.00cases vs. 5.51controls. In the first year post treatment compared to the year prior to treatment, rate ratios indicated a significant (60%) post treatment reduction in absence rates (OR = 0.40, CI = 0.28, 0.57). Absence rates for cases remained lower than controls for the first three years after completion of treatment. Upon discharge from the FAA and company’s three year AOD monitoring program, case’s absence rates increased slightly during the fourth year (controls, x = 0.09, SD = 0.14, cases, x = 0.12, SD = 0.21). However, the following year, their mean absence rates were again below those of the controls (controls, x = 0.08, SD = 0.12, cases, x¯ = 0.06, SD = 0.07). Significant reductions in costs associated with replacing pilots calling in sick, were found to be 60% less, between the year of diagnosis for the cases and the first year after returning to work. A reduction in replacement costs continued over the next two years for the treated employees. ^ Conclusions. This research demonstrates the potential for workplace absences as an active organizational surveillance mechanism to assist managers and supervisors in identifying employees who may be experiencing or at risk of experiencing an alcohol/drug disorder. Currently, many workplaces use only performance problems and ignore the employee’s absence record. A referral to an EAP or alcohol/drug evaluation based on the employee’s absence/sick leave record as incorporated into company policy can provide another useful indicator that may also carry less stigma, thus reducing barriers to seeking help. This research also confirms two conclusions heretofore based only on cross-sectional studies: (1) higher absence rates are associated with employees experiencing an AOD disorder; (2) treatment is associated with lower costs for replacing absent pilots. Due to the uniqueness of the employee population studied (commercial airline pilots) and the organizational documentation of absence, the generalizability of this study to other professions and occupations should be considered limited. ^ Transition to Practice. The odds ratios for the relationship between absence rates and an AOD diagnosis are precise; the OR for year of diagnosis indicates the likelihood of being diagnosed increases 10% for every hour change in sick leave taken. In practice, however, a pilot uses approximately 20 hours of sick leave for one trip, because the replacement will have to be paid the guaranteed minimum of 20 hour. Thus, the rate based on hourly changes is precise but not practical. ^ To provide the organization with practical recommendations the yearly mean absence rates were used. A pilot flies on average, 90 hours a month, 1080 annually. Cases used almost twice the mean rate of sick time the year prior to diagnosis (T-1) compared to controls (cases, x = .11, controls, x = .06). Cases are expected to use on average 119 hours annually (total annual hours*mean annual absence rate), while controls will use 60 hours. The cases’ 60 hours could translate to 3 trips of 20 hours each. Management could use a standard of 80 hours or more of sick time claimed in a year as the threshold for unacceptable absence, a 25% increase over the controls (a cost to the company of approximately of $4000). At the 80-hour mark, the Chief Pilot would be able to call the pilot in for a routine check as to the nature of the pilot’s excessive absence. This management action would be based on a company standard, rather than a behavioral or performance issue. Using absence data in this fashion would make it an active surveillance mechanism. ^

UTILIZATION OF EVALUATION RESULTS

The issue over utilization of evaluation results has been receiving increasing attention lately. However, a continuing problem has been the lack of knowledge of the specific determinants of the utilization of evaluation results, the extent that these results are used, and the absence in the literature of empirical investigations of those determinants. The purposes of the current study were to examine the patterns of utilization of the recommendations made to an organization following an extensive evaluation, and to quantitatively examine some determinants of that utilization. During the course of the ten-month study, a number of important points regarding evaluation studies, and the utilization of their results were identified. Primary among these was the fact that a definition of utilization that requires adoption of recommendations is limited. Tremendous amounts of organizational activity may accrue from the results of an evaluation regardless of whether the results are adopted. In addition, none of the characteristics of the recommendations that were examined proved to be significant predictors of the extent of their utilization other than whether adoption was considered an authority decision or the results of staff concensus. ^

Assessing the impact of electronic health record technology adoption on hospital labor efficiency

Information technology (IT) in the hospital organization is fast becoming a key asset, particularly in light of recent reform legislation in the United States calling for expanding the role of IT in our health care system. Future payment reductions to hospitals included in current health reform are based on expected improvements in hospital operating efficiency. Since over half of hospital expenses are for labor, improved efficiency in use of labor resources can be critical in meeting this challenge. Policy makers have touted the value of IT investments to improve efficiency in response to payment reductions. ^ This study was the first to directly examine the relationship between electronic health record (EHR) technology and staffing efficiency in hospitals. As the hospital has a myriad of outputs for inpatient and outpatient care, efficiency was measured using an industry standard performance metric – full time equivalent employees per adjusted occupied bed (FTE/AOB). Three hypotheses were tested in this study.^ To operationalize EHR technology adoption, we developed three constructs to model adoption, each of which was tested by separate hypotheses. The first hypothesis that a larger number of EHR applications used by a hospital would be associated with greater staffing efficiency (or lower values of FTE/AOB) was not accepted. Association between staffing efficiency and specific EHR applications was the second hypothesis tested and accepted with some applications showing significant impacts on observed values for FTE/AOB. Finally, the hypothesis that the longer an EHR application was used in a hospital would be associated with greater labor efficiency was not accepted as the model showed few statistically significant relationships to FTE/AOB performance. Generally, there does not appear a strong relationship between EHR usage and improved labor efficiency in hospitals.^ While returns on investment from EHR usage may not come from labor efficiencies, they may be better sought using measures of quality, contribution to an efficient and effective local health care system, and improved customer satisfaction through greater patient throughput.^

Effects of a health advocacy program to promote physical activity and nutrition behaviors among adolescents: An evaluation of the Young Leaders for Healthy Change program

Background. This study was designed to evaluate the effects of the Young Leaders for Healthy Change program, an internet-delivered program in the school setting that emphasized health advocacy skills-development, on nutrition and physical activity behaviors among older adolescents (13–18 years). The program consisted of online curricular modules, training modules, social media, peer and parental support, and a community service project. Module content was developed based on Social Cognitive Theory and known determinants of behavior for older adolescents. ^ Methods. Of the 283 students who participated in the fall 2011 YL program, 38 students participated in at least ten of the 12 weeks and were eligible for this study. This study used a single group-only pretest/posttest evaluation design. Participants were 68% female, 58% white/Caucasian, 74% 10th or 11th graders, and 89% mostly A and/or B students. The primary behavioral outcomes for this analysis were participation in 60-minutes of physical activity per day, 20-minutes of vigorous- or moderate- intensity physical activity (MVPA) participation per day, television and computer time, fruit and vegetable (FV) intake, sugar-sweetened beverage intake, and consumption of breakfast, home-cooked meals, and fast food. Other outcomes included knowledge, beliefs, and attitudes related to healthy eating, physical activity, and advocacy skills. ^ Findings. Among the 38 participants, no significant changes in any variables were observed. However, among those who did not previously meet behavioral goals there was an 89% increase in students who participated in more than 20 minutes of MVPA per day and a 58% increase in students who ate home-cooked meals 5–7 days per week. The majority of participants met program goals related to knowledge, beliefs, and attitudes prior to the start of the program. Participants reported either maintaining or improving to the goal at posttest for all items except FV intake knowledge, taste and affordability of healthy foods, interest in teaching others about being healthy, and ease of finding ways to advocate in the community. ^ Conclusions. The results of this evaluation indicated that promoting healthy behaviors requires different strategies than maintaining healthy behaviors among high school students. In the school setting, programs need to target the promotion and maintenance of health behaviors to engage all students who participate in the program as part of a class or club activity. Tailoring the program using screening and modifying strategies to meet the needs of all students may increase the potential reach of the program. The Transtheoretical Model may provide information on how to develop a tailored program. Additional research on how to utilize the constructs of TTM effectively among high school students needs to be conducted. Further evaluation studies should employ a more expansive evaluation to assess the long-term effectiveness of health advocacy programming.^

The Comprehensive Assessment of Leadership for Learning: A Next Generation Formative Evaluation and Feedback System

Recent developments in federal policy have prompted the creation of state evaluation frameworks for principals and teachers that hold educators accountable for effective practices and student outcomes. These changes have created a demand for formative evaluation instruments that reflect current accountability pressures and can be used by schools to focus school improvement and leadership development efforts. The Comprehensive Assessment of Leadership for Learning (CALL) is a next generation, 360-degree on-line assessment and feedback system that reflect best practices in feedback design. Some unique characteristics of CALL include a focus on: leadership distributed throughout the school rather than as carried out by an individual leader; assessment of leadership tasks rather than perceptions of leadership practice; a focus on larger complex systems of middle and high school; and transparency of assessment design. This paper describes research contributing to the design and validation of the CALL survey instrument.