Addressing Multilevel Program Complexity by Evaluation Design

In this paper, we present the evaluation design for a complex multilevel program recently introduced in Switzerland. The evaluation embraces the federal level, the cantonal program level, and the project level where target groups are directly addressed. We employ Pawson and Tilley’s realist evaluation approach, in order to do justice to the varying context factors that impact the cantonal programs leading to varying effectiveness of the implemented activities. The application of the model to the canton of Uri shows that the numerous vertical and horizontal relations play a crucial role for the program’s effectiveness. As a general learning for the evaluation of complex programs, we state that there is a need to consider all affected levels of a program and that no monocausal effects can be singled out in programs where multiple interventions address the same problem. Moreover, considering all affected levels of a program can mean going beyond the borders of the actual program organization and including factors that do not directly interfere with the policy delivery as such. In particular, we found that the relationship between the cantonal and the federal level was a crucial organizational factor influencing the effectiveness of the cantonal program.

The influence of students' prior clinical skills and context characteristics on mini-CEX scores in clerkships - a multilevel analysis.

BACKGROUND In contrast to objective structured clinical examinations (OSCEs), mini-clinical evaluation exercises (mini-CEXs) take place at the clinical workplace. As both mini-CEXs and OSCEs assess clinical skills, but within different contexts, this study aims at analyzing to which degree students' mini-CEX scores can be predicted by their recent OSCE scores and/or context characteristics. METHODS Medical students participated in an end of Year 3 OSCE and in 11 mini-CEXs during 5 different clerkships of Year 4. The students' mean scores of 9 clinical skills OSCE stations and mean 'overall' and 'domain' mini-CEX scores, averaged over all mini-CEXs of each student were computed. Linear regression analyses including random effects were used to predict mini-CEX scores by OSCE performance and characteristics of clinics, trainers, students and assessments. RESULTS A total of 512 trainers in 45 clinics provided 1783 mini-CEX ratings for 165 students; OSCE results were available for 144 students (87 %). Most influential for the prediction of 'overall' mini-CEX scores was the trainers' clinical position with a regression coefficient of 0.55 (95 %-CI: 0.26-0.84; p < .001) for residents compared to heads of department. Highly complex tasks and assessments taking place in large clinics significantly enhanced 'overall' mini-CEX scores, too. In contrast, high OSCE performance did not significantly increase 'overall' mini-CEX scores. CONCLUSION In our study, Mini-CEX scores depended rather on context characteristics than on students' clinical skills as demonstrated in an OSCE. Ways are discussed which focus on either to enhance the scores' validity or to use narrative comments only.

The enduring effects of psychodynamic treatments vis-à-vis alternative treatments: A multilevel longitudinal meta-analysis

Although evidence suggests that the benefits of psychodynamic treatments are sustained over time, presently it is unclear whether these sustained benefits are superior to non-psychodynamic treatments. Additionally, the extant literature comparing the sustained benefits of psychodynamic treatments compared to alternative treatments is limited with methodological shortcomings. The purpose of the current study was to conduct a rigorous test of the growth of the benefits of psychodynamic treatments relative to alternative treatments across distinct domains of change (i.e., all outcome measures, targeted outcome measures, non-targeted outcome measures, and personality outcome measures). To do so, the study employed strict inclusion criteria to identify randomized clinical trials that directly compared at least one bona fide psychodynamic treatment and one bona fide non-psychodynamic treatment. Hierarchical linear modeling (Raudenbush, Bryk, Cheong, Congdon, & du Toit, 2011) was used to longitudinally model the impact of psychodynamic treatments compared to non-psychodynamic treatments at post-treatment and to compare the growth (i.e., slope) of effects beyond treatment completion. Findings from the present meta-analysis indicated that psychodynamic treatments and non-psychodynamic treatments were equally efficacious at post-treatment and at follow-up for combined outcomes (k=20), targeted outcomes (k=19), non-targeted outcomes (k=17), and personality outcomes (k=6). Clinical implications, directions for future research, and limitations are discussed.

Multilevel analysis for a longitudinal clinical trial

This study applies the multilevel analysis technique to longitudinal data of a large clinical trial. The technique accounts for the correlation at different levels when modeling repeated blood pressure measurements taken throughout the trial. This modeling allows for closer inspection of the remaining correlation and non-homogeneity of variance in the data. Three methods of modeling the correlation were compared. ^

Comparison of a traditional and a multilevel Cox Proportional Hazards model

Hierarchically clustered populations are often encountered in public health research, but the traditional methods used in analyzing this type of data are not always adequate. In the case of survival time data, more appropriate methods have only begun to surface in the last couple of decades. Such methods include multilevel statistical techniques which, although more complicated to implement than traditional methods, are more appropriate. ^ One population that is known to exhibit a hierarchical structure is that of patients who utilize the health care system of the Department of Veterans Affairs where patients are grouped not only by hospital, but also by geographic network (VISN). This project analyzes survival time data sets housed at the Houston Veterans Affairs Medical Center Research Department using two different Cox Proportional Hazards regression models, a traditional model and a multilevel model. VISNs that exhibit significantly higher or lower survival rates than the rest are identified separately for each model. ^ In this particular case, although there are differences in the results of the two models, it is not enough to warrant using the more complex multilevel technique. This is shown by the small estimates of variance associated with levels two and three in the multilevel Cox analysis. Much of the differences that are exhibited in identification of VISNs with high or low survival rates is attributable to computer hardware difficulties rather than to any significant improvements in the model. ^

The association of area socioeconomic status and cancer screening: A systematic review and multilevel study

Background. The gap between actual and ideal rates of routine cancer screening in the U.S., particularly for colorectal cancer screening (CRCS) (1;2), is responsible for an unnecessary burden of morbidity and mortality, particularly for disadvantaged groups. Knowledge about the effects of individual and area influences is being advanced by a growing body of research that has examined the association of area socioeconomic status (SES) and cancer screening after controlling for individual SES. The findings from this emerging and heterogeneous research in the cancer screening literature have been mixed. Moreover, multilevel studies in this area have not yet adequately explored the possibility of differential associations by population subgroup, despite some evidence suggesting gender-specific effects. ^ Objectives and methods. This dissertation reports on a systematic review of studies on the association of area SES and cancer screening and a multilevel study of the association between area SES and CRCS. The specific aims of the systematic review are to: (1) describe the study designs, constructs, methods, and measures; (2) describe the association of area SES and cancer screening; and (3) identify neglected areas of research. ^ The empiric study linked a pooled sample of respondents aged ≥50 years without a personal history of colorectal cancer from the 2003 and 2005 California Health Interview Surveys with a comprehensive set of census-tract level area SES measures from the 2000 U.S. Census. Two-level random intercept models were used to test 2 hypotheses: (1) area SES will be associated with adherence to two modalities of CRCS after controlling for individual SES; and (2) gender will moderate the relationship between area socioeconomic status and adherence to both modalities of CRCS. ^ Results. The systematic review identified 19 eligible studies that demonstrated variability in study designs, methods, constructs, and measures. The majority of tested associations were either not statistically significant or significant and in the positive direction, indicating that as area SES increased, the odds of CRCS increased. The multilevel study demonstrated that while multiple aspects of area SES were associated with CRCS after controlling for individual SES, associations differed by screening modality and in the case of endoscopy, they also differed by gender. ^ Conclusions. Conceptual and methodologic heterogeneity and weaknesses in the literature to date limit definitive conclusions about the underlying relationships between area SES and cancer screening. The multilevel study provided partial support for both hypotheses. Future research should continue to explore the role of gender as a moderating influence with the aim of identifying the mechanisms linking area SES and cancer prevention behaviors. ^

Single nucleotide polymorphisms (SNPs) associated with TGF-beta pathway and their significance in systemic sclerosis - A multilevel analysis

Systemic sclerosis (SSc) or Scleroderma is a complex disease and its etiopathogenesis remains unelucidated. Fibrosis in multiple organs is a key feature of SSc and studies have shown that transforming growth factor-β (TGF-β) pathway has a crucial role in fibrotic responses. For a complex disease such as SSc, expression quantitative trait loci (eQTL) analysis is a powerful tool for identifying genetic variations that affect expression of genes involved in this disease. In this study, a multilevel model is described to perform a multivariate eQTL for identifying genetic variation (SNPs) specifically associated with the expression of three members of TGF-β pathway, CTGF, SPARC and COL3A1. The uniqueness of this model is that all three genes were included in one model, rather than one gene being examined at a time. A protein might contribute to multiple pathways and this approach allows the identification of important genetic variations linked to multiple genes belonging to the same pathway. In this study, 29 SNPs were identified and 16 of them located in known genes. Exploring the roles of these genes in TGF-β regulation will help elucidate the etiology of SSc, which will in turn help to better manage this complex disease. ^

HPV vaccine coverage in Texas counties: An application of multilevel, small area estimation

Health departments, research institutions, policy-makers, and healthcare providers are often interested in knowing the health status of their clients/constituents. Without the resources, financially or administratively, to go out into the community and conduct health assessments directly, these entities frequently rely on data from population-based surveys to supply the information they need. Unfortunately, these surveys are ill-equipped for the job due to sample size and privacy concerns. Small area estimation (SAE) techniques have excellent potential in such circumstances, but have been underutilized in public health due to lack of awareness and confidence in applying its methods. The goal of this research is to make model-based SAE accessible to a broad readership using clear, example-based learning. Specifically, we applied the principles of multilevel, unit-level SAE to describe the geographic distribution of HPV vaccine coverage among females aged 11-26 in Texas.^ Multilevel (3 level: individual, county, public health region) random-intercept logit models of HPV vaccination (receipt of ≥ 1 dose Gardasil® ) were fit to data from the 2008 Behavioral Risk Factor Surveillance System (outcome and level 1 covariates) and a number of secondary sources (group-level covariates). Sampling weights were scaled (level 1) or constructed (levels 2 & 3), and incorporated at every level. Using the regression coefficients (and standard errors) from the final models, I simulated 10,000 datasets for each regression coefficient from the normal distribution and applied them to the logit model to estimate HPV vaccine coverage in each county and respective demographic subgroup. For simplicity, I only provide coverage estimates (and 95% confidence intervals) for counties.^ County-level coverage among females aged 11-17 varied from 6.8-29.0%. For females aged 18-26, coverage varied from 1.9%-23.8%. Aggregated to the state level, these values translate to indirect state estimates of 15.5% and 11.4%, respectively; both of which fall within the confidence intervals for the direct estimates of HPV vaccine coverage in Texas (Females 11-17: 17.7%, 95% CI: 13.6, 21.9; Females 18-26: 12.0%, 95% CI: 6.2, 17.7).^ Small area estimation has great potential for informing policy, program development and evaluation, and the provision of health services. Harnessing the flexibility of multilevel, unit-level SAE to estimate HPV vaccine coverage among females aged 11-26 in Texas counties, I have provided (1) practical guidance on how to conceptualize and conduct modelbased SAE, (2) a robust framework that can be applied to other health outcomes or geographic levels of aggregation, and (3) HPV vaccine coverage data that may inform the development of health education programs, the provision of health services, the planning of additional research studies, and the creation of local health policies.^

Bayesian estimation of multilevel item response model with missing data

This study proposed a novel statistical method that modeled the multiple outcomes and missing data process jointly using item response theory. This method follows the "intent-to-treat" principle in clinical trials and accounts for the correlation between outcomes and missing data process. This method may provide a good solution to chronic mental disorder study. ^ The simulation study demonstrated that if the true model is the proposed model with moderate or strong correlation, ignoring the within correlation may lead to overestimate of the treatment effect and result in more type I error than specified level. Even if the within correlation is small, the performance of proposed model is as good as naïve response model. Thus, the proposed model is robust for different correlation settings if the data is generated by the proposed model.^

Automatic Understanding of ATC Speech: Study of Prospectives and Field Experiments for Several Controller Positions

Although there has been a lot of interest in recognizing and understanding air traffic control (ATC) speech, none of the published works have obtained detailed field data results. We have developed a system able to identify the language spoken and recognize and understand sentences in both Spanish and English. We also present field results for several in-tower controller positions. To the best of our knowledge, this is the first time that field ATC speech (not simulated) is captured, processed, and analyzed. The use of stochastic grammars allows variations in the standard phraseology that appear in field data. The robust understanding algorithm developed has 95% concept accuracy from ATC text input. It also allows changes in the presentation order of the concepts and the correction of errors created by the speech recognition engine improving it by 17% and 25%, respectively, absolute in the percentage of fully correctly understood sentences for English and Spanish in relation to the percentages of fully correctly recognized sentences. The analysis of errors due to the spontaneity of the speech and its comparison to read speech is also carried out. A 96% word accuracy for read speech is reduced to 86% word accuracy for field ATC data for Spanish for the "clearances" task confirming that field data is needed to estimate the performance of a system. A literature review and a critical discussion on the possibilities of speech recognition and understanding technology applied to ATC speech are also given.

Pond head level control in a run-of-river hydro power plant using fuzzy controller

The run-of-river hydro power plant usually have low or nil water storage capacity, and therefore an adequate control strategy is required to keep the water level constant in pond. This paper presents a novel technique based on TSK fuzzy controller to maintain the pond head constant. The performance is investigated over a wide range of hill curve of hydro turbine. The results are compared with PI controller as discussed in [1].

ITER fast plant system controller prototype based on PXIe platform

The ITER CODAC design identifies slow and fast plant system controllers (PSC). The gast OSCs are based on embedded technologies, permit sampling rates greater than 1 KHz, meet stringent real-time requirements, and will be devoted to data acquisition tasks and control purposes. CIEMAT and UPM have implemented a prototype of a fast PSC based on commercial off-the-shelf (COTS) technologies with PXI hardware and software based on EPICS

ITER Fast Plant System Controller Prototype Based on Commercial PXI Platform

The ITER CODAC design identifies slow and fast plant system controllers (PSC). The gast OSCs are based on embedded technologies, permit sampling rates greater than 1 KHz, meet stringent real-time requirements, and will be devoted to data acquisition tasks and control purposes. CIEMAT and UPM have implemented a prototype of a fast PSC based on commercial off-the-shelf (COTS) technologies with PXI hardware and software based on EPICS

Digital Control Implementation to Reduce the Cost and Improve the Performance of the Control Stage of an Industrial Switched-Mode Power Supply

The main objective of this work is the design and implementation of the digital control stage of a 280W AC/DC industrial power supply in a single low-cost microcontroller to replace the analog control stage. The switch-mode power supply (SMPS) consists of a PFC boost converter with fixed frequency operation and a variable frequency LLC series resonant DC/DC converter. Input voltage range is 85VRMS-550VRMS and the output voltage range is 24V-28V. A digital controller is especially suitable for this kind of SMPS to implement its multiple functionalities and to keep the efficiency and the performance high over the wide range of input voltages. Additional advantages of the digital control are reliability and size. The optimized design and implementation of the digital control stage it is presented. Experimental results show the stable operation of the controlled system and an estimation of the cost reduction achieved with the digital control stage.