Evaluating impact of clinical guidelines using a realist evaluation framework

RATIONALE, AIMS AND OBJECTIVES: The Remote Primary Health Care Manuals (RPHCM) project team manages the development and publication of clinical protocols and procedures for primary care clinicians practicing in remote Australia. The Central Australian Rural Practitioners Association Standard Treatment Manual, the flagship manual of the RPHCM suite, has been evaluated for accessibility and acceptability in remote clinics three times in its 20-year history. These evaluations did not consider a theory-based framework or a programme theory, resulting in some limitations with the evaluation findings. With the RPHCM having an aim of enabling evidence-based practice in remote clinics and anecdotally reported to do so, testing this empirically for the full suite is vital for both stakeholders and future editions of the RPHCM. METHODS: The project team utilized a realist evaluation framework to assess how, why and for what the RPHCM were being used by remote practitioners. A theory regarding the circumstances in which the manuals have and have not enabled evidence-based practice in the remote clinical context was tested. The project assessed this theory for all the manuals in the RPHCM suite, across government and aboriginal community-controlled clinics, in three regions of Australia. CONCLUSION: Implementing a realist evaluation framework to generate robust findings in this context has required innovation in the evaluation design and adaptation by researchers. This article captures the RPHCM team's experience in designing this evaluation.

Evaluation of the impact of nursing clinics in the rheumatology services

Nursing clinics in rheumatology (NCRs) are organisational care models that provide care centred within the scope of a nurse’s abilities. To analyse the impact of NCR in the rheumatology services, national multicenter observational prospective cohort studied 1-year follow-up, comparing patients attending rheumatology services with and without NCR. NCR was defined by the presence of: (1) office itself; (2) at least one dedicated nurse; and (3) its own appointment schedule. Variables included were (baseline, 6 and 12 months): (a) test to evaluate clinical activity of the disease, research and training, infrastructure of unit and resources of NCR and (b) tests to evaluate socio-demographics, work productivity (WPAI), use of services and treatments and quality of life. A total of 393 rheumatoid arthritis and ankylosing spondylitis patients were included: 181 NCR and 212 not NCR, corresponding to 39 units, 21 with NCR and 18 without NCR (age 53 + 11.8 vs 56 + 13.5 years). Statistically significant differences were found in patients attended in sites without NCR, at some of the visits (baseline, 6 or 12 months), for the following parameters: higher CRP level (5.9 mg/l ± 8.3 vs 4.8 mg/l ± 7.8; p < 0.005), global disease evaluation by the patient (3.6 ± 2.3 vs 3.1 ± 2.4), physician (2.9 ± 2.1 vs 2.3 ± 2.1; p < 0.05), use of primary care consultations (2.7 ± 5.4 vs 1.4 ± 2.3; p < 0.001) and worse work productivity. The presence of NCR in the rheumatology services contributes to improve some clinical outcomes, a lower frequency of primary care consultations and better work productivity of patients with rheumatic diseases.

A Method of Preparing Future Hourly Weather Data for the Study of Global Warming Impact on the Built Environment

The dynamic interaction between building systems and external climate is extremely complex, involving a large number of difficult-to-predict variables. In order to study the impact of global warming on the built environment, the use of building simulation techniques together with forecast weather data are often necessary. Since all building simulation programs require hourly meteorological input data for their thermal comfort and energy evaluation, the provision of suitable weather data becomes critical. Based on a review of the existing weather data generation models, this paper presents an effective method to generate approximate future hourly weather data suitable for the study of the impact of global warming. Depending on the level of information available for the prediction of future weather condition, it is shown that either the method of retaining to current level, constant offset method or diurnal modelling method may be used to generate the future hourly variation of an individual weather parameter. An example of the application of this method to the different global warming scenarios in Australia is presented. Since there is no reliable projection of possible change in air humidity, solar radiation or wind characters, as a first approximation, these parameters have been assumed to remain at the current level. A sensitivity test of their impact on the building energy performance shows that there is generally a good linear relationship between building cooling load and the changes of weather variables of solar radiation, relative humidity or wind speed.

Learning, evaluation, action and reflection for new technologies, empowerment and rural sustainability (The LEARNERS project)

-

Public sector evaluation of festivals and special events

This research project set out to explore Unitary Authority (UA) involvement in festivals and special events across Wales. It considers the level and nature of UA involvement and investigates activity by event purpose; reasons for, and characteristics of, UA engagement; and, crucially, the extent and nature of event evaluation. The study’s aim was to begin the development of a baseline of information for further research into the growing use of festivals and special events as a strategy for local economic development in Wales. A quantitative survey approach facilitated a comprehensive snapshot of UA responses whilst also incorporating discursive elements. A telephone survey was designed and undertaken with representatives of all 22 UA departments responsible for festivals and events in Wales. The research reveals a significant level of festival and special event activity across Wales, supported primarily for its perceived socio-cultural value. However, evaluation would appear to be focused on improving processes and measuring economic outputs rather than assessing whether socio-cultural objectives are being achieved. Whilst overwhelmingly positive about efforts to improve approaches to evaluation, respondents held clear views about the complications most likely to hamper any such efforts. These responses focused upon the need for flexibility, cost effectiveness and comparability across festival and special event typologies.

Evaluation of functional performance in commercial buildings

This report fully summarises a project designed to enhance commercial real estate performance within both operational and investment contexts through the development of a model aimed at supporting improved decision-making. The model is based on a risk adjusted discounted cash flow, providing a valuable toolkit for building managers, owners, and potential investors for evaluating individual building performance in terms of financial, social and environmental criteria over the complete life-cycle of the asset. The ‘triple bottom line’ approach to the evaluation of commercial property has much significance for the administrators of public property portfolios in particular. It also has applications more generally for the wider real estate industry given that the advent of ‘green’ construction requires new methods for evaluating both new and existing building stocks. The research is unique in that it focuses on the accuracy of the input variables required for the model. These key variables were largely determined by market-based research and an extensive literature review, and have been fine-tuned with extensive testing. In essence, the project has considered probability-based risk analysis techniques that required market-based assessment. The projections listed in the partner engineers’ building audit reports of the four case study buildings were fed into the property evaluation model developed by the research team. The results are strongly consistent with previously existing, less robust evaluation techniques. And importantly, this model pioneers an approach for taking full account of the triple bottom line, establishing a benchmark for related research to follow. The project’s industry partners expressed a high degree of satisfaction with the project outcomes at a recent demonstration seminar. The project in its existing form has not been geared towards commercial applications but it is anticipated that QDPW and other industry partners will benefit greatly by using this tool for the performance evaluation of property assets. The project met the objectives of the original proposal as well as all the specified milestones. The project has been completed within budget and on time. This research project has achieved the objective by establishing research foci on the model structure, the key input variable identification, the drivers of the relevant property markets, the determinants of the key variables (Research Engine no.1), the examination of risk measurement, the incorporation of risk simulation exercises (Research Engine no.2), the importance of both environmental and social factors and, finally the impact of the triple bottom line measures on the asset (Research Engine no. 3).

Trust and strong ties in the selection and evaluation of business advisors by small firms

Principal Topic A small firm is unlikely to possess internally the full range of knowledge and skills that it requires or could benefit from for the development of its business. The ability to acquire suitable external expertise - defined as knowledge or competence that is rare in the firm and acquired from the outside - when needed thus becomes a competitive factor in itself. Access to external expertise enables the firm to focus on its core competencies and removes the necessity to internalize every skill and competence. However, research on how small firms access external expertise is still scarce. The present study contributes to this under-developed discussion by analysing the role of trust and strong ties in the small firm's selection and evaluation of sources of external expertise (henceforth referred to as the 'business advisor' or 'advisor'). Granovetter (1973, 1361) defines the strength of a network tie as 'a (probably linear) combination of the amount of time, the emotional intensity, the intimacy (mutual confiding) and the reciprocal services which characterize the tie'. Strong ties in the context of the present investigation refer to sources of external expertise who are well known to the owner-manager, and who may be either informal (e.g., family, friends) or professional advisors (e.g., consultants, enterprise support officers, accountants or solicitors). Previous research has suggested that strong and weak ties have different fortes and the choice of business advisors could thus be critical to business performance) While previous research results suggest that small businesses favour previously well known business advisors, prior studies have also pointed out that an excessive reliance on a network of well known actors might hamper business development, as the range of expertise available through strong ties is limited. But are owner-managers of small businesses aware of this limitation and does it matter to them? Or does working with a well-known advisor compensate for it? Hence, our research model first examines the impact of the strength of tie on the business advisor's perceived performance. Next, we ask what encourages a small business owner-manager to seek advice from a strong tie. A recent exploratory study by Welter and Kautonen (2005) drew attention to the central role of trust in this context. However, while their study found support for the general proposition that trust plays an important role in the choice of advisors, how trust and its different dimensions actually affect this choice remained ambiguous. The present paper develops this discussion by considering the impact of the different dimensions of perceived trustworthiness, defined as benevolence, integrity and ability, on the strength of tie. Further, we suggest that the dimensions of perceived trustworthiness relevant in the choice of a strong tie vary between professional and informal advisors. Methodology/Key Propositions Our propositions are examined empirically based on survey data comprising 153 Finnish small businesses. The data are analysed utilizing the partial least squares (PLS) approach to structural equation modelling with SmartPLS 2.0. Being non-parametric, the PLS algorithm is particularly well-suited to analysing small datasets with non-normally distributed variables. Results and Implications The path model shows that the stronger the tie, the more positively the advisor's performance is perceived. Hypothesis 1, that strong ties will be associated with higher perceptions of performance is clearly supported. Benevolence is clearly the most significant predictor of the choice of a strong tie for external expertise. While ability also reaches a moderate level of statistical significance, integrity does not have a statistically significant impact on the choice of a strong tie. Hence, we found support for two out of three independent variables included in Hypothesis 2. Path coefficients differed between the professional and informal advisor subsamples. The results of the exploratory group comparison show that Hypothesis 3a regarding ability being associated with strong ties more pronouncedly when choosing a professional advisor was not supported. Hypothesis 3b arguing that benevolence is more strongly associated with strong ties in the context of choosing an informal advisor received some support because the path coefficient in the informal advisor subsample was much larger than in the professional advisor subsample. Hypothesis 3c postulating that integrity would be more strongly associated with strong ties in the choice of a professional advisor was supported. Integrity is the most important dimension of trustworthiness in this context. However, integrity is of no concern, or even negative, when using strong ties to choose an informal advisor. The findings of this study have practical relevance to the enterprise support community. First of all, given that the strength of tie has a significant positive impact on the advisor's perceived performance, this implies that small business owners appreciate working with advisors in long-term relationships. Therefore, advisors are well advised to invest into relationship building and maintenance in their work with small firms. Secondly, the results show that, especially in the context of professional advisors, the advisor's perceived integrity and benevolence weigh more than ability. This again emphasizes the need to invest time and effort into building a personal relationship with the owner-manager, rather than merely maintaining a professional image and credentials. Finally, this study demonstrates that the dimensions of perceived trustworthiness are orthogonal with different effects on the strength of tie and ultimately perceived performance. This means that entrepreneurs and advisors should consider the specific dimensions of ability, benevolence and integrity, rather than rely on general perceptions of trustworthiness in their advice relationships.

Incorporating an economic model in the health impact assessment approach

The current policy decision making in Australia regarding non-health public investments (for example, transport/housing/social welfare programmes) does not quantify health benefits and costs systematically. To address this knowledge gap, this study proposes an economic model for quantifying health impacts of public policies in terms of dollar value. The intention is to enable policy-makers in conducting economic evaluation of health effects of non-health policies and in implementing policies those reduce health inequalities as well as enhance positive health gains of the target population. Health Impact Assessment (HIA) provides an appropriate framework for this study since HIA assesses the beneficial and adverse effects of a programme/policy on public health and on health inequalities through the distribution of those effects. However, HIA usually tries to influence the decision making process using its scientific findings, mostly epidemiological and toxicological evidence. In reality, this evidence can not establish causal links between policy and health impacts since it can not explain how an individual or a community reacts to changing circumstances. The proposed economic model addresses this health-policy linkage using a consumer choice approach that can explain changes in group and individual behaviour in a given economic set up. The economic model suggested in this paper links epidemiological findings with economic analysis to estimate the health costs and benefits of public investment policies. That is, estimating dollar impacts when health status of the exposed population group changes by public programmes – for example, transport initiatives to reduce congestion by building new roads/ highways/ tunnels etc. or by imposing congestion taxes. For policy evaluation purposes, the model is incorporated in the HIA framework by establishing association among identified factors, which drive changes in the behaviour of target population group and in turn, in the health outcomes. The economic variables identified to estimate the health inequality and health costs are levels of income, unemployment, education, age groups, disadvantaged population groups, mortality/morbidity etc. However, though the model validation using case studies and/or available database from Australian non-health policy (say, transport) arena is in the future tasks agenda, it is beyond the scope of this current paper.

Property Education : an evaluation of how well undergraduate property students are prepared for commencing their careers

The evolution of property education to adapt to the changing business environment requires changes to course content, method of delivery and assessment. Many universities have a special interest in understanding how the students transition in and transition out of the property programs. The impact of the first year student experience is often easier to assess through students’ progression in the course and performance in their intermediate and advanced units. However, the students’ success in transitioning from university student to property professional is often more difficult to determine. In an environment where many property students commence their professional careers while still completing their undergraduate property qualification, a survey of current final year students was undertaken to identify the students’ perception of their level of preparedness for entry into the professional world. This study has also been informed by feedback received from and informal discussions held with industry representative bodies, alumni and senior members of professional organisations. The QUT UD40 Bachelor of Urban Development, Property Economics course has been designed to achieve graduate capabilities in core technical skills and generic professional skills which are required by property professionals. The results of this study were that some units in the program were perceived to provide direct preparation for students commencing their professional careers whilst the impact of other units was less tangible. Valuable feedback received during the study included an assessment of the relevance of many multi-disciplinary units, the appropriateness of the programming of units within the course and the appropriateness of repetition of content during the course. The further research question arises as to how universities can better assist students in the transition to the professional environment when frequently this occurs prior to completion of the property course.

An evaluation of brief correspondence programs for problem drinkers

The provision of accessible and cost-effective treatment to a large number of problem drinkers is a significant challenge to health services. Previous data suggest that a correspondence intervention may assist in these efforts. We recruited 277 people with alcohol abuse problems and randomly allocated them to immediate cognitive behavioral treatment by correspondence (ICBT), 2 months in a waiting list (WL2-CBT), self-monitoring (SM2-CBT), or extended self-monitoring (SM6-CBT). Everyone received correspondence CBT after the control period. Over 2 months later, no drop in alcohol intake occurred in the waiting list, and CBT had a greater impact than SM. No further gains from SM were seen after 2 months. Effects of CBT were well maintained and were equivalent, whether it was received immediately or after 2 to 6 months of self-monitoring. Weekly alcohol intake fell 48% from pretreatment to 18.6 alcohol units at 12 months. Our results confirmed that correspondence CBT for alcohol abuse was accessible and effective for people with low physical dependence.

Moderating drinking by correspondence : an evaluation of a new method of intervention

Recognizing the need to offer alternative methods of brief interventions, this study developed correspondence treatments for low-dependent problem drinkers and evaluated their impact. One hundred and twenty-one problem drinkers were recruited by media advertisements and were randomly allocated to a full cognitive behavioural treatment programme (CBT) or to a minimal intervention condition (MI) that gave information regarding alcohol misuse and instructions to record drinking. As predicted, CBT was more effective than MI in reducing alcohol consumption over the 4-month controlled trial period. CBT produced a 50% fall in consumption, bringing the average intake of subjects within recommended maximum levels. Treatment gains at 6 months were well maintained to 12 months. High levels of consumer satisfaction, a high representation of women and a substantial participation from isolated rural areas attested to the feasibility of the correspondence programme as an alternative treatment. However, some drinking occasions still involved high intake for a significant subgroup of subjects, and this issue will be addressed in future programmes. The results supported the use of correspondence delivery as a means of promoting early engagement and equity of access between city and country areas.

An evaluation framework for educational reform projects for teacher quality improvements in developing countries : a case study of Egyptian education reform

The role of the evaluation for Official Development Assistance (ODA) enterprises including educational development has become critical after increasing “aid fatigue” experienced by the international community in the 1990s. To date, however, monitoring and evaluating outcomes of the projects has been limited to the project life. Consequently these have been mainly through the international aid agencies. Furthermore, the monitoring and evaluation led by international aid agencies have paid little attention to aspects of the sustainability of technical cooperation in educational development. To sustain the impact of technical cooperation, the reinforcement of evaluation has drawn increasing attention in light of the emerging modalities in international development. Therefore this research was inspired to investigate alternative evaluation frameworks for an educational reform project for teacher quality improvement that may increase possibilities for long term sustainability. Importantly, the new modalities in international development and educational issues provide new options. In addition, the research reviewed theoretical and practical issues surrounding evaluation in general, and highlighted the evaluation of education reform projects. The research reported explored via case studies, the evaluation processes employed by the Egyptian education reform projects implemented by the Japan International Cooperation Agency (JICA) and the United Nations Children’s Fund (UNICEF). The case studies used three data sources (archival and relevant documents, a survey questionnaire and interviews) to illuminate the contextually-embedded evaluation processes. The research found that process evaluation is a potential alternative method since it is likely to be locally institutionalised, which may yield long-term sustainability of the projects.

Extending the IS-Impact Model into the higher education sector

The study addresses known limitations of what may be the most important dependent variable in Information Systems (IS) research; IS-Success or IS-Impact. The study is expected to force a deeper understanding of the broad notions of IS success and impact. The aims of the research are to: (1) enhance the robustness and minimize limitations of the IS-Impact model, and (2) introduce and operationalise a more rigorously validated IS Impact measurement model to Universities, as a reliable model for evaluating different Administrative Systems. In extending and further generalizing the IS-Impact model, the study will address contemporary validation issues.

Validating the IS-Impact Model At Queensland University Of Technology (QUT): Part A

The overall research aims to develop a standardised instrument to measure the impacts resulting from contemporary Information Systems (IS). The research adopts the IS-Impact measurement model, introduced by Gable et al, (2008), as its theoretical foundation, and applies the extension strategy described by Berthon et al. (2002); extending both theory and the context, where the new context is the Human Resource (HR) system. The research will be conducted in two phases, the exploratory phase and the specification phase. The purpose of this paper is to present the findings of the exploratory phase. 134 respondents from a major Australian University were involved in this phase. The findings have supported most of the existing IS-Impact model’s credibility. However, some textual data may suggest new measures for the IS-Impact model, while the low response rate or the averting of some may suggest the elimination of some measures from the model.

The impact of usability on clinician acceptance of a health information system

The two longitudinal case studies that make up this dissertation sought to explain and predict the relationship between usability and clinician acceptance of a health information system. The overall aim of the research study was to determine what role usability plays in the acceptance or rejection of systems used by clinicians in a healthcare context. The focus was on the end users (the clinicians) rather than the views of the system designers and managers responsible for implementation and the clients of the clinicians. A mixed methods approach was adopted that drew on both qualitative and quantitative research methods. This study followed the implementation of a community health information system from early beginnings to its established practice. Users were drawn from different health service departments with distinctly different organisational cultures and attitudes to information and communication technology used in this context. This study provided evidence that a usability analysis in this context would not necessarily be valid when the users have prior reservations on acceptance. Investigation was made on the initial training and post-implementation support together with a study on the nature of the clinicians to determine factors that may influence their attitude. This research identified that acceptance of a system is not necessarily a measure of its quality, capability and usability, is influenced by the user’s attitude which is determined by outside factors, and the nature and quality of training. The need to recognise the limitations of the current methodologies for analysing usability and acceptance was explored to lay the foundations for further research.