Quantitative evaluation in health promotion : evaluation with confidence

Over the past 10 years or so, confidence intervals have become increasingly recognised in program evaluation and quantitative health measurement generally as the preferred way of reporting the accuracy of statistical estimates. Statisticians have found that the more traditional ways of reporting results - using P-values and hypothesis tests - are often very difficult to interpret and can be misleading. This is particularly the case when sample sizes are small and results are 'negative' (ie P>0.05); in these cases, a confidence interval can communicate much more information about the sample and, by inference, about the population. Despite this trend among statisticians and health promotion evaluators towards the use of confidence intervals, it is surprisingly difficult to find succinct and reasonably simple methods to actually compute a confidence interval. This is particularly the case for proportions or percentages. Much of the data which are analysed in health promotion are binary or categorical, rather than the quantities and continuous variables often found in laboratories or other branches of science, so there is a need for health promotion evaluators to be able to present confidence intervals for percentages or proportions. However, the most popular statistical analysis computer package among health promotion professionals, SPSS does not have a routine to compute a simple confidence interval for a proportion! To address this shortcoming, I present in this paper some fairly simple strategies for computing confidence intervals for population percentages, both manually and using the right computer software.

Evaluation in health outcomes research : linking theories, methodologies and practice in health promotion.

Health outcomes research has developed as a means of evaluating the effectiveness of health care interventions and as an approach to informing resource allocation. The use of a health outcomes approach in health promotion has made increasing demands on evaluation methodologies to demonstrate program effectiveness. However, criticism of the contribution of health promotion to outcomes research has made several assumptions about the use of qualitative methodologies and the content of program objectives largely derived from a biomedical approach. In contrast to the measurement of biomedical interventions in clinical health care, health promotion practice involves social phenomena, wide-reaching cultural, psychological, political and ideological problems and issues. The integration of methodologies of health promotion evaluation will inform further conceptualisation of the health outcomes approach with the differentiation of three types of outcomes: health development outcomes; social health outcomes; and biomedical health outcomes. It is concluded that this differentiation moves away from dualist concepts that advocate the replacement of goals and targets with regional and locally based approaches. Rather, the future direction for health promotion evaluation needs to employ a framework that elaborates multiple methodologies and approaches necessary for establishing what relationships exist between morbidity, mortality, health advancement and equity.

Health promotion and evaluation - a programmatic approach

Health promotion has evolved significantly in the past twenty years. Its emphasis has shifted from relatively simple monocausal models focused on behavioural risk factors to a greater emphasis on broader social determinants of health. Single method, single risk factor interventions have increasingly been replaced with multiformat, multiple risk factor interventions and extended campaigns, with whole-of-govemment implications. Health promotion structures have developed from ad hoc single shot activity to large dedicated agencies with continuing responsibilities and a wide ambit.

The development of health promotion research and evaluation has followed these trends. The early epidemiology studies linked behavioural risks such as smoking, diet and physical activity with systemic conditions such as cancer and cardiovascular disease. A raft of small and large scale intervention studies aimed at demonstrating that these behavioural risk factors could be modified and that modification would lead to improved health outcomes followed with mixed results.

More recent evidence suggests that behavioural risks are not the onIy social factors that influence health outcomes. There is now strong evidence that social determinants such as income, education and employment have highly significant direct effects on health outcomes, which are not mediated by behavioural risks, and that behavioural risks are also correlated with these broader determinants.

Health promotion now operates in a variety of ways at different scales and different levels of the health system (and the wider social system). The goals of health promotion, and the measures that assess whether a project, campaign, or general strategy has met its goals, differ accordingly.

Arguably, where local, state and federal governments begin to coordinate their efforts systematically across settings, intervention strategies, health action areas and population groups, health promotion becomes more
programmatic, sustainable and effective. A programmatic approach also integrates knowledge generation, the development of health promotion capacity, practice and evaluation together.

However, programmatic approaches to health promotion are comparatively new. Only recently have governments begun to develop and resource
comprehensive and sustained health promotion programs that address a range of health issues using multiple intervention strategies. The scope of a more programmatic approach and its functions and purposes is still developing.

Although evaluation has a key role to play in this respect, the development of programmatic strategies for health promotion has generally outpaced evaluation theory and practice. While we now have reasonable technologies for measurement of behavioural risks and individual attitudinal and cognitive influences on them, strategies to evaluate organisational and community interventions are still emerging.

Similarly, while new approaches to evaluate small scale community and organisational interventions have been developed, comprehensive models to monitor and evaluate health promotion programs and strategies across multiple intervention sites over extended periods have not yet emerged. Nor have we resolved the methodological problems of teasing out the relative contribution of different intervention strategies to observed change in health outcomes.

More programmatic approaches to health promotion require a more programmatic approach to health promotion evaluation. This paper represents an issues based examination of the evidence base for a more programmatic health promotion and the evaluation issues that arise

Becoming Peer Health Leaders: A Mixed-Methods Evaluation of a Youth Leadership Camp Preparing Peer Leaders for Participation in a Health Promotion Initiative.

Youth are critical partners in health promotion, but the process of training young people to become meaningfully involved is challenging. This mixed-methods evaluation considered the impact of a leadership camp in preparing 42 grade seven students to become peer health leaders in a ‘heart health’ initiative. The experiences of participants and their sense of agency were explored. Data were collected from pre and post camp surveys, focus groups, student journals and researcher observations. Findings indicate that relationships with peers and adults were key to agency development, and participants appeared to broaden their perspectives on the meanings of ‘health’ and ‘leadership.’ Significant changes on two sub-scales of the Harter Perceived Competence Scale for Children were also found. Suggestions for practice and further research are provided.

Quantitative evaluation in health promotion : What's the difference?

This is the third in a series of articles on quantitative evaluation in health promotion written by Damien Jolley for the Health Promotion Journal of Australia. The first of these articles, published in the December 2000 issue, discussed the ideas behind sample surveys and how they can be used to improve evaluation of health promotion initiatives.1 The second, in the April 2001 issue, discussed confidence intervals in more detail and presented some strategies for computing confidence intervals for population percentages, both manually and using appropriate computer software. 2

Impact evaluation of a five-day short course in health promotion: workforce development in action

Issue addressed: This paper reports on impact evaluation of a series of five-day Short Courses in Health Promotion that have been delivered to more than 2,000 people since 2002 as part of a statewide workforce development strategy.

Methods: A triangulated mixed methods research design was selected for the evaluation. Data were collected through a mail survey, key informant interviews, focus groups and organisational case studies. Stakeholder and participant involvement were central to the evaluation.

Results: Organisational change emerged as a key theme. Impacts of the short course were felt in relation to health promotion practice and on organisational capacity to conduct health promotion, while the development of confidence and skills of participants to engage in collaborative opportunities was a not unexpected, but important, benefit of the course.

Conclusions: A short course is effective if attention is given to quality delivery, adult learning methods, participant involvement, appropriate targeting, good planning, and adequate funding. However, respondents commonly report the need for organisational change in order for health promotion practice to be embedded into organisations and for practitioners to be supported in their efforts to re-orient services towards health promotion.

Process evaluation of an integrated health promotion/occupational health model in WellWorks-2

Disparities in chronic disease risk by occupation call for newapproaches to health promotion. Well Works-2 was a randomized, controlled study comparing the effectiveness of a health promotion/occupational health program (HP/OHS) with a standard intervention (HP). Interventions in both studies were based on the same theoretical foundations. Results from process evaluation revealed that a similar number of activities were offered in both conditions and that in the HP/OHS condition there were higher levels of worker participation using three measures: mean participation per activity (HP: 14.2% vs. HP/OHS: 21.2%), mean minutes of worker exposure to the intervention/site (HP: 14.9 vs. HP/OHS: 33.3), and overall mean participation per site (HP: 34.4% vs. HP/ OHS: 45.8%). There were a greater number of contacts with management (HP: 8.8 vs. HP/OHS: 24.9) in the HP/ OHS condition. Addressing occupational health may have contributed to higher levels of worker and management participation and smoking cessation among blue-collar workers.

From 'what' to 'how' : capacity building in health promotion for HIV/AIDS prevention in the Solomon Islands

This paper describes a capacity building process undertaken within the HIV/AIDS prevention project of the Adventist Development and Relief Agency (ADRA) in the Solomon Islands. ADRA HIV/AIDS has recently reoriented its project structure, moving beyond its awareness raising approach to incorporate health promotion frameworks, theories, strategies and assumptions. These have been used to inform project practice in project planning, delivery and evaluation. This paper shares what has worked and not worked in the capacity building process, including a project evaluation of the initial HIV/AIDS awareness raising project and the application of a number of capacity building strategies, including utilising a volunteer Australian Youth Ambassador for Development (AYAD) funded by the Australian Agency for International Development (AusAID). Existing and new projects are outlined. The underlying theme is that any capacity building exercise must include structural support (e.g. management, national frameworks) to ensure the incorporation of new initiatives and approaches. With time this enables ownership by counterparts and external partnerships to develop. The presence of an AYAD volunteer has been an effective strategy to achieve this. Reflections from the evaluators, the AYAD volunteer and the HIV/AIDS team are included.

Evolving directions in health promotion workforce development

PROJECT CONTEXT: Leaders in the fields of public health and health promotion increasingly advocate a socio-ecological approach to meet contemporary and emerging population health challenges. It is essential that health promotion workforce development initiatives mirror the evolving direction of the field to facilitate translation of theory into practice. To date, there has been limited effort to map the socio-ecological approach into tertiary education curricula. PROJECT DESCRIPTION: This project was undertaken as part of the development process for an undergraduate health promotion degree in Queensland, Australia. A review of the health promotion workforce development literature was undertaken. Group processes, key informant interviews and a Delphi technique were used to engage health promotion academics and practitioners, including an International Health Promotion Expert Advisory Panel, and an Industry Advisory Group in defining the components of the program. FINDINGS: The consultative processes facilitated the development of an undergraduate health promotion degree program underpinned by the socio-ecological approach with strong emphases upon the processes or 'how you do it' of health promotion together with evidence-based decision making and practice. CONCLUSIONS: As the basis and practice of health promotion progresses toward a socio-ecological approach, workforce training needs to keep pace with these developments to ensure an appropriately skilled health promotion workforce to meet emerging population health challenges. The reported project and the degree program that has been developed is an example of one step towards achieving this important and necessary shift in health promotion workforce development in Australia.