Assessing the impacts of community participation policy and practice in Scotland and England

Background: Community participation has become an integral part of many areas of public policy over the last two decades. For a variety of reasons, ranging from concerns about social cohesion and unrest to perceived failings in public services, governments in the UK and elsewhere have turned to communities as both a site of intervention and a potential solution. In contemporary policy, the shift to community is exemplified by the UK Government’s Big Society/Localism agenda and the Scottish Government’s emphasis on Community Empowerment. Through such policies, communities have been increasingly encouraged to help themselves in various ways, to work with public agencies in reshaping services, and to become more engaged in the democratic process. These developments have led some theorists to argue that responsibilities are being shifted from the state onto communities, representing a new form of 'government through community' (Rose, 1996; Imrie and Raco, 2003). Despite this policy development, there is surprisingly little evidence which demonstrates the outcomes of the different forms of community participation. This study attempts to address this gap in two ways. Firstly, it explores the ways in which community participation policy in Scotland and England are playing out in practice. And secondly, it assesses the outcomes of different forms of community participation taking place within these broad policy contexts. Methodology: The study employs an innovative combination of the two main theory-based evaluation methodologies, Theories of Change (ToC) and Realist Evaluation (RE), building on ideas generated by earlier applications of each approach (Blamey and Mackenzie, 2007). ToC methodology is used to analyse the national policy frameworks and the general approach of community organisations in six case studies, three in Scotland and three in England. The local evidence from the community organisations’ theories of change is then used to analyse and critique the assumptions which underlie the Localism and Community Empowerment policies. Alongside this, across the six case studies, a RE approach is utilised to examine the specific mechanisms which operate to deliver outcomes from community participation processes, and to explore the contextual factors which influence their operation. Given the innovative methodological approach, the study also engages in some focused reflection on the practicality and usefulness of combining ToC and RE approaches. Findings: The case studies provide significant evidence of the outcomes that community organisations can deliver through directly providing services or facilities, and through influencing public services. Important contextual factors in both countries include particular strengths within communities and positive relationships with at least part of the local state, although this often exists in parallel with elements of conflict. Notably this evidence suggests that the idea of responsibilisation needs to be examined in a more nuanced fashion, incorporating issues of risk and power, as well the active agency of communities and the local state. Thus communities may sometimes willingly take on responsibility in return for power, although this may also engender significant risk, with the balance between these three elements being significantly mediated by local government. The evidence also highlights the impacts of austerity on community participation, with cuts to local government budgets in particular increasing the degree of risk and responsibility for communities and reducing opportunities for power. Furthermore, the case studies demonstrate the importance of inequalities within and between communities, operating through a socio-economic gradient in community capacity. This has the potential to make community participation policy regressive as more affluent communities are more able to take advantage of additional powers and local authorities have less resource to support the capacity of more disadvantaged communities. For Localism in particular, the findings suggest that some of the ‘new community rights’ may provide opportunities for communities to gain power and generate positive social outcomes. However, the English case studies also highlight the substantial risks involved and the extent to which such opportunities are being undermined by austerity. The case studies suggest that cuts to local government budgets have the potential to undermine some aspects of Localism almost entirely, and that the very limited interest in inequalities means that Localism may be both ‘empowering the powerful’ (Hastings and Matthews, 2014) and further disempowering the powerless. For Community Empowerment, the study demonstrates the ways in which community organisations can gain power and deliver positive social outcomes within the broad policy framework. However, whilst Community Empowerment is ostensibly less regressive, there are still significant challenges to be addressed. In particular, the case studies highlight significant constraints on the notion that communities can ‘choose their own level of empowerment’, and the assumption of partnership working between communities and the local state needs to take into account the evidence of very mixed relationships in practice. Most importantly, whilst austerity has had more limited impacts on local government in Scotland so far, the projected cuts in this area may leave Community Empowerment vulnerable to the dangers of regressive impact highlighted for Localism. Methodologically, the study shows that ToC and RE can be practically applied together and that there may be significant benefits of the combination. ToC offers a productive framework for policy analysis and combining this with data derived from local ToCs provides a powerful lens through which to examine and critique the aims and assumptions of national policy. ToC models also provide a useful framework within which to identify specific causal mechanisms, using RE methodology and, again, the data from local ToC work can enable significant learning about ‘what works for whom in what circumstances’ (Pawson and Tilley, 1997).

Supporting self-management: development and evaluation of a digital resource for adults with asthma

The problem: Around 300 million people worldwide have asthma and prevalence is increasing. Support for optimal self-management can be effective in improving a range of outcomes and is cost effective, but is underutilised as a treatment strategy. Supporting optimum self-management using digital technology shows promise, but how best to do this is not clear. Aim: The purpose of this project was to explore the potential role of a digital intervention in promoting optimum self-management in adults with asthma. Methods: Following the MRC Guidance on the Development and Evaluation of Complex Interventions which advocates using theory, evidence, user testing and appropriate modelling and piloting, this project had 3 phases. Phase 1: Examination of the literature to inform phases 2 and 3, using systematic review methods and focussed literature searching. Phase 2: Developing the Living Well with Asthma website. A prototype (paper-based) version of the website was developed iteratively with input from a multidisciplinary expert panel, empirical evidence from the literature (from phase 1), and potential end users via focus groups (adults with asthma and practice nurses). Implementation and behaviour change theories informed this process. The paper-based designs were converted to the website through an iterative user centred process (think aloud studies with adults with asthma). Participants considered contents, layout, and navigation. Development was agile using feedback from the think aloud sessions immediately to inform design and subsequent think aloud sessions. Phase 3: A pilot randomised controlled trial over 12 weeks to evaluate the feasibility of a Phase 3 trial of Living Well with Asthma to support self-management. Primary outcomes were 1) recruitment & retention; 2) website use; 3) Asthma Control Questionnaire (ACQ) score change from baseline; 4) Mini Asthma Quality of Life (AQLQ) score change from baseline. Secondary outcomes were patient activation, adherence, lung function, fractional exhaled nitric oxide (FeNO), generic quality of life measure (EQ-5D), medication use, prescribing and health services contacts. Results: Phase1: Demonstrated that while digital interventions show promise, with some evidence of effectiveness in certain outcomes, participants were poorly characterised, telling us little about the reach of these interventions. The interventions themselves were poorly described making drawing definitive conclusions about what worked and what did not impossible. Phase 2: The literature indicated that important aspects to cover in any self-management intervention (digital or not) included: asthma action plans, regular health professional review, trigger avoidance, psychological functioning, self-monitoring, inhaler technique, and goal setting. The website asked users to aim to be symptom free. Key behaviours targeted to achieve this include: optimising medication use (including inhaler technique); attending primary care asthma reviews; using asthma action plans; increasing physical activity levels; and stopping smoking. The website had 11 sections, plus email reminders, which promoted these behaviours. Feedback during think aloud studies was mainly positive with most changes focussing on clarification of language, order of pages and usability issues mainly relating to navigation difficulties. Phase 3: To achieve our recruitment target 5383 potential participants were invited, leading to 51 participants randomised (25 to intervention group). Age range 16-78 years; 75% female; 28% from most deprived quintile. Nineteen (76%) of the intervention group used the website for an average of 23 minutes. Non-significant improvements in favour of the intervention group observed in the ACQ score (-0.36; 95% confidence interval: -0.96, 0.23; p=0.225), and mini-AQLQ scores (0.38; -0.13, 0.89; p=0.136). A significant improvement was observed in the activity limitation domain of the mini-AQLQ (0.60; 0.05 to 1.15; p = 0.034). Secondary outcomes showed increased patient activation and reduced reliance on reliever medication. There was no significant difference in the remaining secondary outcomes. There were no adverse events. Conclusion: Living Well with Asthma has been shown to be acceptable to potential end users, and has potential for effectiveness. This intervention merits further development, and subsequent evaluation in a Phase III full scale RCT.

Using interaction data for improving the offline and online evaluation of search engines

This thesis investigates how web search evaluation can be improved using historical interaction data. Modern search engines combine offline and online evaluation approaches in a sequence of steps that a tested change needs to pass through to be accepted as an improvement and subsequently deployed. We refer to such a sequence of steps as an evaluation pipeline. In this thesis, we consider the evaluation pipeline to contain three sequential steps: an offline evaluation step, an online evaluation scheduling step, and an online evaluation step. In this thesis we show that historical user interaction data can aid in improving the accuracy or efficiency of each of the steps of the web search evaluation pipeline. As a result of these improvements, the overall efficiency of the entire evaluation pipeline is increased. Firstly, we investigate how user interaction data can be used to build accurate offline evaluation methods for query auto-completion mechanisms. We propose a family of offline evaluation metrics for query auto-completion that represents the effort the user has to spend in order to submit their query. The parameters of our proposed metrics are trained against a set of user interactions recorded in the search engine’s query logs. From our experimental study, we observe that our proposed metrics are significantly more correlated with an online user satisfaction indicator than the metrics proposed in the existing literature. Hence, fewer changes will pass the offline evaluation step to be rejected after the online evaluation step. As a result, this would allow us to achieve a higher efficiency of the entire evaluation pipeline. Secondly, we state the problem of the optimised scheduling of online experiments. We tackle this problem by considering a greedy scheduler that prioritises the evaluation queue according to the predicted likelihood of success of a particular experiment. This predictor is trained on a set of online experiments, and uses a diverse set of features to represent an online experiment. Our study demonstrates that a higher number of successful experiments per unit of time can be achieved by deploying such a scheduler on the second step of the evaluation pipeline. Consequently, we argue that the efficiency of the evaluation pipeline can be increased. Next, to improve the efficiency of the online evaluation step, we propose the Generalised Team Draft interleaving framework. Generalised Team Draft considers both the interleaving policy (how often a particular combination of results is shown) and click scoring (how important each click is) as parameters in a data-driven optimisation of the interleaving sensitivity. Further, Generalised Team Draft is applicable beyond domains with a list-based representation of results, i.e. in domains with a grid-based representation, such as image search. Our study using datasets of interleaving experiments performed both in document and image search domains demonstrates that Generalised Team Draft achieves the highest sensitivity. A higher sensitivity indicates that the interleaving experiments can be deployed for a shorter period of time or use a smaller sample of users. Importantly, Generalised Team Draft optimises the interleaving parameters w.r.t. historical interaction data recorded in the interleaving experiments. Finally, we propose to apply the sequential testing methods to reduce the mean deployment time for the interleaving experiments. We adapt two sequential tests for the interleaving experimentation. We demonstrate that one can achieve a significant decrease in experiment duration by using such sequential testing methods. The highest efficiency is achieved by the sequential tests that adjust their stopping thresholds using historical interaction data recorded in diagnostic experiments. Our further experimental study demonstrates that cumulative gains in the online experimentation efficiency can be achieved by combining the interleaving sensitivity optimisation approaches, including Generalised Team Draft, and the sequential testing approaches. Overall, the central contributions of this thesis are the proposed approaches to improve the accuracy or efficiency of the steps of the evaluation pipeline: the offline evaluation frameworks for the query auto-completion, an approach for the optimised scheduling of online experiments, a general framework for the efficient online interleaving evaluation, and a sequential testing approach for the online search evaluation. The experiments in this thesis are based on massive real-life datasets obtained from Yandex, a leading commercial search engine. These experiments demonstrate the potential of the proposed approaches to improve the efficiency of the evaluation pipeline.