Exploring the Possibilities and Methodological Challenges of Evaluation Practice : A Democratic Question

In our exploration of the methodological possibilities and challenges of evaluation practice that foregrounds a democratic approach, we argue that having sensitivity to participants’ different positions, ideological perspectives, and values, and an understanding of the existing power relationships are paramount for the evaluators if the evaluation is to achieve its goals. This chapter draws on our experiences as evaluators in Australia, where school leaders and teachers are currently experiencing significant curriculum and assessment changes, and now work in a context of heightened accountability. Our evaluation practices with schools are used to illustrate how democratic practices can be incorporated into evaluation in ways that provide opportunities and support for school self-evaluation. It is paramount to recognise that evaluation is a political and ideological practice, and therefore is not a neutral process. Our argument is that it remains the responsibility of evaluators to design spaces that build sustainable relationships with participants and the targets of the programmes being evaluated; ensure that dialogue and deliberation are valued between all stakeholders; represent a range of interests; and give a service back to the evaluated communities by offering understandings and promoting reflective practice and informed decision-making throughout programme implementation.

ewaluacja Nowej Matury

In the frame of the Polish educational reform there are some attempts undertaken to improve the final exam of secondary school. The reformers' proposals undergo an outside analysis and crittique. The author presents two models of democratic evaluation of the so-called in the “matura” exam. One model is a qualitative analysis of the criteria of grading essays in the subject of the Polish language. The other model is a quantitative study of qualitative differences between problem tasks within the exam in mathematics. Both naturalistic approaches are based on evaluating opinions of the subjects of the „matura”, i.e. students who take the exam and teachers who grade the output.

A Construção do processo democrático de auto-avaliação institutucional na UNESP

O desenvolvimento de um processo democrático de avaliação institucional na UNESP, uma universidade multi-campus, envolveu três desafios: garantir o envolvimento de grupos representativos da comunidade no processo de avaliação; atribuir poder e responsabilidade a cada setor acadêmico e administrativo no processo de avaliação; criar uma cultura de auto-avaliação e reflexão que possibilitasse debate crítico e auto-gestão dos projetos acadêmicos. Com base nos termos de referência estabelecidos pela comunidade a CPA desenvolveu uma metodologia de avaliação democrática e investigativa, orientada para a auto-gestão e de natureza quali-quantitativa. Fundamentada em três enfoques teóricos de avaliação, o democrático, o de tomada de decisão e o crítico o processo envolveu três funções: a diagnóstica, a formativo-reflexiva e a de revisão crítica. Atualmente os usos dos resultados da avaliação para orientar políticas e decisões acadêmicas, especialmente nos cursos de graduação, evidenciam avanços no processo de internalização de cultura de avaliação orientada para auto-gestão.

Tracing the Evidence: The Use of Evaluation Results in Direct-Democratic Campaigns

Although research on direct-democratic campaigns in Switzerland has intensified in the last decade, detailed information on the use of evidence in campaigns is still lacking. Our research aims to contribute both to research on direct democracy and to research on evidence-based policy making, by analyzing how evaluation results are used in directdemocratic campaigns. In this conceptual paper, the formulation of our hypothesis is based on a model of evaluation influence that traces the different uses of evaluation results in the process of a direct-democratic campaign. We assume that the policy analytical capacity of individual members in parliament, government and administration in the (pre)-parliamentary process fosters the use of evidence in campaigns. In the course of the campaign, symbolic use of evaluation in the form of justification, persuasion or mobilization prevails. We assume that the media is an important player in making transparent how political actors use evidence to support their positions. Evidence itself often remains ambiguous and uncertain, and evaluations are influenced by the values of the evaluator. To be able to make the right decisions, therefore, citizens should learn about possible interpretations in argumentative processes. For us, the context of direct democracy in Switzerland provides the setting for such a discourse that, besides evidence, brings up different opinions, values and beliefs.

Mediated politics in Australia : towards a qualitative evaluation

The debate about the democratic significance of these trends—a more aggressively inquisitorial media environment, greater public participation in political communication, a more accessible and transparent (at least in appearance) political class—continues, not least in Australia. This essay was written in the first half of 2013, a time of extreme political volatility in Australia, and in the run-up to a general election following three years of minority Labor government. By that stage in the political cycle, Prime Minister Julia Gillard had survived not one but two attempts at leadership “spills”, ministers had resigned or been sacked for disloyalty to the leader, major policy initiatives had been dumped, reversed or quietly dropped, and a Coalition opposition was confidently looking forward to a landslide majority in the election of September that year. Labor’s internal party turmoil, rather than the Coalition’s policy prospectus (which remained sketchy and vague right up to the eve of the election), were widely assumed to be the cause of the former’s poor standing in the opinion polls.

Information preferences for the evaluation of coastal development impacts on ecosystem services: A multi-criteria assessment in the Australian context

Ecosystem based management requires the integration of various types of assessment indicators. Understanding stakeholders' information preferences is important, in selecting those indicators that best support management and policy. Both the preferences of decision-makers and the general public may matter, in democratic participatory management institutions. This paper presents a multi-criteria analysis aimed at quantifying the relative importance to these groups of economic, ecological and socio-economic indicators usually considered when managing ecosystem services in a coastal development context. The Analytic Hierarchy Process (AHP) is applied within two nationwide surveys in Australia, and preferences of both the general public and decision-makers for these indicators are elicited and compared. Results show that, on average across both groups, the priority in assessing a generic coastal development project is for the ecological assessment of its impacts on marine biodiversity. Ecological assessment indicators are globally preferred to both economic and socio-economic indicators regardless of the nature of the impacts studied. These results are observed for a significantly larger proportion of decision-maker than general public respondents, questioning the extent to which the general public's preferences are well reflected in decision-making processes.

Organisational Self-Evaluation and Teacher Education for Community Relations in a Transforming Society?

During 2004, the School of Education at the University of Ulster embarked on an innovative three-year project designed to embed community relations objectives within initial teacher education. With the advent of more peaceful times in Northern Ireland, this was a precipitous time for initial teacher educators to review the preparation given to beginner teachers for teaching in an increasingly pluralist society emerging from conflict. The present paper reports on one very specific and time-limited element of the broader project. That is, development work designed to investigate the possibilities of using processes of self-review and evaluation as a lever for improvements in initial teacher education for community relations. Following a brief contextualisation, the background to, and the development of, a set of materials designed to support rigorous and systematic self-review of all aspects of provision in a university-based initial teacher education department is described. The Community Relations Index for Initial Teacher Education (Cr-ITE) was envisaged as being of use to initial teacher education establishments in order to help teacher educators take responsibility for rigorous learning from their practice, whilst placing inclusive values at the centre of organisational development. The final section includes further critical reflection on the role of organisational self-review in transforming teacher education for inclusion in a society emerging from longstanding communal conflict.

Reforming central government: An evaluation of an accounting innovation

This paper examines a significant accounting innovation in central government accounting – the introduction of Resource Accounting and Budgeting (RAB) in the UK. This innovation is studied through the lens of Rogers diffusion theory. The study setting is the Scotland Parliament. This research shows that, in the terms of diffusion theory, RAB can be classified as an accounting innovation. However, the implementation of RAB is problematic. While the reform of the UK central government system was initially sought as a mechanism to enhance democratic accountability, this paper shows that RAB does not connect with parliamentarians. The introduction of RAB flows as much from a managerial agenda as it does from the aim of democratic accountability.

Evaluation and Evaluating the Community Initiative "URBAN"

The relevance of the relationship between evaluation and learning is widely acknowledged; as well as the importance of this learning process to support policy decision making and to judge the value of “what was done”, “how” and “with what results”. Although the activity of evaluation is not something new, but a field of knowledge with its own theories and practices, it is important to notice that there is great variability in the tradition and cultures of evaluation between countries. This paper presents and compares different conceptual and methodological frameworks created for the assessment of the European initiative Urban II, including the one that was used by the author in the context of an academic evaluation in the city of Porto. The comparative analysis of the results leads us to the recommendation for more democratic processes of evaluation and intervention, in order to improve their quality and accountability and promote the important goal of learning with this type of experimental initiatives.

PANA V : Evaluation of a Literacy Project

PANA V Evaluation of a Literacy ProjectSUMMARY AND CONCLUSIONSThis evaluation set out to explore the impact of the literacy work carried out through PANA V. It focussed on clarifying effects such as empowerment and poverty reduction in relation to the civil society. Two specific objectives were to evaluate the methodological approach and the didactic materials and to evaluate the sustainability of the project.Although the focus of the evaluation has been PANA V, the project has been evaluated in its context, as one in a series of five projects located in Rwanda ten years after the war and genocide. The conclusion will consider future plans in this field.The evaluator has striven to create a holistic picture of the effects of the project, although the given time for the evaluation was short. Only three weeks were spent in the field study and only ten days in the actual field. Although there were some organisational and logistic problems, as is common when carrying out a study like this in a poor country, many literacy sites were visited and quite many participators were interviewed. The overall impression from the study is overwhelmingly positive. So many people commit themselves in this task of teaching Rwandans reading, writing and numeracy. Despite harsh conditions learners strive to learn and group leaders devote themselves to the task. Many leaders on different levels try their very best to manage their difficult and demanding task. The main objective was to explore the impact of the project on poverty reduction, particularly on empowerment and strategies for everyday life. Women were to be regarded particularly. From the results it is clear that the project has a strong, positive impact both on poverty reduction and empowerment of marginalised groups. Among those who have benefited from the alphabetisation are mainly women. Unfortunately, when it comes to leaders in PANA, who may also be said to have benefited from the project, only a small minority is women. This is something that is recommended that it be reconsidered inside the organisation. As a majority of the targeted learners are women, and as the economic and social situation of women in Rwanda is generally weak, this is a question that I recommend the Pentesostal church and ADEPR to look particularly into. With many women being single breadwinners of their households, it is important that also women get access to positions that may bring benefits of different kind.It is also clear that the project has positive effects for the civil society. In the present situation in Rwanda, during the process of reconciliation and rapid progress, basic education for the poor majority is a democratic issue. In a country with a plethora of internet-cafés in the capital and a small minority that use cars and mobile-telephones to communicate nation-wide, it is of outmost importance that the majority acquires basic education, of which literacy is a central part. To strengthen the civil society in Rwanda literacy is important. One central issue is then that Rwanda develops toward becoming a country where literacy is used for the benefit of the citizens and it is a democratic issue that all citizens get an opportunity to participate. Crucial for this is that strong efforts are put into primary schools nation-wide. Literacy projects for adults, like PANA, may only complement these efforts, but they constitute important and necessary complements. Other relevant ways to promote literacy are campaigns in Radio and TV and through cultural events such as festivals, music and theatre. News papers, magazines and books are natural parts of such campaigns as well as adult education. As stated under the results not much can be said about the didactics in this evaluation. On the whole the methodology and the materials fill their function well and receive a high reputation. As people learn to read and write under very simple conditions, obviously the approach is appropriate. A few suggestions may be given from the study:•Focus groups leaders’ attention on clearness, that they show very clearly what is to be read. Good structuring is probably of great importance for many learners.•Make clear what is tested in the tests and consider the possibility to use a holistic test that would be more congruent with the methodology. The possibility to use only one grade, pass, would enable a more practical test, such as reading a short, relevant text, writing something relevant and solving practical mathematic problems. Avoid tests that demand school knowledge.•Avoid using methaphors such as “fight against illiteracy” and connections between illiteracy/literacy and darkness/light. It is not true that illiteracy causes bad things and that literacy only brings good. •Be prepared that it may be more difficult in the future to achieve the goals as it may be the case that the early learners where the ones who achieved easily. The goal of “literacy in six month” in PANA will probably hold only for some learners but also those who do not manage in six months need literacy skills.A third objective was to secure sustainability. As for sustainability of the project in itself, and of the literacy process, the main conclusion is that there is a good potential. The commitment and devotedness among many involved in PANA proves good. One weakness is individual leaders in ADEPR who do not see this as an important task for the Pentecostal church in Rwanda. Other weaknesses are the unwillingness to mention explicitly the wish, for example among group leaders, to get some kind of incentive and the fear of loosing believers by cooperation with other organisations. A higher degree of transparency in this issue would probably solve some irritations and tensions.As for the sustainability of the literacy skills much may be done to improve. The acquired skills seem to be comparably relevant. The level achieved, and the level tested, may be defined as basic literacy skills, consisting of basic reading, writing and numeracy skills. However, these skills are very restricted and there is a high risk that the skills will decline, which means that there is a high risk that people will forget how to read and write because of lack of exercising. From these conclusions a few suggestions for future development will be given.

Impact assessment and quality analysis of corporate social responsability (CSR) programs of mining companies and their contribution to sustainable development in the communities they operate: case study Tenke Fungurume Mining SA in the Democratic Republic of Congo (DRC)

Purpose – This case study presents an impact assessment of Corporate Social Responsibility (CSR) programs of the TFM Company in order to understand how they contribute to the sustainable development of communities in areas in which they operate. Design/Methodology/Approach - Data for this study was collected using qualitative data methods that included semi-structured interviews and Focus Group Discussions most of them audio and video recorded. Documentary analysis and a field visit were also undertaken for the purpose of quality analysis of the CSR programs on the terrain. Data collected was analyzed using the Seven Questions to sustainability (7Qs) framework, an evaluation tool developed by the Mining, Minerals and Sustainable Development (MMSD) North America chapter. Content analysis method was on the other hand used to examine the interviews and FGDs of the study participants. Findings - Results shows that CSR programs of TFM SA do contribute to community development, as there have been notable changes in the communities’ living conditions. But whether they have contributed to sustainable development is not yet the case as programs that enhance the capacity of communities and other stakeholders to support these projects development beyond the implementation stage and the mines operation lifetime need to be considered and implemented. Originality/Value – In DRC, there is paucity of information of research studies that focus on impact assessment of CSR programs in general and specifically those of mining companies and their contribution to sustainable development of local communities. Many of the available studies cover issues of minerals and conflict or conflict minerals as mostly referred to. This study addressees this gap.

How Welfare States Shape the Democratic Public: Policy Feedback, Participation, Voting and Attitudes

This crucial volume significantly advances the study of policy feedbacks. With contributions from many subfields and methodological approaches, it offers both sophisticated theorizing and new empirical examples that show how policies make politics in a variety of ways. Innovative research designs provide more convincing inference than ever. And the normative questions engaged about welfare performance, evaluation, participation, and accountability could not be more important or timely in this era of austerity and discord over the future of welfare states.’

Does a reference to evaluations contribute to discourse quality? The case of Swiss direct-democratic campaigns on school policy: a conceptual framework

This article examines how references to evaluations in school policy debates contribute to discourse quality. The article consists of two parts: First, it presents a descriptive overview of the references to evidence in direct-democratic campaigns. These results are based on a quantitative content analysis of the newspaper coverage and governmental information documents of 103 direct-democratic Swiss school policy votes. In a second step, it discusses these findings in view of the question of whether the incorporation of evaluation results in policy debates contributes to discourse quality. It presents a conceptual framework, including hypotheses and a research design to answer this question.

Evaluation of Tottenham Thinking Space Pilot: Final Report. Research Report 11.

This reports summarises research that began in March 2014 and was completed in October 2015 by an experienced inter-disciplinary research team from the Centre for Social Justice and Change and Psycho-Social Research Group, School of Social Sciences, the University of East London (UEL) and included Dr Yang Li from the Centre for Geo-Information Studies, UEL, for the first phase of the study. Tottenham ‘Thinking Space’ (TTS) was a pilot therapeutic initiative based in local communities and delivered by the Tavistock & Portman NHS Foundation Trust and funded by the London Borough of Haringey Directorate of Public Health. TTS aimed to improve mental health and enable and empower local communities. TTS was situated within a mental health agenda that was integral to Haringey’s Health and Wellbeing Strategy 2012-2015 and aimed to encourage people to help themselves and each other and develop confident communities. On the one hand TTS was well-suited to this agenda, but, on the other, participants were resistant to, and were trying to free themselves from labelling that implied ‘mental health difficulties’. A total of 243 meetings were held and 351 people attended 1,716 times. The majority of participants attended four times or less, and 33 people attended between 5 and 10 times and 39 people attended over 10 times. Attending a small number of times does not necessarily mean that the attendee was not helped. Attendees reflected the ethnic diversity of Tottenham; 29 different ethnic groups attended. The opportunity to meet with people from different cultural backgrounds in a safe space was highly valued by attendees. Similarly, participants valued the wide age range represented and felt that they benefited from listening to inter-generational experiences. The majority of participants were women (72%) and they were instrumental in initiating further Thinking Spaces, topic specific meetings, the summer programme of activities for mothers and young children and training to meet their needs. The community development worker had a key role in implementing the initiative and sustaining its growth throughout the pilot period. We observed that TTS attracted those whose life experiences were marked by personal struggle and trauma. Many participants felt safe enough to disclose mental health difficulties (85% of those who completed a questionnaire). Participants also came seeking a stronger sense of community in their local area. Key features of the meetings are that they are democratic, non-judgemental, respectful, and focussed on encouraging everyone to listen and to try to understand. We found that the therapeutic method was put in place by high quality facilitators and health and personal outcomes for participants were consistent with those predicted by the underpinning psychoanalytical and systemic theories. Outcomes included a reduction in anxieties and improved personal and social functioning; approximately two thirds of those who completed a questionnaire felt better understood, felt more motivated and more hopeful for the future. The overwhelming majority of survey respondents also felt good about contributing to their community, said that they were more able to cooperate with others and accepting of other cultures, and had made new friends. Participants typically had a better understanding of their current situation and how to take positive action; of those who completed a questionnaire, over half felt more confident to seek support for a personal issue and to contact services. Members of TTS supported each other and instilled hope and build community-mindedness that reduced social isolation.