External Evaluation for the BCE-CT Project

Strand one: Effectiveness of project activities and processes

External school evaluation : teachers’ perspectives : two case studies from Northern Portugal

The external evaluation of non-higher education schools in Portugal has been developed by the General Inspectorate of Education since 2006. A first cycle of evaluation was completed, covering all educational units in continental Portugal up to 2011. The model of evaluation has since been subject to alterations, and a second cycle of evaluation is now coming to an end. The current model of evaluation is based on documental analysis, analysis of students’ results, and panel interviews with a variety of representatives of the school community, and addresses three domains: results, provision of educational service and management. This paper is part of an ongoing research project, developed by 6 universities and supported by the Portuguese Foundation for Science and Technology (PTDC/CPE-CED/116674/2010) which intends to analyse the impacts and effects this process of external evaluation has had on Portuguese schools. This project includes a variety of perspectives and methodologies. In particular, we will focus on two case studies undertaken in two schools from the northern region of Portugal, and more specifically on the perspectives expressed by the teachers of those schools. These particular schools were chosen because they have been evaluated twice and represent different educational levels (basic and secondary), contexts and results. These case studies included the analysis of documental data, interviews to key informants and a questionnaire directed to teachers (n = 141) – the latter will be the main focus of this paper. Teachers are essential elements of the school community when considering the impacts of external evaluation, as any changes directed at teaching practices, student evaluation, among others are only possible through their direct action and implication. Therefore, their perceptions on the process and its impacts are crucial to the understanding of what does and does not change in schools as a consequence of external evaluation. Although teachers’ opinions are not homogenous and each school reveals a number of differences when it comes to teachers’ perceptions of School Evaluation, it was possible to stress some areas as the most and as the least consensual. Teachers in both schools agree External School Evaluation (ESE) is useful for the identification of the schools’ strengths and weaknesses, values students’ external evaluation results, imposes a model for schools internal evaluation (and in fact contributes to the very existence of internal evaluation practices), and contributes to schools improvement. However teachers in both schools do not believe ESE contributes to teachers’ autonomy produces changes in how curriculum is managed, or leads to innovative teaching practices. These results point to a greater emphasis on change at the levels of school management, self-evaluation and particularly internal evaluation, but little impact on the teaching practices. We believe the classroom is at the core of school practices and teaching processes are essential to any measure of school quality and to their impacts on student learning.

Wishful Thinking: Lessons from the internal and external evaluations of an innovatory education project in Northern Ireland

From March 1999 to August 2000, the authors were involved in simultaneous internal and external evaluations of the social civic and political education (SCaPE) project in Northern Ireland. This project was a major initiative established by the Citizenship Foundation, the Northern Ireland Council for the Curriculum, Examination and Assessment (CCEA), and the School of Education at the University of Ulster at Coleraine. It was a 2-year project in 25 secondary schools established to design, develop, pilot and evaluate a new programme of social, civic and political education for Northern Ireland. It also aimed to serve as a model for future Citizenship curriculum developments throughout Northern Ireland and elsewhere. This paper describes the background to the project, the design and conduct of the two evaluations, and the links between them. It outlines the main conclusions of each evaluation and describes the way SCaPE has since evolved into a mainstream curriculum development project. The final part of the paper analyses the key opportunities, tensions and challenges involved in running such evaluations at a critical time in the history of Northern Ireland – a time when innovation is both necessary and controversial. It argues that, especially in such circumstances, evaluation cannot be conducted from a neutral, objective standpoint, and that it is incumbent on evaluators to recognise the emotional, personal and political commitment they make to the projects in which they are engaged.

Grouping people in social networks using a weighted multi-constraints clustering method

Grouping users in social networks is an important process that improves matching and recommendation activities in social networks. The data mining methods of clustering can be used in grouping the users in social networks. However, the existing general purpose clustering algorithms perform poorly on the social network data due to the special nature of users' data in social networks. One main reason is the constraints that need to be considered in grouping users in social networks. Another reason is the need of capturing large amount of information about users which imposes computational complexity to an algorithm. In this paper, we propose a scalable and effective constraint-based clustering algorithm based on a global similarity measure that takes into consideration the users' constraints and their importance in social networks. Each constraint's importance is calculated based on the occurrence of this constraint in the dataset. Performance of the algorithm is demonstrated on a dataset obtained from an online dating website using internal and external evaluation measures. Results show that the proposed algorithm is able to increases the accuracy of matching users in social networks by 10% in comparison to other algorithms.

The role of transformative thinking and practice in a property entrepreneur's response to the funding crisis since 2008

This portfolio of exploration explores the role of transformative thinking and practice in a property entrepreneur’s response to the financial crisis which swept over Ireland from 2008. The complexity of this challenge and the mental capacity to meet its demands is at the core of the exploration. This inquiry emerged from the challenges the financial crisis presented to my values, beliefs, assumptions, and theories, i.e. the interpretive lens through which I make meaning of my experiences. Given the issue identified, this inquiry is grounded in aspects of theories of constructive developmental psychology, applied developmental science, and philosophy. Integrating and linking these elements to business practice is the applied element of the Portfolio. As the 2008 crisis unfolded I realised I was at the limits of my way of knowing. I came to understand that the underlying structure of a way of knowing is the ‘subject-object relationship’ i.e. what a way of knowing can reflect upon, look at, have perspective on, in other words, make object, as against what is it embedded in, attached to, identified within, or subject to. My goal became enhancing my awareness of how I made meaning and how new insights, which would transform a way of knowing, are created. The focus was on enhancing my practice. This Portfolio is structured into three essays. Essay One reported on my self-reflection and external evaluation out of which emerged my developmental goals. In Essay Two I undertake a reading for change programme in which different meaning making systems were confronted in order to challenge me as a meaning maker. Essay Three reported on my experiment which concerned the question whether it was possible for me as a property entrepreneur, and for others alike, to retain bank finance in the face of the overwhelming objective of the bank to deleverage their balance sheet of property loans. The output of my research can be grouped into General Developmental and Specific Business Implications. Firstly I address those who are interested in a transformational-based response to the challenges of operating in the property sector in Ireland during a crisis. I outline the apparatus of thought that I used to create insight, and thus transform how I thought, these are Awareness, Subject-object separation, Exploring other’s perspectives from the position of incompleteness, Dialectical thinking and Collingwood’s Questioning activity. Secondly I set out my learnings from the crisis and their impact on entrepreneurial behaviour and the business of property development. I identify ten key insights that have emerged from leading a property company through the crisis. Many of these are grounded in common sense, however, in my experience these were, to borrow Shakespeare’s words, “More honor'd in the breach than the observance” in pre Crisis Ireland. Finally I set out a four-step approach for forging a strategy. This requires my peer practitioners to identify (i) what they are subject to, (ii) Assess the Opportunity or challenge in a Systemic Context, (iii) Explore Multiple Perspectives on the opportunity or Challenge with an Orientation to change how you know and (iv) Using the Questioning Activity to create Knowledge. Based on my experience I conclude that transformative thinking and practice is a key enabler for a property entrepreneur, in responding to a major collapse of traditional (bank debt) funding.

QA and Enhancement Marketplace for HEIs – An Erasmus+ project

Improving European education and training system quality has been set as a key target in Europe’s strategy to become a smart, sustainable and inclusive economy by 2020 (European Commission, 2010). These objectives are more specifically defined in the so called Modernisation Agenda (European Commission, 2011). More specifically it sets a goal to improve the quality and relevance of higher education. In this process external evaluation and
Proceedings of the 11th International CDIO Conference, Chengdu University of Information Technology,
Chengdu, Sichuan, P.R. China, June 8-11, 2015.
self-assessment are seen in a key role! In the CDIO approach the 12 CDIO standards provide a framework for continuous improvement. Each institution/institutional department are encouraged to regularly do the self-evaluation using the CDIO Standards. Eight European universities identified a need for further enhancement of the self-evaluations and creation of processes with peers to reduce the inertia of heavy accreditations/evaluations in HEIs. In September 2014 these universities started an Erasmus+ project (QAEMarketPlace4HEI) aiming at
1. Developing a collaborative, comprehensive and accessible evaluation process model, methods and tools for HEIs to complement the accreditation systems.
2. Promoting, increasing and exploiting further the European collaboration in the evaluation processes and the exchange of best practices.
3. Disseminating the model, best practices and widen the cooperation to new HEIs in Europe through the partner networks.

Avaliação da Escola - Resultados Escolares e a procura da Qualidade

Trabalho de Projeto apresentado para a obtenção do grau de Mestre na especialidade em Administração Educacional. Orientador:Prof. Doutor João Sebastião

Papel da assessoria na auto-avaliação das escolas

Esta tese apresenta uma investigação que procura problematizar o papel da assessoria na auto-avaliação das escolas. Reflecte-se sobre o contributo de assessores externos nas unidades de gestão para encontrar fórmulas que permitam a construção e sustentabilidade da sua auto-avaliação através da concepção, monitorização e avaliação de planos de acção. A investigação consubstanciou-se na análise dos relatórios das 100 unidades orgânicas que participaram na avaliação externa em 2006/2007 promovida pela Inspecção-Geral de Educação (IGE) especificamente no domínio da Capacidade de Auto-Regulação e Melhoria da Escola (CARME), no Questionário a 90 participantes em equipas de auto-avaliação e ainda em todo o trabalho de campo realizado numa escola secundária através de observação participante. Durante a investigação percebeu-se a adaptação de alguns modelos de auto-avaliação de carácter gerencialista à realidade das unidades orgânicas, identificam-se alguns obstáculos que se apresentam à aplicação dos referidos modelos, as dificuldades que as referidas unidades de gestão sentem quando autonomamente se querem auto-avaliar e identificam-se mecanismos utilizados para a concretização da auto-avaliação e sua manutenção ao longo do tempo. Por fim, constata-se que a assessoria externa é um pilar que pode contribuir para a introdução efectiva dos referidos mecanismos de auto-avaliação e sua sustentabilidade.

Impactos do processo RVCC de nível básico : aprender compensou?

Trabalho de projecto de mestrado, Ciências da Educação (Formação de Adultos), Universidade de Lisboa, Instituto de Educação, 2011

Perceções da equipa de avaliação extrerna relativamente às suas práticas de avaliação

Relatório de estágio de mestrado, Ciências da Educação (Administração Educacional), Universidade de Lisboa, Instituto de Educação, 2013

Do currículo de física e química instituído ao aplicado nas escolas : das metodologias de ensino à avaliação

Tese de doutoramento, Educação (Avaliação em Educação), Universidade de Lisboa, Instituto de Educação, 2015

As representações das equipas de avaliação externa da IGEC na dimensão relativa à autoavaliação das escolas

Relatório de estágio de mestrado, Ciências da Educação (Especialização em Administração Educacional), Universidade de Lisboa, Instituto de Educação, 2014

Gestão Escolar e dinâmicas de avaliação externa num agrupamento de escolas

Relatório de estágio de mestrado, Ciências da Educação (Área de especialização em Administração Educacional), Universidade de Lisboa, Instituto de Educação, 2014

Comparing clustering solutions: the use of adjusted paired indices

In the present paper we compare clustering solutions using indices of paired agreement. We propose a new method - IADJUST - to correct indices of paired agreement, excluding agreement by chance. This new method overcomes previous limitations known in the literature as it permits the correction of any index. We illustrate its use in external clustering validation, to measure the accordance between clusters and an a priori known structure. The adjusted indices are intended to provide a realistic measure of clustering performance that excludes agreement by chance with ground truth. We use simulated data sets, under a range of scenarios - considering diverse numbers of clusters, clusters overlaps and balances - to discuss the pertinence and the precision of our proposal. Precision is established based on comparisons with the analytical approach for correction specific indices that can be corrected in this way are used for this purpose. The pertinence of the proposed correction is discussed when making a detailed comparison between the performance of two classical clustering approaches, namely Expectation-Maximization (EM) and K-Means (KM) algorithms. Eight indices of paired agreement are studied and new corrected indices are obtained.

Knowledge Building in Landscape Architecture

The objective of this study was to develop an internet-based seminar framework applicable for landscape architecture education. This process was accompanied by various aims. The basic expectation was to keep the main characteristics of landscape architecture education also in the online format. On top of that, four further objectives were anticipated: (1) training of competences for virtual team work, (2) fostering intercultural competence, (3) creation of equal opportunities for education through internet-based open access and (4) synergy effects and learning processes across institutional boundaries. This work started with the hypothesis that these four expected advantages would compensate for additional organisational efforts caused by the online delivery of the seminars and thus lead to a sustainable integration of this new learning mode into landscape architecture curricula. This rationale was followed by a presentation of four areas of knowledge to which the seminar development was directly related (1) landscape architecture as a subject and its pedagogy, (2) general learning theories, (3) developments in the ICT sector and (4) wider societal driving forces such as global citizenship and the increase of open educational resources. The research design took the shape of a pedagogical action research cycle. This approach was constructive: The author herself is teaching international landscape architecture students so that the model could directly be applied in practice. Seven online seminars were implemented in the period from 2008 to 2013 and this experience represents the core of this study. The seminars were conducted with varying themes while its pedagogy, organisation and the technological tools remained widely identical. The research design is further based on three levels of observation: (1) the seminar design on the basis of theory and methods from the learning sciences, in particular educational constructivism, (2) the seminar evaluation and (3) the evaluation of the seminars’ long term impact. The seminar model itself basically consists of four elements: (1) the taxonomy of learning objectives, (2) ICT tools and their application and pedagogy, (3) process models and (4) the case study framework. The seminar framework was followed by the presentation of the evaluation findings. The major findings of this study can be summed up as follows: Implementing online seminars across educational and national boundaries was possible both in term of organisation and technology. In particular, a high level of cultural diversity among the seminar participants has definitively been achieved. However, there were also obvious obstacles. These were primarily competing study commitments and incompatible schedules among the students attending from different academic programmes, partly even in different time zones. Both factors had negative impact on the individual and working group performances. With respect to the technical framework it can be concluded that the majority of the participants were able to use the tools either directly without any problem or after overcoming some smaller problems. Also the seminar wiki was intensively used for completing the seminar assignments. However, too less truly collaborative text production was observed which could be improved by changing the requirements for the collaborative task. Two different process models have been applied for guiding the collaboration of the small groups and both were in general successful. However, it needs to be said that even if the students were able to follow the collaborative task and to co-construct and compare case studies, most of them were not able to synthesize the knowledge they had compiled. This means that the area of consideration often remained on the level of the case and further reflections, generalisations and critique were largely missing. This shows that the seminar model needs to find better ways for triggering knowledge building and critical reflection. It was also suggested to have a more differentiated group building strategy in future seminars. A comparison of pre- and post seminar concept maps showed that an increase of factual and conceptual knowledge on the individual level was widely recognizable. Also the evaluation of the case studies (the major seminar output) revealed that the students have undergone developments of both the factual and the conceptual knowledge domain. Also their self-assessment with respect to individual learning development showed that the highest consensus was achieved in the field of subject-specific knowledge. The participants were much more doubtful with regard to the progress of generic competences such as analysis, communication and organisation. However, 50% of the participants confirmed that they perceived individual development on all competence areas the survey had asked for. Have the additional four targets been met? Concerning the competences for working in a virtual team it can be concluded that the vast majority was able to use the internet-based tools and to work with them in a target-oriented way. However, there were obvious differences regarding the intensity and activity of participation, both because of external and personal factors. A very positive aspect is the achievement of a high cultural diversity supporting the participants’ intercultural competence. Learning from group members was obviously a success factor for the working groups. Regarding the possibilities for better accessibility of educational opportunities it became clear that a significant number of participants were not able to go abroad during their studies because of financial or personal reasons. They confirmed that the online seminar was to some extent a compensation for not having been abroad for studying. Inter-institutional learning and synergy was achieved in so far that many teachers from different countries contributed with individual lectures. However, those teachers hardly ever followed more than one session. Therefore, the learning effect remained largely within the seminar learning group. Looking back at the research design it can be said that the pedagogical action research cycle was an appropriate and valuable approach allowing for strong interaction between theory and practice. However, some more external evaluation from peers in particular regarding the participants’ products would have been valuable.