Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

A state-wide systems approach to embedding the learning and teaching of sustainability in teacher education

Education systems have a key role to play in preparing future citizens to engage in sustainable living practices and help create a more sustainable world. Many schools throughout Australia have begun to develop whole-school approaches to sustainability education that are supported by national and state policies and curriculum frameworks. Preservice teacher education, however, lags behind in building the capacity of new teachers to initiate and implement such approaches (ARIES, 2010). This proposed project seeks to develop a state-wide systems approach to embedding Education for Sustainability (EfS) in teacher education that is aligned with the Australian National Curriculum and the aspirations for EfS in the Melbourne Declaration and other national documents. Representatives from all teacher education institutions and other agents of change in the Queensland education system will be engaged in a multilevel systems approach, involving collaboration at the state, institutional and course levels, to develop curriculum practices that reflect a shared vision of EfS.

A threshold concepts focus to curriculum design: supporting student learning through application of variation theory

This project develops and evaluates a model of curriculum design that aims to assist student learning of foundational disciplinary ‘Threshold Concepts’. The project uses phenomenographic action research, cross-institutional peer collaboration and the Variation Theory of Learning to develop and trial the model. Two contrasting disciplines (Physics and Law) and four institutions (two research-intensive and two universities of technology) were involved in the project, to ensure broad applicability of the model across different disciplines and contexts. The Threshold Concepts that were selected for curriculum design attention were measurement uncertainty in Physics and legal reasoning in Law. Threshold Concepts are key disciplinary concepts that are inherently troublesome, transformative and integrative in nature. Once understood, such concepts transform students’ views of the discipline because they enable students to coherently integrate what were previously seen as unrelated aspects of the subject, providing new ways of thinking about it (Meyer & Land 2003, 2005, 2006; Land et al. 2008). However, the integrative and transformative nature of such threshold concepts make them inherently difficult for students to learn, with resulting misunderstandings of concepts being prevalent...

Editorial : Action research, action learning and community services

This edition of ALARj has a focus on the contribution of action learning and action research to the development of community services, particularly nonprofits. The landscape of community services has been changing rapidly in recent decades, and can be typified by the notion of complexity. Complexity in the nature of issues that services seek to respond to, complexity in the policy environment and systems of support that have tended to silo and compartmentalise problems and people, and complexity in the institutional location non-profit services occupy in ‘helping’ those who are seen as ‘in need’ or marginalised. In addition to being typified by complexity the environment in which community services are located is dynamic, undergoing profound and ongoing change as neo-liberal approaches to understanding and responding to human need, which emphasise the individualisation of risk, and market principles such as choice, competition and innovation, drive social policy. How can long held values of empowerment, care, inclusivity and benefit to individuals and communities have expression in community services as they grapple with the challenges of being viable and relevant in such a dynamically changing environment? This edition brings together a range of contributions which speak to these challenges. The thematic through these is that processes are needed which engage services and communities in ongoing processes of inquiry about how they can best proceed in contexts typified by complexity and change. Action learning and action research can provide processes of this character.

Wicked problems in designing open online learning

This paper will report on the “wicked” problems encountered when designing an online course with bounded content in an unbounded learning environment. It will describe the dilemmas faced and decisions made by academics in an Australian university challenged by an institutional initiative to design radical, disruptive learning experiences making use of readily available online media. This bounded/unbounded environment demands new roles for instructors in adopting innovative pedagogies and teaching and learning strategies. It also creates changing and challenging roles for course designers as they deal with ill-defined parameters and unknown audiences. In this paper, we propose a novel methodology for making curricular decisions in ill-defined spaces.

EFL teachers' perceptions and experiences of blended learning in a Vietnamese university

This qualitative study investigates English as Foreign Language (EFL) teachers' perceptions and practices of blended learning in a Vietnamese university and influencing factors. Findings revealed that teachers have limited understandings and use of blended learning due to three primary influential factors: (i) the traditional teacher-centred pedagogy, (ii) institutional management and leadership styles, and (iii) fragmented knowledge of technological, pedagogical, and content knowledge (TPACK) for blended learning. To improve the take up and potential benefits of blended learning in EFL education in Vietnamese universities, this study proposes (i) a systematic understanding of blended learning concepts, (ii) a localised TPACK framework, and (iii) a model of teacher professional development program.

An exploratory study of the current assessment practices for improving the learning of English as a foreign language (EFL) in two Vietnamese universities

This case study investigated EFL assessment practices at one public and one private university to explore the support of assessment for English language learning (ELL) within the Vietnamese sociocultural context. Findings demonstrated the potential of assessment to engage students in learning; enhance their understanding of the learning objectives; and facilitate their learning reflection. Findings also identified strong influences of contextual factors such as teachers' language assessment literacy, high-stakes testing and institutional administrative policies on the practices of assessment for ELL. This study contributes to research on Assessment for Learning and EFL education at tertiary level in Vietnam and other similar sociocultural contexts.

The paradox of creative uncertainty in learning and teaching

In an ever-changing and globalised world there is a need for higher education to adapt and evolve its models of learning and teaching. The old industrial model has lost traction, and new patterns of creative engagement are required. These new models potentially increase relevancy and better equip students for the future. Although creativity is recognised as an attribute that can contribute much to the development of these pedagogies, and creativity is valued by universities as a graduate capability, some educators understandably struggle to translate this vision into practice. This paper reports on selected survey findings from a mixed methods research project which aimed to shed light on how creativity can be designed for in higher education learning and teaching settings. A social constructivist epistemology underpinned the research and data was gathered using survey and case study methods. Descriptive statistical methods and informed grounded theory were employed for the analysis reported here. The findings confirm that creativity is valued for its contribution to the development of students’ academic work, employment opportunities and life in general; however, tensions arise between individual educator’s creative pedagogical goals and the provision of institutional support for implementation of those objectives. Designing for creativity becomes, paradoxically, a matter of navigating and limiting complexity and uncertainty, while simultaneously designing for those same states or qualities.

Grant application: QUT Learning and Teaching scheme 2012 Cracking the Code - A checklist to complement CRAs for first year Justice students.

Widening participation brings with it increasing diversity, increased variation in the level of academic preparedness (Clarke, 2011; Nelson, Clarke, & Kift 2010). Cultural capital coupled with negotiating the academic culture creates an environment based on many assumptions about academic writing and university culture. Variations in staff and student expectations relating to the teaching and learning experience is captured in a range of national and institutional data (AUSSE, CEQ, LEX). Nationally, AUSSE data (2009) indicates that communication, writing, speaking and analytic skills, staff expectations are quite a bit higher than students. The research team noted a recognisable shift in the changing cohort of students and their understanding and engagement with feedback and CRAs, as well as variations in teaching staff and student expectations. The current reality of tutor and student roles is that: - Students self select when/how they access lectures and tutorials. - Shorter tutorial times result in reduced opportunity to develop rapport with students. - CRAs are not always used consistently by staff (different marking styles and levels of feedback). - Marking is not always undertaken by the student’s tutor/lecturer. - Student support services might be recommended to students once a poor grade has been given. Students can perceive this as remedial and a further sense of failure. - CRA sheet has a mark /grade attached to it. Stigma attached to low mark. Hard to focus on the CRA feedback with a poor mark etched next to it. - Limited opportunities for sessionals to access professional development to assist with engaging students and feedback. - FYE resources exist, however academic time is a factor in exploring and embedding these resources. Feedback is another area with differing expectations and understandings. Sadler (2009) contends that students are not equipped to decode the statements properly. For students to be able to apply feedback, they need to understand the meaning of the feedback statement. They also need to identify, the particular aspects of their work that need attention. The proposed Checklist/guide would be one page and submitted with each assessment piece thereby providing an interface to engage students and tutors in managing first year understandings and expectations around CRAs, feedback, and academic practice.

Conceptualising recognition of prior learning processes in the age of open learning

This chapter interrogates what recognition of prior learning (RPL) can and does mean in the higher education sector—a sector in the grip of the widening participation agenda and an open access age. The chapter discusses how open learning is making inroads into recognition processes and examines two studies in open learning recognition. A case study relating to e-portfolio-style RPL for entry into a Graduate Certificate in Policy and Governance at a metropolitan university in Queensland is described. In the first instance, candidates who do not possess a relevant Bachelor degree need to demonstrate skills in governmental policy work in order to be eligible to gain entry to a Graduate Certificate (at Australian Qualifications Framework Level 8) (Australian Qualifications Framework Council, 2013, p. 53). The chapter acknowledges the benefits and limitations of recognition in open learning and those of more traditional RPL, anticipating future developments in both (or their convergence).

What counts as English in the internationalised Australian university: Implications for teaching and learning

English is currently ascendant as the language of globalisation, evident in its mediation of interactions and transactions worldwide. For many international students, completion of a degree in English means significant credentialing and increased job prospects. Australian universities are the third largest English-speaking destination for overseas students behind the United States and the United Kingdom. International students comprise one-fifth of the total Australian university population, with 80% coming from Asian countries (ABS, 2010). In this competitive higher education market, English has been identified as a valued ‘good’. Indeed, universities have been critiqued for relentlessly reproducing the “hegemony and homogeneity of English” (Marginson, 2006, p. 37) in order to sustain their advantage in the education market. For international students, English is the gatekeeper to enrolment, the medium of instruction and the mediator of academic success. For these reasons, English is not benign, yet it remains largely taken-for-granted in the mainstream university context. This paper problematises the naturalness of English and reports on a study of an Australian Master of Education course in which English was a focus. The study investigated representations of English as they were articulated across a chain of texts including the university strategic plan, course assessment criteria, student assignments, lecturer feedback, and interviews. Critical Discourse Analysis (CDA) and Foucault’s work on discourse enabled understandings of how a particular English is formed through an apparatus of specifications, exclusionary thresholds, strategies for maintenance (and disruption), and privileged concepts and speaking positions. The findings indicate that English has hegemonic status within the Australian university, with material consequences for students whose proficiency falls outside the thresholds of accepted English practice. Central to the constitution of what counts as English is the relationship of equivalence between standard written English and successful academic writing. International students’ representations of English indicate a discourse that impacts on identities and practices and preoccupies them considerably as they negotiate language and task demands. For the lecturer, there is strategic manoeuvring within the institutional regulative regime to support students’ English language needs using adapted assessment practices, explicit teaching of academic genres and scaffolded classroom interaction. The paper concludes with the implications for university teaching and learning.

The introduction of an Inter-professional learning unit: staff and student perspectives

Background The School of Clinical Sciences comprises a number of health disciplines including podiatry, paramedic science, pharmacy, medical imaging and radiation therapy. A new inter-professional unit was introduced in 2014, which covered key introductory learnings applicable for future health practitioners. This study examined teaching staff and student perspectives about their experience with the new unit for first year students. Methods Qualitative interviews with teaching staff (n=9) and focus group interviews with students (5 groups which ranged in size from 4-30) were conducted. Extensive notes were taken during the interviews Issues emerging from the interviews were identified and organised according to themes and subthemes. Results Four major themes were identified namely: Something new; To be or not to be that is the question; Advantages of the new unit; and Areas for improvement. Previous staff experience with inter-professional learning (IPL) had been ad-hoc, whereas the new unit brought together several disciplines in a planned and deliberate way. There was strong philosophical agreement about the value of IPL but some debate about the extent to which the unit provided IPL experience. The unit was seen as assisting students’ social and academic adjustment to university and provided opportunity for professional socialisation, exposure to macro and micro aspects of the Australian health care system and various types of communication. For podiatry students it was their first opportunity to formally meet and work with other podiatry students and moved their identity from ‘university student’ to ‘podiatry student’. Other positives included providing the opportunity for staff and students to interact at an early stage with the perceived benefit of reducing attrition. Areas for unit improvement included institutional arrangements, unit administration aspects and assessment. Conclusion The unit was seen as beneficial by staff and students however, students were more polarised in their views than staff. There was a tension between feeling apart of and learning about one's own profession and feeling apart of and learning about the roles of other health professionals in relation to patient care and the health care system.

Learning challenges of NGOs in development : Co-operation of Finnish NGOs in Morogoro, Tanzania

Non-governmental organisations (NGOs) have gained an important role in development co-operation during the last two decades. The development funding channelled through NGOs has increased and the number of NGOs engaged in development activities, both North and South, has been growing. Supporting NGOs has been seen as one way to strengthen civil society in the South and to provide potential for enhancing more effective development than the state, and to exercise participatory development and partnership in their North-South relationships. This study focuses on learning in the co-operation practices of small Finnish NGOs in Morogoro, Tanzania. Drawing on the cultural-historical activity theory and the theory of expansive learning, in this study I understand learning as a qualitative change in the actual co-operation practices. The qualitative change, for its part, emerges out of attempts to deal with the contradictions in the present activity. I use the concept of developmental contradiction in exploring the co-operation of the small Finnish NGOs with their Tanzanian counterparts. Developmental contradiction connects learning to actual practice and its historical development. By history, in this study I refer to multiple developmental trajectories, such as trajectories of individual participants, organisations, co-operation practices and the institutional system in which the NGO-development co-operation is embedded. In the empirical chapters I explore the co-operation both in the development co-operation projects and in micro-level interaction between partners taking place within the projects. I analyse the perceptions of the Finnish participants about the different developmental trajectories, the tensions, inclusions and exclusions in the evolving object of co-operation in one project, the construction of power relations in project meetings in three projects, and the collision of explicated partnership with the emerging practice of trusteeship in one project. On the basis of the empirical analyses I elaborate four developmental contradictions and learning challenges for the co-operation. The developmental contradictions include: 1) implementing a ready-made Finnish project idea vs. taking the current activities of Tanzanian NGO as a starting point; 2) gaining experiences and cultural interaction vs. access to outside funding; 3) promoting the official tools of development co-operation in training vs. use of tools and procedures taken from the prior activities of both partners in actual practice; and 4) asymmetric relations between the partners vs. rhetoric of equal partnership. Consequently, on the basis of developmental contradictions four learning challenges are suggested: a shift from legitimation of Finnish ideas to negotiation, transcending the separate objects and finding a partly joint object, developing locally shared tools for the co-operation, and identification and reflection of the power relations in the practice of co-operation. Keywords: activity theory; expansive learning; NGO development co-operation; partnership; power

Creating the reasonable adventurer: The co-evolution of student and learning environment

Purpose: This paper seeks to address the issue of how are graduate skills developed. The focus is not on which skills, but rather what type of learning environments is required within Higher Education to support the development of skills valued and demanded by SMEs within Australia. Approach: This paper takes a step back to consider the underlying issue of how an individual student's habits of thought are altered. In doing so, the past works of Morgan, Dewey, Whitehead, and Tyler are synthesized with the modern work of Baxter Magolda, Heath, and Biggs. Findings: It is argued that that without the development of a student-centred learning environment, most graduates will not develop the types of skills demanded by SMEs in a meaningfully way. That the failure to treat knowledge and skills as equal drivers of curriculum design will result in an imbalance that relegates skill development to a secondary learning outcome. Practical Implications: By removing the distraction of what skills should be developed, a clearer focus is possible regarding how educators should assist students to develop a broad array of generic graduate skills. From this perspective, skills can be viewed as an essential element of the educational process, rather than a new element that must be squeezed in between content. Value of Paper: This paper extends recent discussion of skills development through the use of an evolutionary perspective. Viewed as a process of creating social change, education becomes increasingly connected to a world that lays beyond institutional boundaries, thus promoting the notion of developing graduates for the world that awaits them.

A preliminary investigation into technology and processes facilitating the assurance of learning

This paper reports on the outcomes from a preliminary evaluation of technologies and processes intended to support the Assurance of Learning initiative in the business faculty of an Australian university. The study investigated how existing institutional information systems and operational processes could be used to support direct measures of student learning and the attainment of intended learning goals. The levels at which learning outcomes had been attained were extracted from the University Learning Management System (LMS), based on rubric data for three assessments in two units. Spreadsheets were used to link rubric criteria to the learning goals associated with the assessments as identified in a previous curriculum mapping exercise, and to aggregate the outcomes. Recommendations arising from this preliminary study are made to inform a more comprehensive pilot based on this approach, and manage the quality of student learning experiences in the context of existing processes and reporting structures.