Reframing evaluation of learning and teaching: An approach to change

Evaluation practices in the Higher Education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology (QUT) began a project which aims to reframe the organisation’s approach to the evaluation of learning and teaching. This paper describes the existing evaluation system; the emergence and early development of the project; and formulation of a conceptual framework identifying key dimensions of evaluation. It then compares the draft framework with other conceptualisations and models of evaluation identified in the literature, to determine its validity and suitability for supporting QUT’s plans for the future. Overall, the paper represents a structured evaluation of the REFRAME project at a particular point in its lifecycle. Given that the project follows an evidence based, practice-led process and applies an ongoing action research cycle, the findings are presented in the belief that QUT’s experience is broadly applicable to other institutions which may be contemplating change in relation to evaluation of learning and teaching.

Acquiring search expertise : learning experiences and threshold concepts

Expert searchers engage with information as information brokers, researchers, reference librarians, information architects, faculty who teach advanced search, and in a variety of other information-intensive professions. Their experiences are characterized by a profound understanding of information concepts and skills and they have an agile ability to apply this knowledge to interacting with and having an impact on the information environment. This study explored the learning experiences of searchers to understand the acquisition of search expertise. The research question was: What can be learned about becoming an expert searcher from the learning experiences of proficient novice searchers and highly experienced searchers? The key objectives were: (1) to explore the existence of threshold concepts in search expertise; (2) to improve our understanding of how search expertise is acquired and how novice searchers, intent on becoming experts, can learn to search in more expertlike ways. The participant sample drew from two population groups: (1) highly experienced searchers with a minimum of 20 years of relevant professional experience, including LIS faculty who teach advanced search, information brokers, and search engine developers (11 subjects); and (2) MLIS students who had completed coursework in information retrieval and online searching and demonstrated exceptional ability (9 subjects). Using these two groups allowed a nuanced understanding of the experience of learning to search in expertlike ways, with data from those who search at a very high level as well as those who may be actively developing expertise. The study used semi-structured interviews, search tasks with think-aloud narratives, and talk-after protocols. Searches were screen-captured with simultaneous audio-recording of the think-aloud narrative. Data were coded and analyzed using NVivo9 and manually. Grounded theory allowed categories and themes to emerge from the data. Categories represented conceptual knowledge and attributes of expert searchers. In accord with grounded theory method, once theoretical saturation was achieved, during the final stage of analysis the data were viewed through lenses of existing theoretical frameworks. For this study, threshold concept theory (Meyer & Land, 2003) was used to explore which concepts might be threshold concepts. Threshold concepts have been used to explore transformative learning portals in subjects ranging from economics to mathematics. A threshold concept has five defining characteristics: transformative (causing a shift in perception), irreversible (unlikely to be forgotten), integrative (unifying separate concepts), troublesome (initially counter-intuitive), and may be bounded. Themes that emerged provided evidence of four concepts which had the characteristics of threshold concepts. These were: information environment: the total information environment is perceived and understood; information structures: content, index structures, and retrieval algorithms are understood; information vocabularies: fluency in search behaviors related to language, including natural language, controlled vocabulary, and finesse using proximity, truncation, and other language-based tools. The fourth threshold concept was concept fusion, the integration of the other three threshold concepts and further defined by three properties: visioning (anticipating next moves), being light on one's 'search feet' (dancing property), and profound ontological shift (identity as searcher). In addition to the threshold concepts, findings were reported that were not concept-based, including praxes and traits of expert searchers. A model of search expertise is proposed with the four threshold concepts at its core that also integrates the traits and praxes elicited from the study, attributes which are likewise long recognized in LIS research as present in professional searchers. The research provides a deeper understanding of the transformative learning experiences involved in the acquisition of search expertise. It adds to our understanding of search expertise in the context of today's information environment and has implications for teaching advanced search, for research more broadly within library and information science, and for methodologies used to explore threshold concepts.

Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

Process model comprehension : the effects of cognitive abilities, learning style, and strategy

Process models are used to convey semantics about business operations that are to be supported by an information system. A wide variety of professionals is targeted to use such models, including people who have little modeling or domain expertise. We identify important user characteristics that influence the comprehension of process models. Through a free simulation experiment, we provide evidence that selected cognitive abilities, learning style, and learning strategy influence the development of process model comprehension. These insights draw attention to the importance of research that views process model comprehension as an emergent learning process rather than as an attribute of the models as objects. Based on our findings, we identify a set of organizational intervention strategies that can lead to more successful process modeling workshops.

Theorizing why in e-learning – a frontier for cognitive engagement

Asking why is an important foundation of inquiry and fundamental to the development of reasoning skills and learning. Despite this, and despite the relentless and often disruptive nature of innovations in information and communications technology (ICT), sophisticated tools that directly support this basic act of learning appear to be undeveloped, not yet recognized, or in the very early stages of development. Why is this so? To this question, there is no single satisfactory answer; instead, numerous plausible explanations and related questions arise. After learning something, however, explaining why can be revealing of a person’s understanding (or lack of it). What then differentiates explanation from information; and, explanatory from descriptive content? What ICT scaffolding might support inquiry instigated by why-questioning? What is the role of reflective practice in inquiry-based learning? These and other questions have emerged from this investigation and underscore that why-questions often propagate further questions and are a catalyst for cognitive engagement and dialogue. This paper reports on a multi-disciplinary, theoretical investigation that informs the broad discourse on e-learning and points to a specific frontier for design and development of e-learning tools. Probing why reveals that versatile and ambiguous semantics present the core challenge – asking, learning, knowing, understanding, and explaining why.

Scaffolding reflective inquiry - enabling why-questioning while e-Learning

This paper presents some theoretical and interdisciplinary perspectives that might inform the design and development of information and communications technology (ICT) tools to support reflective inquiry during e-learning. The role of why-questioning provides the focus of discussion and is guided by literature that spans critical thinking, inquiry-based and problem-based learning, storytelling, sense-making, and reflective practice, as well as knowledge management, information science, computational linguistics and automated question generation. It is argued that there exists broad scope for the development of ICT scaffolding targeted at supporting reflective inquiry duringe-learning. Evidence suggests that wiki-based learning tasks, digital storytelling, and e-portfolio tools demonstrate the value of accommodating reflective practice and explanatory content in supporting learning; however, it is also argued that the scope for ICT tools that directly support why-questioning as a key aspect of reflective inquiry is a frontier ready for development.

Opening digital learning to deeper inquiry

This chapter presents an historical narrative on the recent evolution of information and communications technology (ICT) that has been, and is, utilized for purposes of learning. In other words, it presents an account of the development of e-learning supported through the Web and other similar virtual environments. It does not attempt to present a definitive account; as such an exercise is fraught with assumptions, contextual bias, and probable conjecture. The concern here is more with contextualizing the role of inquiry in learning and the evolving digital tools that enable interfaces that promote and support it. In tracking this evolution, both multi-disciplinary and trans-disciplinary research has been pursued. Key historical developments are identified as well as interpretations of the key drivers of e-learning over time and into what might be better described as digital learning. Innovations in the development of digital tools are described as dynamic and emergent, evolving as a consequence of multiple, sometimes hidden drivers of change. But conflating advancements in learning technologies with e-learning seems to be pervasive. As is the push for the “open” agenda – a growing number of initiatives and movements dominated by themes associated with access, intellectual property, public benefit, sharing and technical interoperability. Openness is also explored in this chapter, however, more in terms of what it means when associated with inquiry. By investigating opportunities for the stimulation and support of questioning online – in particular, why-questioning – this chapter is focused on “opening” content – not just for access but for inquiry and deeper learning.

The why dimension - opening frontiers for digital learning

Digital learning has come a long way from the days of simple 'if-then' queries. It is now enabled by countless innovations that support knowledge sharing, openness, flexibility, and independent inquiry. Set against an evolutionary context this study investigated innovations that directly support human inquiry. Specifically, it identified five activities that together are defined as the 'why dimension' – asking, learning, understanding, knowing, and explaining why. Findings highlight deficiencies in mainstream search-based approaches to inquiry, which tend to privilege the retrieval of information as distinct from explanation. Instrumental to sense-making, the 'why dimension' provides a conceptual framework for development of 'sense-making technologies'.

Synthesizing form and data : developing practices for student engagement and learning

We explore the relationship between form and data as a design agenda and learning strategy for novice visual information designers. Our students are university seniors in digital, visual design but novices to information design, manipulation and interpretation. We describe design strategies developed to scaffold sophisticated aesthetic and conceptual engagement despite limited understanding of the domain of designing with information. These revolve around an open-ended design project where students created a physical design from data of their choosing and research. The accompanying learning strategies concern this relationship between data and form to investigate it materially, formally and through ideation. Exemplifying student works that cross media and design domains are described.

Using automated text analysis to evaluate students’ conceptual understanding

Background: A major challenge for assessing students’ conceptual understanding of STEM subjects is the capacity of assessment tools to reliably and robustly evaluate student thinking and reasoning. Multiple-choice tests are typically used to assess student learning and are designed to include distractors that can indicate students’ incomplete understanding of a topic or concept based on which distractor the student selects. However, these tests fail to provide the critical information uncovering the how and why of students’ reasoning for their multiple-choice selections. Open-ended or structured response questions are one method for capturing higher level thinking, but are often costly in terms of time and attention to properly assess student responses. Purpose: The goal of this study is to evaluate methods for automatically assessing open-ended responses, e.g. students’ written explanations and reasoning for multiple-choice selections. Design/Method: We incorporated an open response component for an online signals and systems multiple-choice test to capture written explanations of students’ selections. The effectiveness of an automated approach for identifying and assessing student conceptual understanding was evaluated by comparing results of lexical analysis software packages (Leximancer and NVivo) to expert human analysis of student responses. In order to understand and delineate the process for effectively analysing text provided by students, the researchers evaluated strengths and weakness for both the human and automated approaches. Results: Human and automated analyses revealed both correct and incorrect associations for certain conceptual areas. For some questions, that were not anticipated or included in the distractor selections, showing how multiple-choice questions alone fail to capture the comprehensive picture of student understanding. The comparison of textual analysis methods revealed the capability of automated lexical analysis software to assist in the identification of concepts and their relationships for large textual data sets. We also identified several challenges to using automated analysis as well as the manual and computer-assisted analysis. Conclusions: This study highlighted the usefulness incorporating and analysing students’ reasoning or explanations in understanding how students think about certain conceptual ideas. The ultimate value of automating the evaluation of written explanations is that it can be applied more frequently and at various stages of instruction to formatively evaluate conceptual understanding and engage students in reflective

“I was proud of myself that I didn’t give up and I did it”: Experiences of pride and triumph in learning science

The role that specific emotions, such as pride and triumph, play during instruction in science education is an under-researched field of study. Emotions are recognized as central to learning yet little is known about the way in which they are produced in naturalistic settings, how emotions relate to classroom learning during interactions, and what antecedent factors are associated with emotional experiences during instruction. Data sources for the study include emotion diaries, student written artifacts, video recordings of class interactions, and interviews. Emotions produced in the moment during classroom interactions are analyzed from video data and audio data through a novel theoretical framework related to the sociology of human emotions. These direct observations are compared with students’ recollected emotional experiences reported through emotion diaries and interviews. The study establishes links between pride and triumph within classroom interactions and instructional tasks during learning episodes in a naturalistic setting. We discuss particular classroom activities that are associated with justified feelings of pride and triumph. More specifically, classroom events associated with these emotions were related to understanding science concepts, social interactions, and achieving success on challenging tasks.

Teaching Reflective Learning in Higher Education : A Systematic Approach Using Pedagogic Patterns

This book is about understanding the nature and application of reflection in higher education. It provides a theoretical model to guide the implementation of reflective learning and reflective practice across multiple disciplines and international contexts in higher education. The book presents research into the ways in which reflection is both considered and implemented in different ways across different professional disciplines, while maintaining a common purpose to transform and improve learning and/or practice. Readers will find this book innovative and new in three key ways. First, in its holistic theorisation of reflection within the pedagogic field of higher education; Secondly, in conceptualising reflection in different modes to achieve specific purposes in different disciplines; and finally, in providing conceptual guidance for embedding reflective learning and reflective practice in a systematic way across whole programmes, faculties or institutions in higher education. The book considers important contextual factors that influence the teaching of forms and methods of reflection. It provides a functional analysis of multiple modes of reflection, including written, oral, visual, auditory, and embodied forms. Empirical chapters analyse the application of these modes across disciplines and at different stages of a programme. The theoretical model accounts for students’ stage of development in the disciplinary field, along with progressive and cyclical levels of higher order thinking, and learning and professional practice that are expected within different disciplines and professional fields. The book provides: • A conceptual model for the application of reflection across disciplines in a variety of contexts. • Empirical examples of different modes and pedagogic patterns for reflection. • Guidance and support for embedding systemic pedagogical and curriculum change.

Introduction: Reflective and Reflexive Approaches in Higher Education : A Warrant for Lifelong Learning?

In our complex and incongruous world, where variety produces more variety, and there is no blueprint for dealing with unprecedented change, it is imperative that individuals develop reflexive approaches to life and learning. Higher education has a role to play in guiding students to be self-analysts, with the ability to examine and mediate self and context for improved outcomes. This chapter elucidates the catchphrase of lifelong learning and its enactment in higher education. Theories of reflexivity and personal epistemology are utilised to provide the conceptual tools to understand the ways in which individuals manage competing influences and deliberate about action in their learning journey. The case is made for the integral role of higher education teachers in developing students’ capacities for reflective thinking and reflexive approaches to learning as a life project.

Sustainable Pedagogical Change for Embedding Reflective Learning Across Higher Education Programs

For pedagogical change to be sustained over time, and over the span of higher education courses, it needs to be framed widely, rather than ‘tacked on’. The framing includes curriculum reform and resource provision alongside staff pedagogical development. This is especially true for initiatives (such as reflective writing and assessment) that target broad-based, high-level skills and dispositions. For various reasons, such initiatives can easily become lost because of the discipline-specific focus of a syllabus outweighs the initiative, or because lack of resources compromises a desired approach. Course improvement in higher education contexts is typically difficult and episodic. In such circumstances, we argue that a strategic and trustworthy approach is necessary where practitioner-lead pedagogic development is fostered through trust and communication and is purposefully embedded within key dimensions of curriculum integration and resource provision. This chapter describes an approach to pedagogical change where curriculum, pedagogy and resources are simultaneously and collaboratively orchestrated to provide an effective framework for sustainable and effective change. A robust conceptual model is proposed to guide the implementation of such change.

‘Ngulluck Katitj Wah Koorl Koorliny/ Us mob going along learning to research together’ : drawing on action research to develop a literature review on Indigenous gendered health and wellbeing

This paper describes the collaborative work practices of the Health and Wellbeing Node within the National Indigenous Research and Knowledges Network (NIRAKN). The authors reflect on the processes they used to research and develop a literature review. As a newly established research team, the Health and Wellbeing Node members developed a collaborative approach that was informed by Action Research practices and underpinned by Indigenous ways of working. The authors identify strong links between Action Research and Indigenous processes. They suggest that, through ongoing cycles of research and review, the NIRAKN Health and Wellbeing Node developed a culturally safe, respectful and trulycollaborative way of working together and forming the identity of their work group. In this paper, they describe their developing work processes and explain the way that pictorial conceptual models contributed to their emerging ideas.