Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

A model for research supervision of international students in engineering and information technology disciplines

A large proportion (over 12 per cent) of international and non-English speaking background (NESB) postgraduate research students enrol in engineering and information technology (IT) programs in Australian universities. They find themselves in an advanced research culture, and are technically and scientifically challenged early in their programs. This is in addition to cultural, social and religious isolation and linguistic barriers they have to contend with. The project team surveyed this cohort at QUT and UWA, on the hypothesis that they face challenges that are more discipline-specific. The results of the survey indicate that existing supervisory frameworks which are limited to linguistic contexts are not fully assisting these students and supervisors to achieve high quality research. The goal of this project is to extend these supervisory frameworks to a holistic model that will address the unique needs and supervisory issues these students face in engineering and IT disciplines. The model will be useable by all other Australian universities.

Learner-focused evaluation cycles : facilitating learning using feedforward, concurrent and feedback evaluation

There is a growing trend to offer students learning opportunities that are flexible, innovative and engaging. As educators embrace student-centred agile teaching and learning methodologies, which require continuous reflection and adaptation, the need to evaluate students’ learning in a timely manner has become more pressing. Conventional evaluation surveys currently dominate the evaluation landscape internationally, despite recognition that they are insufficient to effectively evaluate curriculum and teaching quality. Surveys often: (1) fail to address the issues for which educators need feedback, (2) constrain student voice, (3) have low response rates and (4) occur too late to benefit current students. Consequently, this paper explores principles of effective feedback to propose a framework for learner-focused evaluation. We apply a three-stage control model, involving feedforward, concurrent and feedback evaluation, to investigate the intersection of assessment and evaluation in agile learning environments. We conclude that learner-focused evaluation cycles can be used to guide action so that evaluation is not undertaken simply for the benefit of future offerings, but rather to benefit current students by allowing ‘real-time’ learning activities to be adapted in the moment. As a result, students become co-producers of learning and evaluation becomes a meaningful, responsive dialogue between students and their instructors.

An evaluation of different features and learning models for anomalous event detection

The huge amount of CCTV footage available makes it very burdensome to process these videos manually through human operators. This has made automated processing of video footage through computer vision technologies necessary. During the past several years, there has been a large effort to detect abnormal activities through computer vision techniques. Typically, the problem is formulated as a novelty detection task where the system is trained on normal data and is required to detect events which do not fit the learned ‘normal’ model. There is no precise and exact definition for an abnormal activity; it is dependent on the context of the scene. Hence there is a requirement for different feature sets to detect different kinds of abnormal activities. In this work we evaluate the performance of different state of the art features to detect the presence of the abnormal objects in the scene. These include optical flow vectors to detect motion related anomalies, textures of optical flow and image textures to detect the presence of abnormal objects. These extracted features in different combinations are modeled using different state of the art models such as Gaussian mixture model(GMM) and Semi- 2D Hidden Markov model(HMM) to analyse the performances. Further we apply perspective normalization to the extracted features to compensate for perspective distortion due to the distance between the camera and objects of consideration. The proposed approach is evaluated using the publicly available UCSD datasets and we demonstrate improved performance compared to other state of the art methods.

Research and/or learning and teaching : a study of Australian professors' priorities, beliefs and behaviours

This paper presents findings from an empirical study of key aspects of the teaching and research priorities, beliefs and behaviours of 72 professorial and associate professorial academics in Science, Information Technology and Engineering across four faculties in three Australian universities. The academics ranked 16 research activities and 16 matched learning and teaching (L&T) activities from three perspectives: job satisfaction, role model behaviour and perceptions of professional importance. The findings were unequivocally in favour of research in all three areas and remarkably consistent across the universities. The only L&T activity that was ranked consistently well was 'improving student satisfaction ratings for teaching', an area in which academics are increasingly held accountable. Respondents also indicated that their seniors encourage research efforts more than L&T efforts. Recommendations include that higher education rewards for quality L&T are maintained or improved and that recognition of L&T research domains is further strengthened.

Technological tools for science classrooms : choosing and using for productive and sustainable teaching and learning experiences

In this age of rapidly evolving technology, teachers are encouraged to adopt ICTs by government, syllabus, school management, and parents. Indeed, it is an expectation that teachers will incorporate technologies into their classroom teaching practices to enhance the learning experiences and outcomes of their students. In particular, regarding the science classroom, a subject that traditionally incorporates hands-on experiments and practicals, the integration of modern technologies should be a major feature. Although myriad studies report on technologies that enhance students’ learning outcomes in science, there is a dearth of literature on how teachers go about selecting technologies for use in the science classroom. Teachers can feel ill prepared to assess the range of available choices and might feel pressured and somewhat overwhelmed by the avalanche of new developments thrust before them in marketing literature and teaching journals. The consequences of making bad decisions are costly in terms of money, time and teacher confidence. Additionally, no research to date has identified what technologies science teachers use on a regular basis, and whether some purchased technologies have proven to be too problematic, preventing their sustained use and possible wider adoption. The primary aim of this study was to provide research-based guidance to teachers to aid their decision-making in choosing technologies for the science classroom. The study unfolded in several phases. The first phase of the project involved survey and interview data from teachers in relation to the technologies they currently use in their science classrooms and the frequency of their use. These data were coded and analysed using Grounded Theory of Corbin and Strauss, and resulted in the development of a PETTaL model that captured the salient factors of the data. This model incorporated usability theory from the Human Computer Interaction literature, and education theory and models such as Mishra and Koehler’s (2006) TPACK model, where the grounded data indicated these issues. The PETTaL model identifies Power (school management, syllabus etc.), Environment (classroom / learning setting), Teacher (personal characteristics, experience, epistemology), Technology (usability, versatility etc.,) and Learners (academic ability, diversity, behaviour etc.,) as fields that can impact the use of technology in science classrooms. The PETTaL model was used to create a Predictive Evaluation Tool (PET): a tool designed to assist teachers in choosing technologies, particularly for science teaching and learning. The evolution of the PET was cyclical (employing agile development methodology), involving repeated testing with in-service and pre-service teachers at each iteration, and incorporating their comments i ii in subsequent versions. Once no new suggestions were forthcoming, the PET was tested with eight in-service teachers, and the results showed that the PET outcomes obtained by (experienced) teachers concurred with their instinctive evaluations. They felt the PET would be a valuable tool when considering new technology, and it would be particularly useful as a means of communicating perceived value between colleagues and between budget holders and requestors during the acquisition process. It is hoped that the PET could make the tacit knowledge acquired by experienced teachers about technology use in classrooms explicit to novice teachers. Additionally, the PET could be used as a research tool to discover a teachers’ professional development needs. Therefore, the outcomes of this study can aid a teacher in the process of selecting educationally productive and sustainable new technology for their science classrooms. This study has produced an instrument for assisting teachers in the decision-making process associated with the use of new technologies for the science classroom. The instrument is generic in that it can be applied to all subject areas. Further, this study has produced a powerful model that extends the TPACK model, which is currently extensively employed to assess teachers’ use of technology in the classroom. The PETTaL model grounded in data from this study, responds to the calls in the literature for TPACK’s further development. As a theoretical model, PETTaL has the potential to serve as a framework for the development of a teacher’s reflective practice (either self evaluation or critical evaluation of observed teaching practices). Additionally, PETTaL has the potential for aiding the formulation of a teacher’s personal professional development plan. It will be the basis for further studies in this field.

A cognitive processing framework for learning analytics

Incorporating a learner’s level of cognitive processing into Learning Analytics presents opportunities for obtaining rich data on the learning process. We propose a framework called COPA that provides a basis for mapping levels of cognitive operation into a learning analytics system. We utilise Bloom’s taxonomy, a theoretically respected conceptualisation of cognitive processing, and apply it in a flexible structure that can be implemented incrementally and with varying degree of complexity within an educational organisation. We outline how the framework is applied, and its key benefits and limitations. Finally, we apply COPA to a University undergraduate unit, and demonstrate its utility in identifying key missing elements in the structure of the course.

Forging the synergy between anticipation and innovation : the futures action model

This paper presents the theory and practice of the Futures Action Model (FAM). FAM has been in development for over a decade, in a number of contexts and iterations. It is a creative methodology that uses a variety of concepts and tools to guide participants through the conception and modeling of enterprises, services, social innovations and projects in the context of emerging futures. It is used to generate strategic options that people can utilise to build opportunities for value creation as they move into the future. This paper details examples in its development, and provides theoretical and practical guidelines for educators and business facilitators to use the FAM system in their own workplaces.

A state-wide systems approach to embedding the learning and teaching of sustainability in teacher education

Education systems have a key role to play in preparing future citizens to engage in sustainable living practices and help create a more sustainable world. Many schools throughout Australia have begun to develop whole-school approaches to sustainability education that are supported by national and state policies and curriculum frameworks. Preservice teacher education, however, lags behind in building the capacity of new teachers to initiate and implement such approaches (ARIES, 2010). This proposed project seeks to develop a state-wide systems approach to embedding Education for Sustainability (EfS) in teacher education that is aligned with the Australian National Curriculum and the aspirations for EfS in the Melbourne Declaration and other national documents. Representatives from all teacher education institutions and other agents of change in the Queensland education system will be engaged in a multilevel systems approach, involving collaboration at the state, institutional and course levels, to develop curriculum practices that reflect a shared vision of EfS.

Systematic evaluation of GoSoapBox in tertiary education : a student response system for improving learning experiences and outcomes

There is currently a wide range of research into the recent introduction of student response systems in higher education and tertiary settings (Banks 2006; Kay and Le Sange, 2009; Beatty and Gerace 2009; Lantz 2010; Sprague and Dahl 2009). However, most of this pedagogical literature has generated ‘how to’ approaches regarding the use of ‘clickers’, keypads, and similar response technologies. There are currently no systematic reviews on the effectiveness of ‘GoSoapBox’ – a more recent, and increasingly popular student response system – for its capacity to enhance critical thinking, and achieve sustained learning outcomes. With rapid developments in teaching and learning technologies across all undergraduate disciplines, there is a need to obtain comprehensive, evidence-based advice on these types of technologies, their uses, and overall efficacy. This paper addresses this current gap in knowledge. Our teaching team, in an undergraduate Sociology and Public Health unit at the Queensland University of Technology (QUT), introduced GoSoapBox as a mechanism for discussing controversial topics, such as sexuality, gender, economics, religion, and politics during lectures, and to take opinion polls on social and cultural issues affecting human health. We also used this new teaching technology to allow students to interact with each other during class – both on both social and academic topics – and to generate discussions and debates during lectures. The paper reports on a data-driven study into how this interactive online tool worked to improve engagement and the quality of academic work produced by students. This paper will firstly, cover the recent literature reviewing student response systems in tertiary settings. Secondly, it will outline the theoretical framework used to generate this pedagogical research. In keeping with the social and collaborative features of Web 2.0 technologies, Bandura’s Social Learning Theory (SLT) will be applied here to investigate the effectiveness of GoSoapBox as an online tool for improving learning experiences and the quality of academic output by students. Bandura has emphasised the Internet as a tool for ‘self-controlled learning’ (Bandura 2001), as it provides the education sector with an opportunity to reconceptualise the relationship between learning and thinking (Glassman & Kang 2011). Thirdly, we describe the methods used to implement the use of GoSoapBox in our lectures and tutorials, and which aspects of the technology we drew on for learning purposes, as well as the methods for obtaining feedback from the students about the effectiveness or otherwise of this tool. Fourthly, we report cover findings from an examination of all student/staff activity on GoSoapBox as well as reports from students about the benefits and limitations of it as a learning aid. We then display a theoretical model that is produced via an iterative analytical process between SLT and our data analysis for use by academics and teachers across the undergraduate curriculum. The model has implications for all teachers considering the use of student response systems to improve the learning experiences of their students. Finally, we consider some of the negative aspects of GoSoapBox as a learning aid.

A topic based document relevance ranking model

Topic modelling has been widely used in the fields of information retrieval, text mining, machine learning, etc. In this paper, we propose a novel model, Pattern Enhanced Topic Model (PETM), which makes improvements to topic modelling by semantically representing topics with discriminative patterns, and also makes innovative contributions to information filtering by utilising the proposed PETM to determine document relevance based on topics distribution and maximum matched patterns proposed in this paper. Extensive experiments are conducted to evaluate the effectiveness of PETM by using the TREC data collection Reuters Corpus Volume 1. The results show that the proposed model significantly outperforms both state-of-the-art term-based models and pattern-based models.

A little learning is a dangerous thing? The ethics of teaching accidental tourists in higher education

This paper raises questions about the ethical issues that arise for academics and universities when under-graduate students enrol in classes outside of their discipline - classes that are not designed to be multi-disciplinary or introductory. We term these students ‘accidental tourists'. Differences between disciplines in terms of pedagogy, norms, language and understanding may pose challenges for accidental tourists in achieving desired learning outcomes. This paper begins a discussion about whether lecturers and universities have any ethical obligations towards supporting the learning of these students. Recognising that engaging with only one ethical theory leads to a fragmented moral vision, this paper employs a variety of ethical theories to examine any possible moral obligations that may fall upon lecturers and/or universities. In regards to lecturers, the paper critically engages with the ethical theories of utilitarianism, Kantianism and virtue ethics (Aristotle) to determine the extent of any academic duty to accidental tourists. In relation to universities, this paper employs the emerging ethical theory of organisational ethics as a lens through which to critically examine any possible obligations. Organisational ethics stems from the recognition that moral demands also exist for organisations so organisations must be reconceptualised as ethical actors and their policies and practices subject to ethical scrutiny. The analysis in this paper illustrates the challenges faced by lecturers some of whom, we theorise, may experience a form of moral distress facing a conflict between personal beliefs and organisational requirements. It also critically examines the role and responsibilities of universities towards students and towards their staff and the inherent moral tensions between a market model and demands for ‘good' learning experiences. This paper highlights the tensions for academics, between academics and universities and within university policy and indicates the need for greater reflection about this issue, especially given the many constraints facing lecturers and universities.

Embedding EfS in teacher education : an introductory guide to using the systems change model

Welcome to this introductory guide on using a systems change model to embed Education for Sustainability (EfS) into teacher education. Pressing sustainability issues such as climate change, biodiversity loss and depletion of non-renewable resources pose new challenges for education. The importance of education in preparing future citizens to engage in sustainable living practices and help create a more sustainable world is widely acknowledged. As a result many universities around the world are beginning to recognize the need to integrate EfS into their teacher education programs. However, evidence indicates that there is little or no core EfS knowledge or pedagogy in pre-service teacher courses available to student teachers in a thorough and systematic fashion. Instead efforts are fragmented and individually or, at best, institutionally-based and lacking a systems approach to change, an approach that is seen as essential to achieving a sustainable society (Henderson & Tilbury, 2004). The result is new teachers are graduating without the necessary knowledge or skills to teach in ways that enable them to prepare their students to cope well with the new and emerging challenges their communities face. This guide has been prepared as part of a teaching and learning research project that applied a systems change approach to embedding the learning and teaching of sustainability into pre-service teacher education. The processes, outcomes and lessons learnt from this project are presented here as a guide for navigating pathways to systemic change in the journey of re-orienting teacher education towards sustainability. The guide also highlights how a systems change approach can be used to successfully enact change within a teacher education system. If you are curious about how to introduce and embed EfS into teacher education – or have tried other models and are looking for a more encompassing, transformative approach – this guide is designed to help you. The material presented in this guide is designed to be flexible and adaptive. However you choose to use the content, our aim is to help you and your students develop new perspectives, promote discussion and to engage with a system-wide approach to change.

An Artificial Neutral Network (ANN) model for predicting biodiesel kinetic viscosity as a function of temperature and chemical compositions

An Artificial Neural Network (ANN) is a computational modeling tool which has found extensive acceptance in many disciplines for modeling complex real world problems. An ANN can model problems through learning by example, rather than by fully understanding the detailed characteristics and physics of the system. In the present study, the accuracy and predictive power of an ANN was evaluated in predicting kinetic viscosity of biodiesels over a wide range of temperatures typically encountered in diesel engine operation. In this model, temperature and chemical composition of biodiesel were used as input variables. In order to obtain the necessary data for model development, the chemical composition and temperature dependent fuel properties of ten different types of biodiesels were measured experimentally using laboratory standard testing equipments following internationally recognized testing procedures. The Neural Networks Toolbox of MatLab R2012a software was used to train, validate and simulate the ANN model on a personal computer. The network architecture was optimised following a trial and error method to obtain the best prediction of the kinematic viscosity. The predictive performance of the model was determined by calculating the absolute fraction of variance (R2), root mean squared (RMS) and maximum average error percentage (MAEP) between predicted and experimental results. This study found that ANN is highly accurate in predicting the viscosity of biodiesel and demonstrates the ability of the ANN model to find a meaningful relationship between biodiesel chemical composition and fuel properties at different temperature levels. Therefore the model developed in this study can be a useful tool in accurately predict biodiesel fuel properties instead of undertaking costly and time consuming experimental tests.

Myosin light chain kinase regulates synaptic plasticity and fear learning in the lateral amygdala

Learning and memory depend on signaling mole- cules that affect synaptic efficacy. The cytoskeleton has been implicated in regulating synaptic transmission but its role in learning and memory is poorly understood. Fear learning depends on plasticity in the lateral nucleus of the amygdala. We therefore examined whether the cytoskeletal-regulatory protein, myosin light chain kinase, might contribute to fear learning in the rat lateral amygdala. Microinjection of ML-7, a specific inhibitor of myosin light chain kinase, into the lateral nucleus of the amygdala before fear conditioning, but not immediately afterward, enhanced both short-term memory and long-term memory, suggesting that myosin light chain kinase is involved specifically in memory acquisition rather than in posttraining consolidation of memory. Myosin light chain kinase inhibitor had no effect on memory retrieval. Furthermore, ML-7 had no effect on behavior when the train- ing stimuli were presented in a non-associative manner. An- atomical studies showed that myosin light chain kinase is present in cells throughout lateral nucleus of the amygdala and is localized to dendritic shafts and spines that are postsynaptic to the projections from the auditory thalamus to lateral nucleus of the amygdala, a pathway specifically impli- cated in fear learning. Inhibition of myosin light chain kinase enhanced long-term potentiation, a physiological model of learning, in the auditory thalamic pathway to the lateral nu- cleus of the amygdala. When ML-7 was applied without as- sociative tetanic stimulation it had no effect on synaptic responses in lateral nucleus of the amygdala. Thus, myosin light chain kinase activity in lateral nucleus of the amygdala appears to normally suppress synaptic plasticity in the cir- cuits underlying fear learning, suggesting that myosin light chain kinase may help prevent the acquisition of irrelevant fears. Impairment of this mechanism could contribute to pathological fear learning.