Reframing evaluation of learning and teaching: An approach to change

Evaluation practices in the Higher Education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology (QUT) began a project which aims to reframe the organisation’s approach to the evaluation of learning and teaching. This paper describes the existing evaluation system; the emergence and early development of the project; and formulation of a conceptual framework identifying key dimensions of evaluation. It then compares the draft framework with other conceptualisations and models of evaluation identified in the literature, to determine its validity and suitability for supporting QUT’s plans for the future. Overall, the paper represents a structured evaluation of the REFRAME project at a particular point in its lifecycle. Given that the project follows an evidence based, practice-led process and applies an ongoing action research cycle, the findings are presented in the belief that QUT’s experience is broadly applicable to other institutions which may be contemplating change in relation to evaluation of learning and teaching.

Collaboration and competition on a Wiki : the praxis of online social learning to improve academic writing and research in under-graduate students

The number of internet users in Australia has been steadily increasing, with over 10.9 million people currently subscribed to an internet provider (ABS, 2011). Over the past year, the most avid users of the Internet were 15 – 24 year olds, with approximately 95% accessing the internet on a regular basis (ABS, Social Trends, 2011). While the internet, in particularly Web 2.0, has been described as fundamental to higher education students, social and leisure internet tools are also increasingly being used by these students to generate and maintain their social and professional networks and interactions (Duffy & Bruns, 2006). Rapid technological advancements have enabled greater and faster access to information for learning and education (Hemmi et al, 2009; Glassman & Kang, 2011). As such, we sought to integrate interactive, online social media into the assessment profile of a Public Health undergraduate cohort at the Queensland University of Technology (QUT). The aim of this exercise was to engage undergraduate students to both develop and showcase their research on a range of complex, contemporary health issues within the online forum of Wikispaces for review and critique by their peers. We applied Bandura’s Social Learning Theory (SLT) to analyse the interactive processes from which students developed deeper and more sustained learning, and via which their overall academic writing standards were enriched. This paper outlines the assessment task, and the students’ feedback on their learning outcomes in relation to the Attentional, Retentional, Motor Reproduction, and Motivational Processes outlined by Bandura in SLT. We conceptualise the findings in a theoretical model, and discuss the implications for this approach within the broader tertiary environment.

Connecting Indigenous stories with geology : inquiry-based learning in a middle years classroom

One way to integrate indigenous perspectives in junior science is through links between indigenous stories of the local area and science concepts. Using local indigenous stories about landforms, a teacher of year 8 students designed a unit on geology that catered for the diverse student population in his class. This paper reports on the inquiry-based approach structured around the requirements of the Australian Curriculum highlighting the learning and engagement of students during the unit.

Supporting the learning of rapid application development in a database environment

Intelligent Tutoring Systems (ITSs) are computer systems designed to provide individualised help to students, learning in a problem solving context. The difference between an ITS and a Computer Assisted Instruction (CAI) system is that an ITS has a Student Model which allows it to provide a better educational environment. The Student Model contains information on what the student knows, and does not know, about the domain being learnt, as well as other personal characteristics such as preferred learning style. This research has resulted in the design and development of a new ITS: Personal Access Tutor (PAT). PAT is an ITS that helps students to learn Rapid Application Development in a database environment. More specifically, PAT focuses on helping students to learn how to create forms and reports in Microsoft Access. To provide an augmented learning environment, PAT’s architecture is different to most other ITSs. Instead of having a simulation, PAT uses a widelyused database development environment (Microsoft Access). This enables the students to ask for help, while developing real applications using real database software. As part of this research, I designed and created the knowledge base required for PAT. This contains four models: the domain, student, tutoring and exercises models. The Instructional Expert I created for PAT provides individualised help to the students to help them correctly finish each exercise, and also proposes the next exercise that a student should work on. PAT was evaluated by students enrolled in the Databases subject at QUT, and by staff members involved in teaching the subject. The results of the evaluation were positive and are discussed in the thesis.

Hybridized writing for scientific literacy : pedagogy and evidence

Disengagement of students in science and the scientific literacy of young adults are interrelated international concerns. One way to address these concerns is to engage students imaginatively in activities designed to improve their scientific literacy. Our ongoing program of research has focused on the effects of a sequence of activities that require students to transform scientific information on important issues for their communities from government websites into narrative text suitable for a lay reader. These hybridized stories we call BioStories. Students upload their stories for peer review to a dedicated website. Peer reviews are intended to help students refine their stories. Reviewing BioStories also gives students access to a wider range of scientific topics and writing styles. We have conducted separate studies with students from Grade 6, Grade 9 and Grade 12, involving case study and quasi-experimental designs. The results from the 6th grade study support the argument that writing the sequence of stories helped the students become more familiar with the scientific issue, develop a deeper understanding of related biological concepts, and improve their interest in science. Unlike the Grade 6 study, it was not possible to include a control group for the study conducted across eight 9th grade classes. Nevertheless, these results suggest that hybridized writing developed more positive attitudes toward science and science learning, particularly in terms of the students’ interest and enjoyment. In the most recent case study with Grade 12 students, we found that pride, strength, determination, interest and alertness were among the positive emotions most strongly elicited by the writing project. Furthermore, the students expressed enhanced feelings of self-efficacy in successfully writing hybridized scientific narratives in science. In this chapter, we describe the pedagogy of hybridized writing in science, overview the evidence to support this approach, and identify future developments.

Looking Yonder : Longitudinal Research and Evaluation of the Yonder Participative Arts-based Learning Model Report 2010-2012

Since 2007, KITE Arts Education Program @ QPAC has been engaged in a series of arts and drama-based experiences for students in selected primary schools on the edges of Brisbane and in regional Queensland. The in-school workshop experiences of the program have culminated in a performance by the children for their school community, parents and carers at the Queensland Performing Arts Centre or a regional cultural venue. In conducting an analysis of the Yonder project, the researcher aimed to provide evidence of outcomes brought about through participation by schools, school staff, students and their communities in the Yonder project. To develop longitudinal data project initiators, participants were interviewed at six-monthly intervals to establish patterns of engagement and participation. The report analyses arts-based workshops conducted by the teacher artist in edge-city Brisbane and a regional centre; interviews with teachers and school administrators from the participating schools; interviews with teacher artist and professional artists; interviews with community partners; teacher professional development workshops; community-based workshops; performance outcomes that were the culminating events of the workshop program; student work samples and student reflections on the program. This document covers data and project outputs from February 2010 to July 2012. There have been five iterations of the Yonder project since its commencement in mid-2009 — three in regional Queensland (February–April 2010; February–May 2011; February–May 2012) and two in edge-city1 Brisbane (July–September 2010; August–October 2011). This report is a result of a research partnership between Queensland Performing Arts Centre and Queensland University of Technology (QUT) Creative Industries Faculty(Drama).

Learning processes in creative industries initiatives in developing countries : an examination of policy and practice

What does a dance group in Benin that mixes contemporary and ethnic dancing have in common with Mongolian felt producers that want to enter the design market in Europe? These are both examples of learning processes in Creative Industries initiatives in developing countries. Following the concept of sustainable development, I argue that the challenge for developing countries in contemporary society is to meet the very real need of people for economic development and opportunities for income generation, while at the same time avoiding unintended and unwanted consequences of economic development and globalisation. The concept of the Creative Industries may be a way to promote a development that is sustainable and avoids social exclusion of groups-at-risk. In line with this, I argue that the Creative Industries sector could, in fact, link economic development and the continuation and evolution of local traditions and cultural heritage. A pressing question then is: how can education and learning contribute to creating a context in which talent can flourish? This study aims to provide a comprehensive analysis of the research problem of this thesis: what elements are conducive for individual learning processes in creative development initiatives? In this, I argue that it is crucial to determine what ingredients and characteristics contribute to making these initiatives successful, that is, to meet their specific goals, in a developing context. This is explored through a staged analysis: an overview of quantitative data, an inventory and comparative case studies and, finally, the description and analysis of two in-depth case studies – felt design in Mongolia (Asia) and dance in Benin (Africa), in which I was an observer of the action phase of the local interventions. The analysis culminates in practice-related outcomes related to the operation of creative development initiatives, as well as the contribution to the academic debate on issues like the cultural gap between developed and developing countries, transformative learning and the connection of learning spaces.

Problem-based learning : does accounting education need it?

Problem-based learning (PBL) has been used successfully in disciplines such as medicine, nursing, law and engineering. However a review of the literature shows that there has been little use of this approach to learning in accounting. This paper extends the research in accounting education by reporting the findings of a case study of the development and implementation of PBL at the Queensland University of Technology (QUT) in a new Accountancy Capstone unit that began in 2006. The fundamentals of the PBL approach were adhered to. However, one of the essential elements of the approach adopted was to highlight the importance of questioning as a means of gathering the necessary information upon which decisions are made. This approach can be contrasted with the typical ‘give all the facts’ case studies that are commonly used. Another feature was that students worked together in the same group for an entire semester (similar to how teams in the workplace operate) so there was an intended focus on teamwork in solving unstructured, real-world accounting problems presented to students. Based on quantitative and qualitative data collected from student questionnaires over seven semesters, it was found that students perceived PBL to be effective, especially in terms of developing the skills of questioning, teamwork, and problem solving. The effectiveness of questioning is very important as this is a skill that is rarely the focus of development in accounting education. The successful implementation of PBL in accounting through ‘learning by doing’ could be the catalyst for change to bring about better learning outcomes for accounting graduates.

Economic growth and inequality patterns in the presence of costly technology adoption and uncertainty

We develop a stochastic endogenous growth model to explain the diversity in growth and inequality patterns and the non-convergence of incomes in transitional economies where an underdeveloped financial sector imposes an implicit, fixed cost on the diversification of idiosyncratic risk. In the model endogenous growth occurs through physical and human capital deepening, with the latter being the more dominant element. We interpret the fixed cost as a ‘learning by doing’ cost for entrepreneurs who undertake risk in the absence of well developed financial markets and institutions that help diversify such risk. As such, this cost may be interpreted as the implicit returns foregone due to the lack of diversification opportunities that would otherwise have been available, had such institutions been present. The analytical and numerical results of the model suggest three growth outcomes depending on the productivity differences between the projects and the fixed cost associated with the more productive project. We label these outcomes as poverty trap, dual economy and balanced growth. Further analysis of these three outcomes highlights the existence of a diversity within diversity. Specifically, within the ‘poverty trap’ and ‘dual economy’ scenarios growth and inequality patterns differ, depending on the initial conditions. This additional diversity allows the model to capture a richer range of outcomes that are consistent with the empirical experience of several transitional economies.

Intercultural usability of language learning websites

Website usability can be defined as the ease of use of websites. General usability, pedagogical usability, technical usability and intercultural usability can be considered and examined for the understanding of the usability of language learning websites, which requires a discipline-specific approach. In the field of computer-assisted language learning, usability issues have been addressed mainly in terms of evaluation criteria and have been commonly discussed in relation to user expectations and user experiences. In spite of a growing interest in intercultural language learning, however, little research on intercultural usability of language learning websites has been published yet. There is a need to answer the question of how language learning websites integrate the target language and culture for the development of intercultural sensitivity and competence. This article explores intercultural aspects of language learning websites and presents usability guidelines for designing intercultural language learning websites.

Cross-cultural language learning and web design complexity

Accepting the fact that culture and language are interrelated in second language learning (SLL), the web sites should be designed to integrate with the cultural aspects. Yet many SLL web sites fail to integrate with the cultural aspects and/or focus on language acquisition only. This study identified three issues: (1) anthropologists’ cultural models mostly adopted in cross-cultural web user interface have been superficially used; (2) web designers deal with culture as a fixed one which needs to be modeled into interface design elements, so (3) there is a need for a communication framework between educators and design practitioners, which can be utilized in web design processes. This paper discusses what anthropology can contribute to language learning, mediated through web design processes and suggests a cultural user experience framework for web-based SLL by presenting an exemplary matrix. To evaluate the effectiveness of the framework, the key stakeholders (learners, teachers, and designers) participated in a case scenario-based evaluation. The result shows a high possibility that the framework can enhance the effective communication and collaboration for the cultural integration.

Confronting the shortage of skilled and professional workers : A large scale industry-school partnership initiative

In most of the advanced economies, students are losing interest in careers especially in engineering and related industries. Hence, western economies are confronting a critical skilled labour shortage in areas of technology, science and engineering. The aim of this paper is to document how the organisational and institutional elements of one industry-school partnerships initiative – The Gateway Schools Program - contribute to productive knowledge sharing and networking. In particular this paper focuses on an initiative of an Australian State government in response to a perceived crisis around the skills shortage in an economy transitioning from a localised to a global knowledge production economy. The Gateway Schools initiative signals the first sustained attempt in Australia to incorporate schools into production networks through strategic partnerships linking them to partner organisations at the industry level. We provide case examples of how four schools operationalise the partnerships with the mining and energy industries and how these partnerships as knowledge assets impact the delivery of curriculum and capacity building among teachers. A program theory approach to analysis, informed by theoretical perspectives of Bailey (1994), Bagnall (2007) and Walsh (2004) was adopted. Each of these theorists provides a related but different perspective on the establishment, purpose, and effectiveness respectively of partnerships. Our ultimate goal is to define those characteristics of successful partnerships that do contribute to enhanced interest and engagement by students in those careers that are currently experiencing critical shortages.

Do students and lecturers actively use collaboration tools in learning management systems?

In recent years there has been a large emphasis placed on the need to use Learning Management Systems (LMS) in the field of higher education, with many universities mandating their use. An important aspect of these systems is their ability to offer collaboration tools to build a community of learners. This paper reports on a study of the effectiveness of an LMS (Blackboard©) in a higher education setting and whether both lecturers and students voluntarily use collaborative tools for teaching and learning. Interviews were conducted with participants (N=67) from the faculties of Science and Technology, Business, Health and Law. Results from this study indicated that participants often use Blackboard© as an online repository of learning materials and that the collaboration tools of Blackboard© are often not utilised. The study also found that several factors have inhibited the use and uptake of the collaboration tools within Blackboard©. These have included structure and user experience, pedagogical practice, response time and a preference for other tools.

Enhancing opportunities to feed forward ; developing problem solving skills using web based scenarios

Universities often struggle to satisfy students’ need for feedback. This is an area where student satisfaction with courses of study can be low. Yet it is clear that one of the properties of good teaching is giving the highest quality feedback on student work. The term ‘feedback’ though is most commonly associated with summative assessment given by a teacher after work is completed. The student can often be a passive participant in the process. This paper looks at the implementation of a web based interactive scenario completed by students prior to summative assessment. It requires students to participate actively to develop and improve their legal problem solving skills. Traditional delivery of legal education focuses on print and an instructor who conveys the meaning of the written word to students. Today, mixed modes of teaching are often preferred and they can provide enhanced opportunities for feeding forward with greater emphasis on what students do. Web based activities allow for flexible delivery; they are accessible off campus, at a time that suits the student and may be completed by students at their own pace. This paper reports on an online interactive activity which provides valuable formative feedback necessary to allow for successful completion of a final problem solving assignment. It focuses on how the online activity feeds forward and contributes to the development of legal problem solving skills. Introduction to Law is a unit designed and introduced for completion by undergraduate students from faculties other than law but is focused most particularly on students enrolled in the Bachelor of Entertainment Industries degree, a joint initiative of the faculties of Creative Industries, Business and Law at the Queensland University of Technology in Australia. The final (and major) assessment for the unit is an assignment requiring students to explain the legal consequences of particular scenarios. A number of cost effective web based interactive scenarios have been developed to support the unit’s classroom activities. The tool commences with instruction on problem solving method. Students then view the stimulus which is a narrative produced in the form of a music video clip. A series of questions are posed which guide students through the process and they can compare their responses with sample answers provided. The activity clarifies the problem solving method and expectations for the summative assessment and allows students to practise the skill. The paper reports on the approach to teaching and learning taken in the unit including the design process and implementation of the activity. It includes an evaluation of the activity with respect to its effectiveness as a tool to feed forward and reflects on the implications for the teaching of law in higher education.

Dichotomy in the design studio 2.0 : adapting to new blended learning environments

In a study aimed at better understanding how students adapt to new blended studio learning environments, all undergraduate and masters of architecture students at a large school of architecture in Australia, learned a semester of architectural design in newly renovated, technology embedded, design studio environments. The renovations addressed the lessons learned from a 2011 pilot study of a second year architectural design studio learned in a high technology embedded prototype digital laboratory. The new design studios were purpose designed for the architecture students and adapted Student-Centred Active Learning Environment for Undergraduate Programs design principles. At the end of the semester, the students completed a questionnaire about their experiences of learning in the new design studio environments. Using a dual method qualitative approach, the questionnaire data were coded and extrapolated using both thematic analysis and grounded theory methodology. The results from these two approaches were compared, contrasted and finally merged, to reveal five distinct emerging themes, which were instrumental in offering resistance or influencing adaptation to, the new blended studio learning environments. This paper reports on the study, discusses the major contributors to resistance and adaptation, and proposes points for consideration when renovating or designing new blended studio learning environments. This research extends the 2011 pilot study by the same authors: ‘Dichotomy in the design studio: Adapting to new blended learning environments’.