Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

From policy borrowing to implementation : an illuminative evaluation of learning and teaching in higher education in Australia (2002 to 2008)

This study documents and theorises the consequences of the 2003 Australian Government Reform Package focussed on learning and teaching in Higher Education during the period 2002 to 2008. This is achieved through the perspective of program evaluation and the methodology of illuminative evaluation. The findings suggest that the three national initiatives of that time, Learning and Teaching Performance Fund (LTPF), Australian Learning and Teaching Council (ALTC), and Australian Universities Quality Agency (AUQA), were successful in repositioning learning and teaching as a core activity in universities. However, there were unintended consequences brought about by international policy borrowing, when the short-lived nature of LTPF suggests a legacy of quality compliance rather than one of quality enrichment.

Framing student evaluations of university learning and teaching : discursive strategies and textual outcomes

Evaluation in higher education is an evolving social practice, that is, it involves what people, institutions and broader systems do and say, how they do and say it, what they value, the effects of these practices and values, and how meanings are ascribed. The textual products (verbal, written, visual, gestural) that inform and are produced by, for and through evaluative practices are important as they promulgate particular kinds of meanings and values in specific contexts. This paper reports on an exploratory study that sought to investigate, using discourse analysis, the types of evaluative practices that were ascribed value, and the student responses that ensued, in different evaluative instruments. Findings indicate that when a reflective approach is taken to evaluation, students’ responses are more considered, they interrogate their own engagement in the learning context and they are more likely to demonstrate reconstructive thought. These findings have implications for reframing evaluation as reflective learning.

Caught in the headlights: Designing for creative learning and teaching in higher education

This study was an exploration of the different ways that educators conceptualise and approach creative learning and teaching. The research revealed theoretical and practice-based insights, demonstrating that exemplary teachers adopt an ecological approach to designing for student creativity; this approach acknowledges and works with the complexity of the higher education environment and the dynamic relationships between students, peers and teachers. The inquiry confirmed the value of using learning design patterns to uncover hidden creative processes.

Feeling the pressure: Early childhood educators’ reported views about learning and teaching phonics in Australian prior-to-school settings

Early childhood educators’ beliefs about literacy teaching can impact on the types of phonics experiences educators provide for children in prior-to-school settings. The Australian Early Years Learning Framework supports a play-based, intentional approach to teaching phonics, however little is known about what Australian early childhood educators believe is important in teaching phonics in the prior-to-school years. Using a qualitative content analysis, this research study investigates 115 early childhood educators’ views about how phonics should be taught and the use of commercially produced phonics programs (e.g. Jolly Phonics and Letterland) in prior-to-school settings. This study further investigates educators’ perceived pressures to include structured phonics lessons, as a way of addressing parental notions of ‘school readiness’. The results of this study indicate conflicting views were held about how phonics should be taught. Some educators also experienced external pressures to engage in literacy practices that may be in opposition with their own beliefs about how literacy is learnt. This study provides insights into the pedagogical practices early childhood educators believe are appropriate when teaching phonics. The educational implications are discussed.

The paradox of creative uncertainty in learning and teaching

In an ever-changing and globalised world there is a need for higher education to adapt and evolve its models of learning and teaching. The old industrial model has lost traction, and new patterns of creative engagement are required. These new models potentially increase relevancy and better equip students for the future. Although creativity is recognised as an attribute that can contribute much to the development of these pedagogies, and creativity is valued by universities as a graduate capability, some educators understandably struggle to translate this vision into practice. This paper reports on selected survey findings from a mixed methods research project which aimed to shed light on how creativity can be designed for in higher education learning and teaching settings. A social constructivist epistemology underpinned the research and data was gathered using survey and case study methods. Descriptive statistical methods and informed grounded theory were employed for the analysis reported here. The findings confirm that creativity is valued for its contribution to the development of students’ academic work, employment opportunities and life in general; however, tensions arise between individual educator’s creative pedagogical goals and the provision of institutional support for implementation of those objectives. Designing for creativity becomes, paradoxically, a matter of navigating and limiting complexity and uncertainty, while simultaneously designing for those same states or qualities.

The gradual release of responsibility: A case study of teaching science inquiry skills

This thesis investigated how a year-4 teacher used a pedagogical approach referred to as the Gradual Release of Responsibility (GRR) model of instruction for teaching Science Inquiry Skills in a primary classroom. Through scaffolding her students' learning using the GRR, the teacher guided her students towards developing an understanding about Scientific Inquiry leading to the foundations of scientific literacy. A learning environment was established in which students engaged in rich conversations, designed and conducted experiments using fair testing procedures, analysed and offered justifications for results, and negotiated knowledge claims in ways similar to some of those in the scientific community.

Disabled Children Negotiating School Life: Agency, Difference and Teaching Practice

This paper explores the school experiences of seven 11–14 year old disabled children, and focuses on their agency as they negotiated a complex, changing, and often challenging social world at school where “difference” was experienced in negative ways. The paper draws on ethnographic data from a wider three-year study that explores the influence of school experiences on both disabled and non-disabled children’s identity as they make the transition from primary to secondary school in regular New Zealand schools (although the focus of the present paper is only on the experiences of disabled children). The wider study considers how Maori (indigenous people of Aotearoa/New Zealand) and Pakeha (New Zealanders of NZ European descent) disabled children and their non- disabled matched peers (matched for age, gender and classroom) understand their personal identity, and how factors relating to transition (from primary to secondary school); culture; impairment (in the case of disabled children); social relationships; and school experience impact on children’s identities. Data on Maori children’s school experiences is currently being collected, and is not yet available for inclusion in this paper. On the basis of our observations in schools we will illustrate how disabled children felt and were made to feel different through an array of structural barriers such as separate provision for disabled students, and peer and teacher attitudes to diversity. However, we agree with Davis, Watson, Shakespeare and Corker’s (2003) interpretation that disabled children’s rights and participation at school are also under attack from a “deeper cultural division” (p. 205) in schools based on discourses of difference and normality. While disabled students in our study were trying to actively construct and shape their social and educational worlds, our data also show that teachers and peers have the capacity to either support or supplant these attempts to be part of the group of “all children”. We suggest that finding solutions that support disabled children’s full inclusion and participation at school requires a multi-faceted and systemic approach focused on a pedagogy for diverse learners, and on a consistent and explicitly inclusive policy framework centred on children’s rights.

Combinations of Prompt and Target Response Modalities and Their Impact on Emergent Relations and Teaching Efficiency.

In the present study, native Spanish speakers were taught a small English vocabulary (Spanish-to-English intraverbals). Four different training conditions were created by combining textual and echoic prompts with written and vocal target responses. The efficiency of each training condition was examined by analysing emergent relations (i.e., tacts) and the total number of sessions required to reach mastery under each training condition. All combinations of prompt-response modalities generated increases in correct responding on tests for emergent relations but when target responses were written, mastery criterion was reached faster. Results are discussed in terms of efficiency for emergent relations and recommendations for future directions are provided.

An after-school literacy program : investigating the experiences of students with literacy difficulties, their volunteer tutors, and the tutors' transition into the teaching profession

This research responds to a pervasive call for our educational institutions to provide students with literacy skills, and teachers with the instructional supports necessary to facilitate this skill acquisition. Questions were posed to gain information concerning the efficacy ofteaching literacy strategies to students with learning difficulties, the impact of this training on their volunteer tutors, and the influence of this experience on these tutors' ensuing instructional practice as teacher candidates in a preservice education program. Study #1 compared a nontreatment group of students with literacy difficulties who participated in the program and found that program participants were superior at reading letter patterns and at comprehending the elements of story grammar. Concurrently, the second study explored the experiences of 19 volunteer tutors and uncovered that they acquired instructional skills as they established a knowledge base in teaching reading and writing, and they affirmed personal goals to become future teachers. Study #3 tracked 6 volunteer tutors into their pre-service year and identified their constructions, and beliefs about literacy instruction. These teacher candidates discussed how they had intended to teach reading and writing strategies based on their position that effective teaching ofthese skills in the primary grades is integral to academic success. The teacher candidates emphasized the need to build rapport with students, and the need to exercise flexibility in lesson plan delivery while including activities to meet emotional and developmental requirements of students. The teacher candidates entered their pre-service education with an initial cognition set based on the limited teaching context of tutoring. This foundational ii perception represented their prior knowledge of literacy instruction, a perception that appeared untenable once they were immersed in a regular instructional setting. This disparity provoked some of the teacher candidates to denounce their teacher mentors for not consistently employing literacy strategies and individualized instruction. This critical perspective could have been a demonstration of cognitive dissonance. In the end, when the teacher candidates began to look toward the future and how they would manage the demands of an inclusive classroom, they recognized the differences in the contexts. With an appreciation for the need for balance between prior and present knowledge, the teacher candidates remained committed to implementing their tutoring strategies in future teaching positions. This document highlights the need for teacher candidates with instructional experience prior to teacher education, to engage in cognitive negotiations to assimilate newly acquired pedagogies into existing pedagogies.

A Study of role conflict, role ambiguity, locus of control, job satisfaction and participation in decision making among front-line vocational service workers for mentally retarded adults

This study was an investigation of individual and organizational factors, as perceived by front-line vocational service workers from Adult Rehabilitation Centres (ARC Industries) for mentally retarded adults. The specific variables which were measured included role conflict/role ambiguity (role factors), internal/external locus of control (individual differences), job satisfaction with work and supervision (job attitudes) and participation in deci~ion making (organizational factor). The exploration of these constructs was conducted by means of self-report questionnaires which were completed by sixty-nine out of a total of ninety front-line employees. The surveys were distributed in booklet form to nine distinct rehabilitation facilities from St. Catharines, West Lincoln, Greater Niagara, Port Colborne, WeIland, Fort Erie, Hamilton, Guelph and Brantford. The survey data was evaluated by the statisti.cal Package for the Social Sciences (SPSS) which used the Pearson Product Moment Correlation procedure and a compar~son of means test. A comparison of correlation coefficients test was also conducted. This statistical procedure was calculated mathematically. The results obtained from the statistical evaluation confirmed the prediction that self-reported measures of participation in decision making and satisfaction (work and supervision) would be negatively correlated with role conflict and role ambiguity. As well, the speculation that perceived satisfaction (work and supervision) would be positively correlated with participation in decision making was empirically supported. Internal and external locus of control did not contribute to a significant difference in r~sponses to role perceptions (conflict and ambiguity) , satisfaction (work and supervision) or the correlational relationship between participation in decision making and satisfaction (work and supervision). Overall, the findings from this study substantiated the importance of examining employee perceptions in the workplace and the interrelationships among individual and organizational variables. This research was considered a contribution to the general area of occupational stress and to the study of individuals in work organizations.

Sentence combining and thinking : a study of adult ESL learners

Thesis (M.Ed.)--Brock University, 2003.

The association of faculty's teaching philosophy and teaching behaviour, and critical thinking and self- direction in learning

The intent in this study was to investigate in what ways teachers· beliefs about education and teaching are expressed in the specific teaching behaviours they employ, and whether teaching behaviours, as perceived by their students, are correlated with students· critical thinking and self-directed learning. To this end the relationships studied were: among faCUlty members· philosophy of teaching, locus of control orientation, psychological type, and observed teaching behaviour; and among students· psychological type, perceptions of teaching behaviour, self-directed learning readiness, and critical thinking. The overall purpose of the study was to investigate whether the implicit goals of higher education, critical thinking and self-direction, were actually accounted for in the university classroom. The research was set within the context of path-goal theory, adapted from the leadership literature. Within this framework, Mezirow·s work on transformative learning, including the influences of Habermas· writings, was integrated to develop a theoretical perspective upon which to base the research methodology. Both qualitative and quantitative methodologies were incorporated. Four faCUlty and a total of 142 students participated in the study. Philosophy of teaching was described through faCUlty interviews and completion of a repertory grid. Faculty completed a descriptive locus of control scale, and a psychological type test. Observations of their teaching behaviour were conducted. Students completed a Teaching Behaviour Assessment Scale, the Self-Directed Learning Readiness Scale, a psychological type test, and the Watson-Glaser Critical Thinking Appraisal. A small sample of students were interviewed. Follow-up discussions with faculty were used to validate the interview, observation, teaching behaviour, and repertory grid data. Results indicated that some discrepancies existed between faculty's espoused philosophy of teaching and their observed teaching behaviour. Instructors' teaching behaviour, however, was a function of their personal theory of practice. Relationships were found between perceived teaching behaviour and students· self-directed learning and critical thinking, but these varied across situations, as would be predicted from path-goal theory. Psychological type of students and instructor also accounted for some of the variability in the relationships studied. Student psychological type could be shown as a partial predictor of self-directed learning readiness. The results were discussed in terms of theory development and implications for further research and practice.

A case study investigation of the learning needs of the Niagara grape and wine community

The Niagara Grape and Wine Community (NGWC) is an industry that has undergone rapid change and expansion as a result of changes in governmental regulations and consumer preferences. As a result of these changes, the demands of the wine industry workforce have changed to reflect the need to implement new strategies and practices to remain viable and competitive. The influx of people into the community with little or no prior practical experience in grape growing (viticulture) or winemaking (oenology) has created a need for additional training and learning opportunities to meet workforce needs. This case study investigated the learning needs of the members of this community and how these needs are currently being met. The barriers to, and the opportunities for, members acquiring new knowledge and developing skills were also explored. Participants were those involved in all levels of the industry and sectors (viticulture, processing, and retail), and their views on needs and suggestions for programs of study were collected. Through cross analyses of sectors, areas of common and unique interest were identified as well as formats for delivery. A common fundamental component was identified by all sectors - any program must have a significant applied component or demonstration of proficiency and should utilize members as peer instructors, mentors, and collaborators to generate a larger shared collective of knowledge. Through the review of learning organizations, learning communities, communities of practices, and learning networks, the principles for the development of a Grape and Wine Learning Network to meet the learning needs of the NGWC outside of formal institutional or academic programs were developed. The roles and actions of members to make such a network successful are suggested.