Assurance of learning : the role of work integrated learning and industry partners

In the partnering with students and industry it is important for universities to recognize and value the nature of knowledge and learning that emanates from work integrated learning experiences is different to formal university based learning. Learning is not a by-product of work rather learning is fundamental to engaging in work practice. Work integrated learning experiences provide unique opportunities for students to integrate theory and practice through the solving of real world problems. This paper reports findings to date of a project that sought to identify key issues and practices faced by academics, industry partners and students engaged in the provision and experience of work integrated learning within an undergraduate creative industries program at a major metropolitan university. In this paper, those findings are focused on some of the particular qualities and issues related to the assessment of learning at and through the work integrated experience. The findings suggest that the assessment strategies needed to better value the knowledges and practices of the Creative Industries. The paper also makes recommendations about how industry partners might best contribute to the assessment of students’ developing capabilities and to continuous reflection on courses and the assurance of learning agenda.

Hunters & gatherers : strategies for curriculum mapping and data collection for assurance of learning

Assurance of learning is a predominant feature in both quality enhancement and assurance in higher education. Assurance of learning is a process that articulates explicit program outcomes and standards, and systematically gathers evidence to determine the extent to which performance matches expectations. Benefits accrue to the institution through the systematic assessment of whole of program goals. Data may be used for continuous improvement, program development, and to inform external accreditation and evaluation bodies. Recent developments, including the introduction of the Tertiary Education and Quality Standards Agency (TEQSA) will require universities to review the methods they use to assure learning outcomes. This project investigates two critical elements of assurance of learning: 1. the mapping of graduate attributes throughout a program; and 2. the collection of assurance of learning data. An audit was conducted with 25 of the 39 Business Schools in Australian universities to identify current methods of mapping graduate attributes and for collecting assurance of learning data across degree programs, as well as a review of the key challenges faced in these areas. Our findings indicate that external drivers like professional body accreditation (for example: Association to Advance Collegiate Schools of Business (AACSB)) and TEQSA are important motivators for assuring learning, and those who were undertaking AACSB accreditation had more robust assurance of learning systems in place. It was reassuring to see that the majority of institutions (96%) had adopted an embedding approach to assuring learning rather than opting for independent standardised testing. The main challenges that were evident were the development of sustainable processes that were not considered a burden to academic staff, and obtainment of academic buy in to the benefits of assuring learning per se rather than assurance of learning being seen as a tick box exercise. This cultural change is the real challenge in assurance of learning practice.

Communities of practice : building organisational capability through an undergraduate assurance of learning program

This study explores academic perceptions of organizational capability and culture following a project to develop a quality assurance of learning program in a business school. In the project a community of practice structure was established to include academics in the development of an embedded, direct assurance of learning program affecting more than 5000 undergraduate students and 250 academics from nine different disciplines across four discipline based departments. The primary outcome from the newly developed and implemented assurance of learning program was the five year accreditation of the business school’s programs by two international accrediting bodies, EQUIS and AACSB. This study explores a different outcome, namely perceptions of organizational culture and individual capabilities as academics worked together in teaching teams and communities. This study uses a survey and interviews with academics involved, through a retrospective panel design consisting of an experimental group and a control group. Results offer insights into communities of practice as a means of encouraging new individual and organizational capability and strategic culture adaptation.

Australian higher education evaluation through assurance of learning

A collaborative research project conducted by five Australian universities inquired into the philosophy and motivation for Assurance of Learning (AoL) as a process of education evaluation. Associate Deans Teaching and Learning representing Business schools from twenty-five universities across Australia participated in telephone interviews. Data was analysed using NVIVO9. Results indicated that articulated rationale for AoL was both ensuring that students had acquired the attributes and skills the universities claimed they had, and the philosophy of continuous improvement. AoL was motivated both by ritualistic objectives to satisfy accreditation requirements and virtuous agendas for quality improvement. Closing-the-loop was emphasised, but was mostly wishful thinking for next steps beyond data collection and reporting. AoL was conceptualised as one element within the larger context of quality review, but there was no evidence of comprehensive frameworks or strategic plans.

Assurance of learning – are academics and senior management singing from the same hymn sheet?

Assurance of learning (AoL) is an important process in quality education, designed to measure the accomplishment of educational aims at the core of an institution’s programs, whilst encouraging faculty to continuously develop and improve the programs and courses. This paper reports on a study of Australian business schools to investigate current AoL practices through semi structured interviews with senior faculty leaders followed by focus group interviews with groups of senior program leaders and groups of academic teaching staff. Initial findings indicate there are significant challenges in encouraging academic staff to commit to the process and recognise the benefits of assuring learning. The differences in understanding between the various leaders and the academics were highlighted through the different focus groups. Leaders’ stressed strategic issues such as staff engagement and change, while academics focussed on process issues such as teaching graduate attributes and external accreditation. Understanding the differences in the perspectives of leaders and faculty is important, as without a shared understanding between the two groups, there is likely to be limited engagement, which creates difficulties in developing effective assurance of learning processes. Findings indicate that successful strategies developed to foster shared values on assurance of learning include: strong senior leaders’ commitment; developing champions among program and unit level staff; providing professional development opportunities; promoting and celebrating success and effectiveness; and ensuring an inclusive process with academics of all levels collaborating in the development and implementation of the process.

Building sustainable and effective assurance of learning process in a changing higher education environment

In a study of assuring learning in Australian Business Schools, 25 Teaching and Learning Associate Deans were interviewed to identify current issues in developing and measuring the quality of teaching and learning outcomes. Results indicate that for most institutions developing a perspective on graduate attributes and mapping assessments to measure outcomes across an entire program required knowledge creation and the building of new inclusive processes. Common elements of effective practice, namely those which offered consistently superior outcomes, included: inclusive processes; embedded graduate attributes throughout a program; alongside consistent and appropriate assessment. Results indicate that assurance of learning processes are proliferating nationally while quality of teaching and learning outcomes and in the processes for assuring it is increasing as a result.

A preliminary investigation into technology and processes facilitating the assurance of learning

This paper reports on the outcomes from a preliminary evaluation of technologies and processes intended to support the Assurance of Learning initiative in the business faculty of an Australian university. The study investigated how existing institutional information systems and operational processes could be used to support direct measures of student learning and the attainment of intended learning goals. The levels at which learning outcomes had been attained were extracted from the University Learning Management System (LMS), based on rubric data for three assessments in two units. Spreadsheets were used to link rubric criteria to the learning goals associated with the assessments as identified in a previous curriculum mapping exercise, and to aggregate the outcomes. Recommendations arising from this preliminary study are made to inform a more comprehensive pilot based on this approach, and manage the quality of student learning experiences in the context of existing processes and reporting structures.

The Effectiveness of Learning Portfolios: A Study of Quality Assurance Programs of Selected Health Regulatory Colleges in Ontario

Health regulatory colleges promote quality practice and continued competence through Quality Assurance (QA) programs. For many colleges, a QA program includes the use of portfolios that incorporate self-directed learning. The purpose of this study was to determine some of the issues surrounding the effectiveness of QA portfolio programs. The literature review revealed that portfolios are valuable tools, but gaps in knowledge include a comparative analysis of QA programs and the perspective of regulatory college administrators. Data were collected through interviews with 6 administrators and a review of 14 portfolio models described on college websites. The results from the two data sources were applied to Robert Stake's responsive evaluation framework to identify issues related to the portfolio's effectiveness (Stake, 1967). The learning components of portfolios were analyzed through the humanist and constructivist lenses. All 14 portfolio models were found to have 3 main components: self-diagnosis, learning plan and activities, and self-evaluation. However, differences were uncovered in learners' autonomy in selecting learning activities, methods of portfolio evaluation, and the relationship between the portfolio and other QA components. The results revealed a dual philosophy of learning in portfolio models and an apparent contradiction between the needs of the individual learner and the organization. Paths for future research include the tenuous relationship between competence and learning, and the impact of technical approaches on selfdirected learning initiatives. A key recommendation is to acknowledge the unique identity of each profession so that health regulatory colleges can address legislative demands and learner needs.

Student self-assessment of performance : a useful learning tool for students and instructors?

Assurance of Learning is a focus for many Business Schools to meet AACSB requirements. Prior research has examined student self-assessment as a measure of learning in this context, but we propose instead that this is a useful ‘learning tool’ for students and instructors that can help improve student performance and curricula. We examined self-assessment in a third year undergraduate e-marketing course and found a positive, though weak relationship between student self-assessment and instructor ratings on intended learning goals and criteria for a written assignment. Students who self-assessed tended to perform better in the assessment item and course than those who did not self-assess. But less competent students were more inaccurate in their self-assessment, suggesting a need to enhance understanding of marking criteria and performance standards within this group. Implications for educators are discussed.

The practice of quality assuring learning in higher education

There remains a lack of published empirical data on the substantive outcomes of higher learning and the establishment of quality processes for determining them. Studies that do exist are nationally focused with available rankings of institutions reflecting neither the quality of teaching and learning nor the diversity of institutions. This paper describes two studies in which Associate Deans from Australian higher education institutions and focus groups of management and academics identify current issues and practices in the design, development and implementation of processes for assuring the quality of learning and teaching. Results indicate that developing a perspective on graduate attributes and mapping assessments to measure outcomes across an entire program necessitates knowledge creation and new inclusive processes. Common elements supporting consistently superior outcomes included: inclusivity; embedded graduate attributes; consistent and appropriate assessment; digital collection mechanisms; and systematic analysis of outcomes used in program review. Quality measures for assuring learning are proliferating nationally and changing the processes, systems and culture of higher education as a result.

From policy borrowing to implementation : an illuminative evaluation of learning and teaching in higher education in Australia (2002 to 2008)

This study documents and theorises the consequences of the 2003 Australian Government Reform Package focussed on learning and teaching in Higher Education during the period 2002 to 2008. This is achieved through the perspective of program evaluation and the methodology of illuminative evaluation. The findings suggest that the three national initiatives of that time, Learning and Teaching Performance Fund (LTPF), Australian Learning and Teaching Council (ALTC), and Australian Universities Quality Agency (AUQA), were successful in repositioning learning and teaching as a core activity in universities. However, there were unintended consequences brought about by international policy borrowing, when the short-lived nature of LTPF suggests a legacy of quality compliance rather than one of quality enrichment.

A stakeholder analysis of one institution’s assuring learning experience – is the promise and the practice worth the price?

Universities around the world are rushing to implement assurance of learning policies and practices with varying degrees of success. One School investigated its own policy and practice development through the eyes of its key stakeholders to identify whether the practice was worth the price. Findings indicate that although the key stakeholders considered different needs and viewed their experiences differently, value did abound and was in the eye of the beholder.

Quality assurance of assessment and moderation discourses involving sessional staff

Quality assurance is a major agenda in tertiary education. The casualisation of academic work, especially in teaching, is also a quality assurance issue. Casual or sessional staff members teach and assess more than 50% of all university courses in Australia, and yet the research in relation to the role sessional staff play in quality assurance of student assessment outcomes is scarce. Moderation processes are a pivotal part of robust quality assurance measures. Drawing upon previous work surrounding four discourses of moderation, this pilot project reports the results of research into the role and impact of sessional staff in moderation processes at tertiary level. Qualitative data were gathered through focus interviews. Results, in the form of various moderation discourses, indicate that sessional staff impact the formal quality assurance processes in numerous ways.

Evaluation of collaborative assessment of work integrated learning

The international trend towards an increasingly standards-based approach to higher education and the resultant focus on the assurance of learning in tertiary programs have generated a strong emphasis on the assessment of outcomes across the higher education sector. In legal education, curriculum reform is highly prevalent internationally as a result of various reviews of legal education. As legal education focuses more on the attainment of a broader set of outcomes encompassing soft skills, capabilities and attributes, more authentic assessment will need to be developed appropriate to this new environment, meaning that modes of assessment with strong application in real-life settings should be preferred.

The impact of learning outcomes in business education: assessing value, relevance and graduate ability in a multi-country study of employers and business graduates

This paper discusses critical findings from a two-year EU-funded research project involving four European countries: Austria, England, Slovenia and Romania. The project had two primary aims. The first of these was to develop a systematic procedure for assessing the balance between learning outcomes acquired in education and the specific needs of the labour market. The second aim was to develop and test a set of meta-level quality indicators aimed at evaluating the linkages between education and employment. The project was distinctive in that it combined different partners from Higher Education, Vocational Training, Industry and Quality Assurance. One of the key emergent themes identified in exploratory interviews was that employers and recent business graduates in all four countries want a well-rounded education which delivers a broad foundation of key business knowledge across the various disciplines. Both groups also identified the need for personal development in critical skills and competencies. Following the exploratory study, a questionnaire was designed to address five functional business areas, as well as a cluster of 8 business competencies. Within the survey, questions relating to the meta-level quality indicators assessed the impact of these learning outcomes on the workplace, in terms of the following: 1) value, 2) relevance and 3) graduate ability. This paper provides an overview of the study findings from a sample of 900 business graduates and employers. Two theoretical models are proposed as tools for predicting satisfaction with work performance and satisfaction with business education. The implications of the study findings for education, employment and European public policy are discussed.