Occasional paper : Towards the development of an online learning environment (OLE) quality management framework

The project is committed to understanding, recognising and developing various forms of institutionally relevant distributed leadership in developing and trialling various components of a quality management framework for online learning environments in Australian higher education. This paper provides an overview of issues related to the management and improvement of quality, including in the context of higher education. In response to the complex and multi-dimensional nature of both quality and online learning environments (OLEs), the concept of a framework for organising policies, procedures and actions relating to the good governance of OLEs can be found in the literature. Such frameworks vary in scope, format and title, and a (non-exhaustive) sample is presented in summary here. Key learnings that can be drawn from the exemplars frameworks and the related literature include:
- the processes for the design of such frameworks;
- the components of such frameworks;
- the measurement mechanisms and metrics employed in such frameworks; and
- the validation of such frameworks.

Performance of the assessment of quality of life measure in people with hip and knee joint disease and implications for research and clinical use

Objective
To comprehensively evaluate the performance of the Assessment of Quality of Life (AQoL) instrument for measuring health-related quality of life (HRQOL) in people with hip and knee joint disease (arthritis or osteoarthritis).

Methods
Data from 237 individuals were available for analysis from a national cross-sectional, population-based study of hip and knee joint disease in Australia. AQoL-4D data were evaluated using Rasch analysis. A range of measurement properties was explored, including model and item fit, threshold ordering, differential item functioning, and targeting.

Results
Good overall fit of the AQoL with the Rasch model was demonstrated across a range of tests, supporting internal validity. Only 1 item (relating to hearing) showed evidence of misfit. Most AQoL items showed logical sequencing of response option categories, with threshold disordering evident for only 2 of the 12 items (items 4 and 9). Minor issues with potential clinical and research implications include limited options for reporting pain and some evidence of measurement bias between demographic subgroups (including age and sex). Participants' HRQOL was generally better than that represented by the AQoL items (mean ± SD for person abilities −2.15 ± 1.39, mean ± SD for item difficulties 0.00 ± 0.67), indicating ceiling effects that could impact the instrument's ability to detect HRQOL improvement in population-based studies.

Conclusion
The AQoL is a competent tool for assessing HRQOL in people with hip and knee joint disease, although researchers and clinicians should consider the caveats identified when selecting appropriate HRQOL measures for future outcome assessment involving this patient group.

O where are you going? O do you imagine? Reproduction and response : a reflexive sociology of scholarship of teaching and learning in practical legal training

My doctoral research studies Australian PLT practitioners’ engagement with scholarship of teaching and learning. I argue that many PLT practitioners are motivated to engage with scholarship of teaching and learning in their work. There are, however, individual and extra-individual impediments.
PLT practitioners are lawyers that teach in institutional practical legal training (“PLT”). Satisfactory completion of mandatory PLT is an eligibility requirement for admission to the Australian legal profession. The PLT requirement is additional to academic legal qualifications. PLT is undertaken at a post-graduate level with, or after, the academic law degree.
My study investigates PLT practitioners’ motivations and capabilities to engage with scholarship of teaching and learning (“SoTL”). I study organisational symbolic support for SoTL in PLT, and organisational allocation of resources to SoTL in PLT.
The study involves individual and extra-individual domains of PLT practitioners’ work. It considers how social structures (e.g. “the juridical”) are inscribed into individuals’ practices (“teaching”) and, conversely, whether practices influence social structures.
My research adopts qualitative methodologies. These involve inter-disciplinary exchanges between law, legal education, practice research, sociology of law, cultural theory, and theory and practice of teaching and learning. My theoretical framework draws on Pierre Bourdieu’s “reflexive sociology”, and Michel de Certeau’s “heterological science”.
I sourced data from documents, and semi-structured interviews with 36 Australian PLT practitioners. Documentary sources include statutory instruments, speeches, reports, practice directions, histories, and scholarly publications.
To analyse the data I adopted Kelle’s characterisation of “theoretical sensitivity”, drawing on “explicit” and “emergent” analysis strategies derived from “grounded theory”. The explicit strategies were based on my theoretical framework. The emergent strategy involved sensitivity to non-explicit concepts and theories that emerged from the data. Computer-aided qualitative data analysis software expedited these methods.
My findings to date question dominant legal structures’ readiness for change, the implications of this for teaching and learning in PLT, and in particular for PLT practitioners’ engagement with SoTL in PLT.
The espoused rationale for mandatory PLT (in statutes) is improvement for the protection of clients, the administration of justice, and to assure quality legal services. The tacit rationale is improved quality of legal education, and experiences, for lawyers-to-be. My thesis argues dominant structures in legal education impede the espoused and tacit objectives, and impede PLT practitioners’ engagement with scholarship of teaching and learning.

Using hospital standardised mortality ratios to assess quality of care--proceed with extreme caution

Australian Health Ministers have endorsed the hospital standardised mortality ratio (HSMR) as a key indicator of quality and safety, and efforts are currently underway towards its national implementation. In the United Kingdom, Canada, the Netherlands and the United States, the HSMR has been used for several years within organisations to monitor performance and response to various quality and safety programs. In the UK and Canada, the HSMR is also publicly reported and used to compare performance between hospitals. The validity and reliability of the HSMR as a screening tool for distinguishing low-quality from high-quality hospitals remain in doubt, and it has not yet been proven that HSMR reporting necessarily leads to worthwhile improvement in quality of care and patient outcomes.

Principal led school improvement and teacher capacity building in the Barwon South Network : final report, May 2013

The period of interest for this report is the beginning of 2011 to the end of 2012. The period commenced when the Regional Network Leader of the Barwon South Network of schools in the Barwon South Region of the Department of Education and Early Childhood contacted the School of Education at Deakin University, Waurn Ponds Campus Geelong. The Regional Network Leader outlined a desire to engage with Deakin University to research a short-term-cycle model of school improvement to be implemented in the region. While the model was expected to be taken on by all schools in the region the research was limited to the 23 schools in the Barwon South Network with four schools to be investigated more closely for each of two years (2001 & 2012) – eight focus schools in total.

Many positive outcomes flowed from the implementation of short-term-cycle school improvement plans and their associated practices but there was wide variation in the nature and degrees of success and of the perception of the process. The research team asked the following questions of the data:

1. What aspects of the School Improvement Plan (SIP) approach were important for initiating and supporting worthwhile change?
2. What might we take from this, to provide guidance on how best to support change in teaching and learning processes in schools?

The School Improvement Plan (SIP) worked in a range of ways. At one level it was strongly focused on school leadership, and a need to improve principals’ capacity to initiate worthwhile teaching and learning processes in their schools. Underlying this intent, one might think an assumption is operation is that the leadership process involves top down decision-making and a willingness to hold staff accountable for the quality of their practice.

The second strong focus was on the translation into practice and the consequent effect on student learning, involving an emphasis on data and evidence led practice. Hence, along with the leadership focus there was a demand for the process of school improvement to reach down into students and classrooms. Thus, the SIP process inevitably involved a chain of decision-making by which student learning quality drove the intervention, and teachers responsible for this had a common view. The model therefore should not be seen as an intervention only on the principal, but rather on the school decision-making system and focus. Even though it was the principal receiving the SIP planning template, and reporting to the network, the reporting was required to include description of the operation of the school processes, of classroom processes, and of student learning. This of course placed significant constraints on principals, which may help explain the variation in responses and outcomes described above.

The findings from this study are based on multiple data sources: analysis of both open and closed survey questions which all teachers in the 23 schools in the network were invited to complete; interviews with principals, teachers and leaders in the eight case study schools; some interviews with students in the case study schools; and interviews with leaders who worked in the regional network office; and field notes from network meetings including the celebrations days. Celebrations days occurred each school term when groups of principals came together to share and celebrate the improvements and processes happening in their schools. Many of the themes emerging from the analysis of the different data sources were similar or overlapping, providing some confidence in the evidence-base for the findings.

The study, conducted over two years of data collection and analysis, has demonstrated a range of positive outcomes in at the case study schools relating to school communication and collaboration processes, professional learning of principals, leadership teams and classroom teachers. There was evidence in the survey responses and field notes from ‘celebration days’ that these outcomes were also represented in other schools in the network. The key points of change concerned the leadership processes of planning for improvement, and the rigorous attention to student data in framing teaching and learning processes. This latter point of change had the effect of basing SIP processes on a platform of evidence-based change. The research uncovered considerable anecdotal and observational evidence of improvements in student learning, in teacher accounts in interview, and presentations of student work. Interviews with students, although not as representative as the team would have liked, showed evidence of student awareness of learning goals, a key driver in the SIP improvement model. It was, however, not possible over this timescale to collect objective comparative evidence of enhanced learning outcomes.

A number of features of the short-term-cycle SIP were identified that supported positive change across the network. These were: 1) the support structures represented by the network leader and support personnel within schools, 2) the nature of the SIP model – focusing strongly on change leadership but within a collaborative structure that combined top-down and bottom-up elements, 3) the focus on data-led planning and implementation that helped drill down to explicit elements of classroom practice, and 4) the accountability regimes represented by network leader presence, and the celebration days in which principals became effectively accountable to their peers. We found that in the second year of the project, momentum was lost in the case study schools, as the network was dismantled. This raised issues also for the conduct of research in situations of systemic change.

Alongside the finding of evidence of positive outcomes in the case study schools overall, was the finding that the SIP processes and outcomes varied considerably across schools. A number of contextual factors were identified that led to this variation, including school histories of reform, principal management style, and school size and structure that made the short-term-cycle model unmanageable. In some cases there was overt resistance to the SIP model, at least in some part, and this led to an element of performativity in which the language of the SIP was conscripted to other purposes. The study found that even with functioning schools the SIP was understood differently and the processes performed differently, raising the question of whether in the study we are dealing with one SIP or many. The final take home message from the research is that schools are complex institutions, and models of school improvement need to involve both strong principled features, and flexibility in local application, if all schools’ interests in improving teaching and learning processes and outcomes are to be served.

Improving the quality of prediction intervals through optimal aggregation

Neural networks (NNs) are an effective tool to model nonlinear systems. However, their forecasting performance significantly drops in the presence of process uncertainties and disturbances. NN-based prediction intervals (PIs) offer an alternative solution to appropriately quantify uncertainties and disturbances associated with point forecasts. In this paper, an NN ensemble procedure is proposed to construct quality PIs. A recently developed lower-upper bound estimation method is applied to develop NN-based PIs. Then, constructed PIs from the NN ensemble members are combined using a weighted averaging mechanism. Simulated annealing and a genetic algorithm are used to optimally adjust the weights for the aggregation mechanism. The proposed method is examined for three different case studies. Simulation results reveal that the proposed method improves the average PI quality of individual NNs by 22%, 18%, and 78% for the first, second, and third case studies, respectively. The simulation study also demonstrates that a 3%-4% improvement in the quality of PIs can be achieved using the proposed method compared to the simple averaging aggregation method.