An institutional study of the influence of ‘onlineness’ on student evaluation of teaching in a dual mode Australian university

Student evaluation of teaching (SET) is now commonplace in many universities internationally. The most common criticism of SET practices is that they are influenced by a number of non-teaching-related factors. More recently, there has been dramatic growth in online education internationally, but only limited research on the use of SET to evaluate online teaching. This paper presents a large-scale and detailed investigation, using the institutional SET data from an Australian university with a significant offering of wholly online units, and whose institutional SET instrument contains items relating to student perceptions of online technologies in teaching and learning. The relationship between educational technology and SET is not neutral. The mean ratings for the ‗online‘ aspects of SET are influenced by factors in the wider teaching and learning environment, and the overall perception of teaching quality is influenced by whether a unit is offered in wholly online mode or not.

Workplace learning coordinators program : formative evaluation : final report

The workplace learning coordinators program was established to increase the number of young people undertaking workplace learning placements, especially in industries that provide strong vocational outcomes. Additional attention has been paid to the potential benefits of the program for Koorie young people and other students who experience disadvantage. The program also seeks to improve the alignment between VCAL and VET provision and local industry needs, with a focus on indentifying new workplace learning opportunities with each service area’s employers. This formative evaluation sought to monitor the initial implementation of the WLC program during its first twelve months. Over this period the evaluators recorded the early teething challenges faced by WLCs during their first few months, and the eventual consolidation of their practices during the second half of the program’s first year. In reviewing all of the data collected for this evaluation, and the progressive responses of WLCs and stakeholders over the program’s first year, the evaluators consider that the model of service delivery is appropriate for what the program aims to achieve and recommend its continuance to 2013.

A critical evaluation and comparison of two formative measures of system quality using criterion variables

Evaluating the validity of formative variables has presented ongoing challenges for researchers. In this paper we use global criterion measures to compare and critically evaluate two alternative formative measures of System Quality. One model is based on the ISO-9126 software quality standard, and the other is based on a leading information systems research model. We find that despite both models having a strong provenance, many of the items appear to be non-significant in our study. We examine the implications of this by evaluating the quality of the criterion variables we used, and the performance of PLS when evaluating formative models with a large number of items. We find that our respondents had difficulty distinguishing between global criterion variables measuring different aspects of overall System Quality. Also, because formative indicators “compete with one another” in PLS, it may be difficult to develop a set of measures which are all significant for a complex formative construct with a broad scope and a large number of items. Overall, we suggest that there is cautious evidence that both sets of measures are valid and largely equivalent, although questions still remain about the measures, the use of criterion variables, and the use of PLS for this type of model evaluation.

Leading the evaluation of institutional online learning environments for quality enhancement in times of change

This paper reports on findings from a nationally funded project which aims to design and implement a quality management framework for online learning environments (OLEs). Evaluation is a key component of any quality management system and it is this aspect of the framework that is the focus of this paper. In developing the framework initial focus groups were conducted at the five participating institutions. These revealed that, although regarded as important, there did not appear to be a shared understanding of the nature and purpose of evaluation. A second series of focus groups revealed there were multiple perspectives arising from those with a vested interest in online learning. These perspectives will be outlined. Overall, how evaluation was undertaken was highly variable within and across the five institutions reflecting where they were at in relation to the development of their OLE.

Steps in the design, development and formative evaluation of obesity prevention-related behavior change trials

Obesity prevention interventions through dietary and physical activity change have generally not been effective. Limitations on possible program effectiveness are herein identified at every step in the mediating variable model, a generic conceptual framework for understanding how interventions may promote behavior change. To minimize these problems, and thereby enhance likely intervention effectiveness, four sequential types of formative studies are proposed: targeted behavior validation, targeted mediator validation, intervention procedure validation, and pilot feasibility intervention. Implementing these studies would establish the relationships at each step in the mediating variable model, thereby maximizing the likelihood that an intervention would work and its effects would be detected. Building consensus among researchers, funding agencies, and journal editors on distinct intervention development studies should avoid identified limitations and move the field forward.

Formative evaluation of baby makes 3

Baby Makes 3 was originally developed by Whitehorse Community Health Service and the City of Whitehorse as a program which seeks to prevent violence by promoting respect and equality between couples who have recently become parents for the first time. This is a significant but often stressful event in the life of families and it is not uncommon for couples to adopt ways of relating which reflect gendered norms and foster inequality.
By being offered as a program to all first-time parents by maternal and child health staff, the program in Whitehorse managed to engage with couples at this critical time, many of whom found the program enabled them to adopt greater equality in their relationships. As such, the initial Baby Makes 3 program has been found to be an effective and cost-efficient violence prevention strategy (Flynn, 2011).
The introduction of Baby Makes 3 to the Great South Coast Region is the first time this program has been implemented in a non-metropolitan setting. In addition to the original program aims, a number of additional aims and activities have been added, with the new project being known as Baby Makes 3 Plus.
Programs developed in urban contexts often face different challenges when implemented in rural and regional settings. Likewise, the factors for facilitating success may vary from those in an urban setting (Maidment and Bay, 2012). Hence, this formative evaluation aims to provide feedback on the initial implementation of the original Baby Makes 3 program in the Great South Coast Region with the expectation this will enable problems with implementation and/or opportunities for further enhancement of outcomes to be identified and appropriate action taken.

Enhancing quality in online learning : scaffolding planning and design through proactive evaluation

A major issue confronting educators is the extent to which they wish to conform to so-called paradigm shifts in teaching and learning. In the contemporary world of tertiary education these shifts embrace both pedagogy (from instructivist to constructivist) and technology (from classroom to online). As teachers and learners are faced with the potential of these new learning environments, the extent to which learning outcomes are achieved remains a high priority and subject to a wide range of evaluation strategies. Conventionally, evaluation has been positioned at the end of the instructional development cycle, to assess first whether or not the creative effort achieved the original product goals and second whether or not the desired learning outcomes were realized. In the context of online teaching and learning environments, however, the level of understanding teachers, learners and developers have of the medium can impact the ultimate effectiveness of the product. This paper articulates an additional dimension to post-development evaluation processes in proposing proactive evaluation, a framework that identifies critical online learning factors and influences that will better inform the planning, design and development of learning resources. This notion of proactive evaluation advocates resource development being undertaken where all planning activities are assessed against the evaluation criteria that would normally be applied during formative assessment. By performing these evaluation checks proactively, online learning resources will, in principle, work first time as all relevant factors and issues will have been considered and resolved. More importantly, for those participants who are new to online environments, proactive evaluation will perform a scaffolding and professional development role by enhancing online teaching or learning competencies.

Analysis of the potential and problems of new institutional economics for third world development

Purpose – The aim of this paper is to provide a critical evaluation of the potential of new institutional economics (NIE) in third world development.

Design/methodology/approach – The paper reviews various theories under NIE from both conceptual and empirical perspectives. It then reviews the various definitions of institutions and show that institutions are essential to overcome problems of information and uncertainty.

Findings – The review finds that weak institutions can undermine development and hence governments in developing countries should strengthen their institutions to provide greater scope for efficient functioning of markets. Where the market does not work owing to high transactions costs, traditional institutions of collective action and group decision making can work and hence need to be recognised.

Research limitations/implications – The major implications of the paper is that in developing countries, a clear understanding of various institutions such as user groups, inter-linked credit markets, rotational irrigation etc. is needed before they are replaced or modified by other institutions. The main limitations of NIE are that there can be capture by elites of various institutional innovations in rural areas, and that it does not explicitly consider income distribution and uncertainty which are glossed over and hence remain areas for future research.

Originality/value – This paper critically reviews the various institutional environments that developing countries face in addressing development issues.

Evaluation of cervial range of motion and isometric nech muscle strngth: reliability and validity

Objective: To examine the test–retest reliability and construct validity of cervical active range of motion and isometric neck muscle strength as measured by the Multi Cervical Rehabilitation Unit (Hanoun Medical Inc., Ontario).
Design: A cross-sectional study.
Setting: Institutional practice.
Subjects: Twenty-one patients with neck pain and 25 healthy volunteers.
Methods: After a trial-run session, active range of motion (AROM) was measured in the subsequent two sessions, with 2–3 days in between. During each session, three measurements were taken for each direction (flexion, extension, lateral flexions and rotations). The measurement of isometric strength was after a 15-minute break following completion of the measurement of AROM. Three measurements were made for each of the six directions (flexion, extension, lateral flexions, protraction and retraction). The software of the Multi Cervical Rehabilitation Unit automatically recorded and calculated the maximum AROM and isometric strength.
Results: There was a good to high level of reliability in the measurement of AROM for both groups of subjects, with intraclass correlation coefficients (ICCs) ranging from 0.81 to 0.96. Results also demonstrated very good to excellent reliability in isometric strength measurement (ICCs ranged from 0.92 to 0.99). Moreover, there was a significant difference in isometric neck muscle strength (p = 0.001) and in AROM (p = 0.034) between the two groups.
Conclusions: The Multi Cervical Rehabilitation Unit was found to be reliable and valid for testing the cervical active range of motion and isometric neck muscle strength for both normal and patient subjects.

Benchmarking publishing activity of U.S. colleges and universities across the leading journals : a grouped evaluation

This paper examines institutional research productivity across two sets of journals – Hult et al. (1997) and Polonsky and Whitelaw (2006), based on institutions' Carnegie Foundation Classification, funding type (private or public) and whether institutions' offer PhD's in marketing. It is identified that while Research Intensive institutions publish more than other types of organizations across journal types, the type of funding and offering PhD's in Marketing are equally if not more important. As such, institutions seeking to compare themselves to other institutions should choose a 'similar' set of institutions on which to base any comparisons.

Anticipating policy and logics of practice : Australian institutional and academic responses to the globalising "quality research" agenda

Research assessment is now an international trend. This article mobilises a critical policy sociology informed by Bourdieu to unpack the differential effects of research policy shifts in Australia on universities, academics and the field of educational research. It argues in anticipating policy moves - from surveying the logics of practice that have emerged elsewhere from research assessment - that institutional, individual and field responses, while specific to the Australian policy context and mix, have assumed a logic of practice counter productive to "quality" research, education as a field, and equity.

Evaluation of a large group method in an educational Institution : the World café versus Large Group facilitation.

This article reports on the use of a large group method, theWorld Café, compared to large group facilitation in an educational institutional setting in Oman. Change is central to many educational institutions as they aspire to ensure quality procedures and processes are implemented and maintained. One of the difficulties of introducing change within this context is the number of stakeholders involved, as well as the inherent hierarchical nature of these settings. Large group methods (LGM) are one such intervention for change that seeks to involve a maximum number of stakeholders and minimise hierarchies, while at the same time encouraging participation and creating engagement. To evaluate LGM within this setting, the study compared two different workshop techniques: one workshop was run using the LGM theWorld Café, while the other was conducted using large group facilitation (LGF). Results indicated that theWorld Café was superior to LGF in terms of increases in participant knowledge and understanding. Participants who took part in the LGM were also significantly more likely to indicate that the technique was beneficial to their learning, compared to those who took part in the LGF. Qualitative data in the form of comments also provide support for LGM. Further research is needed to assess the applicability of the findings in educational settings in other countries. Moreover, more stringent research is required to assess over time, changes in behaviours occurring following LGM, to provide further evidence as to their value in facilitating change in institutional settings.

Assessing the efficacy and standardization potential of five competing venture capital investment evaluation approaches

The research objective behind this article was to perform a critical evaluation and comparison of five representative business plan evaluation aids (BPEAs) to facilitate constructive discussion of the proposition that greater standardization of venture capital decision-making might be both desirable and possible. The five BPEAs were systematically compared using a structured, taxonomic process. The evidence of this investigation suggests a clear superiority of BPEAs that are based on the researched attributes of successful ventures and use actuarial modeling. Discussion centered on the importance of using BPEAs in a quest for greater consistency during venture capital investment decision-making.

Evaluation of the effectiveness of marine protected areas in temperate waters of Australasia

An evaluation of the effectiveness of Marine Protected Areas (MPAs) in temperate waters of Australasia has been conducted for this thesis. The aim was to identify key elements needed in a strategy for establishment and management of MPA’s in temperate waters of Australasia. This aim was achieved by assessing how effective a sample of MPA’s has been in meeting the conservation objectives for their establishment and by identifying factors that have contributed to success or failure of the MPA’s in meeting these objectives. Particular attention was paid to the objectives of ecological sustainability and biodiversity preservation. A MPA for the purposes of this research was defined as an area of coastal or marine environment, with a substantial subtidal component, set aside by law primarily for conservation purposes. The study region encompassed the coastal zones of Victoria, Tasmania and South Australia (Australia) and New Zealand. The questions posed in order to address the aim of this thesis were; a) Have existing MPA’s been effective in achieving conservation objectives? b) What have been the important influences on effectiveness of existing MPA’s? c) What are the key elements required for implementation of effective MPA’s? The thesis is divided into three parts. Part I is a review of the literature on implementation and effectiveness of MPA’s. Part II presents a detailed evaluation of one MPA: Point Lonsdale Marine Reserve (PLMR), Victoria, Australia. Part in is an evaluation of a number of Australasian temperate MPA’s based on information provided in a survey of people involved in management of MPA’s, and from a variety of published and unpublished documents. The MPA’s are described, evidence about ecological effectiveness is presented and factors that have enhanced and limited the ability of these MPA’s to achieve conservation objectives are derived. A substantial amount of scientific evidence was found for increases in abundance, mean size and size range of fish and invertebrate populations within the boundaries of ‘no-take’ MPA’s, Some evidence was found for ‘spillover’ of adults and juveniles into adjacent fishing grounds. Ecological effects detected within ‘no-take’ MPA’s in Australasia matched those described in the literature. The abundance and mean size of a number of previously exploited species have increased, migration into adjacent fishing grounds has been documented, and species richness has increased in at least one MPA. The PLMR was established primarily to protect the scientifically significant intertidal rock platforms. The results of the case study suggest that this objective has been achieved. Opinions about effectiveness were obtained for 28 MPA’s. Of these 19 were considered to be achieving some objectives and 10 were considered to be performing well in terms of overall ecological effectiveness. Positive effects on biodiversity were generally assumed as a result of reduction of damaging anthropogenic effects on habitat. Many questionnaire respondents noted an increase in community awareness about and support for marine conservation as a result of proclamation of MPA’s, Overall, the results support the value of MPA’s for sustainability of fish stocks and preservation of biodiversity, but there is substantial doubt over whether some of the MPA’s are too small to maintain benefits in the long-term. ‘No-take’ MPA’s, particularly those more remote from the impacts of human activities, have been the most effective in achieving objectives. A number of interacting factors important to eventual success of MPA’s were identified. The most important enhancing factors identified for the PLMR were physical attributes that limit the extent of human use and a strong conservation ethic amongst many of the visitors to this marine reserve. Limiting factors were far more numerous. Of most concern is the inadequacy of at-site management. Almost half of visitors to the area were not aware of its marine reserve status, despite the fact many were frequent visitors. The need for better educational and interpretative material on-site is highlighted by the results of the PLMR visitor survey. A total of 56 factors that have enhanced effectiveness and 46 factors that have limited effectiveness of Australasian temperate MPA’s were identified. A number of factors were important in more than one MPA and this was used to derive a set of critical, or key, factors. For example, a conclusion of all three approaches used in this study is that failure to implement day-to-day management through lack of resources was a major constraint on effectiveness. The importance of MPA’s to marine conservation depends, in part, on how well they are managed. The key factors that influence MPA effectiveness were used as the basis for derivation of the main requirements for implementation of MPA’s that will be capable of meeting the objectives for their establishment. The most important needs are: • that ‘no-take’ areas surrounded by buffer zones form the basis for a system of MPA’s; • that a high level of protection is bestowed by legislation and regulations; • that a minimum size be set for the ‘no-take’ core areas; • that the selection of sites for MPA’s takes into account land-based impacts; • that institutional arrangements are developed specifically for MPA’s; • that funding for MPA’s is increased to enable effective management; • that day-to-day management is implemented in all MPA’s, with enforcement and education programs as priority areas; • that a monitoring program for one or more MPA in each 'State' is established to provide evidence of ecological effects of reservation; • that public and stakeholder involvement in development and implementation of MPA’s is encouraged as this will influence the degree of public support and compliance; • that community-based programs to educate the general public, stakeholders, the media and decision-makers about the value of MPA’s are essential; and • that measures to reduce financial impact on affected stakeholders be implemented.

Developing a scholarly approach to the evaluation of assessment practices

Purpose – The purpose of this paper is to examine and reflect on current assessment practice in a large undergraduate accounting programme delivered both in Australia and offshore, from the perspective of academics in their first semester at a “new-to-them” university.

Design/methodology/approach – The changing higher education environment and the reality of assessment in the current context are considered, as they raise a number of important issues around assessment practice. Some of the often cited literature linking teaching, learning and assessment, including student-centred learning and Confucian heritage culture, is also discussed. A reflective approach is used where Säljö's five categories of student learning are used as the basis for informed reflection of the assessment used in the “new” academics' first semester at the university. The use of empirical evidence to test these reflections would be the next step in this scholarly approach to teaching and learning.

Findings – The reflections reveal a disparity between reality and the ideal in relation to assessment practice. Issues regarding timely feedback to students and timing of assessments can result in summative assessment when it has the potential to be formative. This paper has provided an opportunity for “new” academics to engage with the higher education literature early in their careers.

Originality/value – This paper is a resource for academics beginning to engage with the higher education literature around assessment, teaching and learning and can also be used to inform and improve the teaching and learning practices of many academics in higher education.