Beyond satisfaction surveys: the development of an evaluation process for a postgraduate transferable skills program

Leadership, professional and other transferable skills are embedded in the expected attributes of Australian research postgraduates at the successful completion of their degrees. This paper reports on the development of an evaluation process for a postgraduate transferable skills program at The University of Melbourne, Australia. Existing and emerging evaluation practices and processes are examined in light of the literature on what constitutes 'good' evaluation of the type of program under consideration. The development of the process to go beyond the commonly used participant satisfaction surveys and to improve evaluation practices is described in detail. The results of the evaluation to date are provided and discussed in terms of their usefulness in incorporating particular improvements to the program. The implications for the evaluation of other programs of this type are considered.

A development and evaluation process for mHealth interventions: examples from New Zealand

The authors established a process for the development and testing of mobile phone-based health interventions that has been implemented in several mHealth interventions developed in New Zealand. This process involves a series of steps: conceptualization, formative research to inform the development, pretesting content, pilot study, pragmatic randomized controlled trial, and further qualitative research to inform improvement or implementation. Several themes underlie the entire process, including the integrity of the underlying behavior change theory, allowing for improvements on the basis of participant feedback, and a focus on implementation from the start. The strengths of this process are the involvement of the target audience in the development stages and the use of rigorous research methods to determine effectiveness. The limitations include the time required and potentially a less formalized and randomized approach than some other processes. This article aims to describe the steps and themes in the mHealth development process, using the examples of a mobile phone video messaging smoking cessation intervention and a mobile phone multimedia messaging depression prevention intervention, to stimulate discussion on these and other potential methods.

Process evaluation design and tools used in a kindergarten-based, family-involved intervention to prevent obesity in early childhood. The ToyBox-study

Process evaluation (PE) is used for the in-depth evaluation of the implementation process of health promotion programmes. The aim of the current paper was to present the PE design and tools used in the ToyBox-intervention. The PE design was based on a three-step approach, including the identification of ToyBox-specific PE elements (step 1), the development of PE tools and harmonization of procedures (step 2), and the implementation of PE using standardized protocol and tools across the intervention countries (step 3). Specifically, to evaluate the implementation of the intervention, teachers' monthly logbooks were recorded (dose delivered, fidelity, dose received); post-intervention questionnaires were completed by parents/caregivers and teachers (dose received); participation and attrition rates were recorded (recruitment, reach); and audit questionnaires and retrospective information on weather conditions were collected (physical and social environment within which the intervention was implemented). Regarding the teachers' training sessions, the researchers who performed the trainings completed evaluation forms and documented teachers' attendance after each training (dose delivered, fidelity, dose received) and teachers completed evaluation forms after each training (dose received). The PE performed in the ToyBox-intervention may contribute in the evaluation of its effectiveness, guide the revision of the intervention material and provide insights for future health promotion programmes and public health policy.

Evaluation methodology for work integrated learning - placements : a discussion paper

Background: This paper will address the evaluation of WIL (placements) in order to provide a strategy to improve performance in universities' WIL as benchmarked in the AUSSE, GDS and placement unit feedback.
Overview of issue: Although WIL placements are important and valuable for student engagement, learning, graduate employability and industry partnerships, there are few empirical studies or reviews that inform evaluation methodology for them. The assessment of placement outcomes and the student experience is typically more complex than evaluation of a standard university unit because of the wide variation that occurs with placements. Students are likely to be working in different organisations, and working on different projects within their various disciplines. Adding to this complexity, the organisation supervisor is an additional stakeholder critical to the placement experience, and who ultimately makes the judgment of student performance. Although an organisation supervisor may complete an individual feedback form for their students, generally this information is not strategically aligned within a broader university evaluation process. Initial examination of available literature revealed that the multi-dimensional perspective (organisational supervisor, placement co-ordinator and student) is not usually incorporated into evaluation of placements to inform continuous improvement for example. There are gaps in the evaluation process which could be addressed through more  comprehensive evaluation that could be utilised across Faculties and Institutions. In 2010, we will conduct an inter-faculty project to develop and trial an evaluation methodology for WIL placements. It will incorporate a triangulated approach including student, organisation and university supervisor feedback. Preliminary results gained from this project will be discussed.
Argument: An evaluation methodology that is inclusive of a triangulated approach, would provide university stakeholders with comprehensive feedback that could be used to strategically inform continuous improvement efforts in Institutions in the areas of WIL placements.
Implications: The importance of triangulated data gathering leading to a comprehensive evaluation and subsequent improvement strategies will be concluded. Suggested evaluation process and potential tools will be presented.

Transforming an academic library's spaces : an evaluation study of Deakin University Library at Burwood campus using TEALS discovery

Libraries worldwide are transforming their spaces to better align with the changing needs of their communities. The aim of this paper is to outline the process and outcome of an evaluation study of transformed academic library spaces at the Melbourne Burwood Campus using TEALS. In light of changing higher education practices and students learning preferences, Deakin University has been questioning the balance of informal learning spaces and more formal teaching and academic spaces across its campuses. Commissioned by Deakin University Library, TEALS (Tool for the Evaluation of Academic Library Spaces) was developed to evaluate academic library spaces. The Melbourne Burwood Campus library has undergone several phases of refurbishment to create a library environment that is centred around students’ needs and that supports their individual and group learning experiences. In addition, areas of the library yet to be improved will undergo a major redevelopment over the next year. Given this, carrying out an evaluation of the current spaces is timely to ensure that a better understanding of the impact of changes is achieved. The evaluation process involved: a review of architectural plans and space briefing documents; an observational study of spaces; focus groups with students and library staff; and an online survey of Students’ Library Experience. Use of the TEALS space evaluation tool along with an analysis of data collected during the evaluation process have provided significant insights into various dimensions of the quality of new library spaces. The areas of weakness and strength identified in the study will inform the next phase of Deakin University Library space redevelopment.

A university-wide ePortfolio initiative at Federation University Australia: software analysis, test-to-production, and evaluation phases

This paper describes an ePortfolio implementation strategy at Federation University Australia, Victoria (formerly the University of Ballarat). The authors combined a personal and practical viewpoint to elicit pitfalls, challenges, and recommendations for improvement. The paper is divided into three main areas in order to outline the experiments that occurred. The first section provides a standard literature review around ePortfolio adoption as well as a research-based analysis of available ePortfolio software at Australian universities. The second part depicts the University’s ePortfolio implementation strategy that focused on “test-to-production” and technology dissemination phases. This section is based on the authors’ personal viewpoint of ePortfolio adoption at a university where a “top-down management decision making model” (Slade, Murfin, & Readman, 2013, p. 178) was used. Third, the evaluation strategy is reported, which was based on similar research conducted at Australian universities (Hallam & Creagh, 2010; Hallam, Harper, Hauville, Creagh, & McAllister, 2009). This part is offered as a modest-scoped, mixed methods evaluation process. The paper extends on ePortfolio implementation strategies (Bell & White, 2013; Coffey & Ashford-Rowe, 2014; Jarrott & Gambrel, 2011; Lambert & Corrin, 2007; Ring & Ramirez, 2012; Slade et al., 2013) and software analysis (ACODE, 2011; Slade et al., 2013). Recommendations are made for the careful integration of pre- and post-rollout of ePortfolio programs with face-to-face ePortfolio tutor support, offering online resources and alternative portfolio-making options for students with poor broadband access.

A machine-driven process for human limb length estimation using inertial sensors

The computer based human motion tracking systems are widely used in medicine and sports. The accurate determination of limb lengths is crucial for not only constructing the limb motion trajectories which are used for evaluation process of human kinematics, but also individually recognising human beings. Yet, as the common practice, the limb lengths are measured manually which is inconvenient, time-consuming and requires professional knowledge. In this paper, the estimation process of limb lengths is automated with a novel algorithm calculating curvature using the measurements from inertial sensors. The proposed algorithm was validated with computer simulations and experiments conducted with four healthy subjects. The experiment results show the significantly low root mean squared error percentages such as upper arm - 5.16%, upper limbs - 5.09%, upper leg - 2.56% and lower extremities - 6.64% compared to measured lengths.

Investigating sample bias in student evaluations of teaching

Student evaluations of teaching have increased in importance to universities in Australia over recent years due to changes in government policy. There has been significant debate in the literature as to the validity and usefulness of such evaluations and as to whether students who respond to the evaluations are indeed representative of the student population. A potential invalidating issue is self selection in the evaluation process. In this paper, we consider student evaluations of a large first year business statistics subject that had 1073 eligible students enrolled across four campuses at the time of the evaluation. The study is based on the 373 students (34.8%) who responded to the survey, and their final results. The evaluations were open for a period of six weeks leading up to and just after the final exam. The study looks in detail at the student population identifying such attributes as gender; home campus; course of study; domestic/international; Commonwealth Supported Place/full fee paying, etc. and then mapping these results to those of the students who responded to the survey.

Introducing active support in the United States : staff training developments and research directions

Aim: Active Support, an evidence-based intervention developed in the UK and tested in Australia, enables direct support professionals to engage people with high support needs in meaningful and rewarding every day activities. This paper outlines the practical details of adapting the Active Support curriculum for service providers in the US and outlines an evaluation methodology that will inform both service delivery and future developments of Active Support.

Method: Active Support curricula developed in the UK and Australia were adapted for implementation the US.Train-the-trainer activities were conducted in early 2008 in three group homes serving 12 people with intellectual or developmental disabilities. A formative evaluation process was used to identify changes in the materials and training design needed to support a randomized controlled trial of the intervention in 20 small group homes.

Results: Modifications in the delivery of training are needed to account for differences in scheduling patterns and high turnover rates in US group homes, and to adjust British/Australian terminology to the U.S. context.

Conclusions: The translation of active support into the U.S. context requires modifications to support successful implementation. Research is planned to document the success of the modified curriculum in improving outcomes for persons with disabilities.

Developing a web-based information resource for palliative care : an action-research inspired approach

Background : General Practitioners and community nurses rely on easily accessible, evidence-based online information to guide practice. To date, the methods that underpin the scoping of user-identified online information needs in palliative care have remained under-explored. This paper describes the benefits and challenges of a collaborative approach involving users and experts that informed the first stage of the development of a palliative care website.

Method : The action research-inspired methodology included a panel assessment of an existing palliative care website based in Victoria, Australia; a pre-development survey (n = 197) scoping potential audiences and palliative care information needs; working parties conducting a needs analysis about necessary information content for a redeveloped website targeting health professionals and caregivers/patients; an iterative evaluation process involving users and experts; as well as a final evaluation survey (n = 166).

Results : Involving users in the identification of content and links for a palliative care website is time-consuming and requires initial resources, strong networking skills and commitment. However, user participation provided crucial information that led to the widened the scope of the website audience and guided the development and testing of the website. The needs analysis underpinning the project suggests that palliative care peak bodies need to address three distinct audiences (clinicians, allied health professionals as well as patients and their caregivers).

Conclusion : Web developers should pay close attention to the content, language, and accessibility needs of these groups. Given the substantial cost associated with the maintenance of authoritative health information sites, the paper proposes a more collaborative development in which users can be engaged in the definition of content to ensure relevance and responsiveness, and to eliminate unnecessary detail. Access to volunteer networks forms an integral part of such an approach.

The benefit : a well-written entrepreneurial business plan is to an entrepreneur what a midwife is to an expecting mother

The evaluation of new ventures often involves two key aspects of entrepreneurial business plans: how best to write them and how best to rate (evaluate) them. Ultimately the performance of the venture should be the definitive criterion of quality. Surprisingly, the writing, rating and performance effects of entrepreneurial business plans (EBPs) comprise three related but under researched areas. This article empirically tested principles for writing and rating entrepreneurial business plans to draw inferences on how to improve the private equity investment evaluation process. A simplified perspective of General Systems Theory guided our empirical investigation of the input and outcome of the VC investment decision. Our empirical investigation reveals that entrepreneurial business plans that comport with the writing principles from the literature improve a new venture's likelihood of success.

Is the quality of entrepreneurial business plans related to the outcome of a new venture?

The evaluation of new ventures often involves two key aspects of entrepreneurial business plans: how best to write them and how best to rate (evaluate) them. Ultimately the performance of the venture should be the definitive criterion of quality. Surprisingly, the writing, rating and performance effects of entrepreneurial business plans (EBPs) comprise three related but under researched areas. This paper empirically tested principles for writing and rating entrepreneurial business plans to draw inferences on how to improve the private equity investment evaluation process. A simplified perspective of General Systems Theory guided our empirical investigation of the input and outcome of the VC investment decision. Our empirical investigation reveals that entrepreneurial business plans that comport with the writing principles from the literature improve a new venture’s likelihood of success.

Assessing sustainability : a technical fix or a means of social learning?

The contextual nature of sustainability makes it extremely difficult to measure. Across the world each region has a specific environment and history that has shaped values and perceptions of the local community. In response, a wide range of tools has been developed that employ differing techniques to measure sustainability. These have a range of applications from global to site-specific scales. However, it is yet to be resolved whether assessments made using these tools reflect a technical focus with no close links to the knowledge and perceptions of stakeholders, community and management decisions, or whether the assessments reflect the knowledge and perceptions of local stakeholders and the community. In the southwest region of Victoria, Australia, a sustainability index called AIRS (An Index of Regional Sustainability) has recently been developed. This tool is based on indicators selected by stakeholders and considers relationships between the indicators. The aim of this paper is to report an ex-post evaluation of the AIRS sustainability assessment conducted at a subcatchment scale. The evaluation assesses AIRS's ability to assess regional sustainability and compares and contrasts the subcatchment assessments with the knowledge and perceptions of stakeholders and the community. A participatory approach that acknowledges key stakeholders was used for the evaluation process. Representatives from four stakeholder groups were interviewed to elicit their views of the AIRS assessment, its relevance, quality, applicability and priorities for future development. Strengths and weaknesses of AIRS are revealed and its contribution to social learning is highlighted.

Understanding women's experience of sexual desire to inform scale development

Interviews with women regarding their experiences of sexual desire informed the development of the Female Sexual Desire Questionnaire. A validation sample of 741 women completed this questionnaire, with six interrelated factors found. These components of sexual desire were labelled: Dyadic Desire, Solitary Desire, Resistance, Positive Relationship, Sexual Self-image, and Concern. The portfolio concludes that a standard evaluation process is needed to assess the effectiveness of men's behaviour change groups (MBCGs), and that such programs should comprise only one component of a comprehensive multifaceted intervention strategy against intimate partner violence (IPV). Four case studies are presented.

Proteção dos recursos hídricos de areas urbanas em crescimento : avaliação multi-critérios em sistemas de informações geográficas

Urbanization is one of the greatest manifestations of human activities. Nowadays, world’s explosive urbanization and the problems connected with it pose an important social question. Unbridled urbanization has an expressive tendency to cause dramatic problems, especially on the water resources in terms of quantitative and qualitative changes. Sheltered in a policy delimited in master plan, the planning of the city must incorporate – according to local realities and its particularities –appropriate patterns of landscape in order to achieve sustainable development. The current article aims to establish which zones in Porto Alegre city are more suitable for urbanization. It uses a Multi Criteria Evaluation process in a geographical information systems environment, taking into account human and natural factors and the current Master Plan for the city, with a focus on water resources.