A university-wide ePortfolio initiative at Federation University Australia: software analysis, test-to-production, and evaluation phases

This paper describes an ePortfolio implementation strategy at Federation University Australia, Victoria (formerly the University of Ballarat). The authors combined a personal and practical viewpoint to elicit pitfalls, challenges, and recommendations for improvement. The paper is divided into three main areas in order to outline the experiments that occurred. The first section provides a standard literature review around ePortfolio adoption as well as a research-based analysis of available ePortfolio software at Australian universities. The second part depicts the University’s ePortfolio implementation strategy that focused on “test-to-production” and technology dissemination phases. This section is based on the authors’ personal viewpoint of ePortfolio adoption at a university where a “top-down management decision making model” (Slade, Murfin, & Readman, 2013, p. 178) was used. Third, the evaluation strategy is reported, which was based on similar research conducted at Australian universities (Hallam & Creagh, 2010; Hallam, Harper, Hauville, Creagh, & McAllister, 2009). This part is offered as a modest-scoped, mixed methods evaluation process. The paper extends on ePortfolio implementation strategies (Bell & White, 2013; Coffey & Ashford-Rowe, 2014; Jarrott & Gambrel, 2011; Lambert & Corrin, 2007; Ring & Ramirez, 2012; Slade et al., 2013) and software analysis (ACODE, 2011; Slade et al., 2013). Recommendations are made for the careful integration of pre- and post-rollout of ePortfolio programs with face-to-face ePortfolio tutor support, offering online resources and alternative portfolio-making options for students with poor broadband access.

The role of evaluation in decision-making in the management of health services

The management of health services is a complex administrative practice due to the breadth of the field of health and the need to reconcile individual, corporate and collective interests that are not always convergent. In this context, the evaluation needs to have specific characteristics in order to fulfill its role. The scope of this study was to establish the characteristics that the evaluation for the management of health services should have to contribute to decision-making. Usefulness, opportunity, feasibility, reliability, objectivity and directionality represent the set of principles upon which the evaluation should be based. Evaluations should lead to decisions that guarantee not only their efficiency and effectiveness but also their implementation. The evaluation process should ensure that decisions involve all stakeholders in order to render the implementation of decisions feasible, and take into account the health needs of the population and the goals set for the services. The scope of this article is to elicit a debate among different stakeholders in the evaluation in the hope that it can contribute to the reflection on the real usefulness of evaluations in which the political component in management has been increasingly prevalent.

Use of empowerment evaluation to determine a sustainability plan for a community-based preconception care program

Women With IMPACT (WWI) is a community-based preconception care educational intervention. WWI is being implemented by the Impacting Maternal and Prenatal Care Together (IMPACT) Collaborative and targets zip codes in Harris County, Texas at high risk for infant mortality, low birthweight, and preterm birth. WWI started March 2012 and continues through August 2013. Three workshop series are planned. This study was conducted with participants and facilitators from the first workshop series. This study aimed to 1) evaluate the WWI program using empowerment evaluation, 2) engage all WWI stakeholders in an empowerment evaluation so the method could be adopted as a participatory evaluation process for future IMPACT activities, and 3) develop recommendations for sustainability of the WWI intervention, based on empowerment evaluation findings and results from the pre/post program evaluation completed by WWI participants. Study participants included WWI participants and facilitators and IMPACT Collaborative Steering Committee members. WWI participants were female, 18-35 year-old, non-pregnant residents of zip codes at high risk of adverse birth outcomes. All other study participants were 18 years or older. A two-phased empowerment evaluation (EE) was utilized in this study. Sessions 1-4 were conducted independently of one another – 3 with participants at different sites and one with the facilitators. The fifth session included WWI participant and facilitator representatives, and IMPACT Steering Committee members. Session 5 built upon the work of the other sessions. Observation notes were recorded during each session. Thematic content analysis was conducted on all EE tables and observation notes. Mission statements drafted by each group focused on improvement of physical and mental health through behavior change and empowerment of all participants. The top 5 overall program components were: physical activity, nutrition, self-worth, in-class communication, and stress. Goals for program improvement were set by EE participants for each of these components. Through thematic content analysis of the tables and observation notes, social support emerged as an important theme of the program among all participant groups. Change to a healthy lifestyle emerged as an important theme in terms of program improvement. Two-phased EE provided an opportunity for all program stakeholders to provide feedback regarding important program components and provide suggestions for program improvement. EE, thematic content analysis, pre/post evaluation results, and inherent program knowledge were triangulated to make recommendations to sustain the program once the initial funding ends. ^

Replication Data Management,needs and solutions: an initial evaluation of conceptual approaches for integrating heterogeneous replication study data

Replication Data Management (RDM) aims at enabling the use of data collections from several iterations of an experiment. However, there are several major challenges to RDM from integrating data models and data from empirical study infrastructures that were not designed to cooperate, e.g., data model variation of local data sources. [Objective] In this paper we analyze RDM needs and evaluate conceptual RDM approaches to support replication researchers. [Method] We adapted the ATAM evaluation process to (a) analyze RDM use cases and needs of empirical replication study research groups and (b) compare three conceptual approaches to address these RDM needs: central data repositories with a fixed data model, heterogeneous local repositories, and an empirical ecosystem. [Results] While the central and local approaches have major issues that are hard to resolve in practice, the empirical ecosystem allows bridging current gaps in RDM from heterogeneous data sources. [Conclusions] The empirical ecosystem approach should be explored in diverse empirical environments.

Federating queries in SPARQL 1.1: syntax, semantics and evaluation

Given the sustained growth that we are experiencing in the number of SPARQL endpoints available, the need to be able to send federated SPARQL queries across these has also grown. To address this use case, the W3C SPARQL working group is defining a federation extension for SPARQL 1.1 which allows for combining graph patterns that can be evaluated over several endpoints within a single query. In this paper, we describe the syntax of that extension and formalize its semantics. Additionally, we describe how a query evaluation system can be implemented for that federation extension, describing some static optimization techniques and reusing a query engine used for data-intensive science, so as to deal with large amounts of intermediate and final results. Finally we carry out a series of experiments that show that our optimizations speed up the federated query evaluation process.

A web-based application for the management and evaluation of tutoring requests in PBL-based massive laboratories

One important steps in a successful project-based-learning methodology (PBL) is the process of providing the students with a convenient feedback that allows them to keep on developing their projects or to improve them. However, this task is more difficult in massive courses, especially when the project deadline is close. Besides, the continuous evaluation methodology makes necessary to find ways to objectively and continuously measure students' performance without increasing excessively instructors' work load. In order to alleviate these problems, we have developed a web service that allows students to request personal tutoring assistance during the laboratory sessions by specifying the kind of problem they have and the person who could help them to solve it. This service provides tools for the staff to manage the laboratory, for performing continuous evaluation for all students and for the student collaborators, and to prioritize tutoring according to the progress of the student's project. Additionally, the application provides objective metrics which can be used at the end of the subject during the evaluation process in order to support some students' final scores. Different usability statistics and the results of a subjective evaluation with more than 330 students confirm the success of the proposed application.

Energy efficiency evaluation of zero energy houses

Given the global energy and environmental situation, the European Union has been issuing directives with increasingly demanding requirements in term of the energy efficiency in buildings. The international competition of sustainable houses, Solar Decathlon Europe (SDE), is aligned with these European objectives. SDE houses are low energy solar buildings that must reach the near to zero energy houses’ goal. In the 2012 edition, in order to emphasize its significance, the Energy Efficiency Contest was added. SDE houses’ interior comfort, functioning and energy performance is monitored. The monitoring data can give an idea about the efficiency of the houses. However, a jury comprised by international experts is responsible for carrying out the houses energy efficiency evaluation. Passive strategies and houses services are analyzed. Additionally, the jury's assessment has been compared with the behavior of the houses during the monitoring period. Comparative studies make emphasis on the energy aspects, houses functioning and their interior comfort. Conclusions include thoughts related with the evaluation process, the results of the comparative studies and suggestions for the next competitions.

Power generation investment opportunities evaluation: A comparison between net present value and real options approach

To evaluate an investment project in the competitive electricity market, there are several key factors that affects the project's value: the present value that the project could bring to investor, the possible future course of actions that investor has and the project's management flexibility. The traditional net present value (NPV) criteria has the ability to capture the present value of the project's future cash flow, but it fails to assess the value brought by market uncertainty and management flexibility. By contrast with NPV, the real options approach (ROA) method has the advantage to combining the uncertainty and flexibility in evaluation process. In this paper, a framework for using ROA to evaluate the generation investment opportunity has been proposed. By given a detailed case study, the proposed framework is compared with NPV and showing a different results

Understanding consumer evaluation criteria for repurchase in a performing arts context

The purpose of this paper is to examine consumers' experience of a performing arts service to identify the predictors of audience behaviour especially as related to positive repurchase intention. Experiential service settings such as the performing arts have been cited in recent research as service contexts that may challenge current theory that repurchase intention is driven by service quality and customer satisfaction. It is posited that consumer emotions and the hedonic nature of the consumption experience may complicate the evaluation process to repurchase intention in a setting such as the performing arts. Qualitative semi-structured indepth interviews were undertaken of twenty-six performing arts consumers using a pool of questions and prompts developed from a review of the extant literature. Transcribed field notes were examined for key words and phrases and data was divided into the main emergent themes related to each of the questions and also coded for confirmation and is-confirmation of the extant literature constructs and relationships. The dimensions of service experience,price, service quality, target goal-directed emotions and non-target appraisal emotions were identified as driving repurchase intention in a performing arts setting. Customer satisfaction in this setting appears to result from emotional factors rather than expectancy dis-confirmation. This research supports the notion that an experiential consumption experience such as the performing arts will challenge the current theory of the drivers of repurchase intention and suggests that a more thorough large scale examination of these dimensions in this service setting is warranted.

The Application of Geographic Information Systems to climate change & land evaluation : a study of the effects of climate change upon crop productivity in a sub-tropical environment

This research investigates the contribution that Geographic Information Systems (GIS) can make to the land suitability process used to determine the effects of a climate change scenario. The research is intended to redress the severe under representation of Developing countries within the literature examining the impacts of climatic change upon crop productivity. The methodology adopts some of the Intergovernmental Panel on Climate Change (IPCC) estimates for regional climate variations, based upon General Circulation Model predictions (GCMs) and applies them to a baseline climate for Bangladesh. Utilising the United Nations Food & Agricultural Organisation's Agro-ecological Zones land suitability methodology and crop yield model, the effects of the scenario upon agricultural productivity on 14 crops are determined. A Geographic Information System (IDRISI) is adopted in order to facilitate the methodology, in conjunction with a specially designed spreadsheet, used to determine the yield and suitability rating for each crop. A simple optimisation routine using the GIS is incorporated to provide an indication of the 'maximum theoretical' yield available to the country, should the most calorifically significant crops be cultivated on each land unit both before and after the climate change scenario. This routine will provide an estimate of the theoretical population supporting capacity of the country, both now and in the future, to assist with planning strategies and research. The research evaluates the utility of this alternative GIS based methodology for the land evaluation process and determines the relative changes in crop yields that may result from changes in temperature, photosynthesis and flooding hazard frequency. In summary, the combination of a GIS and a spreadsheet was successful, the yield prediction model indicates that the application of the climate change scenario will have a deleterious effect upon the yields of the study crops. Any yield reductions will have severe implications for agricultural practices. The optimisation routine suggests that the 'theoretical maximum' population supporting capacity is well in excess of current and future population figures. If this agricultural potential could be realised however, it may provide some amelioration from the effects of climate change.

Computer automation of land evaluation procedures

This study was concerned with the computer automation of land evaluation. This is a broad subject with many issues to be resolved, so the study concentrated on three key problems: knowledge based programming; the integration of spatial information from remote sensing and other sources; and the inclusion of socio-economic information into the land evaluation analysis. Land evaluation and land use planning were considered in the context of overseas projects in the developing world. Knowledge based systems were found to provide significant advantages over conventional programming techniques for some aspects of the land evaluation process. Declarative languages, in particular Prolog, were ideally suited to integration of social information which changes with every situation. Rule-based expert system shells were also found to be suitable for this role, including knowledge acquisition at the interview stage. All the expert system shells examined suffered from very limited constraints to problem size, but new products now overcome this. Inductive expert system shells were useful as a guide to knowledge gaps and possible relationships, but the number of examples required was unrealistic for typical land use planning situations. The accuracy of classified satellite imagery was significantly enhanced by integrating spatial information on soil distribution for Thailand data. Estimates of the rice producing area were substantially improved (30% change in area) by the addition of soil information. Image processing work on Mozambique showed that satellite remote sensing was a useful tool in stratifying vegetation cover at provincial level to identify key development areas, but its full utility could not be realised on typical planning projects, without treatment as part of a complete spatial information system.

Whole life performance bid evaluation in the Korean public sector

Korea has increasingly adopted design-build for public construction projects in the last few years. There is a much greater awareness of the need to change a system based on ‘Value for Money’ which is high on the government's agenda. A whole life performance bid evaluation model is proposed to aid decision makers in the selection of a design-builder. This is based on the integration of a framework using an analytic hierarchy process as the bid awarding system is being changed from one based on lowest price, to one based on best value over the life-cycle. Key criteria like whole life cost, service life planning and design quality are important through the key stages of evaluation process. The model uses a systematic and holistic approach which enables a public sector to make better decisions in design-builder selection, which will deliver whole life benefits, based on long term cost-effectiveness and whole life.

Accommodating special needs users in the evaluation of an m-Learning application:a case study

Adult illiteracy rates are alarmingly high worldwide. The portability, affordability, and ease of use of mobile (or handheld) devices offer a realistic opportunity to provide novel, context-sensitive literacy resources to adults with limited literacy skills. To this end, we developed the concept of ALEX – a mobile Adult Literacy support application for EXperiential learning (Lumsden et al., 2005). On the basis of a medium-fidelity prototype of this application, we conducted an evaluation of ALEX using participants from our in tended user group. This evaluation had two goals: (a) to assess the usefulness of the ALEX concept and the usability of its current design; and (b) to reflect on the appropriateness of our evaluation process given the literacy-related needs of our participants. This paper outlines our approach to this evaluation as well as the results we obtained and our reflections on the process.

Team Primacy Concept-based employee evaluation and job performance

This qualitative case study explored how employees learn from Team Primacy Concept (TPC)-based employee evaluation and how they apply the knowledge in their job performance. Kolb's experiential learning model (1974) served as a conceptual framework for the study to reveal the process of how employees learn from TPC evaluation, namely, how they experience, reflect, conceptualize and act on performance feedback. TPC based evaluation is a form of multirater evaluation that consists of three components: self-feedback, supervisor's feedback, and peer feedback. The distinctive characteristic of TPC based evaluation is the team evaluation component during which the employee's professional performance is discussed by one's peers in a face-to-face team setting, while other forms of multirater evaluation are usually conducted in a confidential and anonymous manner.^ Case study formed the methodological framework. The case was the Southeastern Virginia (SEVA) region of the Institute for Family Centered Services, and the participants were eight employees of the SEVA region. Findings showed that the evaluation process was anxiety producing for employees, especially the process of peer evaluation in a team setting. Preparation was found to be an important phase of TPC evaluation. Overall, the positive feedback delivered in a team setting made team members feel acknowledged. The study participants felt that honesty in providing feedback and openness to hearing challenges were significant prerequisites to the TPC evaluation process. Further, in the planning phase, employees strove to develop goals for themselves that were meaningful. Also, the catalyst for feedback implementation appeared to stem from one's accountability to self and to the client or community. Generally, the participants identified a number of performance improvement goals that they attained during their employment with IFCS, which were supported by their developmental plans.^ In conclusion, the study identified the process by which employees learned from TPC-based employee evaluation and the ways in which they used the knowledge to improve their job performance. Specifically, the study examined how participants felt and what they thought about TPC-based feedback, in what ways they reflected and made meaning of the feedback, and how they used the feedback to improve their job performance.^

New Principles in Planning Evaluation

Maintaining and enhancing living conditions in cities through a combination of physical planning and environmental management is a newly emerging focus of governments around the world. For example, local governments seek to insulate sensitive land uses such as residential areas from environmentally intrusive activities such as major transport facilities and manufacturing. Regional governments protect water quality and natural habitat by enforcing pollution controls and regulating the location of growth. Some national governments fund acquisition of strategically important sites, facilitate the renewal of brown fields, and even develop integrated environmental quality plans. This book provides recently developed and tested methods for assessing the strengths and weaknesses of planning and policy options. Several contributions focus on new substantive areas of concern in planning evaluation, including environmental justice and sustainable urban development. Applications of evaluation in several planning contexts are demonstrated, and special problems that these pose are assessed. Several chapters address how to communicate the process and results to several stakeholder groups, and how to engage these groups in the evaluation process. Each chapter employs a realworld case in practice, thus dealing with the complexity of applying planning evaluation, and providing practical advice useful in similar situations.