Evaluation framework to assess orthopaedic procedures

Introduction & aims The demand for evidence of efficacy of treatments in general and orthopaedic surgical procedures in particular is ever increasing in Australia and worldwide. The aim of this study is to share the key elements of an evaluation framework recently implemented in Australia to determine the efficacy of bone-anchored prostheses. Method The proposed evaluation framework to determine the benefit and harms of bone-anchored prostheses for individuals with limb loss was extracted from a systematic review of the literature including seminal studies focusing on clinical benefits and safety of procedures involving screw-type implant (e.g., OPRA) and press-fit fixations (e.g., EEFT, ILP, OPL). [1-64] Results The literature review highlighted that a standard and replicable evaluation framework should focus on: • The clinical benefits with a systematic recording of health-related quality of life (e.g., SF-26, Q-TFA), mobility predictor (e.g., AMPRO), ambulation abilities (e.g., TUG, 6MWT), walking abilities (e.g., characteristic spatio-temporal) and actual activity level at baseline and follow-up post Stage 2 surgery, • The potential harms with systematic recording of residuum care, infection, implant stability, implant integrity, injuries (e.g., falls) after Stage 1 surgery. There was a general consensus around the instruments to monitor most of the benefits and harms. The benefits could be assessed using a wide spectrum of complementary assessments ranging from subjective patient self-reporting to objective measurements of physical activity. However, this latter was assessed using a broad range of measurements (e.g., pedometer, load cell, energy consumption). More importantly, the lack of consistent grading of infections was sufficiently noticeable to impede cross-fixation comparisons. Clearly, a more universal grading system is needed. Conclusions Investigators are encouraged to implement an evaluation framework featuring the domains and instruments proposed above using a single database to facilitate robust prospective studies about potential benefits and harms of their procedure. This work is also a milestone in the development of national and international clinical outcome registries.

Review: "Context : A framework for its Influence on Evaluation Practice (New Directions for Evaluation No. 135)" edited by Rog, D. J.; Fitzpatrick, J. L. and Conner, R. F.

This book offers a framework for the influence of context on evaluation practice and is applied to three case studies: environmental context; indigenous context and political context; and finishes with a process for implementation.

Evaluation of texture and geometry for dimensional facial expression recognition

Facial expression recognition (FER) algorithms mainly focus on classification into a small discrete set of emotions or representation of emotions using facial action units (AUs). Dimensional representation of emotions as continuous values in an arousal-valence space is relatively less investigated. It is not fully known whether fusion of geometric and texture features will result in better dimensional representation of spontaneous emotions. Moreover, the performance of many previously proposed approaches to dimensional representation has not been evaluated thoroughly on publicly available databases. To address these limitations, this paper presents an evaluation framework for dimensional representation of spontaneous facial expressions using texture and geometric features. SIFT, Gabor and LBP features are extracted around facial fiducial points and fused with FAP distance features. The CFS algorithm is adopted for discriminative texture feature selection. Experimental results evaluated on the publicly accessible NVIE database demonstrate that fusion of texture and geometry does not lead to a much better performance than using texture alone, but does result in a significant performance improvement over geometry alone. LBP features perform the best when fused with geometric features. Distributions of arousal and valence for different emotions obtained via the feature extraction process are compared with those obtained from subjective ground truth values assigned by viewers. Predicted valence is found to have a more similar distribution to ground truth than arousal in terms of covariance or Bhattacharya distance, but it shows a greater distance between the means.

Strategic project evaluation for open pit mining ventures using real options and allied econometric techniques

Open pit mine operations are complex businesses that demand a constant assessment of risk. This is because the value of a mine project is typically influenced by many underlying economic and physical uncertainties, such as metal prices, metal grades, costs, schedules, quantities, and environmental issues, among others, which are not known with much certainty at the beginning of the project. Hence, mining projects present a considerable challenge to those involved in associated investment decisions, such as the owners of the mine and other stakeholders. In general terms, when an option exists to acquire a new or operating mining project, , the owners and stock holders of the mine project need to know the value of the mining project, which is the fundamental criterion for making final decisions about going ahead with the venture capital. However, obtaining the mine project’s value is not an easy task. The reason for this is that sophisticated valuation and mine optimisation techniques, which combine advanced theories in geostatistics, statistics, engineering, economics and finance, among others, need to be used by the mine analyst or mine planner in order to assess and quantify the existing uncertainty and, consequently, the risk involved in the project investment. Furthermore, current valuation and mine optimisation techniques do not complement each other. That is valuation techniques based on real options (RO) analysis assume an expected (constant) metal grade and ore tonnage during a specified period, while mine optimisation (MO) techniques assume expected (constant) metal prices and mining costs. These assumptions are not totally correct since both sources of uncertainty—that of the orebody (metal grade and reserves of mineral), and that about the future behaviour of metal prices and mining costs—are the ones that have great impact on the value of any mining project. Consequently, the key objective of this thesis is twofold. The first objective consists of analysing and understanding the main sources of uncertainty in an open pit mining project, such as the orebody (in situ metal grade), mining costs and metal price uncertainties, and their effect on the final project value. The second objective consists of breaking down the wall of isolation between economic valuation and mine optimisation techniques in order to generate a novel open pit mine evaluation framework called the ―Integrated Valuation / Optimisation Framework (IVOF)‖. One important characteristic of this new framework is that it incorporates the RO and MO valuation techniques into a single integrated process that quantifies and describes uncertainty and risk in a mine project evaluation process, giving a more realistic estimate of the project’s value. To achieve this, novel and advanced engineering and econometric methods are used to integrate financial and geological uncertainty into dynamic risk forecasting measures. The proposed mine valuation/optimisation technique is then applied to a real gold disseminated open pit mine deposit to estimate its value in the face of orebody, mining costs and metal price uncertainties.

Technology transfer evaluation in the high technology industry : an interdisciplinary perspective

The underlying objective of this study was to develop a novel approach to evaluate the potential for commercialisation of a new technology. More specifically, this study examined the 'ex-ante'. evaluation of the technology transfer process. For this purpose, a technology originating from the high technology sector was used. The technology relates to the application of software for the detection of weak signals from space, which is an established method of signal processing in the field of radio astronomy. This technology has the potential to be used in commercial and industrial areas other than astronomy, such as detecting water leakages in pipes. Its applicability to detecting water leakage was chosen owing to several problems with detection in the industry as well as the impact it can have on saving water in the environment. This study, therefore, will demonstrate the importance of interdisciplinary technology transfer. The study employed both technical and business evaluation methods including laboratory experiments and the Delphi technique to address the research questions. There are several findings from this study. Firstly, scientific experiments were conducted and these resulted in a proof of concept stage of the chosen technology. Secondly, validation as well as refinement of criteria from literature that can be used for „ex-ante. evaluation of technology transfer has been undertaken. Additionally, after testing the chosen technology.s overall transfer potential using the modified set of criteria, it was found that the technology is still in its early stages and will require further development for it to be commercialised. Furthermore, a final evaluation framework was developed encompassing all the criteria found to be important. This framework can help in assessing the overall readiness of the technology for transfer as well as in recommending a viable mechanism for commercialisation. On the whole, the commercial potential of the chosen technology was tested through expert opinion, thereby focusing on the impact of a new technology and the feasibility of alternate applications and potential future applications.

Using the Donabedian framework to examine the quality and safety of nursing service innovation

Aims and objectives To evaluate the safety and quality of nurse practitioner service using the audit framework of Structure,Process and Outcome. Background Health service and workforce reform are on the agenda of governments and other service providers seeking to contain healthcare costs whilst providing safe and effective health care to communities. The nurse practitioner service is one health workforce innovation that has been adopted globally to improve timely access to clinical care, but there is scant literature reporting evaluation of the quality of this service innovation. Design. A mixed-methods design within the Donabedian evaluation framework was used. Methods The Donabedian framework was used to evaluate the Structure, Process and Outcome of nurse practitioner service. A range of data collection approaches was used, including stakeholder survey (n=36), in-depth interviews (11 patients and 13 nurse practitioners) and health records data on service processes. Results The study identified that adequate and detailed preparation of Structure and Process is essential for the successful implementation of a service innovation. The multidisciplinary team was accepting of the addition of nurse practitioner service, and nurse practitioner clinical care was shown to be effective, satisfactory and safe from the perspective of the clinician stakeholders and patients. Conclusions This study demonstrated that the Donabedian framework of Structure, Process and Outcome evaluation is a valuable and validated approach to examine the safety and quality of a service innovation. Furthermore, in this study, specific Structure elements were shown to influence the quality of service processes further validating the framework and the interdependence of the Structure, Process and Outcome components. Relevance to clinical practice Understanding the structure and process requirements for establishing nursing service innovation lays the foundation for safe, effective and patient-centred clinical care.

Spatio temporal feature evaluation for action recognition

Spatio-Temporal interest points are the most popular feature representation in the field of action recognition. A variety of methods have been proposed to detect and describe local patches in video with several techniques reporting state of the art performance for action recognition. However, the reported results are obtained under different experimental settings with different datasets, making it difficult to compare the various approaches. As a result of this, we seek to comprehensively evaluate state of the art spatio- temporal features under a common evaluation framework with popular benchmark datasets (KTH, Weizmann) and more challenging datasets such as Hollywood2. The purpose of this work is to provide guidance for researchers, when selecting features for different applications with different environmental conditions. In this work we evaluate four popular descriptors (HOG, HOF, HOG/HOF, HOG3D) using a popular bag of visual features representation, and Support Vector Machines (SVM)for classification. Moreover, we provide an in-depth analysis of local feature descriptors and optimize the codebook sizes for different datasets with different descriptors. In this paper, we demonstrate that motion based features offer better performance than those that rely solely on spatial information, while features that combine both types of data are more consistent across a variety of conditions, but typically require a larger codebook for optimal performance.

Will the application of spatial multi criteria evaluation technique enhance the quality of decision-making to resolve boundary conflicts in the Philippines?

Systematic studies that evaluate the quality of decision-making processes are relatively rare. Using the literature on decision quality, this research develops a framework to assess the quality of decision-making processes for resolving boundary conflicts in the Philippines. The evaluation framework breaks down the decision-making process into three components (the decision procedure, the decision method, and the decision unit) and is applied to two ex-post (one resolved and one unresolved) and one ex-ante cases. The evaluation results from the resolved and the unresolved cases show that the choice of decision method plays a minor role in resolving boundary conflicts whereas the choice of decision procedure is more influential. In the end, a decision unit can choose a simple method to resolve the conflict. The ex-ante case presents a follow-up intended to resolve the unresolved case for a changing decision-making process in which the associated decision unit plans to apply the spatial multi criteria evaluation (SMCE) tool as a decision method. The evaluation results from the ex-ante case confirm that the SMCE has the potential to enhance the decision quality because: a) it provides high quality as a decision method in this changing process, and b) the weaknesses associated with the decision unit and the decision procedure of the unresolved case were found to be eliminated in this process.

Revealing performance heterogeneity through knowledge management maturity evaluation : a capability-based approach

The dynamic capabilities view (DCV) focuses on renewal of firms’ strategic knowledge resources so as to sustain competitive advantage within turbulent markets. Within the context of the DCV, the focus of knowledge management (KM) is to develop the KMC through deploying knowledge governance mechanisms that are conducive to facilitating knowledge processes so as to produce superior business performance over time. The essence of KM performance evaluation is to assess how well the KMC is configured with knowledge governance mechanisms and processes that enable a firm to achieve superior performance through matching its knowledge base with market needs. However, little research has been undertaken to evaluate KM performance from the DCV perspective. This study employed a survey study design and adopted hypothesis-testing approaches to develop a capability-based KM evaluation framework (CKMEF) that upholds the basic assertions of the DCV. Under the governance of the framework, a KM index (KMI) and a KM maturity model (KMMM) were derived not only to indicate the extent to which a firm’s KM implementations fulfill its strategic objectives, and to identify the evolutionary phase of its KMC, but also to bench-mark the KMC in the research population. The research design ensured that the evaluation framework and instruments have statistical significance and good generalizabilty to be applied in the research population, namely construction firms operating in the dynamic Hong Kong construction market. The study demonstrated the feasibility of quantitatively evaluating the development of the KMC and revealing the performance heterogeneity associated with the development.

Kaleidoscope : Reframing learning and teaching evaluation through a shared stakeholder lens

Sector wide interest in Reframe: QUT’s Evaluation Framework continues with a number of institutions requesting finer details as QUT embeds the new approach to evaluation across the university in 2013. This interest, both nationally and internationally has warranted QUT’s collegial response to draw upon its experiences from developing Reframe into distilling and offering Kaleidoscope back to the sector. The word Reframe is a relevant reference for QUT’s specific re-evaluation, reframing and adoption of a new approach to evaluation; whereas Kaleidoscope reflects the unique lens through which any other institution will need to view their own cultural specificity and local context through an extensive user-led stakeholder engagement approach when introducing new approaches to learning and teaching evaluation. Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sector. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own evaluation projects and maximise impact. Kaleidoscope offers an institutional case study of widespread organisational change underpinned by Reframe’s (i) evidence-based methodology; (ii) research including published environmental scan, literature review (Alderman, et al., 2012), development of a conceptual model (Alderman, et al., in press 2013), project management principles (Alderman & Melanie, 2012) and national conference peer reviews; and (iii) year-long strategic project with national outreach to collaboratively engage the development of a draft set of National Guidelines. Kaleidoscope’s aims are to inform Higher Education evaluation policy development through national stakeholder engagement, the finalisation of proposed National Guidelines. In correlation with the conference paper, the authors will present a Draft Guidelines and Framework ready for external peer review by evaluation practitioners from the Higher Education sector, as part of Kaleidoscope’s dissemination strategy (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and ongoing discussions (Shapiro, et al., 1983; Jacobs, 2000). The initial National Guidelines will be distilled from the Reframe: QUT’s Evaluation Framework’s Policy, Protocols, and incorporated Business Rules. It is intended that the outcomes of Kaleidoscope are owned by and reflect sectoral engagement, including iterative evaluation through multiple avenues of dissemination and collaboration including the Higher Education sector. The dissemination strategy with the inclusion of Illuminative Evaluation methodology provides an inclusive opportunity for other institutions and stakeholders across the Higher Education sector to give voice through the information-gathering component of evaluating the draft Guidelines, providing a comprehensive understanding of the complex realities experienced across the Higher Education sector, and thereby ‘illuminating’ both the shared and unique lenses and contexts. This process will enable any final guidelines developed to have broader applicability, greater acceptance, enhanced sustainability and additional relevance benefiting the Higher Education sector, and the adoption and adaption by any single institution for their local contexts.

Kaleidoscope : illuminating learning and teaching evaluation for a shared stakeholder lens

Within the Australian higher education sector, institutions are required to evaluate teaching, units and courses to assure the quality of the student learning experience, however with hardly any regulatory parameters guiding institutions, and with disparate practices, there are few opportunities to benchmakr across institutions or the sector. QUT has received interest and requests from national and international universities on accessing Reframe: QUT's Evaluation Framework.

Evaluation of robust autonomy and implications on UAS certification and design

As the number of potential applications of Unmanned Aircraft Systems (UAS) grows in civilian operations and national security, National Airworthiness Authorities are under increasing pressure to provide a path for certification and allow UAS integration into the national airspace. The success of this integration depends on developments in improved UAS reliability and safety, regulations for certification, and technologies for operational performance and safety assessment. This paper focusses on the latter and describes the use of a framework for evaluating robust autonomy of UAS, namely, the autonomous system’s ability to either continue operation in the presence of faults or safely shut down. The paper draws parallels between the proposed evaluation framework and the evaluation of pilots during the licensing process. It also discusses how the data from the proposed evaluation can be uses as an aid for decision making in certification and UAS designs.

Integrating understandability in the evaluation of consumer health search engines

In this paper we propose a method that integrates the no- tion of understandability, as a factor of document relevance, into the evaluation of information retrieval systems for con- sumer health search. We consider the gain-discount evaluation framework (RBP, nDCG, ERR) and propose two understandability-based variants (uRBP) of rank biased precision, characterised by an estimation of understandability based on document readability and by different models of how readability influences user understanding of document content. The proposed uRBP measures are empirically contrasted to RBP by comparing system rankings obtained with each measure. The findings suggest that considering understandability along with topicality in the evaluation of in- formation retrieval systems lead to different claims about systems effectiveness than considering topicality alone.

Validation of a framework for measuring hospital disaster resilience using factor analysis

Hospital disaster resilience can be defined as “the ability of hospitals to resist, absorb, and respond to the shock of disasters while maintaining and surging essential health services, and then to recover to its original state or adapt to a new one.” This article aims to provide a framework which can be used to comprehensively measure hospital disaster resilience. An evaluation framework for assessing hospital resilience was initially proposed through a systematic literature review and Modified-Delphi consultation. Eight key domains were identified: hospital safety, command, communication and cooperation system, disaster plan, resource stockpile, staff capability, disaster training and drills, emergency services and surge capability, and recovery and adaptation. The data for this study were collected from 41 tertiary hospitals in Shandong Province in China, using a specially designed questionnaire. Factor analysis was conducted to determine the underpinning structure of the framework. It identified a four-factor structure of hospital resilience, namely, emergency medical response capability (F1), disaster management mechanisms (F2), hospital infrastructural safety (F3), and disaster resources (F4). These factors displayed good internal consistency. The overall level of hospital disaster resilience (F) was calculated using the scoring model: F = 0.615F1 + 0.202F2 + 0.103F3 + 0.080F4. This validated framework provides a new way to operationalise the concept of hospital resilience, and it is also a foundation for the further development of the measurement instrument in future studies.