How training evaluation can be used to potentiate employee development in an organization

A Work Project, presented as part of the requirements for the Award of a Masters Degree in Management from the NOVA – School of Business and Economics

NASS investigator training - evaluation of the first training cycle. Final report.

National Highway Traffic Safety Administration, Washington, D.C.

Motorcycle operator training evaluation plan /

Cover title varies: Motorcycle rider training evaluation plan.

An Approach for a more Objective Evaluation of Practical Projects, Used in the Training Process

Well–prepared, adaptive and sustainably developing specialists are an important competitive advantage, but also one of the main challenges for businesses. One option of the education system for creation and development of staff adequate to the needs is the development of pro jects with topics from real economy ("Practical Projects"). The objective assessment is an essential driver and motivator, and is based on a system of well-chosen, well-defined and specific criteria and indicators. An approach to a more objective evaluation of practical projects is finding more objective weights of the criteria. A natural and reasonable approach is the accumulation of opinions of proven experts and subsequent bringing out the weights from the accumulated data. The preparation and conduction of a survey among recognized experts in the field of project-based learning in mathematics, informatics and information technologies is described. The processing of the data accumulated by applying AHP, allowed us to objectively determine weights of evaluation criteria and hence to achieve the desired objectiveness. ACM Computing Classification System (1998): K.3.2.

How should a vocational education and training course be evaluated?

The course evaluation process used by a large VET provider was evaluated using guidelines suggested by the course evaluation literature and feedback obtained from multiple stakeholders. A modified model is presented as an exemplar for course evaluation in the VET sector.

Measurement of affect during simulation-based training

Training designed to support and strengthen higher-order mental abilities now often involves immersion in Virtual Reality where dangerous real world scenarios can be safely replicated. However despite the growing popularity of advanced training simulations, methods for evaluating their use rely heavily on subjective measures or analysis of final outcomes. Without dynamic, objective performance measures the outcome of training in terms of impact on cognitive skills and ability to transfer newly acquired skills to the real world is unknown. The relationship between affective intensity and cognitive learning provides a potential new approach to ensure the processing of cognitions which occur prior to final outcomes, such as problem-solving and decision-making, are adequately evaluated. This paper describes the technical aspects of pilot work recently undertaken to develop a new measurement tool designed to objectively track individual affect levels during simulation-based training.

Development and evaluation of the Clinician Suicide Risk Assessment Checklist

This paper describes the development and evaluation of a new instrument - the Clinician Suicide Risk Assessment Checklist (CSRAC). The instrument assesses the clinician's competency in three areas: clinical interviewing, assessment of specific suicide risk factors, and formulating a management plan. A draft checklist was constructed by integrating information from 1) literature review 2) expert clinician focus group and 3) consultation with experts. It was utilised in a simulated clinical scenario with clinician trainees and a trained actor in order to test for inter-rater agreement. Agreement was calculated and the checklist was re-drafted with the aim of maximising agreement. A second phase of simulated clinical scenarios was then conducted and inter-rater agreement was calculated for the revised checklist. In the first phase of the study, 18 of 35 items had inadequate inter-rater agreement (60%>), while in the second phase, using the revised version, only 3 of 39 items failed to achieve adequate inter-rater agreement. Further evidence of reliability and validity are required. Continued development of the CSRAC will be necessary before it can be utilised to assess the effectiveness of risk assessment training programs.

Webinar Training: an acceptable, feasible and effective approach for multi-site medical record abstraction: the BOWII experience

Background Abstractor training is a key element in creating valid and reliable data collection procedures. The choice between in-person vs. remote or simultaneous vs. sequential abstractor training has considerable consequences for time and resource utilization. We conducted a web-based (webinar) abstractor training session to standardize training across six individual Cancer Research Network (CRN) sites for a study of breast cancer treatment effects in older women (BOWII). The goals of this manuscript are to describe the training session, its participants and participants' evaluation of webinar technology for abstraction training. Findings A webinar was held for all six sites with the primary purpose of simultaneously training staff and ensuring consistent abstraction across sites. The training session involved sequential review of over 600 data elements outlined in the coding manual in conjunction with the display of data entry fields in the study's electronic data collection system. Post-training evaluation was conducted via Survey Monkey©. Inter-rater reliability measures for abstractors within each site were conducted three months after the commencement of data collection. Ten of the 16 people who participated in the training completed the online survey. Almost all (90%) of the 10 trainees had previous medical record abstraction experience and nearly two-thirds reported over 10 years of experience. Half of the respondents had previously participated in a webinar, among which three had participated in a webinar for training purposes. All rated the knowledge and information delivered through the webinar as useful and reported it adequately prepared them for data collection. Moreover, all participants would recommend this platform for multi-site abstraction training. Consistent with participant-reported training effectiveness, results of data collection inter-rater agreement within sites ranged from 89 to 98%, with a weighted average of 95% agreement across sites. Conclusions Conducting training via web-based technology was an acceptable and effective approach to standardizing medical record review across multiple sites for this group of experienced abstractors. Given the substantial time and cost savings achieved with the webinar, coupled with participants' positive evaluation of the training session, researchers should consider this instructional method as part of training efforts to ensure high quality data collection in multi-site studies.

Training in intensive care medicine. A challenge within reach.

The medical training model is currently immersed in a process of change. The new paradigm is intended to be more effective, more integrated within the healthcare system, and strongly oriented towards the direct application of knowledge to clinical practice. Compared with the established training system based on certification of the completion of a series or rotations and stays in certain healthcare units, the new model proposes a more structured training process based on the gradual acquisition of specific competences, in which residents must play an active role in designing their own training program. Training based on competences guarantees more transparent, updated and homogeneous learning of objective quality, and which can be homologated internationally. The tutors play a key role as the main directors of the process, and institutional commitment to their work is crucial. In this context, tutors should receive time and specific formation to allow the evaluation of training as the cornerstone of the new model. New forms of objective summative and training evaluation should be introduced to guarantee that the predefined competences and skills are effectively acquired. The free movement of specialists within Europe is very desirable and implies that training quality must be high and amenable to homologation among the different countries. The Competency Based training in Intensive Care Medicine in Europe program is our main reference for achieving this goal. Scientific societies in turn must impulse and facilitate all those initiatives destined to improve healthcare quality and therefore specialist training. They have the mission of designing strategies and processes that favor training, accreditation and advisory activities with the government authorities.

Interprofessional Education in Anatomy: Learning Together in Medical and Nursing Training

Teamwork and the interprofessional collaboration of all health professions are a guarantee of patient safety and highly qualified treatment in patient care. In the daily clinical routine, physicians and nurses must work together, but the education of the different health professions occurs separately in various places, mostly without interrelated contact. Such training abets mutual misunderstanding and cements professional protectionism, which is why interprofessional education can play an important role in dismantling such barriers to future cooperation. In this article, a pilot project in interprofessional education involving both medical and nursing students is presented, and the concept and the course of training are described in detail. The report illustrates how nursing topics and anatomy lectures can be combined for interprofessional learning in an early phase of training. Evaluation of the course showed that the students were highly satisfied with the collaborative training and believed interprofessional education (IPE) to be an important experience for their future profession and understanding of other health professionals. The results show that the IPE teaching concept, which combines anatomy and nursing topics, provides an optimal setting for learning together and helps nurses and doctors in training to gain knowledge about other health professionals’ roles, thus evolving mutual understanding.

Evaluation forum /

"A review journal for employment and training professionals focusing on policy analysis and evaluation research in education, occupational training, work and welfare, and economic development."

An empirical study of Kirkpatrick's evaluation model in the hospitality industry

This study examined Kirkpatrick’s training evaluation model (Kirkpatrick & Kirkpatrick, 2006) by assessing a sales training program conducted at an organization in the hospitality industry. The study assessed the employees’ training outcomes of knowledge and skills, job performance, and the impact of the training upon the organization. By assessing these training outcomes and their relationships, the study demonstrated whether Kirkpatrick’s theories are supported and the lower evaluation levels can be used to predict organizational impact. The population for this study was a group of reservations sales agents from a leading luxury hotel chain’s reservations center. During the study period from January 2005 to May 2007, there were 335 reservations sales agents employed in this Global Reservations Center (GRC). The number of reservations sales agents who had completed a sales training program/intervention during this period and had data available for at least two months pre and post training composed the sample for this study. The number of agents was 69 ( N = 69). Four hypotheses were tested through paired-samples t tests, correlation, and hierarchical regression analytic procedures. Results from the analyses supported the hypotheses in this study. The significant improvement in the call score supported hypothesis one that the reservations sales agents who completed the training improved their knowledge of content and required skills in handling calls (Level 2). Hypothesis two was accepted in part as there was significant improvement in call conversion, but there was no significant improvement of time usage. The significant improvement in the sales per call supported hypothesis three that the reservations agents who completed the training contributed to increased organizational impact (Level 4), i.e., made significantly more sales. Last, findings supported hypothesis four that Level 2 and Level 3 variables can be used for predicting Level 4 organizational impact. The findings supported the theory of Kirkpatrick’s evaluation model that in order to expect organizational results, a positive change in behavior (job performance) and learning must occur. The examinations of Levels 2 and 3 helped to partially explain and predict Level 4 results.

Effectiveness of Depression-Suicidal Behaviour Gatekeeper training among police officers in three European regions: outcomes of the Optimising Suicide Prevention Programmes and their Implementation in Europe (OSPI-Europe) study

Background: Gatekeeper training for community facilitators, to identify and respond to those at risk of suicide, forms an important part of multi-level community-based suicide prevention programmes. Aims: This study examined the effects of gatekeeper training on attitudes, knowledge and confidence of police officers in dealing with persons at risk of suicide. Methods: A total of 828 police officers across three European regions participated in a 4-hour training programme which addressed the epidemiology of depression and suicidal behaviour, symptoms of depression, warning signs and risk factors associated with suicidal behaviour, motivating help-seeking behaviour, dealing with acute suicidal crisis and informing bereaved relatives. Participants completed internationally validated questionnaires assessing stigmatising attitudes, knowledge about depression and confidence in dealing with suicidal persons pre- and post-training. Results: There were significant differences among countries in terms of previous exposure to suicidal persons and extent of previous training. Post-training evaluation demonstrated significant improvements in stigmatising attitudes, knowledge and confidence in all three countries. Conclusion: The consistently positive effects of gatekeeper training of police officers across different regions support inclusion of this type of training as a fundamental part of multi-level community-based suicide prevention programmes and roll-out, nationally and internationally.

The impact of usability on clinician acceptance of a health information system

The two longitudinal case studies that make up this dissertation sought to explain and predict the relationship between usability and clinician acceptance of a health information system. The overall aim of the research study was to determine what role usability plays in the acceptance or rejection of systems used by clinicians in a healthcare context. The focus was on the end users (the clinicians) rather than the views of the system designers and managers responsible for implementation and the clients of the clinicians. A mixed methods approach was adopted that drew on both qualitative and quantitative research methods. This study followed the implementation of a community health information system from early beginnings to its established practice. Users were drawn from different health service departments with distinctly different organisational cultures and attitudes to information and communication technology used in this context. This study provided evidence that a usability analysis in this context would not necessarily be valid when the users have prior reservations on acceptance. Investigation was made on the initial training and post-implementation support together with a study on the nature of the clinicians to determine factors that may influence their attitude. This research identified that acceptance of a system is not necessarily a measure of its quality, capability and usability, is influenced by the user’s attitude which is determined by outside factors, and the nature and quality of training. The need to recognise the limitations of the current methodologies for analysing usability and acceptance was explored to lay the foundations for further research.

Critical success elements for the design and implementation of organisational e-learning

Organisations are engaging in e-learning as a mechanism for delivering flexible learning to meet the needs of individuals and organisations. In light of the increasing use and organisational investment in e-learning, the need for methods to evaluate the success of its design and implementation seems more important than ever. To date, developing a standard for the evaluation of e-learning appears to have eluded both academics and practitioners. The currently accepted evaluation methods for e-learning are traditional learning and development models, such as Kirkpatrick’s model (1976). Due to the technical nature of e-learning it is important to broaden the scope and consider other evaluation models or techniques, such as the DeLone and McLean Information Success Model, that may be applicable to the e-learning domain. Research into the use of e-learning courses has largely avoided considering the applicability of information systems research. Given this observation, it is reasonable to conclude that e-learning implementation decisions and practice could be overlooking useful or additional viewpoints. This research investigated how existing evaluation models apply in the context of organisational e-learning, and resulted in an Organisational E-learning success Framework, which identifies the critical elements for success in an e-learning environment. In particular this thesis highlights the critical importance of three e-learning system creation elements; system quality, information quality, and support quality. These elements were explored in depth and the nature of each element is described in detail. In addition, two further elements were identified as factors integral to the success of an e-learning system; learner preferences and change management. Overall, this research has demonstrated the need for a holistic approach to e-learning evaluation. Furthermore, it has shown that the application of both traditional training evaluation approaches and the D&M IS Success Model are appropriate to the organisational e-learning context, and when combined can provide this holistic approach. Practically, this thesis has reported the need for organisations to consider evaluation at all stages of e-learning from design through to implementation.