GLMP for automatic assessment of DFS algorithm learning

We describe how to use a Granular Linguistic Model of a Phenomenon (GLMP) to assess e-learning processes. We apply this technique to evaluate algorithm learning using the GRAPHs learning environment.

Statistical Machine Learning for Automatic Assessment of Physical Activity Intensity Using Multi-axial Accelerometry and Heart Rate

This work explores the automatic recognition of physical activity intensity patterns from multi-axial accelerometry and heart rate signals. Data collection was carried out in free-living conditions and in three controlled gymnasium circuits, for a total amount of 179.80 h of data divided into: sedentary situations (65.5%), light-to-moderate activity (17.6%) and vigorous exercise (16.9%). The proposed machine learning algorithms comprise the following steps: time-domain feature definition, standardization and PCA projection, unsupervised clustering (by k-means and GMM) and a HMM to account for long-term temporal trends. Performance was evaluated by 30 runs of a 10-fold cross-validation. Both k-means and GMM-based approaches yielded high overall accuracy (86.97% and 85.03%, respectively) and, given the imbalance of the dataset, meritorious F-measures (up to 77.88%) for non-sedentary cases. Classification errors tended to be concentrated around transients, what constrains their practical impact. Hence, we consider our proposal to be suitable for 24 h-based monitoring of physical activity in ambulatory scenarios and a first step towards intensity-specific energy expenditure estimators

An approach to automatic learning assessment based on the computational theory of perceptions

E-learning systems output a huge quantity of data on a learning process. However, it takes a lot of specialist human resources to manually process these data and generate an assessment report. Additionally, for formative assessment, the report should state the attainment level of the learning goals defined by the instructor. This paper describes the use of the granular linguistic model of a phenomenon (GLMP) to model the assessment of the learning process and implement the automated generation of an assessment report. GLMP is based on fuzzy logic and the computational theory of perceptions. This technique is useful for implementing complex assessment criteria using inference systems based on linguistic rules. Apart from the grade, the model also generates a detailed natural language progress report on the achieved proficiency level, based exclusively on the objective data gathered from correct and incorrect responses. This is illustrated by applying the model to the assessment of Dijkstra’s algorithm learning using a visual simulation-based graph algorithm learning environment, called GRAPHs

Assessment of learning outcomes in computing studies

The assessment of learning outcomes is a key concept in the European Credit Transfer and Accumulation System (ECTS) since credits are awarded when the assessment shows the competences which were aimed at have been developed at an appropriate level. This paper describes a study which was first part of the project of the Bologna Experts Team-Spain and then developed as an independent study. It was carried out with the overall goal to gain experience in the assessment of learning outcomes. More specifically it aimed at 1) designing procedures for the assessment of learning outcomes related to these compulsory generic competences; 2) testing some basic psychometric features that an assessment device with some consequences for the subjects being evaluated needs to prove; 3) testing different procedures of standard setting, and 4) using assessment results as orienting feedback to students and their tutors. The process of development of tests to carry out the assessment of learning outcomes related to these competences, as well as some basic features regarding their reliability and validity is described and first results on the comparison of results achieved at two academic levels, will also be described at a later stage.

Methods and Tools for Objective Assessment of Psychomotor Skills in Laparoscopic Surgery

Training and assessment paradigms for laparoscopic surgical skills are evolving from traditional mentor–trainee tutorship towards structured, more objective and safer programs. Accreditation of surgeons requires reaching a consensus on metrics and tasks used to assess surgeons’ psychomotor skills. Ongoing development of tracking systems and software solutions has allowed for the expansion of novel training and assessment means in laparoscopy. The current challenge is to adapt and include these systems within training programs, and to exploit their possibilities for evaluation purposes. This paper describes the state of the art in research on measuring and assessing psychomotor laparoscopic skills. It gives an overview on tracking systems as well as on metrics and advanced statistical and machine learning techniques employed for evaluation purposes. The later ones have a potential to be used as an aid in deciding on the surgical competence level, which is an important aspect when accreditation of the surgeons in particular, and patient safety in general, are considered. The prospective of these methods and tools make them complementary means for surgical assessment of motor skills, especially in the early stages of training. Successful examples such as the Fundamentals of Laparoscopic Surgery should help drive a paradigm change to structured curricula based on objective parameters. These may improve the accreditation of new surgeons, as well as optimize their already overloaded training schedules.

Development and Assessment of the Competence Creativity Applied to Technical Drawing

The results obtained after incorporating the competence “creativity” to the subject Technical Drawing of the first course of the Degree in Forestry, Technical University of Madrid, are presented in this study.At first, learning activities which could serve two functions at the same time -developing students’ creativity and developing other specific competences of the subject- were considered. Besides, changes in the assessment procedure were made and a method which analyzes two aspects of the assessment of the competence creativity was established. On the one hand, the products are evaluated by analyzing the outcomes obtained by students in the essays suggested and by establishing a parameter to assess the creativity expressed in those essays. On the other, an assessment of the student is directly carried out through a psychometric test which has been previously chosen by the team.Moreover, these results can be applied to similar or could be of general application

Assessment of work performed by paired students: a pedagogical experience

“Teamwork” is one of the abilities most valued by employers. In [16] we describe the process of adapting to the ECTS methodologies (for ongoing assessment), a course in computer programming for students in a technical degree (Marine Engineering, UPM) not specifically dedicated to computing. As a further step in this process we have emphasized cooperative learning. For this, the students were paired and the work of each pair was evaluated via surprise tests taken and graded jointly, and constituting a substantial part of the final grade. Here we document this experience, discussing methodological aspects, describing indicators for measuring the impact of these methodologies on the educational experience, and reporting on the students’ opinion of it.

Competence assessment of final year projects for undergraduate telecommunication students

The value of project-based learning has lead to the inclusion of project development activities in engineering courses, being the Final Year Project (FYP) the most remarkable one. Several approaches have been proposed for assessing and grading FYPs but, among them, rubrics are becoming a standard for such type of assessment. However, due to the different characteristics and orientations of the projects (some are more practically oriented, some more theoretically), and the high amount of different competences to be evaluated (knowledge, working capability, communication skills, etc.), the definition of one unique rubric suitable for the evaluation of all FYPs presented in different degree programs, is a big challenge. In a former work, the educational outcomes expected from the FYP were defined and resulted in a proposal for their assessment. Afterwards, the proposal has been tested during one year within an educational innovation-project at the Universidad Politécnica de Madrid which involved the follow-up of 8 undergraduate telecommunication students elaborating their FYPs. In this publication, our experience will be described, based on the emerging work taking place through the formalisation of the process which consisted in the following steps: i) establishment of a schedule for the whole process (publication of FYPs topics, selection of applying students and their enrolment, assignation of a jury to each FYP, elaboration and follow-up of FYPs, final report submission, oral presentation, etc.); ii) design of rubrics for each of three assessment parts: working process, final report and oral presentation; and iii) follow-up and evaluation of the involved FYPs. Finally, problems that appeared during this experience (e.g. administrative aspects, criticisms and suggestions from the students, tutors and juries involved) are discussed and some modifications in the assessment system will be proposed in order to solve or minimize these problems.

Assessment of problem solving in computing studies

The assessment of learning outcomes is a key concept in the European Credit Transfer and Accumulation System (ECTS) since credits are awarded when the assessment shows the competences which were aimed at have been developed at an appropriate level. This paper describes a study which was first part of the Bologna Experts Team-Spain project and then developed as an independent study. It was carried out with the overall goal to gain experience in the assessment of learning outcomes. More specifically it aimed at 1) designing procedures for the assessment of learning outcomes related to these compulsory generic competences; 2) testing some basic psychometric features that an assessment device with some consequences for the subjects being evaluated needs to prove; 3) testing different procedures of standard setting, and 4) using assessment results as orienting feedback to students and their tutors. The process of development of tests to carry out the assessment of learning outcomes is described as well as some basic features regarding their reliability and validity. First conclusions on the comparison of the results achieved at two academic levels are also presented.

Large-scale assessment of language proficiency: theoretical and pedagogical reflections on the use of multiple-choice tests

The new requirement placed on students in tertiary settings in Spain to demonstrate a B1 or a B2 proficiency level of English, in accordance with the Common European Framework of Reference for Languages (CEFRL), has led most Spanish universities to develop a program of certification or accreditation of the required level. The first part of this paper aims to provide a rationale for the type of test that has been developed at the Universidad Politécnica de Madrid for the accreditation of a B2 level, a multiple choice version, and to describe how it was constructed and validated. Then, in the second part of the paper, the results from its application to 924 students enrolled in different degree courses at a variety of schools and faculties at the university are analyzed based on a final test version item analysis. To conclude, some theoretical as well as practical conclusions about testing grammar that affect the teaching and learning process are drawn. RESUMEN. Las nuevas exigencias sobre niveles de competencia B1 y B2 en inglés según el Marco Común Europeo de Referencia para las Lenguas (MCERL) que se imponen sobre los estudiantes de grado y posgrado han llevado a la mayoría de las universidades españolas a desarrollar programas de acreditación o de certificación de estos niveles. La primera parte de este trabajo trata sobre las razones que fundamentan la elección de un tipo concreto de examen para la acreditación del nivel B2 de lengua inglesa en la Universidad Politécnica de Madrid. Se trata de un test de opción múltiple y en esta parte del trabajo se describe cómo fue diseñado y validado. En la segunda parte, se analizan los resultados de la aplicación del test a gran escala a un total de 924 estudiantes matriculados en varias escuelas y Facultades de la Universidad. Para terminar, se apuntan una serie de conclusiones teóricas y prácticas sobre la evaluación de la gramática y de qué modo influye en los procesos de enseñanza y aprendizaje.

A multidisciplinary reference framework to support innovative design, implementation and assessment of chronic care models

Hoy en día, por primera vez en la historia, la mayor parte de la población podrá vivir hasta los sesenta años y más (United Nations, 2015). Sin embargo, todavía existe poca evidencia que demuestre que las personas mayores, estén viviendo con mejor salud que sus padres, a la misma edad, ya que la mayoría de los problemas de salud en edades avanzadas están asociados a las enfermedades crónicas (WHO, 2015). Los sistemas sanitarios de los países desarrollados funcionan adecuadamente cuando se trata del cuidado de enfermedades agudas, pero no son lo suficientemente eficaces en la gestión de las enfermedades crónicas. Durante la última década, se han realizado esfuerzos para mejorar esta gestión, por medio de la utilización de estrategias de prevención y de reenfoque de la provisión de los servicios de atención para la salud (Kane et al. 2005). Según una revisión sistemática de modelos de cuidado de salud, comisionada por el sistema nacional de salud Británico, pocos modelos han conceptualizado cuáles son los componentes que hay que utilizar para proporcionar un cuidado crónico efectivo, y estos componentes no han sido suficientemente estructurados y articulados. Por lo tanto, no hay suficiente evidencia sobre el impacto real de cualquier modelo existente en la actualidad (Ham, 2006). Las innovaciones podrían ayudar a conseguir mejores diagnósticos, tratamientos y gestión de pacientes crónicos, así como a dar soporte a los profesionales y a los pacientes en el cuidado. Sin embargo, la forma en las que estas innovaciones se proporcionan no es lo suficientemente eficiente, efectiva y amigable para el usuario. Para mejorar esto, hace falta crear equipos de trabajo y estrategias multidisciplinares. En conclusión, hacen falta actividades que permitan conseguir que las innovaciones sean utilizadas en los sistemas de salud que quieren mejorar la gestión del cuidado crónico, para que sea posible: 1) traducir la “atención sanitaria basada en la evidencia” en “conocimiento factible”; 2) hacer frente a la complejidad de la atención sanitaria a través de una investigación multidisciplinaria; 3) identificar una aproximación sistemática para que se establezcan intervenciones innovadoras en el cuidado de salud. El marco de referencia desarrollado en este trabajo de investigación es un intento de aportar estas mejoras. Las siguientes hipótesis han sido propuestas: Hipótesis 1: es posible definir un proceso de traducción que convierta un modelo de cuidado crónico en una descripción estructurada de objetivos, requisitos e indicadores clave de rendimiento. Hipótesis 2: el proceso de traducción, si se ejecuta a través de elementos basados en la evidencia, multidisciplinares y de orientación económica, puede convertir un modelo de cuidado crónico en un marco descriptivo, que define el ciclo de vida de soluciones innovadoras para el cuidado de enfermedades crónicas. Hipótesis 3: es posible definir un método para evaluar procesos, resultados y capacidad de desarrollar habilidades, y asistir equipos multidisciplinares en la creación de soluciones innovadoras para el cuidado crónico. Hipótesis 4: es posible dar soporte al desarrollo de soluciones innovadoras para el cuidado crónico a través de un marco de referencia y conseguir efectos positivos, medidos en indicadores clave de rendimiento. Para verificar las hipótesis, se ha definido una aproximación metodológica compuesta de cuatro Fases, cada una asociada a una hipótesis. Antes de esto, se ha llevado a cabo una “Fase 0”, donde se han analizado los antecedentes sobre el problema (i.e. adopción sistemática de la innovación en el cuidado crónico) desde una perspectiva multi-dominio y multi-disciplinar. Durante la fase 1, se ha desarrollado un Proceso de Traducción del Conocimiento, elaborado a partir del JBI Joanna Briggs Institute (JBI) model of evidence-based healthcare (Pearson, 2005), y sobre el cual se han definido cuatro Bloques de Innovación. Estos bloques consisten en una descripción de elementos innovadores, definidos en la fase 0, que han sido añadidos a los cuatros elementos que componen el modelo JBI. El trabajo llevado a cabo en esta fase ha servido también para definir los materiales que el proceso de traducción tiene que ejecutar. La traducción que se ha llevado a cabo en la fase 2, y que traduce la mejor evidencia disponible de cuidado crónico en acción: resultado de este proceso de traducción es la parte descriptiva del marco de referencia, que consiste en una descripción de un modelo de cuidado crónico (se ha elegido el Chronic Care Model, Wagner, 1996) en términos de objetivos, especificaciones e indicadores clave de rendimiento y organizada en tres ciclos de innovación (diseño, implementación y evaluación). Este resultado ha permitido verificar la segunda hipótesis. Durante la fase 3, para demostrar la tercera hipótesis, se ha desarrollado un método-mixto de evaluación de equipos multidisciplinares que trabajan en innovaciones para el cuidado crónico. Este método se ha creado a partir del método mixto usado para la evaluación de equipo multidisciplinares translacionales (Wooden, 2013). El método creado añade una dimensión procedural al marco. El resultado de esta fase consiste, por lo tanto, en una primera versión del marco de referencia, lista para ser experimentada. En la fase 4, se ha validado el marco a través de un caso de estudio multinivel y con técnicas de observación-participante como método de recolección de datos. Como caso de estudio se han elegido las actividades de investigación que el grupo de investigación LifeStech ha desarrollado desde el 2008 para mejorar la gestión de la diabetes, actividades realizadas en un contexto internacional. Los resultados demuestran que el marco ha permitido mejorar las actividades de trabajo en distintos niveles: 1) la calidad y cantidad de las publicaciones; 2) se han conseguido dos contratos de investigación sobre diabetes: el primero es un proyecto de investigación aplicada, el segundo es un proyecto financiado para acelerar las innovaciones en el mercado; 3) a través de los indicadores claves de rendimiento propuestos en el marco, una prueba de concepto de un prototipo desarrollado en un proyecto de investigación ha sido transformada en una evaluación temprana de una intervención eHealth para el manejo de la diabetes, que ha sido recientemente incluida en Repositorio de prácticas innovadoras del Partenariado de Innovación Europeo en Envejecimiento saludable y activo. La verificación de las 4 hipótesis ha permitido demonstrar la hipótesis principal de este trabajo de investigación: es posible contribuir a crear un puente entre la atención sanitaria y la innovación y, por lo tanto, mejorar la manera en que el cuidado crónico sea procurado en los sistemas sanitarios. ABSTRACT Nowadays, for the first time in history, most people can expect to live into their sixties and beyond (United Nations, 2015). However, little evidence suggests that older people are experiencing better health than their parents, and most of the health problems of older age are linked to Chronic Diseases (WHO, 2015). The established health care systems in developed countries are well suited to the treatment of acute diseases but are mostly inadequate for dealing with CDs. Healthcare systems are challenging the burden of chronic diseases by putting more emphasis on the prevention of disease and by looking for new ways to reorient the provision of care (Kane et al., 2005). According to an evidence-based review commissioned by the British NHS Institute, few models have conceptualized effective components of care for CDs and these components have been not structured and articulated. “Consequently, there is limited evidence about the real impact of any of the existing models” (Ham, 2006). Innovations could support to achieve better diagnosis, treatment and management for patients across the continuum of care, by supporting health professionals and empowering patients to take responsibility. However, the way they are delivered is not sufficiently efficient, effective and consumer friendly. The improvement of innovation delivery, involves the creation of multidisciplinary research teams and taskforces, rather than just working teams. There are several actions to improve the adoption of innovations from healthcare systems that are tackling the epidemics of CDs: 1) Translate Evidence-Based Healthcare (EBH) into actionable knowledge; 2) Face the complexity of healthcare through multidisciplinary research; 3) Identify a systematic approach to support effective implementation of healthcare interventions through innovation. The framework proposed in this research work is an attempt to provide these improvements. The following hypotheses have been drafted: Hypothesis 1: it is possible to define a translation process to convert a model of chronic care into a structured description of goals, requirements and key performance indicators. Hypothesis 2: a translation process, if executed through evidence-based, multidisciplinary, holistic and business-oriented elements, can convert a model of chronic care in a descriptive framework, which defines the whole development cycle of innovative solutions for chronic disease management. Hypothesis 3: it is possible to design a method to evaluate processes, outcomes and skill acquisition capacities, and assist multidisciplinary research teams in the creation of innovative solutions for chronic disease management. Hypothesis 4: it is possible to assist the development of innovative solutions for chronic disease management through a reference framework and produce positive effects, measured through key performance indicators. In order to verify the hypotheses, a methodological approach, composed of four Phases that correspond to each one of the stated hypothesis, was defined. Prior to this, a “Phase 0”, consisting in a multi-domain and multi-disciplinary background analysis of the problem (i.e.: systematic adoption of innovation to chronic care), was carried out. During phase 1, in order to verify the first hypothesis, a Knowledge Translation Process (KTP) was developed, starting from the JBI Joanna Briggs Institute (JBI) model of evidence-based healthcare was used (Pearson, 2005) and adding Four Innovation Blocks. These blocks represent an enriched description, added to the JBI model, to accelerate the transformation of evidence-healthcare through innovation; the innovation blocks are built on top of the conclusions drawn after Phase 0. The background analysis gave also indication on the materials and methods to be used for the execution of the KTP, carried out during phase 2, that translates the actual best available evidence for chronic care into action: this resulted in a descriptive Framework, which is a description of a model of chronic care (the Chronic Care Model was chosen, Wagner, 1996) in terms of goals, specified requirements and Key Performance Indicators, and articulated in the three development cycles of innovation (i.e. design, implementation and evaluation). Thanks to this result the second hypothesis was verified. During phase 3, in order to verify the third hypothesis, a mixed-method to evaluate multidisciplinary teams working on innovations for chronic care, was created, based on a mixed-method used for the evaluation of Multidisciplinary Translational Teams (Wooden, 2013). This method adds a procedural dimension to the descriptive component of the Framework, The result of this phase consisted in a draft version of the framework, ready to be tested in a real scenario. During phase 4, a single and multilevel case study, with participant-observation data collection, was carried out, in order to have a complete but at the same time multi-sectorial evaluation of the framework. The activities that the LifeStech research group carried out since 2008 to improve the management of diabetes have been selected as case study. The results achieved showed that the framework allowed to improve the research activities in different directions: the quality and quantity of the research publications that LifeStech has issued, have increased substantially; 2 project grants to improve the management of diabetes, have been assigned: the first is a grant funding applied research while the second is about accelerating innovations into the market; by using the assessment KPIs of the framework, the proof of concept validation of a prototype developed in a research project was transformed into an early stage assessment of innovative eHealth intervention for Diabetes Management, which has been recently included in the repository of innovative practice of the European Innovation Partnership on Active and Health Ageing initiative. The verification of the 4 hypotheses lead to verify the main hypothesis of this research work: it is possible to contribute to bridge the gap between healthcare and innovation and, in turn, improve the way chronic care is delivered by healthcare systems.