An approach to automatic learning assessment based on the computational theory of perceptions

E-learning systems output a huge quantity of data on a learning process. However, it takes a lot of specialist human resources to manually process these data and generate an assessment report. Additionally, for formative assessment, the report should state the attainment level of the learning goals defined by the instructor. This paper describes the use of the granular linguistic model of a phenomenon (GLMP) to model the assessment of the learning process and implement the automated generation of an assessment report. GLMP is based on fuzzy logic and the computational theory of perceptions. This technique is useful for implementing complex assessment criteria using inference systems based on linguistic rules. Apart from the grade, the model also generates a detailed natural language progress report on the achieved proficiency level, based exclusively on the objective data gathered from correct and incorrect responses. This is illustrated by applying the model to the assessment of Dijkstra’s algorithm learning using a visual simulation-based graph algorithm learning environment, called GRAPHs

Peer Assessment and Self-assessment: Effective Learning Tools in Higher Education.

When used appropriately, self- and peer-assessment are very effective learning tools. In the present work, instructor formative assessment and feedback, self-assessment (SA), and peer-assessment (PA) have been compared. During the first part of a semester, the students followed a continuous formative assessment. Subsequently, they were divided into two subgroups based on similar performances. One subgroup performed SAs, and the other followedPAduring the last part of the course. The performances of the two groups in solving problems were compared. Results suggest that PA is a more effective learning tool than SA, and both are more effective than instructor formative assessment. However, a survey that was conducted at the end of the experiment showed higher student confidence in instructor assessment than in PA. The students recognized the usefulness of acting as peer assessors, but believed that SA helped them more than PA.

Evaluación para el aprendizaje en la asignatura sistemas operativos en modalidad online: un estudio cualitativo basado en la taxonomía de Bloom

La asignatura Sistemas Operativos presenta dificultades para su aprendizaje, pero poco se conoce acerca de las mismas, ya que no han sido determinadas ni estudiadas por la literatura. Asimismo, los trabajos existentes sobre la enseñanza y aprendizaje de Sistemas Operativos se limitan a proponer distintos enfoques para impartir la asignatura y en general no evalúan el aprendizaje de los estudiantes para comprobar la eficacia del método propuesto ni usan metodologías de investigación rigurosas. Por otra parte, la impartición de la asignatura Sistemas Operativos en modalidad online ha sido escasamente estudiada y podría tener dificultades adicionales a las de la modalidad presencial, ya que el contexto online impone una serie de restricciones tanto para el profesor como para el estudiante. En la presente tesis se ha llevado a cabo una evaluación formativa en la asignatura Sistemas Operativos, perteneciente al Grado de Ingeniería Informática de una universidad online. El objetivo inicial de la evaluación era descubrir las dificultades de los estudiantes para la comprensión de los conceptos de la asignatura. Posteriormente y, dada la buena aceptación de la evaluación por parte de los estudiantes, se ampliaron los objetivos del trabajo para explorar los efectos de la evaluación realizada sobre el aprendizaje. La evaluación formativa diseñada está basada en la taxonomía revisada de Bloom y sus principales objetivos son: (a) promover el aprendizaje significativo y (b) hacer a los estudiantes conscientes de su proceso de aprendizaje. La metodología de investigación utilizada es el estudio de caso cualitativo y la muestra está constituida por 9 estudiantes del total de 13 matriculados en la asignatura. Los datos cualitativos analizados proceden de las pruebas de evaluación formativa llevadas a cabo por los estudiantes durante la impartición de la asignatura. Los conceptos de sistemas operativos que han resultado más difíciles de comprender en el curso online estudiado han sido las interrupciones y los semáforos. Además, alrededor de estos conceptos se han identificado las dificultades específicas y sus posibles causas. Las dificultades descubiertas acerca de los semáforos corroboran las investigaciones existentes en el área de programación concurrente. El resto de las dificultades identificadas no habían sido determinadas por la literatura existente. En cuanto a los efectos de la evaluación formativa sobre el aprendizaje, la evidencia empírica muestra que ésta ha provocado en los estudiantes una reflexión profunda sobre los conceptos de la asignatura y sobre su propio proceso de aprendizaje. El estudio de caso presentado puede ayudar a los profesores del área de ingeniería a crear evaluaciones formativas en cursos online. La tesis, por tanto, realiza aportaciones relevantes en las áreas de enseñanza y aprendizaje de sistemas operativos, evaluación formativa, metodologías cualitativas y educación online. ABSTRACT Operating Systems is a difficult subject to learn; however little is known about said difficulties, as they have not been studied nor determined by the relevant literature. Existing studies on teaching and learning the subject of operating systems are limited to presenting different approaches for teaching the subject and generally do not evaluate students’ learning to verify the effectiveness of the proposed methods, nor do they use rigorous research methodologies. On the other hand, there are very few studies on teaching operating systems online, which may inherently present more difficulties than the in-person format, since an online context imposes a series of restrictions on both professors and students, such as not having face-to-face interaction for communications. This thesis studies a formative assessment of the subject of operating systems, as part of the Degree in Information Technology Engineering for an online university. The initial objective of this assessment was to determine the students’ difficulties in conceptual comprehension for this subject. Once students had accepted the assessment, the study’s objectives were expanded to include an investigation of the effects of the assessment on learning. The designed formative assessment was based on Revised Bloom’s Taxonomy with the following main objectives: (a) to promote meaningful learning and (b) (b) to make students aware of their learning process. The research methodology involves a qualitative case study with a sample consisting of 9 of the total 13 students registered for this course. The qualitative data analyzed comes from the formative assessment tests taken by these students during the course. The most difficult operating systems concepts for students in the online course were interrupts and semaphores. Additionally, the specific difficulties and their possible causes have been identified. The students’ comprehension difficulties with semaphores corroborate the existing research in the area of concurrent programming. The other identified difficulties were not discussed in the existing literature. Regarding the effects of the formative assessment on learning, the empirical evidence shows that it causes students to reflect carefully on the subject’s concepts as well as their own learning process. The presented case study can help professors in the area of engineering to create formative assessments for online courses. This thesis, therefore, makes relevant contributions to the areas of teaching and learning operating systems, formative assessment, qualitative methodologies, and online education.

Especificaciones eMathTeacher, creación del Modelo Granular Lingüístico de la evaluación del aprendizaje humano, y diseño de sistemas que implementan estos conceptos

El concepto de algoritmo es básico en informática, por lo que es crucial que los alumnos profundicen en él desde el inicio de su formación. Por tanto, contar con una herramienta que guíe a los estudiantes en su aprendizaje puede suponer una gran ayuda en su formación. La mayoría de los autores coinciden en que, para determinar la eficacia de una herramienta de visualización de algoritmos, es esencial cómo se utiliza. Así, los estudiantes que participan activamente en la visualización superan claramente a los que la contemplan de forma pasiva. Por ello, pensamos que uno de los mejores ejercicios para un alumno consiste en simular la ejecución del algoritmo que desea aprender mediante el uso de una herramienta de visualización, i. e. consiste en realizar una simulación visual de dicho algoritmo. La primera parte de esta tesis presenta los resultados de una profunda investigación sobre las características que debe reunir una herramienta de ayuda al aprendizaje de algoritmos y conceptos matemáticos para optimizar su efectividad: el conjunto de especificaciones eMathTeacher, además de un entorno de aprendizaje que integra herramientas que las cumplen: GRAPHs. Hemos estudiado cuáles son las cualidades esenciales para potenciar la eficacia de un sistema e-learning de este tipo. Esto nos ha llevado a la definición del concepto eMathTeacher, que se ha materializado en el conjunto de especificaciones eMathTeacher. Una herramienta e-learning cumple las especificaciones eMathTeacher si actúa como un profesor virtual de matemáticas, i. e. si es una herramienta de autoevaluación que ayuda a los alumnos a aprender de forma activa y autónoma conceptos o algoritmos matemáticos, corrigiendo sus errores y proporcionando pistas para encontrar la respuesta correcta, pero sin dársela explícitamente. En estas herramientas, la simulación del algoritmo no continúa hasta que el usuario introduce la respuesta correcta. Para poder reunir en un único entorno una colección de herramientas que cumplan las especificaciones eMathTeacher hemos creado GRAPHs, un entorno ampliable, basado en simulación visual, diseñado para el aprendizaje activo e independiente de los algoritmos de grafos y creado para que en él se integren simuladores de diferentes algoritmos. Además de las opciones de creación y edición del grafo y la visualización de los cambios producidos en él durante la simulación, el entorno incluye corrección paso a paso, animación del pseudocódigo del algoritmo, preguntas emergentes, manejo de las estructuras de datos del algoritmo y creación de un log de interacción en XML. Otro problema que nos planteamos en este trabajo, por su importancia en el proceso de aprendizaje, es el de la evaluación formativa. El uso de ciertos entornos e-learning genera gran cantidad de datos que deben ser interpretados para llegar a una evaluación que no se limite a un recuento de errores. Esto incluye el establecimiento de relaciones entre los datos disponibles y la generación de descripciones lingüísticas que informen al alumno sobre la evolución de su aprendizaje. Hasta ahora sólo un experto humano era capaz de hacer este tipo de evaluación. Nuestro objetivo ha sido crear un modelo computacional que simule el razonamiento del profesor y genere un informe sobre la evolución del aprendizaje que especifique el nivel de logro de cada uno de los objetivos definidos por el profesor. Como resultado del trabajo realizado, la segunda parte de esta tesis presenta el modelo granular lingüístico de la evaluación del aprendizaje, capaz de modelizar la evaluación y generar automáticamente informes de evaluación formativa. Este modelo es una particularización del modelo granular lingüístico de un fenómeno (GLMP), en cuyo desarrollo y formalización colaboramos, basado en la lógica borrosa y en la teoría computacional de las percepciones. Esta técnica, que utiliza sistemas de inferencia basados en reglas lingüísticas y es capaz de implementar criterios de evaluación complejos, se ha aplicado a dos casos: la evaluación, basada en criterios, de logs de interacción generados por GRAPHs y de cuestionarios de Moodle. Como consecuencia, se han implementado, probado y utilizado en el aula sistemas expertos que evalúan ambos tipos de ejercicios. Además de la calificación numérica, los sistemas generan informes de evaluación, en lenguaje natural, sobre los niveles de competencia alcanzados, usando sólo datos objetivos de respuestas correctas e incorrectas. Además, se han desarrollado dos aplicaciones capaces de ser configuradas para implementar los sistemas expertos mencionados. Una procesa los archivos producidos por GRAPHs y la otra, integrable en Moodle, evalúa basándose en los resultados de los cuestionarios. ABSTRACT The concept of algorithm is one of the core subjects in computer science. It is extremely important, then, for students to get a good grasp of this concept from the very start of their training. In this respect, having a tool that helps and shepherds students through the process of learning this concept can make a huge difference to their instruction. Much has been written about how helpful algorithm visualization tools can be. Most authors agree that the most important part of the learning process is how students use the visualization tool. Learners who are actively involved in visualization consistently outperform other learners who view the algorithms passively. Therefore we think that one of the best exercises to learn an algorithm is for the user to simulate the algorithm execution while using a visualization tool, thus performing a visual algorithm simulation. The first part of this thesis presents the eMathTeacher set of requirements together with an eMathTeacher-compliant tool called GRAPHs. For some years, we have been developing a theory about what the key features of an effective e-learning system for teaching mathematical concepts and algorithms are. This led to the definition of eMathTeacher concept, which has materialized in the eMathTeacher set of requirements. An e-learning tool is eMathTeacher compliant if it works as a virtual math trainer. In other words, it has to be an on-line self-assessment tool that helps students to actively and autonomously learn math concepts or algorithms, correcting their mistakes and providing them with clues to find the right answer. In an eMathTeacher-compliant tool, algorithm simulation does not continue until the user enters the correct answer. GRAPHs is an extendible environment designed for active and independent visual simulation-based learning of graph algorithms, set up to integrate tools to help the user simulate the execution of different algorithms. Apart from the options of creating and editing the graph, and visualizing the changes made to the graph during simulation, the environment also includes step-by-step correction, algorithm pseudo-code animation, pop-up questions, data structure handling and XML-based interaction log creation features. On the other hand, assessment is a key part of any learning process. Through the use of e-learning environments huge amounts of data can be output about this process. Nevertheless, this information has to be interpreted and represented in a practical way to arrive at a sound assessment that is not confined to merely counting mistakes. This includes establishing relationships between the available data and also providing instructive linguistic descriptions about learning evolution. Additionally, formative assessment should specify the level of attainment of the learning goals defined by the instructor. Till now, only human experts were capable of making such assessments. While facing this problem, our goal has been to create a computational model that simulates the instructor’s reasoning and generates an enlightening learning evolution report in natural language. The second part of this thesis presents the granular linguistic model of learning assessment to model the assessment of the learning process and implement the automated generation of a formative assessment report. The model is a particularization of the granular linguistic model of a phenomenon (GLMP) paradigm, based on fuzzy logic and the computational theory of perceptions, to the assessment phenomenon. This technique, useful for implementing complex assessment criteria using inference systems based on linguistic rules, has been applied to two particular cases: the assessment of the interaction logs generated by GRAPHs and the criterion-based assessment of Moodle quizzes. As a consequence, several expert systems to assess different algorithm simulations and Moodle quizzes have been implemented, tested and used in the classroom. Apart from the grade, the designed expert systems also generate natural language progress reports on the achieved proficiency level, based exclusively on the objective data gathered from correct and incorrect responses. In addition, two applications, capable of being configured to implement the expert systems, have been developed. One is geared up to process the files output by GRAPHs and the other one is a Moodle plug-in set up to perform the assessment based on the quizzes results.

La evaluación del aprendizaje de los deportes colectivos en educación física a través del modelo de enseñanza comprensivo

El presente trabajo tiene como objetivo evaluar el aprendizaje de los deportes colectivos en el alumnado de segundo ciclo de Educación Secundaria Obligatoria (ESO) utilizando una herramienta denominada Game Performance Assessment Instrument (GPAI). Para ello se ha realizado una revisión de dos modelos de enseñanza existentes: el modelo técnico y el modelo comprensivo. Así mismo hemos revisado dos tipos de procesos de evaluación; por un lado la evaluación sumativa, más relacionada con el modelo técnico; por otro lado la evaluación formativa, más próxima al modelo comprensivo. A continuación se ha llevado a cabo una intervención didáctica aplicando el modelo de enseñanza comprensivo (TGfU) y el instrumento mencionado anteriormente (GPAI). Los resultados obtenidos han sido utilizados para proponer un sistema de evaluación basado en el modelo comprensivo con la utilización del GPAI y a través de la coevaluación como forma de asentar el aprendizaje en el alumnado. Se concluye que el modelo de enseñanza comprensivo (TGfU) con el GPAI y una evaluación entre iguales es bastante óptimo para favorecer el proceso de enseñanzaaprendizaje de los deportes colectivos en el alumnado de segundo ciclo de Educación Secundaria Obligatoria (ESO). ABSTRACT The present work has the aim to assess the learning of collective sports in Secondary Education students using a tool called Game Performance Assessment Instrument (GPAI). For that we have used to revision of two teaching models: the technical model and Teaching Games for Understanding (TGfU). At the same time, we have reviewed two kinds of assessment process; on one hand summative assessment which is related to technical model; on the other hand formative assessment nearest to Teaching Games for Understanding (TGfU). Next it has been done a didactic intervention using the Teaching Games for Understanding (TGfU) and that instrument mentioned beforehand. The results obtained have been used to suggest an assessment system based on Teaching Games for Understanding (TGfU) using the GPAI through the peer assessment as a way of teaching.

Competence assessment of final year projects for undergraduate telecommunication students

The value of project-based learning has lead to the inclusion of project development activities in engineering courses, being the Final Year Project (FYP) the most remarkable one. Several approaches have been proposed for assessing and grading FYPs but, among them, rubrics are becoming a standard for such type of assessment. However, due to the different characteristics and orientations of the projects (some are more practically oriented, some more theoretically), and the high amount of different competences to be evaluated (knowledge, working capability, communication skills, etc.), the definition of one unique rubric suitable for the evaluation of all FYPs presented in different degree programs, is a big challenge. In a former work, the educational outcomes expected from the FYP were defined and resulted in a proposal for their assessment. Afterwards, the proposal has been tested during one year within an educational innovation-project at the Universidad Politécnica de Madrid which involved the follow-up of 8 undergraduate telecommunication students elaborating their FYPs. In this publication, our experience will be described, based on the emerging work taking place through the formalisation of the process which consisted in the following steps: i) establishment of a schedule for the whole process (publication of FYPs topics, selection of applying students and their enrolment, assignation of a jury to each FYP, elaboration and follow-up of FYPs, final report submission, oral presentation, etc.); ii) design of rubrics for each of three assessment parts: working process, final report and oral presentation; and iii) follow-up and evaluation of the involved FYPs. Finally, problems that appeared during this experience (e.g. administrative aspects, criticisms and suggestions from the students, tutors and juries involved) are discussed and some modifications in the assessment system will be proposed in order to solve or minimize these problems.

Development of a method of assessment of the problem-solving competence at the technical University of Madrid.

The competence evaluation promoted by the European High Education Area entails a very important methodological change that requires guiding support to help lecturers carry out this new and complex task. In this regard, the Technical University of Madrid (UPM, by its Spanish acronym) has financed a series of coordinated projects with the objective of developing a model for teaching and evaluating core competences and providing support to lecturers. This paper deals with the problem-solving competence. The first step has been to elaborate a guide for teachers to provide a homogeneous way to asses this competence. This guide considers several levels of acquisition of the competence and provides the rubrics to be applied for each one. The guide has been subsequently validated with several pilot experiences. In this paper we will explain the problem-solving assessment guide for teachers and will show the pilot experiences that has been carried out. We will finally justify the validity of the method to assess the problem-solving competence.

Development of a method of assessment of the problem-solving competency at the Technical University of Madrid

The competence evaluation promoted by the European High Education Area entails a very important methodological change that requires guiding support to help lecturers carry out this new and complex task. In this regard, the Technical University of Madrid (UPM, by its Spanish acronym) has financed a series of coordinated projects with the objective of developing a model for teaching and evaluating core competences and providing support to lecturers. This paper deals with the problem solving competence. The first step has been to elaborate a guide for teachers to provide an homogeneous way to asses this competence. This guide considers several levels of acquisition of the competence and provided the rubrics to be applied for each one. The guide has been subsequently validated with several pilot experiences. In this paper we will explain the problem-solving assessment guide for teachers and will show the pilot experiences that has been carried out. We will finally justify the validity of the method to assess the problem solving competence.