Large-scale assessment of language proficiency: theoretical and pedagogical reflections on the use of multiple-choice tests

The new requirement placed on students in tertiary settings in Spain to demonstrate a B1 or a B2 proficiency level of English, in accordance with the Common European Framework of Reference for Languages (CEFRL), has led most Spanish universities to develop a program of certification or accreditation of the required level. The first part of this paper aims to provide a rationale for the type of test that has been developed at the Universidad Politécnica de Madrid for the accreditation of a B2 level, a multiple choice version, and to describe how it was constructed and validated. Then, in the second part of the paper, the results from its application to 924 students enrolled in different degree courses at a variety of schools and faculties at the university are analyzed based on a final test version item analysis. To conclude, some theoretical as well as practical conclusions about testing grammar that affect the teaching and learning process are drawn. RESUMEN. Las nuevas exigencias sobre niveles de competencia B1 y B2 en inglés según el Marco Común Europeo de Referencia para las Lenguas (MCERL) que se imponen sobre los estudiantes de grado y posgrado han llevado a la mayoría de las universidades españolas a desarrollar programas de acreditación o de certificación de estos niveles. La primera parte de este trabajo trata sobre las razones que fundamentan la elección de un tipo concreto de examen para la acreditación del nivel B2 de lengua inglesa en la Universidad Politécnica de Madrid. Se trata de un test de opción múltiple y en esta parte del trabajo se describe cómo fue diseñado y validado. En la segunda parte, se analizan los resultados de la aplicación del test a gran escala a un total de 924 estudiantes matriculados en varias escuelas y Facultades de la Universidad. Para terminar, se apuntan una serie de conclusiones teóricas y prácticas sobre la evaluación de la gramática y de qué modo influye en los procesos de enseñanza y aprendizaje.

Evolution and implementation of CDIO initiatives at ETSII-UPM

The School of Industrial Engineering at Universidad Politécnica de Madrid (ETSII-UPM) has been promoting student-centred teaching-learning activities, according to the aims of the Bologna Declaration, well before the official establishment of the European Area of Higher Education. Such student-centred teaching-learning experiences led us to the conviction that project based learning is rewarding, both for students and academics, and should be additionally promoted in our new engineering programmes, adapted to the Grade-Master structure. The level of commitment of our teachers with these activities is noteworthy, as the teaching innovation experiences carried out in the last ten years have led to the foundation of 17 Teaching Innovation Groups at ETSII-UPM, hence leading the ranking of teaching innovation among all UPM centres. Among interesting CDIO activities our students have taken part in especially complex projects, including the Formula Student, linked to the complete development of a competition car, and the Cybertech competition, aimed at the design, construction and operation of robots for different purposes. Additional project-based learning teamwork activities have been linked to toy design, to the development of medical devices, to the implementation of virtual laboratories, to the design of complete industrial installations and factories, among other activities detailed in present study. The implementation of Bologna process will culminate at ETSII-UPM with the beginning of the Master’s Degree in Industrial Engineering, in academic year 2014-15. The program has been successfully approved by the Spanish Agency for Accreditation (ANECA), with the inclusion of a set of subjects based upon the CDIO methodology denominated generally “INGENIA”, linked to the Spanish “ingeniar” (to provide ingenious solutions), also related etymologically in Spanish with “ingeniero”, engineer. INGENIA students will live through the complete development process of a complex product or system and there will be different kind of projects covering most of the engineering majors at ETSII-UPM.

Student academic performance stochastic simulator based on the Monte Carlo method

In this paper, a computer-based tool is developed to analyze student performance along a given curriculum. The proposed software makes use of historical data to compute passing/failing probabilities and simulates future student academic performance based on stochastic programming methods (MonteCarlo) according to the specific university regulations. This allows to compute the academic performance rates for the specific subjects of the curriculum for each semester, as well as the overall rates (the set of subjects in the semester), which are the efficiency rate and the success rate. Additionally, we compute the rates for the Bachelors degree, which are the graduation rate measured as the percentage of students who finish as scheduled or taking an extra year and the efficiency rate (measured as the percentage of credits of the curriculum with respect to the credits really taken). In Spain, these metrics have been defined by the National Quality Evaluation and Accreditation Agency (ANECA). Moreover, the sensitivity of the performance metrics to some of the parameters of the simulator is analyzed using statistical tools (Design of Experiments). The simulator has been adapted to the curriculum characteristics of the Bachelor in Engineering Technologies at the Technical University of Madrid(UPM).

Estudio de los criterios de calidad, expectativas y satisfacción de los usuarios de las cartas de servicio de la Administración General del Estado aplicados al sector marítimo. Análisis de las necesidades de desarrollo de futuro

La aproximación de las organizaciones a la mejora de sus procesos ha venido por distintos caminos. Muchas Administraciones Públicas se acercaron a este universo a través del modelo EFQM de calidad y excelencia que surgió en la década de los 80 y que sirvió como referente en el ámbito de la Unión Europea como vía de autoevaluación y determinación de procesos de mejora continua. Está basado en la identificación de los puntos fuertes y débiles aplicados a diferentes ámbitos de la organización, siendo éstos el punto de partida para el proceso de mejora continua. Se trata de un modelo en el que puedes decir que estás en calidad aunque tu puntuación sea muy pequeña, por lo que, por imagen, muchas empresas públicas empezaron a implantarlo. La empresa privada sin embargo se decantó por los sistemas de calidad basados en normas ISO. En estos sistemas has de tener un nivel mínimo para poder exhibir una certificación o acreditación del sistema de calidad. La más extendida es la ISO 9001:2008 ya que es válida para todo tipo de empresa. Este sistema se centra en la satisfacción del cliente y está basada en gran medida en el PDCA, acrónimo de Plan, Do, Check, Act (Planificar, Hacer, Verificar y Actuar). Al tratarse de sistemas documentados, pasados los años se llega a la misma conclusión, aquellas empresas que simplemente tienen un certificado colgado en la pared y que arreglan los papeles antes de la auditoría no tienen nada que aporte valor añadido a la empresa y se autoengañan. La potencia de todo sistema de gestión de calidad reside en aprovechar el potencial de sus recursos humanos dirigiendo los recursos de la empresa de forma eficiente y haciendo participe de los objetivos de la organización a su personal para que se impliquen y sepan que se espera de cada uno. La formación de ingeniero siempre nos hace ser críticos con los modelos existentes y tratar de buscar caminos alternativos que sean más eficientes. Para ello es necesario tener un conocimiento muy preciso de la organización. Por ello, después de más de cinco años trabajando en la Organización, desempeñando diversos cometidos, analizando diferentes esferas de actuación y tras estudiar informes de diferentes organizaciones que supervisan el funcionamiento de la empresa me di cuenta que la Dirección General de la Marina Mercante podía cambiar de forma de actuar para modernizarse y ser más transparente, eficaz y eficiente. Esta tesis versa sobre la posibilidad de implantar un nuevo servicio en la Dirección General de la Marina Mercante que le permita mejorar su competitividad a nivel mundial, como estado de abanderamiento, y que, dentro de nuestras fronteras, haga que sus servicios se reorienten aprovechando el conocimiento de su personal, teniendo en cuenta las necesidades de sus usuarios y los recursos de la Organización. Las cartas de servicio permiten acercar al ciudadano al funcionamiento de la organización. Le informa de las condiciones en las que se presta el servicio, los compromisos de la empresa y la forma en la que puede participar para mejorarlos, entre otros. Por otra parte, la empresa no necesita previamente tener ningún sistema de calidad implantado, aunque, como veremos en el capítulo tres y cuatro siempre ayuda a la hora de tener sistemas de aseguramiento implantados. En el capítulo seis se detallan los objetivos que se lograrían con la implantación de las cartas de servicio en la DGMM y en el capítulo siete se discuten dichos resultados y conclusiones. ABSTRACT Different ways have been used by organizations to approach process improvement. Many Public Administrations chose quality and excellence EFQM model for that approachment. This quality program began in 80 decade and that it was the Europe Unión reference to continuous improvement autoevaluation and determination. It is based on strong and weak points of different organization fields, and they are considered as starting point for continuous improvement. This model allows enterprises to say that they are working on a quality scheme even though their score is very little, and this was why a lot of Public Administrations began using it. Nevertheless private enterprises chose quality management systems based on ISO standards. In these systems there is a threshold you must have to be able to have a certification or an accreditation of quality management system. ISO 9001:2008 is the standard most used because of it can be applied to a great range of enterprises. This system is focused on customer satisfaction and it is based on PDCA, Plan, Do, Check, Act. All these systems are documented ones, so once time goes by the same conclusion is reached: enterprises that have the certificate hung on the wall and that papers are fixed for audits have nothing that give them added value and they self-delusion. Quality management system power is related to the usage of human resources potential to lead enterprise resources efficiently and to make them participate in organization objectives. Naval architect training makes them to be critic with existing models and to try to find alternative ways to be more efficient. To achieve this goal, a precise knowledge of the organization is needed. That is the reason why, after five years in quality related issues in the Organization, in different chores, analyzing our scope and reports of organizations that supervise our operation, I realized that Merchant Marine Directorate could change the way of operation to modernize and be more transparent, efficient and effective. This thesis is about the possibility of implantation of a new service in Merchant Marine Directorate that will make it possible to improve their worldwide competitiveness as Flag State, and that to reorient all services taking into account citizens needs and Organization resources. Citizen’s charters able approachment to organization operation. It gives the following information: which are terms in which service is given, enterprise compromises, ways in which citizen can collaborate to improve them, and etc. Additionally, no quality management system is needed to be implemented. Although we’ll see in chapter three and four that having it is of great help. In chapter six are detailed goals achieved if citizen’s charters are implemented in Merchant Marine Directorate, and in chapter seven conclusions and results are discussed.

Relevance evaluation of engineering masters program in Peru

In a context of mass higher education, it is necessary to ensure not only quality but also the relevance of engineering master's programs, namely the appropriateness of the objectives and outcomes to the needs and interests of the program beneficiaries. After a literature review we analyzed the evaluation models of three organizations in Peru: the Board of Evaluation, Accreditation and Certification of the University Education Quality CONEAU, the Institute of Quality and Accreditation of Computing, Engineering and Technology - ICACIT and the Pontificia Universidad Catolica del Peru. The result of this study is a model for relevance evaluation for an engineering master¿s program in Peru.

Relevance of Higher Education. How to measure it? The case of masters of engineering in Peru

Existing evaluation models for higher education have, mainly, accreditation purposes, and evaluate the efficiency of training programs, that is to say, the degree of suitability between the educational results and the objectives of the program. However, it is not guaranteed that those objectives adequate to the needs and real interests of students and stakeholders, that is to say, they do not assess the relevance of the programs, a very important aspect in developing countries. From the review of experiences, this paper proposes a model for evaluating the relevance of engineering masters program, and applies it to the case of a master?s degree at the University of Piura, Peru. We conclude that the proposed model is applicable to other masters program, offers an objective way for determining is a training program keep being relevant, and identifies improvement opportunities

Evaluación de la pertinencia de maestrías en ingeniería : aplicación en la Universidad de Piura, Perú

La formación de postgrado en ingeniería es muy importante para mejorar la competitividad y lograr el desarrollo en los países. Para ello es necesaria una fuerte vinculación de la universidad con su entorno socio económico de modo que los objetivos que se plantea en sus programas formativos sean coherentes con las necesidades reales de los beneficiarios: los estudiantes, la universidad y la comunidad. Es decir, los programas deben ser pertinentes. Y en los países en vías de desarrollo este tema es aún más importante. Se necesita modelos de evaluación que midan este grado de adecuación entre los objetivos de los programas con las necesidades de los estudiantes y las partes interesadas. Sin embargo, los modelos de evaluación existentes tienen principalmente fines de acreditación y están diseñados para evaluar la eficacia, es decir si los resultados obtenidos están de acuerdo con la misión y los objetivos planteados. Su objetivo no es medir la pertinencia. Esta investigación tiene como objetivo diseñar un modelo de evaluación de la pertinencia de maestrías en ingeniería y aplicarlo a un caso concreto. Se trata de maestrías que ya están en funcionamiento y son dictadas en una universidad en un país en desarrollo. Para diseñar el modelo se define primero el concepto de pertinencia de una maestría en ingeniería haciendo una revisión bibliográfica y consultando a expertos en los temas de pertinencia de la educación superior y formación en postgrado en ingeniería. Se utiliza una definición operativa que facilita luego la identificación de factores e indicadores de evaluación. Se identifica dos tipos de pertinencia: local y global. La pertinencia global está relacionada con la inserción de la maestría en el sistema global de producción de conocimiento. La pertinencia local tiene tres dimensiones: la personal, relacionada con la satisfacción de necesidades de los estudiantes, la institucional, relacionada con las necesidades e intereses de la universidad que acoge a la maestría y la pertinencia social, ligada a la satisfacción de necesidades y demandas de la comunidad local y nacional. El modelo diseñado es aplicado en la maestría en Ingeniería Civil con mención en Ingeniería Vial de la Universidad de Piura, Perú lo que permite obtener conclusiones para su aplicación en otras maestrías. ABSTRACT Graduate engineering education is very important to improve competitiveness and achieve development in countries. It is necessary a strong linkage between university and its socio economic environment, so that programs objectives are consistent with the real needs of the students, university and community. That is to say programs must be relevant. And in developing countries this issue is very important. Evaluation models to measure the degree of adequacy between the programs objectives with the needs of students and stakeholders is needed. However, existing evaluation models have mainly the purpose of accreditation and are designed to evaluate the efficacy. They evaluate if the results are consistent with the mission and objectives. Their goal is not to measure the relevance. This work aimed to design a model for evaluating the relevance of master's degrees in engineering and applied to a specific case. They must be masters already in operation and are taught at a university in a developing country. In order to build the model, first concept of relevance of a master's degree in engineering was defined. Literature was reviewed and we consulted experts on issues of relevance of higher education and graduate engineering education. An operational definition is used to facilitate the identification of factors and evaluation indicators. Local and global: two types of relevance were identified. The global relevance is related to the inclusion of Master in the global system of knowledge production. The local relevance has three dimensions: personal, related to meeting students' needs, institutional, related to the needs and interests of university that houses the Master and social relevance, linked to the satisfaction of needs and demands of local and national community. The designed model is applied to the Master degree in Civil Engineering with a major in Traffic Engineering of Universidad de Piura, Peru which allowed to obtain conclusions for application in other masters.